Remove RAG group

[jonahdc]

This removes the rag group from the project configuration.

[patched-admin]

File Changed: .github/workflows/test.yml

Rule 1: Do not ignore potential bugs in the code
Details: Removal of the rag-test job without any replacement could introduce potential bugs if this testing was critical for RAG functionality. Even though it's commented as "disabled because this currently takes too long", removing test coverage completely rather than optimizing it could lead to undetected issues.

Affected Code Snippet:

  rag-test:
    # disabled because this currently takes too long
    if: false
    runs-on: ubuntu-latest

Start Line: 121
End Line: 170

---

Rule 2: Do not overlook possible security vulnerabilities
Details: The removed code contained sensitive API key usage (PATCHED_API_KEY and SCM_GITHUB_KEY). While removing this job doesn't directly introduce vulnerabilities, ensure these secrets aren't needed elsewhere and that their removal won't affect other dependent processes. Additionally, any cached data in the GitHub Actions cache should be properly cleaned up after removing this job.

Affected Code Snippet:

          --patched_api_key=$https://github.com/patched-codes/patchwork/pull/1328/files#diff-9a4e1b5185aca60ac378253dd90a488e3b1858536e09d72d694490c065248833 \
          --github_api_key=$https://github.com/patched-codes/patchwork/pull/1328/files#diff-dcfdde616c57a27ef1c8dcb483375e99d5bc36af829f9834e372718bdf91a893 \

Start Line: 159
End Line: 160

File Changed: patchwork/common/utils/dependency.py

Rule 1: Do not ignore potential bugs in the code

Details: Potential bug identified. The removal of the "rag" dependency group and the chromadb() function could lead to broken dependencies if there are other parts of the codebase still relying on these functions. This change should be accompanied by a full dependency audit.

Affected Code Snippet:

__DEPENDENCY_GROUPS = {
-    "rag": ["chromadb"],
     "security": ["semgrep", "depscan"],
     "notification": ["slack_sdk"],
}

-def chromadb():
-    return import_with_dependency_group("chromadb")

Start Line: 3
End Line: 6

File Changed: patchwork/common/utils/utils.py

Rule 1: Do not ignore potential bugs in the code

Details: The removal of the get_vector_db_path() function may cause issues if there are dependencies elsewhere in the codebase that rely on this function for ChromaDB path configuration. The code doesn't show any checks for these dependencies being properly handled.

Affected Code Snippet:

def get_vector_db_path() -> str:
    CHROMA_DB_PATH = HOME_FOLDER / "chroma.db"
    if CHROMA_DB_PATH:
        return str(CHROMA_DB_PATH)
    else:
        return ".chroma.db"

Start Line: 132
End Line: 138

---

Rule 2: Do not overlook possible security vulnerabilities

Details: The removal of OpenAI and HuggingFace embedding functions includes API key handling code. If these functions are being moved elsewhere, there needs to be verification that the new location maintains proper API key security practices. Additionally, the removal of HOME_FOLDER import might affect secure file path handling.

Affected Code Snippet:

def openai_embedding_model(
    inputs: dict,
) -> "chromadb.api.types.EmbeddingFunction"["chromadb.api.types.Documents"] | None:
    model = inputs.get(openai_embedding_model.__name__)
    if model is None:
        return None

    api_key = inputs.get("openai_api_key")
    if api_key is None:
        raise ValueError("Missing required input data: 'openai_api_key'")

    return chromadb().utils.embedding_functions.OpenAIEmbeddingFunction(
        api_key=api_key,
        model_name=model,
    )

Start Line: 141
End Line: 156

File Changed: patchwork/steps/GenerateCodeRepositoryEmbeddings/GenerateCodeRepositoryEmbeddings.py

Rule 1: Do not ignore potential bugs in the code

Details: Several potential bugs identified in the code that should be addressed:

1. No error handling for file operations after exception
2. Unclosed file handle in open_with_chardet operation
3. Undefined 'text' variable when file read fails

Affected Code Snippet:

try:
    with open_with_chardet(file, "r") as fp:
        text = fp.read()
except Exception as e:
    logger.warning(f"Error reading file {file}: {e}")

if len(text.strip()) == 0:
    continue

Start Line: 93
End Line: 100

---

Rule 2: Do not overlook possible security vulnerabilities

Details: Several security concerns identified:

1. Path traversal vulnerability in file operations
2. Unsafe direct file reading without size limits
3. Use of relative paths without validation

Affected Code Snippet:

for file in files:
    try:
        with open_with_chardet(file, "r") as fp:
            text = fp.read()
    except Exception as e:
        logger.warning(f"Error reading file {file}: {e}")

Start Line: 92
End Line: 96

File Changed: patchwork/steps/GenerateCodeRepositoryEmbeddings/README.md

Rule 1: Do not ignore potential bugs in the code

Details: The code documentation being removed appears to have important information about code functionality, including inputs, description, and outputs. Its removal could lead to bugs due to missing documentation about expected behavior, especially around:

• File filtering and handling of blacklisted directories
• Hash generation for code files
• Database interactions
• Batch processing functionality

The removal of documentation makes the codebase more prone to bugs due to lack of clear specifications and expectations.

Affected Code Snippet:

### Description
- The code is responsible for generating embeddings for code files in a code repository.
- It uses the `git` Python package to interact with the Git repository where the code files are stored.
- The code provides a function `filter_files` that takes an iterable of file paths and filters out files based on directory blacklists.
- The code filters out specific file types based on a whitelist and ignores certain directories based on a blacklist.

Start Line: 7
End Line: 11

---

Rule 2: Do not overlook possible security vulnerabilities introduced by code modifications

Details: Removing documentation about security-sensitive operations could lead to misuse and security vulnerabilities, particularly around:

• SHA-1 hash generation process
• Git repository interaction
• File filtering mechanisms that might be security-related
• Database interaction patterns

The removal of this documentation could lead to improper implementation or usage of these security-sensitive features.

Affected Code Snippet:

- The `hash_text` function generates a SHA-1 hash for the text content of code files.
- The `GenerateCodeRepositoryEmbeddings` class manages the process of generating embeddings for the code repository.
- It fetches code files, reads their content, generates hashes, and interacts with the ChromaDB database to store embeddings and related metadata.

Start Line: 12
End Line: 14

File Changed: patchwork/steps/GenerateCodeRepositoryEmbeddings/filter_lists.py

Rule 1: Do not ignore potential bugs in the code

Details: The code has potential bugs related to duplicate file extensions and inconsistent naming patterns.

Affected Code Snippet:

_EXTENSION_WHITELIST = [
    # TypeScript appears twice
    ".ts",  # Line 3
    # ...
    ".ts",  # Line 46
    
    # Visual Basic appears twice
    ".vb",  # Line 48
    # ...
    ".vb",  # Line 86
    
    # SQL appears twice
    ".sql",  # Line 54
    # ...
    ".sql",  # Line 82
]

Start Line: 1
End Line: 90

Details: The _DIRECTORY_BLACKLIST contains a bug in filename patterns.

Affected Code Snippet:

_DIRECTORY_BLACKLIST = [
    "test",
    "tests",
    "___init__.py",  # Should be "__init__.py" (3 underscores instead of 2)
    "___main__.py",  # Should be "__main__.py" (3 underscores instead of 2)
    "__pycache__",
]

Start Line: 91
End Line: 97

File Changed: patchwork/steps/GenerateCodeRepositoryEmbeddings/typed.py

Rule 2: Do not overlook possible security vulnerabilities

Details: Potential security concern in documents field type definition. Using a plain str type for documents could potentially allow for unsafe content if not properly sanitized. Consider using a more specific type or adding validation constraints.

Affected Code Snippet:

class GenerateCodeRepositoryEmbeddingsOutputs(TypedDict):
    embedding_name: str
    documents: str

Start Line: 10

End Line: 12

File Changed: patchwork/steps/GenerateEmbeddings/GenerateEmbeddings.py

Rule 1: Do not ignore potential bugs in the code

Details: Several potential bugs were identified in the code:

1. Undefined variable reference in embeddings check
2. Potential memory issues with large documents
3. No validation of chunk and overlap sizes

Affected Code Snippet:

elif embeddings is not None:  # Line 79
    embedding_ids.append(str(document.get("id")))
    embeddings.append(embedding)

Start Line: 79
End Line: 81

Additional Issue:

def split_text(document_text: str, chunk_size: int, overlap: int) -> list[str]:
    char_length = len(document_text)
    chunks = []
    for i in range(0, char_length, chunk_size - overlap):
        chunk = "".join(document_text[i : i + chunk_size])

Start Line: 15
End Line: 19

---

Rule 2: Do not overlook possible security vulnerabilities

Details: Several security concerns were identified:

1. No input validation for document content
2. Potential for path traversal in vector_db_path
3. No size limits on metadata dictionary

Affected Code Snippet:

metadata = {key: value for key, value in document.items() if key not in ["id", "document"]}
metadata["original_document"] = document_text
metadata["original_id"] = doc_id

Start Line: 75
End Line: 77

File Changed: patchwork/steps/GenerateEmbeddings/README.md

Rule 3: Do not deviate from the original coding standards

Details: The complete removal of README documentation represents a deviation from coding standards if documentation is part of the established standards. Documentation is typically a critical part of code maintenance and understanding.

Affected Code Snippet:

# Summary of `GenerateEmbeddings.py` 
...
## Usage:
1. Import the `GenerateEmbeddings` class from the module.
2. Create an instance of `GenerateEmbeddings` with the required input dictionary.
3. Call the `run` method to generate embeddings for the provided documents and store them in the database collection.
4. Receive the output dictionary indicating the completion of the embeddings generation process.

Start Line: 1
End Line: 19

File Changed: patchwork/steps/GenerateEmbeddings/typed.py

Rule 1: Do not ignore potential bugs in the code

Details: Potential bug found - The GenerateEmbeddingsOutputs TypedDict is empty which could lead to runtime errors if the function is expected to return any values. An empty output type definition suggests either missing output specifications or incorrect interface definition.

Affected Code Snippet:

class GenerateEmbeddingsOutputs(TypedDict):
    pass

Start Line: 15
End Line: 16

---

Rule 2: Do not overlook possible security vulnerabilities

Details: Security concern identified - The documents field accepts List[Dict[str, Any]] which allows arbitrary dictionary content. This untyped approach could lead to injection vulnerabilities if the input is not properly sanitized before processing.

Affected Code Snippet:

class __GenerateEmbeddingsRequiredInputs(TypedDict):
    embedding_name: Annotated[str, StepTypeConfig(is_config=True)]
    documents: List[Dict[str, Any]]

Start Line: 6
End Line: 8

File Changed: patchwork/steps/QueryEmbeddings/QueryEmbeddings.py

Rule 1: Do not ignore potential bugs in the code

Details: There is a potential bug in error handling where input validation may be insufficient. The code checks for required keys but doesn't validate the type or content of those keys. Additionally, the token counting logic could lead to unexpected behavior if metadata["original_document"] contains invalid content.

Affected Code Snippet:

def __init__(self, inputs: dict):
    super().__init__(inputs)
    if not all(key in inputs.keys() for key in self.required_keys):
        raise ValueError(f'Missing required data: "{self.required_keys}"')

    client = chromadb().PersistentClient(path=get_vector_db_path())
    embedding_function = get_embedding_function(inputs)
    self.collection = client.get_collection(name=inputs["embedding_name"], embedding_function=embedding_function)

    self.texts: list[str] = inputs["texts"]
    self.top_k = inputs.get("top_k", 10)
    self.token_limit = inputs.get("token_limit", 4096)

Start Line: 14
End Line: 25

---

Rule 2: Do not overlook possible security vulnerabilities

Details: Potential security vulnerability exists in handling unvalidated input data that goes directly into the ChromaDB client. The code doesn't sanitize the embedding_name parameter which could potentially lead to path traversal or injection attacks depending on how ChromaDB handles collection names.

Affected Code Snippet:

client = chromadb().PersistentClient(path=get_vector_db_path())
embedding_function = get_embedding_function(inputs)
self.collection = client.get_collection(name=inputs["embedding_name"], embedding_function=embedding_function)

Start Line: 20
End Line: 22

File Changed: patchwork/steps/QueryEmbeddings/README.md

Rule 1: Do not ignore potential bugs in the code

Details: Documentation removes important information about token limits mentioned in the last line without providing alternative documentation. This could lead to potential bugs if developers are unaware of token limitations.

Affected Code Snippet:

- The `run` method executes the query process and returns the dictionary with the embedding results sorted by distance, respecting the specified token limit.

Start Line: 23
End Line: 23

---

Rule 2: Do not overlook possible security vulnerabilities

Details: The documentation removal eliminates crucial information about embedding functions and vector database paths, which could lead to security issues if developers implement these features without proper guidance on secure implementation.

Affected Code Snippet:

-  - `get_embedding_function` from `patchwork.common.utils.utils`
-  - `get_vector_db_path` from `patchwork.common.utils.utils`

Start Line: 4
End Line: 5

File Changed: patchwork/steps/QueryEmbeddings/typed.py

Rule 1: Do not ignore potential bugs in the code

Details: Potential bug identified in type definitions. The token_limit and top_k parameters are marked as optional (due to total=False), but there are no default values specified. This could lead to runtime errors if these configuration parameters are not provided.

Affected Code Snippet:

class QueryEmbeddingsInputs(__QueryEmbeddingsRequiredInputs, total=False):
    top_k: Annotated[int, StepTypeConfig(is_config=True)]
    token_limit: Annotated[int, StepTypeConfig(is_config=True)]

Start Line: 10
End Line: 12

---

Rule 2: Do not overlook possible security vulnerabilities

Details: Potential security vulnerability identified. The texts parameter accepts a list of strings without any input validation or sanitization. This could potentially lead to injection attacks or memory issues if extremely large texts are provided.

Affected Code Snippet:

class __QueryEmbeddingsRequiredInputs(TypedDict):
    embedding_name: str
    texts: List[str]

Start Line: 6
End Line: 8

File Changed: patchwork/steps/__init__.py

Rule 1 - Do not ignore potential bugs in the code

Details: Removing embedding-related modules without ensuring all their references are removed could lead to runtime errors if these modules are still being used elsewhere in the codebase. This should be verified with a comprehensive dependency check.

Affected Code Snippet:

from patchwork.steps.GenerateCodeRepositoryEmbeddings.GenerateCodeRepositoryEmbeddings import (
    GenerateCodeRepositoryEmbeddings,
)
from patchwork.steps.GenerateEmbeddings.GenerateEmbeddings import GenerateEmbeddings
...
from patchwork.steps.QueryEmbeddings.QueryEmbeddings import QueryEmbeddings

Start Line: 26
End Line: 40

File Changed: pyproject.toml

Rule 1: Do not ignore potential bugs in the code

Details: Potential bug risk due to removal of pinned transitive dependencies. The removal of explicitly pinned dependencies (opentelemetry-exporter-otlp-proto-grpc, onnxruntime) that were previously managed could lead to version conflicts or unexpected behavior in dependent packages.

Affected Code Snippet:

# rag - pinning transitive dependencies
opentelemetry-exporter-otlp-proto-grpc = { version = "1.25.0", optional = true }
onnxruntime = { version = "1.19.2", optional = true }

Start Line: 9
End Line: 10

---

Rule 2: Do not overlook possible security vulnerabilities introduced by code modifications

Details: Security concern due to removal of version constraints. The removal of the RAG dependencies without proper deprecation or replacement strategy could lead to undefined behavior if any part of the codebase still references these packages. Additionally, removing explicit version pinning of transitive dependencies may lead to pulling in potentially vulnerable versions of these packages if they're still required by other dependencies.

Affected Code Snippet:

# rag
chromadb = { version = "~0.4.24", optional = true }
sentence-transformers = { version = "~2.7.0", optional = true }

# rag - pinning transitive dependencies
opentelemetry-exporter-otlp-proto-grpc = { version = "1.25.0", optional = true }
onnxruntime = { version = "1.19.2", optional = true }
torch = { version = "~2.5.1", optional = true }
orjson = { version = "~3.9.15", optional = true }

Start Line: 5
End Line: 12

File Changed: tests/steps/test_GenerateEmbeddings.py

Rule 1: Do not ignore potential bugs in the code

Details: Duplicate test function names found. There are two test functions named test_generate_embeddings_run() which could lead to unpredictable test execution and confusion.

Affected Code Snippet:

def test_generate_embeddings_run():
    inputs = {"embedding_name": "test", "documents": [{"document": "test document"}]}
    step = GenerateEmbeddings(inputs)
    result = step.run()
    assert result == {}

Start Line: 19, 39
End Line: 23, 44

Additional Details: Testing result == {} may be too permissive and could miss bugs. Tests should verify the actual expected output structure rather than just checking for an empty dictionary.

File Changed: tests/steps/test_QueryEmbeddings.py

Rule 1: Do not ignore potential bugs in the code

Details: The test suite has potential bugs in error handling and edge cases that are not being tested. The QueryEmbeddings class is not being tested for empty input lists, malformed metadata, or invalid collection names.

Affected Code Snippet:

def test_query_results(setup_collection):
    # Test that the query results are processed correctly
    collection = setup_collection
    inputs = {"embedding_name": collection.name, "texts": ["text1", "text2"]}
    query_embeddings = QueryEmbeddings(inputs)
    results = query_embeddings.run()
    assert isinstance(results, dict)
    assert "embedding_results" in results

Start Line: 41
End Line: 48

---

Rule 2: Do not overlook possible security vulnerabilities

Details: The test fixture uses UUID for document IDs but doesn't validate input text content, which could potentially lead to injection vulnerabilities in a production environment. Additionally, the persistent storage path is not properly sanitized.

Affected Code Snippet:

client = chromadb.PersistentClient(path=get_vector_db_path())
collection = client.get_or_create_collection(_TEST_COLLECTION)
collection.upsert(
    ids=[str(uuid.uuid1()), str(uuid.uuid1()), str(uuid.uuid1())],
    documents=["text1", "text2", "text3"],
    metadatas=[
        {"original_id": "1", "original_document": "text1"},
        {"original_id": "2", "original_document": "text2"},
        {"original_id": "3", "original_document": "text3"},
    ],
)

Start Line: 14
End Line: 25