Ingestion Pipeline

Understand how Rephole processes your codebase from repository to searchable vectors.

Pipeline Overview

Repository → Clone → Parse → Chunk → Embed → Store → Index

When you submit a repository for ingestion, Rephole automatically:

1. Clones your repository to local storage
2. Parses code files using AST (Abstract Syntax Tree) analysis
3. Chunks code intelligently at function/class level
4. Generates embeddings using OpenAI's text-embedding-3-small model
5. Stores vectors in ChromaDB and metadata in PostgreSQL
6. Indexes for fast retrieval

Supported Languages

Rephole supports 20+ programming languages through Tree-sitter parsing:

Category	Languages
Web	JavaScript, TypeScript, HTML, CSS
Backend	Python, Java, Go, Rust, C#, Ruby, PHP
Systems	C, C++, Rust
Data	SQL, JSON, YAML
Mobile	Swift, Kotlin
Other	Bash, Markdown

Chunking Strategy

Rephole uses intelligent code chunking rather than simple text splitting:

Function-Level Chunking

// Each function becomes a separate chunk
function authenticate(user, password) {  // ← Chunk 1
  // ...
}

function validateToken(token) {  // ← Chunk 2
  // ...
}

Class-Level Chunking

// Classes are chunked with their methods
class UserService {  // ← Single chunk with context
  constructor() { }
  findOne(id) { }
  create(data) { }
}

Parent-Child Retrieval

Rephole implements a parent-child retrieval strategy:

1. Child Chunks: Small, precise code segments for accurate matching
2. Parent Documents: Full file content for complete context

When you search:

• The query matches against child chunks for precision
• Results return the parent document for full context

This ensures you get both accurate matches AND understand the surrounding code.

Embedding Model

Rephole uses OpenAI's text-embedding-3-small model:

Property	Value
Dimensions	1536
Max Tokens	8191
Cost	$0.00002 / 1K tokens

Processing Flow

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Client    │────▶│  API Server │────▶│ Redis Queue │
└─────────────┘     └─────────────┘     └──────┬──────┘
                                               │
                    ┌─────────────┐     ┌──────▼──────┐
                    │  ChromaDB   │◀────│   Worker    │
                    └─────────────┘     └──────┬──────┘
                                               │
                    ┌─────────────┐     ┌──────▼──────┘
                    │ PostgreSQL  │◀────┘
                    └─────────────┘

Next Steps

• Learn about the Query Flow to understand how searches work
• See the Architecture Overview for system design