Data Flow

Detailed sequence diagrams showing how data flows through Rephole.

Repository Ingestion Flow

When you submit a repository for ingestion:

┌────────┐     ┌───────────┐     ┌─────────────┐     ┌────────┐
│ Client │────▶│ API Server│────▶│ Redis Queue │────▶│ Worker │
└────────┘     └───────────┘     └─────────────┘     └────┬───┘
                                                          │
                                                          ▼
┌────────────┐     ┌────────────┐     ┌───────────────────────┐
│ PostgreSQL │◀────│  ChromaDB  │◀────│  Clone → Parse → Embed│
└────────────┘     └────────────┘     └───────────────────────┘

Step by Step

1. Client sends POST /ingestions/repository
2. API Server validates request and creates job
3. Redis Queue stores job for processing
4. API Server returns jobId immediately
5. Worker picks up job from queue
6. Worker clones repository to local storage
7. Worker parses code files using Tree-sitter
8. Worker generates embeddings via OpenAI API
9. Worker stores vectors in ChromaDB
10. Worker stores file content in PostgreSQL
11. Worker marks job complete in Redis

Semantic Search Flow

When you perform a search:

┌────────┐     ┌───────────┐     ┌────────────┐
│ Client │────▶│ API Server│────▶│ OpenAI API │
└────────┘     └───────────┘     └─────┬──────┘
                    │                   │
                    │◀──────────────────┘
                    │      (embedding)
                    ▼
              ┌───────────┐
              │ ChromaDB  │
              └─────┬─────┘
                    │ (child chunk IDs)
                    ▼
              ┌────────────┐
              │ PostgreSQL │
              └─────┬──────┘
                    │ (parent content)
                    ▼
              ┌───────────┐
              │  Response │
              └───────────┘

Step by Step

1. Client sends POST /queries/search with prompt
2. API Server sends prompt to OpenAI for embedding
3. OpenAI returns 1536-dimensional vector
4. API Server queries ChromaDB for similar child chunks
5. ChromaDB returns matching chunk IDs (k × 3)
6. API Server groups chunks by parent document
7. API Server fetches parent content from PostgreSQL
8. API Server formats and returns results to client

Data Storage

ChromaDB (Vectors)

Collection: rephole-collection
├── Document ID: chunk-001
│   ├── Vector: [0.123, -0.456, ...]  (1536 dims)
│   └── Metadata: {parent_id, file_path, repo_id}
├── Document ID: chunk-002
│   └── ...

PostgreSQL (Content)

Table: files
├── id: ulid
├── repo_id: string
├── path: string
├── content: text (full file)
├── created_at: timestamp
└── updated_at: timestamp

Queue Structure

BullMQ Job

{
  "name": "repo-ingestion",
  "data": {
    "repoUrl": "https://github.com/...",
    "ref": "main",
    "token": "...",
    "userId": "...",
    "repoId": "..."
  },
  "opts": {
    "attempts": 3,
    "backoff": {
      "type": "exponential",
      "delay": 1000
    }
  }
}