Indexing Engine

How Synapse's two-path deterministic AST indexing works.

Synapse uses a two-path deterministic indexing architecture — one path for first-run full scans, another for sub-5ms incremental Git delta updates.

Two-Path Design

                   ┌───────────────────────┐
                   │   Git State Monitor   │
                   │  (HEAD / index / refs)│
                   └──────────┬────────────┘
                              │
                 First run?   │   Commit shift?
           ┌──────────────────┴──────────────────┐
           │                                     │
           ▼                                     ▼
  ┌─────────────────────┐             ┌────────────────────┐
  │  Full Index         │             │  Incremental Index │
  │  (synap init)       │             │  (daemon watcher)  │
  │                     │             │                    │
  │  ProcessPoolExecutor│             │  git diff-tree     │
  │  500 files/chunk    │             │  (delta only)      │
  │  All CPU cores      │             │  Changed files     │
  └──────────┬──────────┘             └─────────┬──────────┘
             │                                  │
             └──────────────┬───────────────────┘
                            ▼
               ┌────────────────────────┐
               │   SQLite WAL Database  │
               │   (.synap/synap.db)    │
               │                        │
               │  ● symbols             │
               │  ● edges               │
               │  ● files               │
               │  ● symbols_fts (FTS5)  │
               │  ● wiki_queue          │
               └────────────────────────┘

Path 1: Full Indexing (synap init)

Triggered on first initialization or when --force is passed.

Scanner Discovery

Walks the entire repository tree, respecting .gitignore rules via a custom glob-to-regex matcher. Skips:

• .venv/, node_modules/, dist/, build/, __pycache__/
• Files larger than max_file_bytes (default: 1MB)
• Binary files (detected via control character ratio check on first 1KB)

Parallel Processing

Files are split into 500-file chunks and distributed across all CPU cores using ProcessPoolExecutor. Each worker runs Tree-sitter in a subprocess with no GIL contention.

Pass 1 — AST Symbol Extraction

For each file, Tree-sitter generates a concrete syntax tree. The extractor pulls:

• function_definition, class_definition, method_definition
• import_statement, import_from_statement
• Module-level variable assignments

Each symbol gets a stable, collision-free ID:

symbol_id = sha256(f"{path}:{name}:{kind}:{start_line}").hexdigest()[:16]

Pass 2 — Edge Resolution

After all symbols are inserted, a second pass resolves import paths to actual file paths and inserts depends_on edges into the edges table. Batched via executemany transactions.

Wiki Enqueuing

Files and project-level summaries (overview.md, architecture.md, schema.md) are enqueued into wiki_queue for background async generation.

Path 2: Incremental Indexing (Daemon)

Triggered automatically by the background daemon when a Git commit shift is detected.

Git Delta Detection

Instead of reading all files, Synapse executes:

git diff-tree -r --no-commit-id --name-status <prev_commit> <current_commit>

This immediately returns only the files that changed — typically 2–20 files per commit.

Selective AST Update

Only the changed files are re-parsed. Stale symbols for modified or deleted files are removed atomically before new symbols are inserted — preventing ghost symbols.

Incremental Edge Re-resolution

Only edges referencing changed-file symbols are re-computed. The rest of the dependency graph is preserved unchanged.

Typical incremental latency: 3–7ms total.

SQLite Optimizations

| Optimization | Detail | |---|---| | WAL Mode | PRAGMA journal_mode=WAL — concurrent readers never block the writer | | Synchronous Normal | PRAGMA synchronous=NORMAL — safe crash recovery without fsync on every write | | Batch Inserts | All symbol inserts use executemany in a single transaction | | FTS5 Virtual Table | symbols_fts for sub-millisecond BM25 keyword search | | Module Key Index | Pre-computed module_key (e.g. src.auth.service) for O(1) module lookups | | LRU Cache | Vector dot-products cached in memory to avoid recomputing embeddings |

Supported Languages

Tree-sitter grammars are bundled for: Python, TypeScript, TSX, JavaScript, JSX, Rust, Go, C, C++, C#, Java, Kotlin, Swift, Scala, Ruby, PHP, Bash, SQL, Markdown, JSON, TOML, YAML, HTML, and CSS.