Architecture

Crawling & Indexing

Learn how Finkkle discovers web content, populates the search tables, and processes index calculations.

Indexer reference · Finkkle documentation

Storage Models

Finkkle utilizes a SQL schema optimized for fast FTS text querying and vector queries. Two environments are supported:

• Local SQLite (search.db): Used for local prototyping, testing, and single-node Node.js executions.
• Turso / libSQL Database: Remote, replicated serverless database used during production server operations and Vercel serverless deployments.

Text indices are maintained in a pages_fts virtual table to execute rapid prefix, token, and proximity lookups.

The Crawl Stack

Finkkle includes two separate crawler setups:

Classic Crawler

Found under src/crawler/. Provides straightforward, sequential scraping loops. Reads target sitemaps and crawls static HTML documents, respecting standard robots.txt limitations.

Phase 1 Crawler

Located in src/crawler/phase1/. Designed as a distributed crawl coordinator and worker system. It leverages playbooks, sitemaps, worker queues, and runs Playwright browser instances to capture dynamic JavaScript-rendered pages.

Crawl commands include:

• npm run crawl: Executes classic crawler.
• npm run crawl:phase1: Runs the modern Phase 1 crawler.
• npm run crawl-coordinator: Launches the queue coordinator.
• npm run crawl-worker: Starts a crawl worker process.

Index Maintenance Scripts

Following ingestion, several scripts must run to update relevance scores, authority ranks, and embedding data:

PageRank

Calculates document relationship linkages. Run with npm run pagerank.

Domain Authority

Assigns score factors based on domain host credibility. Run with npm run domain-authority.

Vector Embeddings

Creates vector snapshots of crawled text segments. Run with npm run embed.

Post-Crawl Pipeline

Executes PageRank, authority scoring, and embedding generations sequentially. Run with npm run post-crawl.