How SARR finds packages

By the numbers

• 166,819 PyPI packages indexed as 384-d embeddings in Qdrant
• 18 ms Warm search, server p50 (p95 65 ms); 172 ms with rerank
• 15 ms Warm GraphQL search, server p50 (p95 16 ms, 30 queries)
• 718 Packages refreshed in one 15-minute Lambda run, 0 failures
• 701 / 718 Refreshed packages with a last-commit date; the old data had none (the other 17 repos are deleted)
• 142 Renamed GitHub repos corrected; 111 archived and 17 deleted flagged
• 333 / 718 Refreshed packages with releases after 2018, where the old data stopped
• 621 / 718 Refreshed packages with 30-day download counts (13.7B in total)

The problem

Package registries are easy to search when you already know a name. They are harder when you only know the job: an HTTP client, a dataframe tool, something for retries. Keyword indexes reward exact tokens; they under-serve descriptive queries and rarely encode whether a project is widely used or still maintained.

SARR (Semantic Artifacts Retrieval and Ranking) indexes PyPI packages as semantic documents and returns a ranked list for natural-language queries. The live demo covers 166,819 packages from public Libraries.io metadata joined with repository signals. That dump stops at December 2018, so a scheduled health collector now refreshes the most-starred packages from GitHub and PyPI every night (see below).

System design

Indexing is a rare GPU job (Colab, PyTorch). Serving only embeds the query, retrieves neighbors from Qdrant, optionally reranks, and blends in adoption and recency. If you check LLM, a second path asks Vertex Gemini for a grounded top-3 from those retrieved packages only. On Lambda both the bi-encoder and the cross-encoder run as ONNX — no PyTorch import on the request path. Both paths share the same BAAI/bge-small-en-v1.5 checkpoint and document format.

End-to-end path

1. Index offline. A GPU batch job pulls PyPI rows from BigQuery, builds a structured text document per package, embeds it with a PyTorch bi-encoder, and upserts vectors plus metadata into Qdrant. Runs are watermarked so a disconnect can resume without duplicating points.
2. Encode the query online. The same embedding checkpoint used at index time maps your search string into the same 384-d space. Hosted Lambda runs a baked ONNX graph; local dev can still use sentence-transformers when no ONNX files are present.
3. Retrieve neighbors. Approximate nearest-neighbor search returns the closest packages by cosine similarity over the full collection.
4. Refine when needed. If you check Rerank, the UI sends rerank: true and an ONNX MiniLM cross-encoder rescores the short top‑k list. No PyTorch import on Lambda. This checkbox does not call Gemini.
5. Blend for usefulness. The final score mixes semantic relevance with adoption (stars, 30-day downloads or dependents, SourceRank) and recency (last commit, else latest release). Popularity stays in structured payload fields so it influences rank without polluting the embedding text.
6. Grounded recommendations (optional). If you check LLM, the UI calls POST /v1/rag. Gemini writes a top-3 from the retrieved names and descriptions only. Packages not in that set are dropped. If generation fails, the ranked list is still the result. Leave LLM unchecked for search without an LLM call.

Grounded RAG

Search and generation are separate. Classic search still uses POST /v1/search. Checking LLM on the demo calls POST /v1/rag and streams Server-Sent Events. Checking Rerank only runs the ONNX cross-encoder; it does not call Gemini.

1. Retrieve 50 neighbors from Qdrant (same bi-encoder as search).
2. If Rerank is on, the MiniLM cross-encoder scores those 50; otherwise ranking uses cosine plus the popularity/recency blend. The API keeps the top 8 for the prompt.
3. Emit ranked_list immediately. That list is a complete result if Gemini is slow or fails.
4. Vertex Gemini (gemini-2.5-flash-lite, fallback gemini-2.5-flash) sees only package name and description. It returns JSON with exactly three recommendations, each with package, reason and cited_snippet.
5. The backend drops any package not in the eight retrieved names, then emits llm_done (or llm_error). The UI marks a Gemini icon on that block so generated text is distinct from ranked hits.

On the hosted demo, API Gateway may deliver the stream in one block, so the ranked list and the top-3 can appear together. A labeled eval set (precision@k, citation accuracy, faithfulness) is still future work.

Compared with PyPI.org search

Try “text to speech” or “async HTTP client with retries” on the search page. PyPI.org often ranks packages whose names contain the query words (including mismatched descriptions). SARR can surface a widely used library such as google-cloud-texttospeech even when the name is not a literal copy of the query — and can also over-retrieve near-misses such as speech-to-text vs text-to-speech.

Models and systems choices

The MVP uses a compact bi-encoder (BAAI/bge-small-en-v1.5, 384 dimensions) so indexing ~168k documents on a Colab T4 stays practical. ETL embeds the corpus with PyTorch / sentence-transformers. The Lambda image exports those weights to ONNX at build time so query embedding on CPU stays well under API Gateway’s 30s client wait. Reranking uses cross-encoder/ms-marco-MiniLM-L-6-v2 as ONNX on the retrieved shortlist rather than scoring the whole corpus — cheap recall, then selective precision.

Serving keeps the heavy lift out of the request path: corpus embeddings are precomputed. Each hosted search request only runs small ONNX graphs for the query (and the cross-encoder when rerank is on), hits Qdrant, and blends scores. Gemini runs only when the caller uses /v1/rag (the demo LLM checkbox). The same FastAPI app runs locally (sentence-transformers when no ONNX graph is present) and on Lambda (ONNX Runtime, no PyTorch), with shared schemas and document formatting between ETL and search.

What “fast enough” looks like today

Server times (took_ms) measure the search itself. Client times add the network round trip to us-east-1. Cold starts are listed separately rather than mixed into the percentiles.

• Full corpus load: 167,619 rows (PyTorch embed + Qdrant upsert on a Colab T4) into 166,819 unique packages.
• GraphQL, warm: server p50 15 ms, p95 16 ms (30 sequential queries, 9 Oct 2026); client p50 ~149 ms.
• Cold Lambda: about 5 s without rerank, about 16 s with rerank (loads ONNX sessions).
• Warm, rerank=false: server p50 18 ms, p95 65 ms, p99 67 ms. Warm, rerank=true: p50 172 ms, p95 257 ms (~154 ms extra for the cross-encoder; client p50 ~271 ms from this region).
• Local RAG (warm, LLM + rerank on): ranked list 472 ms, Gemini top-3 1.05 s (llm_ms) for async HTTP client.
• Hosted RAG (warm, LLM + rerank on Lambda): ranked list ~350–520 ms server took_ms, Gemini ~650 ms–1.2 s (llm_ms). API Gateway may deliver SSE in one block rather than streaming incrementally.

Search percentiles come from 50 sequential mixed queries against the live API after one warmup request (20 Aug 2026).

MCP for agents

Model Context Protocol lets coding agents (Cursor, Claude Desktop, etc.) call SARR as tools instead of guessing PyPI names. The MCP server is a thin local process — it does not run on Lambda and does not load ONNX or Gemini. It forwards HTTP to the same FastAPI app the UI uses.

1. search_packages → POST /v1/search. Cheap semantic lookup with query, limit, and optional rerank.
2. recommend_packages → POST /v1/rag. The server reads the SSE stream (ranked_list, then llm_done or llm_error) and returns one JSON object — ranked hits, grounded top-3, timings, and error if Gemini failed.
3. health → GET /healthz.

Agents should only cite package names returned by these tools; citation guardrails stay on the API. Setup and the full tool schema are in the repo README.

GraphQL API

POST /graphql (Strawberry on the same FastAPI app) runs the exact search path as /v1/search, but the client chooses the fields: a UI card can ask for name and stars only, an agent can add license and timings. Filters (minimum stars, license, Python version) map to Qdrant payload indexes. Queries are capped at depth 6 and 5 aliases so one request can't fan out into many searches.

{ search(query: "async HTTP client", limit: 3, rerank: true) {
    tookMs
    packages { name stars license }
} }

Package health collector

Libraries.io stopped at 2018: every package had an empty last-commit date and no download counts, and star counts were years old. sarr-collect refreshes them from the GitHub REST API, the PyPI JSON API and pypistats, adds a 0–1 health_score (push and release recency, release cadence, issues per star, recent commits; archived repos capped), and writes with Qdrant partial payload updates, so nothing is re-embedded.

1. Prioritize. A SQLite frontier orders packages by staleness × log(stars + SourceRank), so stale, popular packages go first. Three consecutive failures dead-letter a package.
2. Fetch politely. One HTTP client spaces requests per host, sleeps until the rate-limit reset when quota runs low, honors Retry-After, backs off on 5xx, and sends ETags so unchanged GitHub repos return 304 without costing quota. Renamed repos (301) are followed and their URL fixed.
3. Write only changes. A hash of the new fields skips writes when nothing changed; the score is hashed in 0.05 steps so time decay alone doesn't trigger rewrites.

The first scheduled run refreshed 718 of the most-starred packages in 13 minutes with no failures. 701 now have a last-commit date (441 committed in 2026), 621 have 30-day downloads (13.7 billion in total), and 333 show releases after the 2018 cutoff. It corrected 142 renamed repos and flagged 111 archived and 17 deleted ones. Hosted search ranks with the refreshed stars and dates immediately, because they are the same payload fields, and the search page shows them as health badges on each checked package.

Infrastructure as code

The collector runs as its own Lambda, defined in Terraform (infra/terraform/collector, 24 resources), separate from the SAM-deployed search API.

• Schedules: EventBridge runs a nightly refresh and a weekly backfill that adds newly popular packages.
• State: Lambda storage is wiped between runs, so the frontier lives in a versioned, encrypted S3 bucket. A conditional-write lock (If-None-Match: *) keeps overlapping runs from clobbering it.
• Secrets: the GitHub token and Qdrant key are SSM SecureStrings; Terraform creates placeholders and never stores the real values in its state.
• Least privilege: the role can write only its own logs and state prefix and read only its two parameters.
• Observability: each run emits CloudWatch metrics (processed, written, failed, …); alarms cover errors, a missed daily run and mass package failures.

Stack at a glance

• BigQuery → sentence-transformers / PyTorch → Qdrant Cloud
• FastAPI + Pydantic for the search contract; Mangum on Lambda
• ONNX Runtime for hosted query embeddings and rerank (no PyTorch import)
• POST /v1/rag + SSE: retrieve 50 → top 8 → citation-checked Gemini top-3 (demo LLM checkbox)
• Vertex Gemini (gemini-2.5-flash-lite, fallback gemini-2.5-flash) on GCP; service account in AWS Secrets Manager on Lambda
• MCP stdio server: search_packages, recommend_packages, health → HTTP API (local or hosted)
• GraphQL (Strawberry) on /graphql with depth and alias limits
• Health collector: httpx + SQLite frontier → Qdrant partial updates, on a scheduled Lambda (EventBridge, S3, SSM, CloudWatch, SNS) provisioned with Terraform
• Vite demo (Search, How it works, Contact)
• 127 unit tests, Ruff and Terraform validation in CI