codekb quality benchmark — external backend¶

Date: 2026-10-06

Type: internal benchmark, not a published parity claim
Target: a real, previously-unseen production backend service (read-only, untouched throughout) — not a repo this tool was built or tuned against
Binary under test: v2/kl4a-rs/target/release/codekb.exe (release build, built for this benchmark)
Note on naming: specific file names, route paths, model/schema names, and business terms from the target codebase are generalized below to avoid revealing internal details of someone else's project; the counts, structure, and findings are real

What the codebase is — a Python web backend built on:

FastAPI — route decorators, with multiple APIRouter modules mounted onto a main app
SQLAlchemy — declarative ORM models, including single-table inheritance
Pydantic — request/response schema classes, including multi-hop derived subclasses
pytest — a test suite alongside the HTTP surface
A handful of standalone CLI/script entry points outside the HTTP surface

This is exactly the stack codekb's architecture detection is designed to read (see Architecture Detection), which is why it was chosen for this benchmark.

Repo stats¶

224 real files (excluding dependency/VCS/cache directories): 189 Python source files, plus a handful of SQL, JSON, config, and infrastructure files
Total LOC: 28,149 across the 189 Python files (raw line-count sum)
Python is the only language codekb treats as "source" here — the bundle's config scopes parsing to .py by design

1. Parse coverage — 100%, 0 errors¶

190 sources ingested: 189 Python files + one dependency-manifest file (deliberately tagged as repo-config context, not a parse target)
189/189 files parsed successfully, 0 errors, 0 warnings, no inventory warnings
Nothing missing: counts reconcile exactly against an independent file count; non-Python files are absent from the module/symbol output, which is in-scope-by-config, not a gap
Open item: the build summary reports a separate bundle-validation-level warning counter (several thousand, static and hybrid runs differ slightly) that wasn't independently decomposed this session — flagged for a follow-up pass if warning content/severity matters

Verdict: clean, complete parse coverage on this repo — nothing to debug here.

2. Architecture detection accuracy — verified by hand, 100% precision and recall on everything checked¶

Method: hand-counted directly from the real route, model, and schema modules in the target repo (not the bundle), before looking at the tool's output — then checked both directions against the bundle's architecture output

HTTP endpoints — recall 28/28 (100%), precision: 0 fabrications found¶

What was hand-counted, chosen to hit the hard cases the detection design calls out (effective path = mount prefix + router prefix + route path):

A root route and a health-check route at the application entrypoint
Six route modules covering typical CRUD/listing/lookup endpoints, status-count endpoints, state-transition endpoints (e.g. escalate, retry, discard), and a sample-upload endpoint — together roughly 30 of the 28+ hand-verified routes, each under its own URL prefix
Edge case 1 — prefix applied at registration, not at router definition: an auth module defines its router with no prefix; the application's entrypoint module registers that router with a prefix at include_router() time. A route inside that module must resolve to the entrypoint-assigned prefix plus its own path, not an empty prefix.
Edge case 2 — dead router reassignment: a route module assigns a router variable once (unused — no routes ever attached), then immediately reassigns the same variable name to a second router object with a real prefix. Only the second object's route decorator is live and gets included.

Results:

All 28 matched exactly on method, effective path, handler name and line number, including both edge cases — the auth-module route correctly resolved with the prefix attributed to the include-site rather than the router definition, and the reassigned-router route correctly picked up the live router's prefix
Three more route modules verified end-to-end (every endpoint, not a subset): one with 6/6 routes including two "specific-path-before-catchall" routes next to a generic /{id}-style route, which codekb did not confuse; one with 2/2 nested sub-path routes; one with 7/7 routes including two specific paths correctly ordered ahead of a generic /{id} route
A handful of modules whose routes genuinely omit the application's otherwise-standard URL prefix were confirmed correctly reproduced — not a bug, just an inconsistency in the real repo that codekb reproduced faithfully rather than "fixing"
Total: 89 HTTP endpoints + 23 CLI/script entries (correctly kept out of the HTTP view) = 112 endpoints, matching the bundle's summary block. The full list of all 89 HTTP endpoint paths/methods/handlers was scanned for anything implausible — naming, prefixes and line numbers are internally consistent throughout; nothing looked fabricated

Database models — recall and precision both 19/19 (100%), the full set, not a sample¶

The models directory has exactly 19 model classes; all 19 appear in the architecture output's data-models view, each with correct table name, base classes, columns and relationships
Single-table inheritance correctly handled: two models share a base model and correctly show no table name of their own (they inherit the base's table) — spot-checked against source

Data schemas — recall and precision both verified against the full real set¶

The schemas directory has exactly 94 real classes; the architecture output's schema list also has exactly 94 entries, every one mapping 1:1 to a real class at the right file/line
Transitively-derived subclasses (the thing the detection design specifically calls out) captured correctly at multiple inheritance depths:
Three schemas deriving from one shared base schema (one hop)
One schema deriving from another schema which itself derives from the framework's base model (two hops, not flattened incorrectly)
Four schemas deriving from two sibling schemas rather than the base model directly
All of these showed up with the correct base-class field

Two minor bugs found — both pre-existing, not specific to this port¶

A comparison run against the same repo confirmed both predate this codebase entirely:

Phantom router-wiring entry for a dead reassigned router. The router-wiring summary view lists the user-creation route module twice — once for the dead, unused router assignment, once for the real one, both marked as mounted. The actual endpoints list is not affected (exactly one real endpoint there, correctly attributed). A pre-existing limitation in how router identity is keyed (by variable name + each router-construction call site, rather than by final bound value).
Schema count off by one from the actual schema list length. The summary block reports one more schema than the schema list itself actually contains, even though the list's 94 entries exactly match the real repo's 94 classes — per-item detection is fully accurate, only the aggregate count field is miscounting by one.

Both are real but minor (a summary count and a secondary wiring view, not the endpoint/model/schema records a consumer would actually act on) — a follow-up item, not scoped to this verification pass.

Aggregate counts: 112 endpoints, 28 routers, 19 models, 95 (summary count) schemas, 14 dependencies. Combined with the line-by-line hand verification above, this is a solid result on architecture detection for this repo.

3. Relations — 40.4% resolved (4,018/9,936), and the unresolved 59.6% looks correct¶

Schema: relation records carry a resolution status (exact / inferred / unresolved), a predicate (calls/imports/inherits-from/decorated-by/defines/etc.), and a target name or expression
Totals: 9,936 relations — exact 3,592 (36.2%), inferred 426 (4.3%), unresolved 5,918 (59.6%). Resolved = 4,018/9,936 = 40.4%
By predicate: defines, raises, reads, covers, tested_by, writes are all 100% resolved. The unresolved mass concentrates in calls (76% unresolved), imports (67% unresolved), inherits_from (83% unresolved), decorated_by (100% unresolved)

Sample of ~15–20 unresolved relations, checked against real source:

decorated_by unresolved (339/339): test-mocking decorators, instance-bound router method decorators, test-fixture/marker decorators, ORM/validation framework decorators — all external-library or instance-bound, no in-repo symbol to resolve to. Correctly unresolved.
imports unresolved: standard-library and third-party framework symbols (typing, ORM session types, logging, UUID, web-framework and validation-library base classes) — 100% stdlib/third-party in the sample. Correctly unresolved.
inherits_from unresolved (111): the large majority are the validation framework's base model plus stdlib enum/exception base classes (external, correct) — and 17 cases targeting a module-level variable bound to an ORM declarative-base factory call rather than an AST class node, so there's genuinely nothing to resolve to. A real but minor gap — not a resolver bug, more a "doesn't track factory-call-aliased base classes" limitation worth a future enhancement.
calls unresolved (4,329): the large majority are attribute/method calls needing type inference a static pass doesn't attempt. A small number of bare-name calls were cross-checked against all in-repo symbol names — every match had multiple same-named candidates in different modules, genuinely ambiguous. Correctly unresolved for static mining.

Tally: 0 clear hard gaps found in the sample (no unambiguous in-repo symbol that should have resolved but didn't); one soft/structural limitation (factory-call-aliased base classes, 17 relations) worth flagging as a future enhancement.

4. Claim groundedness (static bundle) — 100% exact anchoring, 9/10 sample pass¶

870 knowledge items, each carrying an anchor status. 870/870 (100%) are exact — no item fell back to coarse/approximate grounding; each item's evidence resolves to a precise line-range span, spot-verified by hand
10-item random sample, each checked against the real cited source lines: 9 passed (docstring/behavior matches, exception-handling branches confirmed, multi-step call sequences verified at their real call sites); 1 was noise rather than a factual error — a generic "needs human review" heuristic fired on a trivial, undocumented private test-helper method, apparently triggered by its enclosing class name rather than anything in the method itself
Tally: 9/10 pass, 1/10 noisy/over-triggered heuristic — not a grounding error (the line range was correct in all 10/10), a precision gap in the "flag for human review" heuristic specifically
Coverage note: this check covered the static bundle's 870 items only — the hybrid run's additional 1,230 LLM-sourced claims (below) were not separately spot-checked for accuracy this session

5. Hybrid mining — tested, and it worked¶

What hybrid mining actually adds: static mining only extracts what a rule-based pass over the syntax tree can mechanically spot — a function raises this exception, a route is decorated with that method. Hybrid mining takes that same static output and sends the highest-signal symbols to an LLM to interpret, producing claims about what a piece of code is actually for or does in plain language — the kind of thing a rule-based pattern match can't articulate, but a human reviewer would otherwise have to write by hand. It isn't free-form, though: a hybrid claim is only accepted if it cites a symbol or relation id that static extraction already produced (claims citing invented ids are dropped outright) and quotes real code verbatim from that symbol's own lines (a claim whose quote can't be located is kept but demoted to a review-required tier, not silently trusted). So "it worked" here means something specific: not just that the LLM returned text, but that the tool's own citation and quote-verification checks accepted it as grounded.

Credential resolution: checked the settings resolution order (saved setting > env var > .env). A plausible environment-variable credential was present but superseded by a saved provider configuration, which takes precedence
Cheap test first: built a 1-file, 9-symbol mini-repo copy and ran a hybrid build against it — 6/6 LLM calls completed, 22 claims accepted, 0 rejected — confirmed the credential actually authenticates before committing to a full run
Full hybrid build on all 190 sources completed successfully: 2,135 knowledge items total (870 static + 1,230 LLM-enriched claims from 250 symbols, 39 rejected, via per-symbol enrichment + 35 more claims from 50 procedure-clusters, 15 rejected), validation 0 errors / 7,161 warnings. Runtime a few minutes (concurrent calls, 5 at a time)
What "39 rejected" and "15 rejected" mean: not failed LLM calls — these are claims the LLM proposed that failed the citation or verbatim-quote check above and were discarded rather than kept on trust. A non-zero rejection count here is a sign the safety check is actually doing something, not a sign of a broken run
Caveat carried over from the Bottom line: these 1,230 LLM-enriched claims were confirmed to exist, be well-formed, and pass the citation/quote checks — but unlike the static claims above, they were not independently spot-checked by a human for whether the interpretation itself is accurate, only that it's grounded in real code

Bottom line — why this result is trustworthy¶

The headline numbers, all independently hand-verified

0 parse errors across all 189 real Python files — nothing skipped, nothing silently dropped
100% precision and recall on architecture detection: all 28 hand-picked endpoints, all 19 real database models, and all 94 real data schemas came back correct — right method, right full URL path, right handler
Both trickiest edge cases handled correctly: a route prefix only added where a router is registered, not where it's defined; and a route module defining two different router objects under the same variable name
100% of mined claims (870/870) point at an exact, verifiable source line — not an approximation
9 of 10 independently spot-checked claims were confirmed true and useful against real source
A real LLM connection was confirmed working end to end, nearly tripling the number of mined insights (870 → 2,135) in a full enrichment pass

That's the trust case. In the interest of not cherry-picking, here's what keeps this from being a perfect score:

Relation resolution sits at 40% — not a shortfall, since the other 60% was correctly declined (third-party library calls, genuinely ambiguous same-named functions) rather than guessed at, but it does mean roughly 60% of relationships require a human or an LLM pass to fill in, not a static one
The one claim that didn't pass wasn't a wrong fact — a review-flagging heuristic over-triggered on a trivial test helper — but it's still a real (minor) precision gap
Two small bookkeeping bugs were found (a dead router double-listed in a summary table, one summary count off by one) — neither affects the actual endpoint/model/schema records a consumer would act on, and both turned out to be pre-existing limitations, not something new here
The hybrid run's 1,230 additional LLM-sourced claims were confirmed to exist and be well-formed, but weren't separately spot-checked for accuracy the way the static claims were