evals: fix false-negative regex checks with the agentic LLM judge#95440
Draft
gaojude wants to merge 3 commits into
Draft
evals: fix false-negative regex checks with the agentic LLM judge#95440gaojude wants to merge 3 commits into
gaojude wants to merge 3 commits into
Conversation
Contributor
Stats from current PR🟢 1 improvement
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URLCommit: c4f7be1 |
Contributor
Tests PassedCommit: c4f7be1 |
…t-eval 1.3 Bump @vercel/agent-eval 0.9.5 -> 1.3.0 (ships the agentic judge runtime) and adapt run-evals.js to the 1.3 CLI: `run baseline agents-md --force`, with --dry mapping to the read-only `status` command. Generated experiments now use vercel-ai-gateway/claude-code (default model claude-opus-4-8), so the OIDC token from `vc env pull` is the only credential needed — sandbox, codegen model, and judge. The judge is pinned to claude-haiku-4-5: a cheap fixed grader, identical for every run regardless of the model under test.
d2f29a5 to
a0641c6
Compare
agent-034-async-cookies: /await\s+cookies\(\)/ only matched the naive form and rejected the correct `await Promise.all([cookies(), headers()])`, and the no-sync-call lookbehind wrongly flagged the bare cookies() inside that array. The four await-mechanics tests collapse into one judge criterion (the promises must actually be awaited before use, in any correct form); the theme-cookie and Accept-Language content checks stay regex. agent-030-app-router-migration-hard: the home-page test rejected solutions that didn't match one exact shape (e.g. fetching extracted to a helper means no literal `fetch(` in page.tsx). Its body becomes one judge criterion; the file-existence check and the other seven tests stay deterministic. Both fixtures exclude EVAL.ts from tsconfig: the @vercel/agent-eval/eval import has no type declarations, and `next build` (run by experiments with scripts: ['build']) type-checks the fixture — same reason EVAL.tsx was already excluded.
a0641c6 to
58a13bb
Compare
Eval fixtures are sandbox workspaces with their own package.json/tsconfig, not repo code. Their EVAL.ts files may import modules that only resolve inside the sandbox (@vercel/agent-eval/eval) and are excluded from the fixture tsconfig, so type-aware linting with `project: true` fails to place them in a project. Both ignore lists (eslint.cli.config.mjs and its mirror block) stay in sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the false-negative regex assertions identified in the kindle-alpha eval report by grading those specific checks with
@vercel/agent-eval's agentic LLM judge instead. Only the assertions with demonstrated false negatives change; every deterministic check stays.agent-034-async-cookies:/await\s+cookies\s*\(\s*\)/only matched the naiveawait cookies()and rejected the correct (arguably better)await Promise.all([cookies(), headers()])— 5 of 6 kindle-alpha runs failed on it — and the no-sync-call lookbehind wrongly flagged the barecookies()inside the array. The four await-mechanics tests collapse into one judge criterion: the promises must actually be awaited before use, in any correct form. The theme-cookie and Accept-Language content checks stay regex.agent-030-app-router-migration-hard: the home-page test rejected solutions that didn't match one exact shape (e.g. data fetching extracted to a helper means no literalfetch(inpage.tsx). Its body becomes one judge criterion (async Server Component, fetches during server render, nogetServerSideProps); the file-existence check and the other seven tests stay deterministic.The judge is pinned to
claude-haiku-4-5in the generated experiment configs — a cheap fixed grader, identical for every run regardless of the model under test. Because the grader is a small model, each criterion spells out reference correct and wrong solutions to compare against rather than asking it to reason from first principles (this also cut judge time roughly in half in testing).Supporting changes: bumps
@vercel/agent-eval0.9.5→1.3.0(ships the judge runtime) and adaptsrun-evals.jsto the 1.3 CLI (run baseline agents-md --force;--drymaps tostatus); generated experiments now run through the Vercel AI Gateway sovc env pullis the only credential setup; converted fixtures excludeEVAL.tsfrom tsconfig (the@vercel/agent-eval/evalimport has no type declarations — vercel-labs/agent-eval#166 — andnext buildtype-checks the fixture, so experiments runningscripts: ['build']would otherwise fail on it, same reasonEVAL.tsxwas already excluded). Not converted:agent-040(already redesigned upstream in #94578) andagent-041(no diagnosed false negative in the report).Verified end-to-end from this repo: converted
agent-034with codegenclaude-sonnet-4-5via the AI Gateway in a Vercel sandbox, judge pinned toclaude-haiku-4-5— passes, with the judge clause executing in-sandbox.Running it: