Skip to content

evals: fix false-negative regex checks with the agentic LLM judge#95440

Draft
gaojude wants to merge 3 commits into
canaryfrom
jude/eval-agent-024-agentic-judge
Draft

evals: fix false-negative regex checks with the agentic LLM judge#95440
gaojude wants to merge 3 commits into
canaryfrom
jude/eval-agent-024-agentic-judge

Conversation

@gaojude

@gaojude gaojude commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes the false-negative regex assertions identified in the kindle-alpha eval report by grading those specific checks with @vercel/agent-eval's agentic LLM judge instead. Only the assertions with demonstrated false negatives change; every deterministic check stays.

  • agent-034-async-cookies: /await\s+cookies\s*\(\s*\)/ only matched the naive await cookies() and rejected the correct (arguably better) await Promise.all([cookies(), headers()]) — 5 of 6 kindle-alpha runs failed on it — and the no-sync-call lookbehind wrongly flagged the bare cookies() inside the array. The four await-mechanics tests collapse into one judge criterion: the promises must actually be awaited before use, in any correct form. The theme-cookie and Accept-Language content checks stay regex.
  • agent-030-app-router-migration-hard: the home-page test rejected solutions that didn't match one exact shape (e.g. data fetching extracted to a helper means no literal fetch( in page.tsx). Its body becomes one judge criterion (async Server Component, fetches during server render, no getServerSideProps); the file-existence check and the other seven tests stay deterministic.

The judge is pinned to claude-haiku-4-5 in the generated experiment configs — a cheap fixed grader, identical for every run regardless of the model under test. Because the grader is a small model, each criterion spells out reference correct and wrong solutions to compare against rather than asking it to reason from first principles (this also cut judge time roughly in half in testing).

Supporting changes: bumps @vercel/agent-eval 0.9.5 → 1.3.0 (ships the judge runtime) and adapts run-evals.js to the 1.3 CLI (run baseline agents-md --force; --dry maps to status); generated experiments now run through the Vercel AI Gateway so vc env pull is the only credential setup; converted fixtures exclude EVAL.ts from tsconfig (the @vercel/agent-eval/eval import has no type declarations — vercel-labs/agent-eval#166 — and next build type-checks the fixture, so experiments running scripts: ['build'] would otherwise fail on it, same reason EVAL.tsx was already excluded). Not converted: agent-040 (already redesigned upstream in #94578) and agent-041 (no diagnosed false negative in the report).

Verified end-to-end from this repo: converted agent-034 with codegen claude-sonnet-4-5 via the AI Gateway in a Vercel sandbox, judge pinned to claude-haiku-4-5 — passes, with the judge clause executing in-sandbox.

Running it:

pnpm install                 # picks up @vercel/agent-eval 1.3.0
pnpm --filter=next build     # pnpm eval packs the locally built next
vc env pull .env.local       # the only credential: sandbox + AI Gateway (codegen and judge)

pnpm eval agent-034-async-cookies         # baseline + AGENTS.md variants
pnpm eval agent-034-async-cookies --dry   # preview only

@github-actions

github-actions Bot commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

Stats from current PR

🟢 1 improvement

Metric Canary PR Change Trend
Webpack Build Time 24.080s 23.414s 🟢 666ms (-3%) ▇█▄▆▄
📊 All Metrics
📖 Metrics Glossary

Dev Server Metrics:

  • Listen = TCP port starts accepting connections
  • First Request = HTTP server returns successful response
  • Cold = Fresh build (no cache)
  • Warm = With cached build artifacts

Build Metrics:

  • Fresh = Clean build (no .next directory)
  • Cached = With existing .next directory

Change Thresholds:

  • Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
  • Size: Changes < 1KB AND < 1% are insignificant
  • All other changes are flagged to catch regressions

⚡ Dev Server

Metric Canary PR Change Trend
Cold (Listen) 814ms 814ms ✓ ▁▂▁▂▁
Cold (Ready in log) 800ms 810ms ✓ ▁▁▁▁▁
Cold (First Request) 1.375s 1.391s ✓ ▂▂▁▂▂
Warm (Listen) 814ms 815ms ✓ ▂▂▁▂▁
Warm (Ready in log) 806ms 809ms ✓ ▁▁▁▁▁
Warm (First Request) 640ms 641ms ✓ ▁▁▁▁▂
📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric Canary PR Change Trend
Cold (Listen) 760ms 759ms ✓ ██▁██
Cold (Ready in log) 742ms 739ms ✓ ██▅▇▆
Cold (First Request) 3.195s 3.189s ✓ ▇█▄▆▅
Warm (Listen) 760ms 759ms ✓ ██▁██
Warm (Ready in log) 742ms 739ms ✓ ██▃▆█
Warm (First Request) 3.221s 3.255s ✓ ▇▅▁▄▆

⚡ Production Builds

Metric Canary PR Change Trend
Fresh Build 5.611s 5.572s ✓ ▁▁▁▂▁
Cached Build 5.900s 5.879s ✓ ▁▁▁▂▁
📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric Canary PR Change Trend
Fresh Build 24.080s 23.414s 🟢 666ms (-3%) ▇█▄▆▄
Cached Build 24.305s 24.003s ✓ ▇█▃▅▆
node_modules Size 514 MB 514 MB ✓ █████
📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles
Canary PR Change
034zmn7v4tsj3.js gzip 53.5 kB N/A -
03gmyxr7dukvs.js gzip 10.3 kB N/A -
03tt4thpkf_py.js gzip 156 B N/A -
040t1q17lo593.js gzip 8.77 kB N/A -
0bb0vtp8dwsez.js gzip 450 B N/A -
0cz1d0mv5g_q7.js gzip 39.4 kB 39.4 kB ✓
0nxqz3q165m2l.js gzip 220 B N/A -
0qr-1tnb7_hg5.js gzip 5.72 kB N/A -
0uilzvklme4ea.js gzip 8.76 kB N/A -
0v0wmwk2-r35h.js gzip 160 B N/A -
0z0y83mfx-l-h.js gzip 10 kB N/A -
13mqkxitd_mp-.js gzip 156 B N/A -
15ok8ydf274z0.js gzip 10.6 kB N/A -
16cqqcigykfea.js gzip 13.6 kB N/A -
17h18zxt1915l.js gzip 1.46 kB N/A -
1elt1qium-r2m.css gzip 115 B 115 B ✓
1fq1h7dxad4cv.js gzip 9.44 kB N/A -
1fym-adbk_69_.js gzip 13.1 kB N/A -
1gav7nfh4y8ln.js gzip 8.76 kB N/A -
1gvnrogz-nk26.js gzip 157 B N/A -
1nm0awhqye5p8.js gzip 154 B N/A -
1nwugjw0i93us.js gzip 156 B N/A -
1rhk5y2vvnyud.js gzip 169 B N/A -
1s8oolynvqjk6.js gzip 155 B N/A -
1zxeepg375efo.js gzip 14.3 kB N/A -
2_1mg3cv86xn-.js gzip 156 B N/A -
29du04jy1wjq7.js gzip 7.65 kB N/A -
2ayx44yp1i407.js gzip 71 kB N/A -
2rypqffktcxqh.js gzip 8.69 kB N/A -
2uqsizxtehgqf.js gzip 8.79 kB N/A -
2vzktdqwbijqc.js gzip 8.74 kB N/A -
2y2m2aowi7lgb.js gzip 161 B N/A -
38wdlbypioegf.js gzip 153 B N/A -
3c244akvx_-be.js gzip 8.74 kB N/A -
3czh4acsysft-.js gzip 155 B N/A -
3dmlemitx3ef3.js gzip 2.29 kB N/A -
3j3wt_5y9cpjx.js gzip 65.5 kB N/A -
3oq7la2gelh4n.js gzip 155 B N/A -
3umfpv6uzk6cg.js gzip 13.2 kB N/A -
4237vdj6eyslt.js gzip 8.7 kB N/A -
turbopack-0_..0e0k.js gzip 3.74 kB N/A -
turbopack-00..ji86.js gzip 3.74 kB N/A -
turbopack-09..5el4.js gzip 3.74 kB N/A -
turbopack-0o..iy-c.js gzip 3.74 kB N/A -
turbopack-0s..7rtz.js gzip 3.74 kB N/A -
turbopack-0v..ep2g.js gzip 3.74 kB N/A -
turbopack-0x..-y72.js gzip 3.74 kB N/A -
turbopack-11..68z6.js gzip 3.75 kB N/A -
turbopack-1w..lt0k.js gzip 3.74 kB N/A -
turbopack-2b..8rwo.js gzip 3.72 kB N/A -
turbopack-2l..nquv.js gzip 3.74 kB N/A -
turbopack-2w..sjcd.js gzip 3.74 kB N/A -
turbopack-32..ss4y.js gzip 3.74 kB N/A -
turbopack-3t..7tg5.js gzip 3.74 kB N/A -
00tc16j5-eos9.js gzip N/A 1.46 kB -
04k1dnw6y-sgq.js gzip N/A 160 B -
0gaw1gmpwuwkn.js gzip N/A 8.8 kB -
0n8vgrgx6gsx2.js gzip N/A 65.5 kB -
0rbtcmwiuz-3x.js gzip N/A 157 B -
1_jfzs24krjbo.js gzip N/A 156 B -
12zyjjmq4oad1.js gzip N/A 5.72 kB -
1cfirxwnd7m9h.js gzip N/A 161 B -
1d14swk91dglf.js gzip N/A 157 B -
1k9cc4-gpa2cx.js gzip N/A 13.1 kB -
1n59uogejo81s.js gzip N/A 10.3 kB -
1npei3hlexdv3.js gzip N/A 9.99 kB -
1rvrfyu27qc4g.js gzip N/A 53.5 kB -
1ssuiwj_rnpc5.js gzip N/A 8.77 kB -
1wb808_e_-w3u.js gzip N/A 155 B -
2-phc9obd3nyd.js gzip N/A 8.77 kB -
23gji-p3v7u-m.js gzip N/A 14.3 kB -
24utq57r3l1cw.js gzip N/A 8.7 kB -
28r3lfbanlqsz.js gzip N/A 154 B -
2fhjq6ek3srmr.js gzip N/A 8.69 kB -
2g0t7upz_v-hf.js gzip N/A 10.6 kB -
2ia9qqy99aie9.js gzip N/A 13.6 kB -
2oza28kdqvoay.js gzip N/A 8.74 kB -
2petq82fjj2-5.js gzip N/A 71 kB -
2rv500jdndh2l.js gzip N/A 156 B -
2ty8muxbqewy-.js gzip N/A 154 B -
2zh8-mmhujy8o.js gzip N/A 2.29 kB -
30h_m0irj-q6p.js gzip N/A 221 B -
37ejbfx1u9lpo.js gzip N/A 8.75 kB -
38c63ct4i1cy7.js gzip N/A 449 B -
396nht7itx-uq.js gzip N/A 153 B -
3fospa9508caf.js gzip N/A 155 B -
3hd1us4ordyc9.js gzip N/A 13.2 kB -
3jimpal3lvz5e.js gzip N/A 9.44 kB -
3rc2knzmbwc3u.js gzip N/A 7.65 kB -
3sopw705rkml5.js gzip N/A 157 B -
3urjgcevdqj17.js gzip N/A 169 B -
3wb4pe4z4-aim.js gzip N/A 8.77 kB -
turbopack-05..bfnr.js gzip N/A 3.74 kB -
turbopack-07..idrr.js gzip N/A 3.74 kB -
turbopack-08..mpl7.js gzip N/A 3.74 kB -
turbopack-0l..kcaf.js gzip N/A 3.72 kB -
turbopack-0s..944p.js gzip N/A 3.74 kB -
turbopack-1k..sc_y.js gzip N/A 3.74 kB -
turbopack-2_..ibpn.js gzip N/A 3.74 kB -
turbopack-2l..lm-v.js gzip N/A 3.74 kB -
turbopack-37..fdi3.js gzip N/A 3.75 kB -
turbopack-39..5fjh.js gzip N/A 3.74 kB -
turbopack-3o..3o_l.js gzip N/A 3.74 kB -
turbopack-3t..fc26.js gzip N/A 3.74 kB -
turbopack-3u..0ekr.js gzip N/A 3.74 kB -
turbopack-40..fson.js gzip N/A 3.74 kB -
Total 466 kB 466 kB ⚠️ +3 B

Server

Middleware
Canary PR Change
middleware-b..fest.js gzip 731 B 723 B 🟢 8 B (-1%)
Total 731 B 723 B ✅ -8 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 433 B 436 B ✓
Total 433 B 436 B ⚠️ +3 B

📦 Webpack

Client

Main Bundles
Canary PR Change
2486.HASH.js gzip 169 B N/A -
3146-HASH.js gzip 63.6 kB N/A -
39fcf99b-HASH.js gzip 62.9 kB N/A -
8443-HASH.js gzip 4.68 kB N/A -
9431-HASH.js gzip 5.62 kB N/A -
framework-HASH.js gzip 59.8 kB 59.8 kB ✓
main-app-HASH.js gzip 255 B 253 B ✓
main-HASH.js gzip 39.6 kB 40.1 kB 🔴 +455 B (+1%)
webpack-HASH.js gzip 1.68 kB 1.68 kB ✓
6105-HASH.js gzip N/A 5.63 kB -
764.HASH.js gzip N/A 169 B -
8898-HASH.js gzip N/A 63.1 kB -
9597-HASH.js gzip N/A 4.65 kB -
e1ccab69-HASH.js gzip N/A 62.9 kB -
Total 238 kB 238 kB ✅ -68 B
Polyfills
Canary PR Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB ✓
Total 39.4 kB 39.4 kB ✓
Pages
Canary PR Change
_app-HASH.js gzip 194 B 194 B ✓
_error-HASH.js gzip 183 B 182 B ✓
css-HASH.js gzip 335 B 335 B ✓
dynamic-HASH.js gzip 1.8 kB 1.8 kB ✓
edge-ssr-HASH.js gzip 255 B 254 B ✓
head-HASH.js gzip 351 B 349 B ✓
hooks-HASH.js gzip 384 B 384 B ✓
image-HASH.js gzip 580 B 581 B ✓
index-HASH.js gzip 259 B 259 B ✓
link-HASH.js gzip 2.49 kB 2.49 kB ✓
routerDirect..HASH.js gzip 319 B 319 B ✓
script-HASH.js gzip 386 B 386 B ✓
withRouter-HASH.js gzip 315 B 313 B ✓
1afbb74e6ecf..834.css gzip 106 B 106 B ✓
Total 7.96 kB 7.95 kB ✅ -6 B

Server

Edge SSR
Canary PR Change
edge-ssr.js gzip 127 kB 127 kB ✓
page.js gzip 284 kB 284 kB ✓
Total 410 kB 411 kB ⚠️ +962 B
Middleware
Canary PR Change
middleware-b..fest.js gzip 615 B 616 B ✓
middleware-r..fest.js gzip 156 B 155 B ✓
middleware.js gzip 45 kB 44.9 kB ✓
edge-runtime..pack.js gzip 842 B 842 B ✓
Total 46.6 kB 46.5 kB ✅ -104 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 719 B 719 B ✓
Total 719 B 719 B ✓
Build Cache
Canary PR Change
0.pack gzip 4.75 MB 4.73 MB 🟢 11.1 kB (0%)
index.pack gzip 117 kB 117 kB ✓
index.pack.old gzip 116 kB 116 kB ✓
Total 4.98 MB 4.97 MB ✅ -11.2 kB

🔄 Shared (bundler-independent)

Runtimes
Canary PR Change
app-page-exp...dev.js gzip 364 kB 364 kB ✓
app-page-exp..prod.js gzip 200 kB 200 kB ✓
app-page-tur...dev.js gzip 364 kB 364 kB ✓
app-page-tur..prod.js gzip 200 kB 200 kB ✓
app-page-tur...dev.js gzip 360 kB 360 kB ✓
app-page-tur..prod.js gzip 198 kB 198 kB ✓
app-page.run...dev.js gzip 360 kB 360 kB ✓
app-page.run..prod.js gzip 198 kB 198 kB ✓
app-route-ex...dev.js gzip 79.1 kB 79.1 kB ✓
app-route-ex..prod.js gzip 53.6 kB 53.6 kB ✓
app-route-tu...dev.js gzip 79.1 kB 79.1 kB ✓
app-route-tu..prod.js gzip 53.6 kB 53.6 kB ✓
app-route-tu...dev.js gzip 78.7 kB 78.7 kB ✓
app-route-tu..prod.js gzip 53.4 kB 53.4 kB ✓
app-route.ru...dev.js gzip 78.7 kB 78.7 kB ✓
app-route.ru..prod.js gzip 53.4 kB 53.4 kB ✓
dist_client_...dev.js gzip 324 B 324 B ✓
dist_client_...dev.js gzip 326 B 326 B ✓
dist_client_...dev.js gzip 318 B 318 B ✓
dist_client_...dev.js gzip 317 B 317 B ✓
pages-api-tu...dev.js gzip 45.2 kB 45.2 kB ✓
pages-api-tu..prod.js gzip 33.9 kB 33.9 kB ✓
pages-api.ru...dev.js gzip 45.2 kB 45.2 kB ✓
pages-api.ru..prod.js gzip 33.9 kB 33.9 kB ✓
pages-turbo....dev.js gzip 54.7 kB 54.7 kB ✓
pages-turbo...prod.js gzip 39.6 kB 39.6 kB ✓
pages.runtim...dev.js gzip 54.6 kB 54.6 kB ✓
pages.runtim..prod.js gzip 39.5 kB 39.5 kB ✓
server.runti..prod.js gzip 65 kB 65 kB ✓
use-cache-pr...dev.js gzip 71.3 kB 71.3 kB ✓
use-cache-pr...dev.js gzip 71.3 kB 71.3 kB ✓
use-cache-pr...dev.js gzip 69.6 kB 69.6 kB ✓
use-cache-pr...dev.js gzip 69.6 kB 69.6 kB ✓
Total 3.47 MB 3.47 MB ⚠️ +1 B
📎 Tarball URL
https://vercel-packages.vercel.app/next/commits/c4f7be15e7fe7e700dbcfb8fb33796ce63acb871/next

Commit: c4f7be1

@github-actions

github-actions Bot commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

Tests Passed

Commit: c4f7be1

…t-eval 1.3

Bump @vercel/agent-eval 0.9.5 -> 1.3.0 (ships the agentic judge runtime) and
adapt run-evals.js to the 1.3 CLI: `run baseline agents-md --force`, with
--dry mapping to the read-only `status` command.

Generated experiments now use vercel-ai-gateway/claude-code (default model
claude-opus-4-8), so the OIDC token from `vc env pull` is the only credential
needed — sandbox, codegen model, and judge. The judge is pinned to
claude-haiku-4-5: a cheap fixed grader, identical for every run regardless of
the model under test.
@gaojude gaojude force-pushed the jude/eval-agent-024-agentic-judge branch from d2f29a5 to a0641c6 Compare July 3, 2026 03:03
@gaojude gaojude changed the title evals: convert agent-024 to the agentic LLM judge evals: fix false-negative regex checks with the agentic LLM judge Jul 3, 2026
agent-034-async-cookies: /await\s+cookies\(\)/ only matched the naive form and
rejected the correct `await Promise.all([cookies(), headers()])`, and the
no-sync-call lookbehind wrongly flagged the bare cookies() inside that array.
The four await-mechanics tests collapse into one judge criterion (the promises
must actually be awaited before use, in any correct form); the theme-cookie
and Accept-Language content checks stay regex.

agent-030-app-router-migration-hard: the home-page test rejected solutions
that didn't match one exact shape (e.g. fetching extracted to a helper means
no literal `fetch(` in page.tsx). Its body becomes one judge criterion; the
file-existence check and the other seven tests stay deterministic.

Both fixtures exclude EVAL.ts from tsconfig: the @vercel/agent-eval/eval
import has no type declarations, and `next build` (run by experiments with
scripts: ['build']) type-checks the fixture — same reason EVAL.tsx was
already excluded.
@gaojude gaojude force-pushed the jude/eval-agent-024-agentic-judge branch from a0641c6 to 58a13bb Compare July 3, 2026 03:07
Eval fixtures are sandbox workspaces with their own package.json/tsconfig, not
repo code. Their EVAL.ts files may import modules that only resolve inside the
sandbox (@vercel/agent-eval/eval) and are excluded from the fixture tsconfig,
so type-aware linting with `project: true` fails to place them in a project.
Both ignore lists (eslint.cli.config.mjs and its mirror block) stay in sync.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant