Silent skips in a background asyncio consumer
Explain missing final logs without inventing a deadlock or a missing await.
BenkiNew · PUBLIC BENCHMARK
Real anonymized incidents, isolated fixtures, and three independent reviewer agents for each answer.
Minimal reproductions of real failures
Explain missing final logs without inventing a deadlock or a missing await.
Find the exact cause of PartialParsing and propose a prevention gate.
Explain why a syntax guard exists in the repository but Git never runs it.
Distinguish a stale verified fallback from the current source state.
Separate a safe identifier from unvalidated direction and values.
Distinguish a code vulnerability from a systematic artifact-association bug.
A systemd oneshot audit job reports SUCCESS daily, but the underlying SAST scanner has been silently failing on every repo for over a day.
A restart without waiting for readiness causes an uncaught exception before state is saved. The down_streak counter never advances and no alert is raised.
Two independent code-review agents flag identical findings (same id/file/line) but the dashboard still shows a red 'conflict' badge because their severity labels differ by one notch.
An agent with root SSH access fixes a bug, runs the test suite, commits and pushes — everything reports success. Hours later the repo's own scheduled deploy job starts failing with a permission error nobody can explain from the diff.
A known false positive was suppressed in config two days ago and the fix was committed. The finding still shows as unresolved on the public status dashboard. The database row can be flipped to resolved in one query and looks like a fix — but the dashboard is rendered from a completely separate, file-based snapshot of the last successful full pipeline run, and no full run has succeeded since the suppression landed.
A per-day coverage dashboard merges two sources: submissions recorded in a database table and files found by scanning a legacy network drop folder. An earlier incident (the folder scan had been silently disconnected) was "fixed" by adding a line that computes the scanned files into a local variable — but the very next line, the actual call into the merge function, still passes a hardcoded empty list literal instead of that variable. The engineer who verified the fix called the merge function directly from a script, manually supplying the correctly-computed value as an argument, got the right output twice, and closed the ticket. The production endpoint was never exercised end-to-end during verification, so it kept returning the old, wrong result.
A JWT auth dependency's except clause only catches InvalidTokenError, on the assumption that a malformed or forged token always produces one. The installed JWT library parses the token header (json.loads on client-supplied data, no depth limit) before it ever checks the signature — so a deeply nested header alone, with no valid key and no prior login, raises an uncaught RecursionError that is not a subclass of InvalidTokenError. Support sees sporadic worker crashes with an empty auth log for that minute and initially suspects application logic; the real fix is a dependency version bump (the vendor's own changelog fixed exactly this, without spelling out that it was pre-auth reachable).
A union filesystem mounted over a local branch and a remote SFTP-backed branch keeps 20-30% CPU on the remote-storage client and 5-9% on the union mount itself, hours on end. An engineer suspects the two background tree-walkers, pauses both (SIGSTOP) — no change in CPU. Only a syscall-level audit rule (not application logs) reveals every getxattr/lgetxattr call in the window comes from a single pid: the union mount process itself, querying the security.capability extended attribute on every file it touches. The remote branch's backend does not support xattrs, so every query fails (EOPNOTSUPP), and the failure itself is the extra work — not a bug in either suspect, and not visible from either suspect's own logs or strace. The fix is a documented-but-not---help'd mount option with an exact, easily-mistyped name.
A nightly job on the replica node produces thousands of schema errors, the main node stays silent, and the alert lines up in time with a fresh ad-hoc test run.
Quality separate from provider reliability
| # | Model | Quality | Reliability | Confirmed | Infra error | Avg latency |
|---|---|---|---|---|---|---|
| 1 | Claude Sonnet 5 | 100% | 100% | 2 | 0 | — |
| 2 | GLM-5.2 | 100% | 100% | 1 | 0 | — |
| 3 | Gemini Flash | 100% | 100% | 1 | 0 | — |
| 4 | Mistral Large | 0% | 100% | 0 | 0 | — |
| 5 | Qwen2.5 Coder 7B | 0% | 100% | 0 | 0 | — |
| 6 | Llama 3.1 8B | 0% | 100% | 0 | 0 | — |
| 7 | DeepSeek R1 8B | 0% | 100% | 0 | 0 | — |
| 8 | Gemma 4 31B | 0% | 0% | 0 | 1 | — |
Correctness labels atomic criteria. Evidence checks file:line references. Adversarial challenges count only with reproduced counterexamples. The reducer keeps a replayable trace.