Post-Training a Professional Chip Coding Model
We post-trained a coding model for chip design and evaluated it on NVIDIA’s CVDP benchmark. This post shows the results, the failures we found by reading every failed run against the hidden tests, and how we train with SFT and RL.
- Our model, CVDP pass rate
- 88.3%
- Testbench checkers & SVA
- +3.5 pt
- Cost & latency per problem
- $0.02 · 11s
mean of 8 categories; +2.2 points above Claude Opus 5.5 (pass@1)
widest lead over frontier generalist LLMs on cycle-accurate verification
120× cheaper and 4× faster than Opus 5.5 ($2.40 · 45s); private on-premise inference ready
The benchmark & results
NVIDIA’s Comprehensive Verilog Design Problems (CVDP, Pinckney et al., 2025) has 783 problems written by chip designers in 13 categories: RTL from spec, RTL changes, CDC and lint clean-up, bug fixing, testbench stimulus, SystemVerilog Assertions (SVA) and testbench checkers.
Our model scores higher than Claude Opus 5.5 and GLM 5.3 in all eight categories we evaluated. The widest margin is on testbench checkers, and it runs at a fraction of the cost and latency.
CVDP pass rate
mean of 8 industrial categories · higher is better
Our model
Opus 5.5
GLM 5.3Verification tasks
stimulus, checkers, SVA assertions · higher is better
Our modelOpus 5.5GLM 5.3Inference latency
seconds per problem · lower is better
Our modelGLM 5.3Opus 5.5Inference cost
USD per problem · lower is better
Our modelGLM 5.3Opus 5.5| Task | GLM 5.3 | Opus 5.5 | Our model |
|---|---|---|---|
Write and change RTLSpec to RTL cid03 | 87.4% | 89.1% | 92.1% |
Modify existing RTL cid04 | 85.9% | 86.8% | 88.2% |
Lint and QoR clean-up cid07 | 89.6% | 90.3% | 91.4% |
Complete partial RTL cid02 | 83.8% | 85.2% | 87.5% |
DebugBug fixing cid16 | 90.7% | 92.1% | 93.8% |
VerifyTestbench stimulus cid12 | 81.6% | 83.4% | 85.1% |
Assertions (SVA) cid14 | 79.5% | 81.7% | 84.3% |
Testbench checker cid13 | 77.8% | 80.2% | 83.7% |
Quality improvement was applied only to the CVDP benchmark, because too many of its tasks are defective.
Why frontier coding models struggle with chip code
Coding models got good on software because software gives fast, honest feedback: run the tests, read the stack trace, try again. Chip code weakens each part of that loop.
- Sequential logic is where the gap opens. Models that sit close together on ordinary software separate once the code has to hold state across clock edges.
- A passing testbench is weak evidence. A bench can accept a design that is functionally wrong. The check is the hard part.
- The simulator, not the editor, drives progress. A run moves forward when it simulates, and stalls when it only searches and edits. Software agent habits do not transfer as they are.
- More simulation does not close the gap; cycle-exact checking does. Extra simulation still fails when the bench never counts latency, or takes its expected values from the model’s own reading of the spec instead of the numbers the spec gives.
- The grader is scarcer than the data. The RTL, specs and tool logs sit behind NDAs, and so do the checks needed to grade an agent. Sign-off judgment lives in engineers’ heads, and turning it into machine-checkable rules is a large engineering job of its own.
What is special about chip tasks
Five properties of hardware have no close match in software, and each one shows up in how models fail.
- Time is part of the answer. A design with the right datapath and one extra register stage is wrong. Latency and handshakes are checked cycle by cycle.
- The bench runs alongside the design. A testbench that drives and samples on the same clock edge races the design it checks. When that happens, the fix belongs in the bench, not the RTL.
- Simulators disagree. 2-state and 4-state simulation, event ordering and tool versions all differ. A design is only correct with respect to the tool that signs it off.
- “No behavior change” is a strict promise. A lint or clean-up change must keep every cycle the same, including resets and pipeline depth.
- The interface is fixed by the rest of the chip. Port names, file names, the skeleton’s registers and the style of sibling modules are a contract with code the model never sees.
How frontier models fail on each, with examples from our runs
| Failure | What happens | Seen in our runs |
|---|---|---|
| Pipeline latency off by one cycle | The datapath is right, but an output or handshake lands one cycle early or late: an extra register stage, a dropped stage, or always_comb where the spec needs a flop. | MSHR: allocate_ready drops one cycle late (17 vs 16). DES: one extra stage on o_done (49 edges vs 48). Booth multiplier: the model’s own testbench counts latency one short, so it adds a stage to correct RTL (6 → 7) and ignores the spec’s “actual = 5”. |
| RTL bent to fit a testbench race | The visible testbench drives and samples on the same edge. Instead of suspecting the bench, the model changes the clock edge, the assignment type or the register count until it passes. | Universal shift register: RTL rewritten to negedge with blocking assignments to pass a racy bench; the hidden test fails. low_power_channel: GLM adds rd_data_q and fails; Sonnet calls the race, leaves the RTL alone, and passes. |
| Simulator semantics hide the bug | The model checks in one simulator only. 2-state simulation has no X, event ordering differs between tools, and each tool version accepts a different dialect. | A = 1'b0 initializer on a reset delay chain removes the falling edge of an async reset, so done never comes; Verilator cannot show it. iverilog 13 rejects code that 12 accepts; a lint warning appears only in the grader’s newer Verilator. |
| Behavior change in a no-behavior-change task | On lint and cleanup tasks, the model changes cycle behavior while clearing warnings, or adds resets and “robustness” the original never had. | Image rotate (lint only): valid pipeline widened from [5:0] to [6:0], latency 7 → 8. A reset added to a packet-ID register the original design leaves unreset. |
| Interface contract broken | Port names, file names, the skeleton’s registers and sibling modules are the contract. The model treats them as suggestions. | interrupt_enable renamed to intr_enable; .v written where the flow expects .sv; immediates zero-extended where the ISA sign-extends. The run that passed copied the o_ce style of an untouched sibling decoder. |
One pattern cuts across the table. In 4 of the 11 tasks the teacher solved and the student did not, an earlier snapshot of the student’s RTL already passed the hidden test. The model then broke it: to fit a racy bench, to match a miscounted self-test, or with an optional refactor just before the time limit.
How we train it
Most of the work goes into the training signal. If the grader or the task is wrong, the model learns the error. Matching the grader and auditing the tasks take most of the engineering time.
Same simulators and an isolated workspace for every run, so a pass in training means a pass at grading.
Broken tasks are fixed or removed before any model learns from them.
Passing runs from strong models, with the bad turns masked out.
Reward is the hidden test, pass or fail.
Groundwork
- Grade exactly like the grader. Our sandbox runs the same simulator builds as NVIDIA’s grading image, and every grading run has its own workspace. Before we fixed both, one evaluation had 19 false fails and 4 false passes.
- Audit the tasks. Broken tasks are fixed or removed before any model trains or is scored on them.1
1 Each unsolved task is checked by hand against its hidden test; the split is by design family, and a formal equivalence check backs up simulation where a golden design exists. We will open-source the CVDP tasks we fixed. Link coming soon.
SFT data
The traces matter. So does where they come from: a benchmark whose checks are real, and tasks that are actually hard. A pass on an easy or broken task is a bad lesson.
- A broken task teaches the wrong thing. When the hidden test is wrong, ambiguous, or too weak to reject a bad design, SFT and RL both learn that mistake. The audit above keeps those tasks out of the data.
- The hard tasks are the ones that count. RTL that compiles is the easy end of this benchmark. The hard end is a checker or an assertion that fails on a wrong design, and an edit to existing RTL that keeps every cycle the same. Easy passes are common. They do not get to crowd out the hard categories.
- A green run can still be a bad teacher. The hidden test can pass after the model weakens its own bench, copies expected values out of the RTL, or changes the design to fit a race. We train on the turns where the check is one a designer would trust. A turn that fails that check stays in context, and the model does not learn it.
RL
The reward is the hidden test, pass or fail. The model never sees that test and cannot edit it.
- What this reward moves. About three quarters of the gain came from compile failures, which fell from 25% to 7%. Functional errors barely moved.
- Pass or fail does not say what failed. Latency, a handshake or a broken contract can all hide inside the same bit. More RL on this reward keeps repairing runs that were close to compiling. Cycle-exact behavior has to already be in the SFT data.
On a chip team’s own flow, the same loop runs on-premise. The grader becomes the team’s simulators, formal and lint checks, and the tasks come from its RTL and regression history.
Sources
- Comprehensive Verilog Design Problems, Pinckney et al., NVIDIA, 2025.
Want to see this loop on your own flow? Book a call
All research