When Zero-Knowledge Proof Benchmarks Don't Account for Witness Generation

Benchmarks lie. Not intentionally—they just leave out the messy part. When you compare zero-knowledge proof systems by looking at prove time and verify time, you're missing the step that often eats the most clock cycles: witness generation. The witness is the secret input that satisfies the circuit, but in production it has to be assembled from scattered data, parsed, and checked. That's real work. And it's different for every application.

I've watched teams pick a scheme based on a slick benchmark dashboard, only to find their actual throughput is a fraction of what they expected. The culprit? They assumed the witness was free. This article walks through where witness generation hides, why it varies wildly, and how to build benchmarks that reflect reality.

Where Witness Generation Hits Real Work

Private transaction pools and mempool reconstruction

Most teams optimize for the proving phase—how fast can you generate a zk-SNARK? Then they deploy to mainnet and discover that the real bottleneck lives somewhere else entirely. I have seen a privacy-focused DEX spend three weeks tuning their Groth16 prover only to realize that every trade required the sequencer to reconstruct the entire private mempool from scratch. That reconstruction is witness generation. The DEX stored encrypted order data off-chain, but the zk-circuit needed the full set of unopened orders to prove correct matching. Each new block meant deserializing hundreds of ciphertexts, verifying each decryption key's validity, then ordering the revealed values. That sequence alone ate 4.2 seconds—the actual proof took 0.8. The team shipped on testnet anyway. Latency killed them. A competing centralized aggregator with no privacy guarantees settled orders in 200 milliseconds. Users noticed.

The tricky part is that mempool reconstruction doesn't scale linearly with transaction count. It scales with the number of contending transactions—the ones that could influence ordering within a time window. Double that number and witness generation can triple, because each new candidate forces re-checks against earlier commitments. Most benchmark papers assume a flat file of already-processed witnesses. Real life gives you a torrent of raw, unauthenticated data that must be parsed, validated, and bound into the exact structure the circuit expects. Wrong order? The constraint system rejects the whole block. Not yet.

Identity credential aggregation in verifiable credentials

Consider a healthcare credential aggregator: a mobile app collects three separate identity attestations—a medical license, a board certification, and a hospital affiliation. The proving system should output a single zero-knowledge proof that the user holds all three without revealing any individual credential's details. That sounds fine until you actually build it. Each credential arrives in a different format—JSON-LD, raw X.509, a custom binary blob—and each must be parsed, cryptographically verified against its issuer's public key, and then serialized into the circuit's constraint-friendly representation.

We fixed this by writing a pre-compilation layer that normalizes all inputs into a shared intermediate representation. The catch: normalization introduced its own overhead. One credential had embedded timestamps in a locale-specific date format. The parser failed silently, the circuit treated the missing field as zero, and the proof validated against garbage data. That was a six-hour debugging session. Honestly, the proof generation itself was never the problem—it ran in 700 milliseconds. Witness assembly and validation took 3.1 seconds, and that's before any aggregation logic began. Most teams stop measuring after the proof lands. They don't log the time spent in witness assembly. That hides the real cost.

“We cut proving latency by 60% in two weeks, then lost three months to a witness bug that only surfaced under mainnet load.”

— A lead engineer at a medical-credential startup, during a post-mortem I attended.

Rollup state proof assembly in zk-rollups

The bottleneck shifts again. In a zk-rollup, the state transition requires the prover to read the entire Merkle-Patricia trie of account balances—hundreds of thousands of leaf nodes. The proving system can handle 10,000 constraints per second. Witness generation, however, demands that every leaf be fetched from disk, hashed, and inserted into the correct position in a sparse Merkle tree. That's I/O bound. On a typical cloud instance, disk reads for a 2 GB state trie cost 800 milliseconds per batch. The proof itself finishes in 400. Double the trie depth and witness generation jumps to 1.6 seconds while proving barely moves.

What usually breaks first is the sequential dependency: the circuit expects a sorted array of account updates, but the raw transaction log arrives interleaved. Sorting 50,000 updates to match the trie order takes 1.1 seconds on a single core. A naive parallel sort violates the ordering assumptions the circuit's constraints depend on, producing an invalid witness. That hurts. One rollup team I consulted for tried to parallelize by hashing subtrees independently, then stitching them together. It worked—until a hash collision between two subtrees created a duplicate leaf in the witness. The proof verified. The state was wrong. They reverted to a centralized sequencer for three months while rewriting the witness assembly pipeline.

The through-line across these three scenarios is identical: everyone optimizes the prover first. The prover is the sexy part. Witness generation is plumbing. But plumbing leaks first. Measure it. Log it. If your witness assembly takes more than 60% of your total proving time, the fix is not a faster GPU—it's a leaner, more I/O-conscious data pipeline. Run that experiment tomorrow.

What Most People Get Wrong About Witnesses

Witness vs. circuit vs. instance: the triad

Most teams can recite the ZK trilogy: setup, prove, verify. They memorize it for interviews, diagram it in RFCs, and then ship a system where witness generation is just 'that thing we do before proving.' Wrong order. Witness generation sits between the instance — the public statement everyone agrees on — and the circuit, the fixed arithmetic representation of your computation. The triad isn't parallel; it's serial. Instance is static, circuit is rigid, but the witness is the only piece that moves with every single execution. I have seen teams treat a witness like a bag of bytes you can grab off a database — and then wonder why their prover stalls for forty seconds. That hurts.

The distinction bites hardest when you change proof systems. Groth16 expects a witness structured as a single field-element vector, flattened and ordered to match the circuit's constraint layout. Switch to PLONK and your witness becomes a set of polynomials evaluated over a domain — same logical data, completely different physical representation. The tricky part is that your application code probably serializes the witness as JSON or protobuf, and then the ZK layer expects it as raw finite-field elements. That translation? That is witness generation, and it's where most benchmarks lie. Prove time stays fast; witness prep time explodes.

Why witness structure changes between proof systems

Take a Merkle proof. In a vanilla SNARK, your witness might be a vector of hashes plus the leaf index — simple, flat, easy to feed into a constraint. In a recursive proof system, that same Merkle witness becomes a sequence of hash-circuit sub-witnesses, each one itself a full execution trace. The structure inverts: instead of a single flat vector, you get nested blobs that demand recursive generation. The catch is that most teams pick their proof system for peak prover speed, benchmark that peak, and never measure the cost of building those nested witnesses. Then they hit production and the seam blows out — witness generation takes longer than proving itself.

One concrete pattern I've debugged: a team used Groth16 for a cross-chain bridge, with a witness that required fetching 400 storage proofs from a light client. Their benchmark proved in 1.2 seconds. Witness generation? No benchmark. In production, each storage proof required three RPC round trips, one Merkle verification, and a re-serialization step — total 14 seconds per proof, times 400. The prover sat idle for 93 minutes before it even started. That's not a proving problem; that's a witness-structure blindness problem.

'Witness generation is where real-world data meets mathematical abstraction — and abstraction never pays the network bill.'

— Engineer at a zk-rollup team, after their third hotfix

Reality check: name the technology owner or stop.

The myth of the 'free' witness in trusted setups

Trusted setups create a dangerous assumption: because the setup ceremony is expensive and one-time, everything downstream must be cheap. Not yet. The setup produces a structured reference string (SRS) that defines the circuit's shape, but it says nothing about how you populate the witness. That SRS is a fixed public key; your witness is a dynamic, application-layer payload. Confusing the two costs you a day of debugging when your Groth16 prover rejects a witness because your field-element conversion truncated a 256-bit number.

What usually breaks first is type alignment. The circuit expects field elements modulo a specific prime. Your data comes from Ethereum logs — bytes32, address, uint256. The witness must coerce those types into the prime field without overflow, mapping variable-length bytes into fixed-width elements. Do it wrong and the proof verifies but encodes garbage. Do it right and you've just rewritten half your data pipeline. The myth of the 'free' witness persists because benchmarks use synthetic data — pre-aligned, pre-flattened, pre-blessed. Real witnesses come from databases, APIs, and user input. That gap is where most teams revert to centralized solutions, not because proving is hard, but because witness generation is a hidden tax they never budgeted for.

Patterns That Keep Witness Generation Fast

Incremental Witness Building with Cached Intermediates

The trick that saves teams weeks is hiding in plain sight — don't rebuild the witness from scratch every time. I have watched engineers recompute the same Merkle inclusion proofs dozens of times per transaction, burning CPU cycles they'd never get back. Instead, design your witness construction as a series of checkpoints. Cache the intermediate hashes, the partial openings, even the raw byte offsets of useful data. When a user changes one field in a transaction, you only recompute the path from that leaf to the root. The rest stays frozen. That sounds obvious, but most frameworks push you toward stateless, pure functions that forget everything after each call. Wrong order. You need state — not global mutable state, but a session-scoped accumulator that lives as long as the proving request.

What usually breaks first is the caching layer itself. Teams dump everything into a hashmap and call it done. Then memory blows up, or stale intermediates poison the next proof. The pattern that works: lease-based caches tied to the proving session's lifecycle. When the session dies, the cache dies. No manual eviction, no silent corruption. One concrete fix we applied: store only the last three layers of a Sparse Merkle Tree path, not the entire tree. The seam blew out once, but after that the witness generation dropped from 12 seconds to under 900 milliseconds. Not bad for a three-line change.

Parallelizable Witness Construction Using Domain-Specific Data Layouts

Most people get this wrong: they serialize the witness as one big blob, then try to parallelize the circuit's constraint evaluation. That's backward. The bottleneck is data movement, not computation. If your witness requires iterating over ten thousand account states, and you store them in a single flat array, you can't split the work across cores without contention. The fix is domain-specific layouts — group related witness elements into independent chunks. Account balances, nonces, code hashes each get their own structure. Each chunk can be built in parallel, then stitched together with a lightweight coordinator that does almost nothing.

The catch is that this requires your circuit to tolerate non-contiguous witness input. Many constraint systems (especially Rank-1) assume a single, dense vector. That hurts. You either pad aggressively or switch to a Plonkish arithmetization where the witness can be split across multiple columns. I have seen teams fight this for three months before conceding that the circuit design itself forces sequential witness building. If your benchmark only measures prover time, you will never catch this — the witness generation stage simply never finishes in the first place. Parallel layouts work, but only if the circuit architecture buys into them.

'We shaved 70% off witness time by splitting the data into three independent tracks. The prover didn't care. The witness builder finally breathed.'

— Senior engineer at a DeFi protocol, after switching from a single-threaded validator loop to domain-partitioned witness chunks

Choosing Circuits That Minimize Witness Size

This is the architectural choice most teams defer until it's too late. Rank-1 constraint systems (R1CS) generate compact proofs but fat witnesses — every intermediate variable must be materialized as a full field element. Plonkish circuits, with their custom gates and lookup tables, can compress the witness significantly because many constraints are folded into selectors and precomputed polynomials. The trade-off: Plonkish setups are more complex to audit and harder to debug. But if your witness generation routinely eats 60% of the total proving latency, the complexity pays for itself.

A concrete pattern: use lookup arguments for repetitive operations like range checks or boolean constraints. Instead of emitting twenty witness rows for a simple comparison, a lookup commits to a single witness element and verifies against a table. The witness shrinks; the prover does more work on the backend, but the bottleneck shifts from I/O-bound data assembly to compute-bound proving. That's usually the right trade, because computation scales with hardware upgrades while data shuffling doesn't. However — and this is the pitfall — Plonkish circuits with heavy lookups cause witness generation to become memory-bandwidth limited. You need to benchmark end-to-end, not just the proving phase. One team I know hard-coded their lookup tables as on-chain constants, then wondered why the witness builder spent 40 seconds loading tables from disk. Move them to a memory-mapped file. Instant win.

A rhetorical question worth asking: why does your witness generation need to re-derive data that never changes? If your circuit includes public inputs that are static across all proofs (like a contract address or a governance parameter), precompute those witness elements once, not per proof. Many frameworks don't expose this optimization naturally — you have to hack it in. But the payoff is a steady 15–20% reduction in total latency, with zero impact on security. Not earth-shattering, but consistent. And in production, consistency beats spiky benchmarks every time.

Anti-Patterns That Make Teams Revert to Centralized Solutions

Over-caching witness data — and creating staleness you can't see

The easiest trap in ZK engineering is caching the witness struct too aggressively. I have seen teams serialize the entire witness tree once, store it in Redis, and then serve it for days. That sounds efficient until constraints change mid-deployment — which they always do. The cached witness still validates against the old circuit hash, sure, but the actual prover expects updated fields. Suddenly proofs fail silently, or worse, they pass with stale data that corrupts downstream logic. You lose a day debugging why a zero-knowledge proof verifies on testnet but breaks in production. The fix is brutal: flush the cache on every circuit update. Most teams skip this because it feels wasteful. But the alternative — reverting to a plain database query without ZK — is the real regression.

Assuming witness shape is static when constraints change

The tricky part is that witness generation code lives at the intersection of software engineering and cryptography. Your smart contract team tweaks a Merkle proof depth from 32 to 40 levels. No big deal on-chain. But your witness builder now silently pads or truncates data, adding 200 milliseconds to every proof. Or worse — it breaks silently because the deserialization assumes fixed array lengths. I have watched three different projects abandon ZK mid-sprint for exactly this reason.

Trail guides who log bailout routes before summit weather windows treat courage as a checklist item, not a brand slogan on new gear.

They couldn't trace why proof generation started taking eight seconds instead of two. The cause? A constraint parameter shifted, the witness shape bent, and no test caught it. One team reverted to a centralized REST endpoint within two weeks. What usually breaks first is the assumption that circuit changes are backward compatible. They're not. Never assume a constraint change is witness-neutral.

Ignoring serialization and deserialization overhead

Most performance audits focus on proving time — the Groth16 or PLONK computation itself. But witness generation often spends 40% of its wall-clock time moving bytes through JSON or Protobuf pipelines. That hurts. Especially when your input data comes from heterogeneous sources: a MySQL row, an S3 blob, a gRPC stream. Each hop adds serialization tax. The anti-pattern is wrapping every field in a generic String or Vec<u8> and parsing later. That approach inflates witness memory by 3x and pushes generation time past acceptable latency. One concrete anecdote: a payments team we advised was passing 2MB of transaction history through Base64 twice — once in transit, once in the witness builder. Shaving that single redundant decode cut generation time by 370ms. Not flashy. But enough to keep them from ditching ZK for a centralized sequencer.

Reality check: name the technology owner or stop.

Skiking incremental witness updates for full recomputation

Wrong order. Many teams regenerate the entire witness on every proof cycle, even when only one leaf changed. That's like recompiling a whole kernel module because you edited a comment. The result? Generation time scales with the complete dataset, not the delta. When datasets hit 100k+ records, that approach collapses. Teams then look at the profiler, see 6-second witness generation, and conclude "ZK is too slow." They revert to a centralized solution — usually a signed hash from a trusted server — within two sprints. The overlooked fix is incremental witness construction: track which inputs changed, rebuild only the affected subtrees, and reuse the rest. We fixed this by adding a dirty-flag table per witness segment. Generation time dropped from 5.2s to 0.9s. That's the difference between shipping decentralized or bolting on a trusted third party.

'Every time I hear "we'll just regenerate the whole witness," I know someone is two weeks away from a centralized fallback.'

— ZK engineer at a custody protocol, after watching three architecture reviews end the same way

Long-Term Costs: Drift, Maintenance, and Decay

Circuit upgrades that break witness builders

The moment you ship a circuit change—adding a constraint, swapping a hash function, even reordering inputs—your witness generation code becomes a liability. I have watched teams spend two weeks hand-tuning a witness builder, only to discover that the new circuit expects a field element in a different byte order. Suddenly every proving run fails silently at the constraint-check stage. What makes this insidious? The circuit still compiles. The prover still runs. But the witness is structurally malformed. That hurts.

Most teams skip version-locking the witness builder to the exact circuit hash. They treat witness generation as a generic input pipeline—it isn't. One concrete anecdote: a production system at a DeFi protocol upgraded their Poseidon hash parameters from 2-to-1 to 3-to-1 compression. The circuit compiled fine. The prover accepted witness data. But the proof failed verification after six hours of batch computation. The root cause? The witness builder still used the old sponge configuration. Circuit upgrades break witness builders before they break circuits—the failure surface is upstream, not inside the prover. Version-pin your witness artifacts; treat every circuit commit as a potential witness-breaking event.

Data source schema changes that silently invalidate witnesses

The trickier cost is drift from the data side. Your witness builder reads from an API, a database, or a blockchain event log. That source changes schema—a field gets renamed, a timestamp shifts from Unix milliseconds to nanoseconds, a null value appears where an integer used to live. No one notices until the prover returns 'ConstraintUnsatisfied' with zero context. The catch: the error message tells you which constraint broke, not why the witness data is wrong. You lose a day tracing field mappings.

I have seen teams treat schema changes as unrelated infrastructure work. Wrong order. A column rename in Postgres can silently invalidate every Merkle proof your prover generates for the next week. The maintenance tax here is hidden—no compile error, no CI failure, just a slow degradation of proof success rates. One fix: run a lightweight validation step that checks witness structure against a known schema hash before feeding data to the prover. That validation costs microseconds. The debugging cycle it saves costs days.

'We spent three months building a witness pipeline, then six months debugging it. The circuit was never the problem—the witness always was.'

— Lead engineer at a zk-rollup startup, after their mainnet launch

Prover hardware upgrades that don't help witness generation

Here is the asymmetry most people miss: you can throw GPUs at the prover, double your RAM, switch to a custom ASIC—witness generation stays single-threaded on the CPU. The bottleneck doesn't shift. Teams budget $50k for prover hardware and $0 for witness optimization, then wonder why their end-to-end latency barely improves. That sounds fine until you realize witness generation can account for 40–60% of total proving time in non-trivial circuits. I have seen a zero-knowledge identity system where witness building took 22 seconds and proving took 18—all the GPU investment shaved 4 seconds off the back half. The front half remained glacial.

The real long-term cost is decay: your witness generator, untended for six months, falls behind both circuit updates and data source changes. Meanwhile your prover hardware is current and underutilized. The imbalance grows. A team I advised fixed this by separating witness generation into its own pipeline with explicit version manifests and regeneration scripts that run on every schema deployment. Not glamorous—but it stopped the silent rot. Maintain witness code like you maintain a database migration: version it, test it, and expect it to break when anything upstream changes. The alternative is a centralized fallback—exactly the outcome zero-knowledge proofs were supposed to eliminate.

When You Should Not Optimize for Witness Generation

Very short-lived proofs (e.g., one-time use)

You're building a proof that lives for three seconds. A single-use credential, maybe, or a one-shot authentication token that burns after verification. Should you care whether witness generation takes 40ms or 400ms? Probably not. The proof itself vanishes before anyone measures the latency. I have watched teams spend two weeks micro-optimizing witness construction for proofs that expire faster than a coffee break — and the real bottleneck was network round trips, not witness prep. The trap is engineer pride: you see a slow benchmark and reflexively optimize, even when the proof's lifetime is shorter than a single TCP retransmit. That hurts.

Here is the hard rule: if your proof lives for fewer than ten verification cycles, or if the verifier discards the proof immediately after checking, witness generation speed is noise. The system's total throughput is dominated by serialization, transport, and the verifier's CPU cycles — not how fast you assembled the witness on the producer side. Most teams skip this: they profile the wrong phase. Run a flame graph on the full pipeline first. I have seen profiles where witness generation accounted for 2% of wall-clock time, yet the team was rewriting data structures to shave 1ms off it. That energy belongs somewhere else — maybe in reducing proof size or batching verifications.

Wrong order. Optimize for witness generation only after you have confirmed it's the actual drag. If the proof is ephemeral, let it be fast enough, not fast.

Protocols where witness is provided externally (e.g., client-supplied)

The witness arrives over the wire. You didn't build it. You don't maintain the code that built it. In protocols like anonymous credentials or delegated proving, the client creates the witness on their device and ships it alongside the proof statement. Your server's job is verification, not construction. So why are you benchmarking witness generation? The catch is subtle: many teams still run internal witness generation to validate the client's submission, turning a single-phase protocol into a double-phase drag. I fixed this by simply trusting the client's witness format and skipping local reconstruction — you lose a day debugging mismatches, but you gain an order of magnitude in throughput.

That sounds fine until you realize the client might send malformed witnesses. The trade-off is clear: you accept a small risk of invalid proofs (caught at verification time anyway) versus eating a constant overhead on every request. For high-volume APIs — think 10,000 proofs per second from mobile clients — the cost of locally regenerating the witness is a death sentence. Drop it. Design the protocol so the prover supplies a structured witness, and your verifier checks its integrity directly. This shifts the optimization burden to the client, where it belongs. If the client is a browser or a thin IoT device, sure, help them with SDK tooling — but don't mirror their work on the server just because your benchmark framework "feels" incomplete.

Honestly—most benchmarks lie here because they assume the prover and witness generator are the same process. In real systems, they often are not. Make the distinction explicit in your test harness.

Flag this for blockchain: shortcuts cost a day.

Systems where proving time dwarfs witness gen (e.g., high-complexity circuits)

Your circuit has 10 million constraints. The proving step takes 90 seconds. Witness generation takes 300 milliseconds. The ratio is 300:1. Optimizing the witness path at that point is rearranging deck chairs on a sinking ship — actually, on a perfectly fine ship, because the hull is the prover. I have seen teams deploy new witness serialization formats that saved 80ms while the prover was still chewing through polynomial commitments for another 45 seconds. Nobody noticed. The user certainly didn't.

The tricky bit is that high-complexity circuits often have complex witness generation too — but complexity is not the same as runtime. A witness with 200 fields of Merkle proofs might look expensive but completes in linear time against the circuit's depth. Meanwhile, the prover is doing multi-scalar multiplications across large elliptic curve groups. Do the math: if proving time is 100× worse than witness time, any witness optimization yields at most a 1% improvement. That's noise. Worse, it distracts from the real lever — reducing constraint counts, switching to faster proving systems (e.g., Groth16 instead of PLONK), or batching proofs.

'We spent three months optimizing witness generation. Then we switched to a different curve and proving time dropped 40×. The witness optimization was irrelevant.'

— lead engineer at a zk-rollup startup, after the rewrite

The lesson: measure the full pipeline end-to-end before committing to witness work. If proving time dominates by an order of magnitude, your optimization budget goes there. Not to witness generation. Not until the bottleneck shifts. That day may come — as proof systems accelerate, witness generation will eventually become the slow step again. But today is not that day for most teams. Run the experiment: profile with proof time stripped to zero (dummy prover), then add real proving back. The delta tells you where to spend your Monday mornings.

Open Questions and Practical FAQs

Can we benchmark witness generation separately?

Most teams skip this: they measure proving time down to the microsecond but treat witness generation as a black box that 'just happens.' The honest answer is yes, you can isolate it—but the tools don't make it easy. Standard ZKP frameworks like Gnark, Halo2, or Circom expose a prove() call that bundles witness computation with constraint building and proof creation. To separate them you have to fork the pipeline or insert timing hooks at the circuit boundary. I have seen teams burn two weeks wiring custom profilers only to discover that witness generation consumed 40% of their end-to-end latency—not the prover. The catch is that isolated benchmarks can mislead: a fast witness function in a vacuum might produce constraints that blow up proving time. So benchmark both separately and end-to-end. Without that dual view, you're optimizing in the dark.

Do recursive proofs change the trade-off?

Recursive proofs sound like the magic wand—verify one proof inside another and collapse verification cost. But here is the rarely-discussed sting: recursive composition multiplies witness generation because each inner proof requires its own witness, plus an extra layer to verify the outer proof. That means your witness function runs N+1 times instead of once. Worse, the witness for the recursive verifier circuit often involves hashing the previous proof's public inputs—a non-trivial operation that serialises poorly. I have watched a team cut proving time by 65% using recursion, only to have witness generation jump from 80ms to 450ms. Their '10x faster' benchmark was a lie because they benchmarked only the prover. Recursive schemes demand a separate budget for witness work; treat them as a joint optimisation, not a free lunch.

How do hardware accelerators (FPGAs, GPUs) affect witness gen vs. proving?

Hardware accelerators are excellent at the heavy arithmetic of MSM and NTT—the proving side. Witness generation, however, is mostly control flow: database lookups, hash chains, Merkle path construction, and conditional branches. That stuff doesn't map well to GPUs or FPGAs. The typical outcome: GPU proving drops from 10 seconds to 300 milliseconds, while witness generation stays stuck on the CPU at 2 seconds. Suddenly your bottleneck flips. The fastest prover in the world is useless if the witness takes longer to build than the proof.

— observation from a production ZKP pipeline I consulted on, where the team swapped GPU accelerators three times before realising the witness CPU was the real wedge.

The practical fix is to precompute witness components ahead of time—build Merkle trees during idle CPU cycles, cache expensive hash evaluations, and pipeline the witness function so it starts before the proving request arrives. That said, off-the-shelf FPGA boards for witness work are rare; most devs end up writing custom C++ or Rust modules that run on the same CPU cores as the prover. Not elegant, but it works. One concrete next action: profile your witness function with perf or a flamegraph before buying any accelerator hardware. You might find that a simple caching layer beats a $3000 FPGA board.

Summary and Next Experiments to Run

Instrument your own pipeline: measure witness gen vs. prove time

Stop guessing which phase dominates your wall clock. I have watched teams spend weeks optimizing a prover that was already sub-100ms while witness generation silently sat at 3.8 seconds—hidden behind a single log line that lumped everything into 'preprocessing.' Grab a stopwatch. Or better: wrap your witness builder, your constraint compiler, and your prover in separate timers. The first run will sting—call it a rite of passage. That raw split tells you whether Groth16's fast proving matters when your witness generation folds JSON blobs of 40 MB.

The catch is subtle: witness generation doesn't scale linearly with circuit size. A 10× input blob can produce a 50× spike in allocation-and-copy overhead. Measure at three different payload sizes, not just the happy path. One engineering team I worked with traced a 2.3-second witness build to a single quadratic memory reallocation—fixed with a pre-sized vector, zero architecture changes. You can't fix what you haven't measured. Do it this week.

Test with multiple witness sizes and structures

Flat arrays are not real data. Most benchmarks ship with witness sizes of 1 KB and call it done. Real workloads throw nested dictionaries, sparse vectors, or Merkleized state that must be walked recursively. Run your pipeline against a 10 MB witness, then a 200 KB one. Watch the variance: some backends choke on large discrete logarithms but fly through scalar-heavy structures. Plonky2, for instance, hates irregular memory access patterns—its recursive verifier stalls if your witness triggers frequent cache misses.

That hurts worse under STARKs, where the witness is not just a bag of field elements but includes trace rows that must be consistent across columns. Change one structure—swap a list for a binary tree—and the proving time can triple without any circuit modification. I have seen a team revert to a centralized API because their Halo2 witness builder allocated 16 GB for a 2 MB input. Wrong structure. Flat hash maps would have saved them.

A practical experiment: fork your existing circuit, swap the witness format from dense structs to lazy-loading iterators, and re-run the benchmark. Is the improvement dramatic? If yes, your witness generation was the bottleneck all along. If no, then the prover is your real constraint—and you can relax.

‘We optimized the prover for three months. Then we timed witness generation. It was nine seconds. We cried. Then we fixed it in two days.’

— Principal engineer, private audit conversation

Compare Groth16, Plonky2, and STARKs under realistic witness workloads

Pick one scheme and you're flying blind. Groth16's single trusted setup and fast verification look seductive—until your witness has 50 branching paths that require separate circuit generations. Plonky2's recursive composition shines when you can amortize witness generation across many proofs; it sinks when your witness changes every block and you recompute the whole trace. STARKs? No trusted setup, but your witness must be transcribed into execution traces with uniform width—variable-length fields cost you padding bytes that explode memory.

Run the same three witness workloads across all three backends. Not the textbook examples—your actual JSON, your actual Merkle proofs, your actual state deltas. What usually breaks first is the witness serialization layer: the library that turns your domain objects into the fixed-point field elements each scheme expects. One format will align with your data's natural shape; the other two will force copies, conversions, and heap allocations that kill throughput.

The next step is brutal but necessary: instrument the whole system in production, not just a microbenchmark. Capture one week of real witness traces. Replay them against each proving backend. You will likely find that Groth16 wins on pure proving speed but loses on setup iteration, while STARKs tolerate noisy witness sizes but punish you with 15-second proof generation. Pick the loser whose pain you can afford—and measure again next quarter. Your data will drift; so must your choice.