Most model evaluations answer the wrong question for enterprises. They simply measure if a model can complete a task or follow some leaderboard theatrics on some math and golf problems. The harder question, and the one that decides whether an agentic system reaches production, is what happens to that model under sustained concurrency against 40,000-token enterprise context windows.
That is where multi-agent systems break. Latency compounds across turns. KV-cache pressure builds until the serving layer starts queuing. A model that looks capable in a single-request demo becomes economically unworkable at the volume real business processes demand.
Uniphore’s Business AI Cloud is model-agnostic by design. Our Model Layer routes each task to the best-fit model based on accuracy, latency, and cost, which means we run continuous evaluation of open models as candidates for high-volume production paths. When NVIDIA gave us early access to Nemotron 3.5 Lightning, a 30B-3A open model, we put it through the same four-suite benchmark we use for every model that enters our stack.
We compared it against Gemma 4 31B IT, our current open-weight reference model for these workloads. Both ran in BF16. At 30B and 31B parameters, this is a near-identical size class, so any difference in throughput reflects architecture and serving efficiency rather than a smaller model doing less work.
How we tested
We ran four suites in sequence, from synthetic to fully applied
- Machine benchmark (Locust / vLLM). No business logic. Pure serving characteristics across three input/output token profiles.
- Enterprise Q&A. Knowledge Graph question answering against two production knowledge bases, measured for both accuracy and speed, before and after fine-tuning.
- Agentic execution. Two production agents running instruction sequences at concurrency levels from 1 to 25.
- Conversational summarization. High-volume summarization against real conversation transcripts. These are specialized workloads since these carry higher input and output token counts.
Knowledge bases of two production use cases carried through the Q&A and agentic suites.
Contract leakage. Enterprises with large active contract portfolios lose revenue quietly. Entitlements go unbilled. Escalation clauses pass their trigger date without ever being applied. Terms drift out of compliance as the commercial relationship changes, and nobody reconciles the paperwork. Catching any of them means reading every contract against every invoice, which is why most organizations do not. We built an agent that works across a knowledge base of contract and invoice data.
Legal review of marketing material. Semiconductor companies produce a constant stream of marketing collateral, and every piece has to clear legal before it publishes. Reviewers are looking for patent infringement exposure and performance claims the company cannot substantiate. The work is high volume and high judgment at the same time, which is a combination that scales badly with headcount. The agent performs that review against a knowledge base of marketing material from a semiconductor manufacturer.
For readability, the results below refer to these as KB1 (contract and invoice data) and KB2 (semiconductor marketing material).
Test Environment
All tests ran from the same class of machines – a 4 vCPU core machine with 16GB RAM as the host machine with the models loaded on a 2-H100 GPU cluster. All tests were run on the same machine. If coming thru an external ingress controller the actual metrics do vary a little, but the relative difference stays the same.
Test 1: Machine benchmark
We started with no business logic at all to establish a clean serving baseline across three token profiles.
| Input / output tokens | Metric | Gemma 4 31B | Nemotron 3.5 Lightning |
|---|---|---|---|
| 3,048 / 128 | Total throughput | 4,039 tok/s | 19,790 tok/s |
| Requests per minute | 76.2 | 373.4 | |
| Median TTFT | 3.30 s | 0.67 s | |
| 4,096 / 1,024 | Total throughput | 1,943 tok/s | 9,522 tok/s |
| Requests per minute | 22.8 | 111.7 | |
| Median TTFT | 4.69 s | 0.96 s | |
| 8,192 / 3,072 | Total throughput | 1,481 tok/s | 7,257 tok/s |
| Requests per minute | 7.8 | 38.2 | |
| Median TTFT | 7.10 s | 1.45 s |
The consistency is the interesting part. Nemotron 3.5 Lightning held a 4.9x throughput advantage and a 4.9x time-to-first-token advantage across all three profiles, from short-context classification shapes to long-context generation.
Median time per output token was 3.7x lower at every profile. An advantage that stable across input and output shapes points to the serving architecture rather than a profile the model happens to be tuned for.
Test 2: Enterprise Q&A, and what fine-tuning changed
This is the suite where we measured quality, and it is the most honest result in the set.
Out of the box, base Nemotron 3.5 Lightning scored below a finetuned Gemma 4 31B on accuracy for both knowledge bases: 0.781 against 0.840 on contracts, 0.578 against 0.663 on legal review. It already led on answer relevancy in both cases, which suggested the model was reasoning well over retrieved context and losing points elsewhere.
So, we fine-tuned it. We ran supervised fine-tuning (SFT) against the base model using NVIDIA NeMo Megatron Bridge with a PEFT-style LoRA adapter, trained on domain data from each knowledge base.
| Metric | KB1 Gemma | KB1 Nemotron | KB1 Nemotron + SFT |
|---|---|---|---|
| Accuracy | 0.840 | 0.781 | 0.901 |
| Faithfulness | 0.940 | 0.898 | 0.932 |
| Answer relevancy | 0.365 | 0.468 | 0.534 |
| Answer completeness | 0.533 | 0.513 | 0.620 |
| Context precision | 0.514 | 0.516 | 0.599 |
| Context recall | 0.916 | 0.913 | 0.980 |
| Metric | KB2 Gemma | KB2 Nemotron | KB2 Nemotron + SFT |
|---|---|---|---|
| Accuracy | 0.663 | 0.578 | 0.753 |
| Faithfulness | 0.870 | 0.876 | 0.890 |
| Answer relevancy | 0.227 | 0.394 | 0.431 |
| Answer completeness | 0.347 | 0.305 | 0.440 |
| Context precision | 0.486 | 0.379 | 0.533 |
| Context recall | 0.587 | 0.596 | 0.626 |
Accuracy improved 15% on KB1 and 30% on KB2 relative to the base model. On the legal review knowledge base, the fine-tuned model beat Gemma on every metric we track. On contracts it led on five of six, with faithfulness landing within a point.
Two things matter here. First, the fine-tuning path was straightforward and fully supported, which is the practical argument for building on an open weights model rather than accepting whatever a closed endpoint gives you. Second, fine-tuning changed accuracy and left throughput untouched:
| Metric | Gemma 4 31B | Nemotron 3.5 Lightning |
|---|---|---|
| Requests per minute | 17.8 | 79.8 |
| Median TTFT | 8,278 ms | 1,378 ms |
| TTFT P95 | 16,070 ms | 2,339 ms |
| Full suite duration | 337.5 s | 75.2 s |
A 4.5x request rate and a 6.9x improvement at P95 time-to-first-token, with accuracy now ahead. That combination is what moves a workload from pilot to production.
Test 3: Agentic execution under concurrency
The agentic suite is the closest thing we have to a production stress test: approximately 40,000 input tokens per request, 200 requested output tokens, streaming enabled, prefix cache hit rate at zero, run across escalating concurrency.
| Concurrency | Request rate | Output throughput | TTFT P50 | End-to-end P50 | KV-cache usage |
|---|---|---|---|---|---|
| 1 | 3.9x higher | 3.9x higher | 4.2x faster | 3.8x faster | 86.5% lower |
| 5 | 4.4x higher | 4.4x higher | 5.2x faster | 4.4x faster | 79.7% lower |
| 20 | 4.9x higher | 4.8x higher | 4.9x faster | 4.8x faster | 72.0% lower |
| 25 | 6.8x higher | 6.8x higher | 5.8x faster | 5.3x faster | 67.1% lower |
The advantage widens as load increases, and the KV-cache column explains why. Gemma reached 88.5% cache utilization at concurrency 25. It stopped scaling after concurrency 20 and began degrading, with request rate falling from 6.0 per minute to 4.5 and P95 end-to-end latency climbing from 39.8 seconds to 113.1 seconds.
Nemotron 3.5 Lightning was still at 29.1% cache utilization at concurrency 25, still increasing throughput, and held P95 end-to-end latency at 17.6 seconds. It completed 100% of requests at concurrency 1 through 20 and 98.9% at 25, with zero preemptions throughout.
For a multi-agent system, headroom of that kind is the difference between provisioning for peak and provisioning for average.
Test 4: Conversational summarization
We ran our highest-volume workload class against real conversation transcripts, averaging 5,779 input tokens.
| Metric | Gemma 4 31B | Nemotron 3.5 Lightning |
|---|---|---|
| Requests per minute | 50.9 | 124.5 |
| Median TTFT | 1.68 s | 758 ms |
| Median completion latency | 9.02 s | 3.67 s |
| Prefill throughput | 4,906 tok/s | 11,504 tok/s |
| Total throughput | 5,175 tok/s | 12,194 tok/s |
The margin here is narrower than the other three suites, in the 2.2x to 2.5x range rather than 5x. Summarization is prefill-dominated with modest output lengths, which compresses the gap. We report it because it is a real result and because it shows the advantage is workload-shaped rather than uniform. We did not run an accuracy comparison on this suite.
What this means for enterprise AI
There are three conclusions we are carrying forward:
1. Throughput is the constraint on agentic scale.
Enterprises rarely abandon agentic programs because a model cannot do the task. They abandon them because the inference bill at production volume does not survive the business case. A 5x throughput advantage by Nemotron 3.5 Lightning at equivalent parameter count changes the economics because the same model instance can now handle more load, thereby reducing the cost and latency for enterprise agentic systems and without impacting accuracy.
2. An open model with a real tuning path beats a marginally better closed one.
Base Nemotron 3.5 Lightning trailed on accuracy. That mattered less than the fact that we could fix it ourselves, quickly, with NeMo Megatron Bridge and supervised fine-tuning, on our own data. Closed endpoints do not offer that. This is the same principle behind our SLM Factory, where we distill large models into efficient domain-specific ones inside the Business AI Cloud Knowledge Layer.
3. Open weights are a sovereignty requirement, not a preference.
Our customers in financial services, insurance, and healthcare need inference to run inside their own boundary, on their own infrastructure, under their own governance. Open-weight models of this quality are what make that architecturally possible. Sovereign, composable, and secure is how we describe the Business AI Cloud, and the model layer has to hold up its end of that.
What we would like to see next
There are two areas we have raised with the NVIDIA team. We would like deeper published guidance on serving multiple domain-tuned variants (multiLoRa) of the same base model concurrently, since that is the pattern a multi-tenant enterprise platform needs. MoE models typically don’t have widespread availability of finetuning options across neoclouds and hyperscalers, which will cause restrictions in usage of the Lightning model. Currently, the NeMo Megatron Bridge is an NVIDIA-provided toolkit, which makes it restrictive to have a full-blown solution where autonomous fine-tuning can happen on any inference platform. We are also interested in prefix caching behavior at the 40,000-token context lengths our agents operate at, where our tests deliberately ran with a 0% hit rate to establish a worst case.
Both are the kind of questions that only surface once a model is carrying production traffic, which is the point of working with NVIDIA this early in a release cycle. Nemotron 3.5 Lightning is entering evaluation for high-volume paths across the Uniphore Business AI Cloud, and we will publish what we learn.
Try it yourself
Nemotron 3.5 Lightning is available now as an open-weight model from NVIDIA, along with the datasets and tooling behind it. If you are building agentic systems where inference economics decide what you can afford to automate, the Uniphore Business AI Cloud gives you the layer around the model: task-level routing across open and proprietary models, domain fine-tuning through our SLM Factory, and the guardrails and governance to run all of it inside your own environment. Talk to the Uniphore team about benchmarking against your own workload. If you run this evaluation yourself, we would like to see your numbers.