NVIDIA Nemotron 3.5 Lightning Activates 3B of 30B Parameters: The Self-Hosted Agent Cost Math

NVIDIA's Nemotron 3.5 Lightning combines roughly 30 billion total parameters with about 3 billion active parameters per token. That sparse architecture is attractive for always-on coding agents because active compute can resemble a much smaller model—but memory, operations, and utilization still determine whether self-hosting beats an API.

NVIDIA released BF16 and NVFP4 variants in August under its Open Model Developer License. The official model card describes the BF16 weights as the customization reference and directs latency-sensitive users to NVFP4. AWS subsequently made the model available in SageMaker JumpStart and reports up to four times higher throughput and up to 30% faster task completion for always-on agents in its tested comparisons. Treat those as vendor measurements, not universal guarantees.

Active Parameters Are Compute, Total Parameters Are Memory

In a mixture-of-experts model, a router selects only part of the network for each token. About 3B active parameters can reduce arithmetic work compared with running all 30B on every token. It does not make the other weights disappear. The serving system still needs access to the full model, so memory capacity and bandwidth remain closer to the total-parameter footprint.

Quantization changes that footprint substantially. A rough weight-only estimate is 60 GB at 16 bits for 30B parameters and 15 GB at 4 bits, before runtime overhead, caches, buffers, and implementation details. NVIDIA's NVFP4 package is designed for its supported hardware and software path; a filename is not proof that any 16 GB consumer device will deliver production throughput.

The Break-Even Formula

For an API, monthly cost is input tokens times the input rate plus output tokens times the output rate. For self-hosting, use monthly hardware amortization, electricity, hosting, engineering operations, observability, and idle capacity. Then divide that total by successful agent tasks, not generated tokens. Sparse activation improves the throughput side of the equation, but poor utilization can erase the benefit.

Imagine dedicated equipment and operations total $1,000 per month. At 10,000 accepted lightweight tasks, infrastructure is $0.10 per task before human review. At 500 accepted tasks, it is $2.00. The hardware is identical; utilization changes the unit economics by 20 times. This is why a model aimed at always-on agents is most compelling for steady queues, not occasional interactive coding.

Measure Completion, Not Tokens per Second

Fast decoding helps only if the model completes the work. A weak agent that emits code quickly, fails tests, and retries can be slower and more expensive per accepted patch. Build an evaluation set from your repository: bug fixes, small features, test repairs, review comments, and tool calls. Track success rate, median wall time, tokens, retries, and reviewer minutes.

The official model card positions Nemotron as an agent model, but benchmark fit is not repository fit. Test its instruction following, tool schema adherence, long-session stability, and license requirements. Compare the quantized runtime you will actually deploy rather than borrowing results from a different precision or accelerator.

Measure energy and capacity at the service level. Record wall power during idle and sustained generation, tokens per second at realistic concurrency, and the context lengths your agents actually use. Include time spent loading weights and recovering workers. A peak benchmark on one warm prompt overstates useful throughput when production jobs vary in length and the queue repeatedly transitions between idle and busy.

Run the same tests after driver, runtime, and quantization updates. A serving upgrade can change memory headroom, output quality, and throughput enough to move the break-even point even when the model name stays constant.

Where a 3B-Active Model Fits

• High-volume triage: classify issues, summarize diffs, and route work before a frontier model is called.
• Bounded coding: repetitive edits with deterministic tests and short tool chains can exploit throughput.
• Private environments: local inference can keep code inside an approved boundary, although logs and tools still need governance.
• Hybrid routing: let the local model attempt routine tasks and escalate low-confidence failures to a stronger API.

Do Not Record a Fictional API Price

An open-weight release can appear through multiple hosting services, each with instance, endpoint, or token pricing. There is no single universal Nemotron 3.5 Lightning API rate to place next to a direct vendor token price. Verify the exact deployment region, hardware, minimum uptime, autoscaling behavior, and managed-service fee. For a local deployment, price the machine you can actually buy and the electricity you actually pay.

Nemotron 3.5 Lightning strengthens the case for a cheap local worker tier, not the claim that inference becomes free. Sparse activation lowers the compute burden; total weights, idle hours, validation, and operations remain. Compare a measured self-hosted task cost with frontier API scenarios in our AI Cost Calculator.