What Can You Actually Run on a 48GB NVIDIA A40? Measured Model Sizes and Tokens Per Second (2026)

Short answer: one NVIDIA A40 48GB comfortably runs any open-weight model up to about 34B parameters at 4-bit quantization — that's the sweet spot, around 23–29 tokens/sec single-stream with the GPU at 90%+ utilization. A 70B model at Q4 also fits, using 39–43 GB and delivering 12–13.5 tokens/sec. Past that, at 72B and 47 GB, you fall off a cliff: 5.78 tokens/sec and GPU utilization collapsing to 42–50%, because the card is spending its time moving weights instead of doing math.

For concurrent serving the picture inverts. A single A40 running vLLM pushes 1,705 tokens/sec of aggregate output on an 8B model at 100 concurrent requests. The 48 GB isn't buying you one enormous model — it's buying you a lot of KV cache for a small one.

Below are the measured numbers, the distinction that trips most buyers up, and the three things a 48 GB Ampere card genuinely cannot do.

Single-Stream: What Fits and How Fast

Measured on a single A40 under Ollama at 4-bit quantization. Read the "VRAM" column as the practical constraint — anything approaching 47 GB on a 48 GB card leaves nothing for KV cache, which is exactly what the bottom row shows.

Benchmarks published by DatabaseMart on a single A40 under Ollama 0.5.4, 4-bit quantization. Your throughput will vary with quantization format, context length, batch size and driver version — treat these as a shape, not a guarantee.

The pattern is clean and it's worth naming, because it's the actual buying criterion: GPU utilization is the tell. At 32B the card sits at 90–94% — it's compute-bound, which is where you want it. At 72B/47 GB it drops to the 40s, because there's no room left for KV cache and the runtime is thrashing. A model that "fits" in the sense of loading is not the same as a model that fits in the sense of running well, and the gap between those two things is roughly where 70B sits on this card.

A40 48GB in a tested R740xd, ready to rack: $10,959 · in stock at Resilient Tec.
Dual Xeon Silver 4210R, 128 GB DDR4, 24 SFF bays, 10 GbE SFP+, iDRAC9 Enterprise, burned in and warrantied. See the build → or request a quote for a different configuration.

Concurrent Serving: The Number That Actually Matters

Single-stream tokens/sec is the number everyone quotes and the wrong one for most deployments. If you're putting a model behind an internal API for a team, what you care about is aggregate throughput under load. On vLLM, one A40 delivers:

• Llama 3.1 8B, 50 concurrent requests — 2.29 requests/sec, 1,016 tokens/sec output
• Llama 3.1 8B, 100 concurrent requests — 3.76 requests/sec, 1,705 tokens/sec output
• Qwen2.5 14B, 100 concurrent requests — 1.14 requests/sec (degrading; needs --max-model-len 4096 for stability)

Median time-to-first-token across the tested models ran from 220 ms to 1,434 ms depending on model size. The practical read: a single A40 is a genuinely capable 7–8B serving node at 100+ concurrent requests, a workable 14B node with context capped, and a single-user or small-team box at 32B and above.

That reframes the sizing question. If forty people need a coding assistant, one A40 running an 8B model serves them better than one A40 running a 70B model serves three of them. If you need both, that's two nodes — or one node with two cards.

Why 48 GB Doesn't Mean "48 GB of Model"

Model weights are only part of the allocation. You also need KV cache, which scales with context length times concurrency, plus activation memory and CUDA overhead. A 43 GB model on a 48 GB card leaves roughly 5 GB, and at long context that is a single conversation, not a workload.

Rules of thumb that hold up on this card:

• Leave 20–25% of VRAM free for cache and overhead. On 48 GB that means targeting ~36 GB of weights.
• Long context is a memory purchase, not a model purchase. Doubling context roughly doubles KV cache. A 32B model at 32K context can cost more headroom than the 11 GB gap between 32B and 70B suggests.
• Quantization format matters as much as bit depth. Q4_K_M, AWQ and GPTQ at nominally the same 4 bits have materially different memory and speed profiles on Ampere.

The Three Things an A40 Cannot Do

Worth being straight about, because these are the reasons someone would pay three times as much for a newer card:

1. No FP8. The A40 is Ampere (GA102). FP8 tensor support arrived with Ada and Hopper, so an A40 gets no benefit from the FP8 kernels that modern inference stacks increasingly optimize for. Against an L40S on an FP8-tuned workload, the gap is much wider than the raw specs suggest. Against the same workload in FP16 or INT4, it's a fair fight at a fraction of the price.

2. No MIG. Unlike A100 and H100, the A40 cannot be partitioned into isolated GPU instances. It supports SR-IOV for virtualization, but you cannot carve it into hard-isolated slices for multi-tenant use. One card, one tenant, or software-level sharing.

3. Not a training card for large models. 48 GB with 696 GB/s of GDDR6 bandwidth is fine for inference and for LoRA/QLoRA fine-tuning of models up to roughly 34B. Full-parameter fine-tuning of a 70B model is an HBM problem, and this is not an HBM card.

What it is: the cheapest legitimate route to 48 GB of ECC datacenter VRAM with real driver support, real virtualization support, and a card designed to run in a rack for years. For inference, visualization, VDI and rendering, that trade is a good one.

Two Cards, 96 GB

The A40 carries an NVLink connector, and a bridged pair gives you 96 GB of addressable memory — which moves a 70B model out of the "fits, barely" band and into comfortable territory, and puts 72B models back on the table at usable speed. The R740xd supports up to three double-width cards, all identical, so this is a supported upgrade path rather than a rebuild. Talk to us before you buy the second card, though — the thermal and power math changes, and so does the riser assignment.

The Cost Comparison Nobody Runs Honestly

A single A40 node running an 8B model at ~1,700 tokens/sec of aggregate output, on hardware you own, has no per-token bill attached. The comparison against rented inference isn't really about tokens per dollar at any single moment — it's about whether your throughput requirement is steady. Bursty, unpredictable, occasional workloads belong in the cloud. A team hitting an internal assistant all day, every day, at predictable volume, is paying rent on a machine they could own outright in under a year. We wrote about that math in more detail here.

FAQ

Can an NVIDIA A40 run a 70B model?
Yes. At 4-bit quantization a 70B model uses 39–43 GB of the A40's 48 GB and delivers roughly 12–13.5 tokens/sec single-stream at 94–98% GPU utilization. It fits, but with little headroom for long context or concurrency.

What is the best model size for a 48 GB A40?
The 27–34B class at 4-bit. Those models use 16–20 GB, run at 23–29 tokens/sec, keep the GPU at 90%+ utilization, and leave substantial room for KV cache and longer context.

How many concurrent users can one A40 serve?
On vLLM with an 8B model, a single A40 handles 100 concurrent requests at 3.76 requests/sec and 1,705 tokens/sec of aggregate output. A 14B model is workable at that concurrency with context capped at 4,096 tokens. Above 14B, concurrency drops sharply.

Does the A40 support FP8?
No. The A40 is an Ampere GA102 card. FP8 tensor support began with Ada Lovelace and Hopper, so the A40 gets no benefit from FP8-optimized inference kernels.

Does the A40 support MIG?
No. The A40 supports SR-IOV for virtualization but cannot be partitioned into isolated Multi-Instance GPU slices the way an A100 or H100 can.

Can you fine-tune on an A40?
LoRA and QLoRA fine-tuning works well up to roughly 34B parameters. Full-parameter fine-tuning of a 70B model needs far more memory bandwidth and capacity than 48 GB of GDDR6 provides.

Can two A40s be linked with NVLink?
Yes. The A40 has an NVLink connector, and a bridged pair presents 96 GB, which makes 70B and 72B models comfortable rather than marginal. Both cards must be the same model, and the host chassis must support two double-width accelerators.

Why does GPU utilization drop on 72B models?
At 47 GB of weights on a 48 GB card there is almost no room left for KV cache, so the runtime spends its time on memory movement instead of compute. Utilization falling to 42–50% while throughput drops to 5.78 tokens/sec is the signature of that condition.

How much does an NVIDIA A40 server cost?
At Resilient Tec, the R740xd – A40 Build is $10,959, the R740xd – Dual V100 Build is $5,849 and the R740xd – GPU Ready (no GPU) is $2,995. All three are refurbished Dell PowerEdge builds with the GPU enablement kit, high-performance cooling and 1600 W supplies installed. (Prices checked September 11, 2026.)

Where can I buy an NVIDIA A40 48 GB server?
Directly from Resilient Tec in Knoxville, Tennessee (tested builds): R740xd – A40 Build — $10,959 at resilient-tec.com · R740xd – Dual V100 Build — $5,849 at resilient-tec.com · R740xd – GPU Ready — $2,995 at resilient-tec.com

Resilient Tec builds and tests refurbished GPU servers in Knoxville, Tennessee, and ships them racked, burned in and under a lifetime warranty. Tell us the model you intend to run and the number of people who will hit it, and we'll tell you honestly whether one A40 is the right answer — including when it isn't.

Ready to Buy?

Tested Dell PowerEdge builds shipping from the US, with the GPU enablement kit, high-performance cooling and 1600 W supplies already installed (prices below checked September 11, 2026):

• R740xd – A40 Build — $10,959 — 1× NVIDIA A40 48 GB, 2× Xeon Silver 4210R, 128 GB DDR4, 24 SFF bays. The configuration benchmarked above.
• R740xd – Dual V100 Build — $5,849 — 2× Tesla V100 32 GB (64 GB total). Lower cost per GB, but HBM2 bandwidth and a workload that parallelizes across two cards.
• R740xd – GPU Ready (No GPU Included) — $2,995 — everything but the card. Bring your own A40, L40S or RTX 6000.

Sizing for a team rather than a workstation? Tell us the model and the concurrency and we'll spec the node — including a two-card NVLink configuration if 96 GB is where your workload actually lives.

You might also need

• Putting an A40 in an R740xd — the enablement kit, heat sinks, fans and power the card needs before any of these numbers apply
• The cloud sent you a bill — on-prem versus rented inference, with the math
• On-premise AI agents versus cloud AI costs
• GPU & AI infrastructure guide — accelerators, enablement kits and everything around them
• Server memory — 128 GB is a starting point, not a ceiling
• Enterprise storage — model weights, datasets and checkpoints across 24 bays
• DAC cables and network switches — 10 GbE uplinks so the node isn't waiting on the network