Short answer: if your people are already pasting company documents into ChatGPT, you have an AI policy whether you wrote one or not. Running your own large language model (LLM) on hardware you own keeps that data in the building, replaces per-seat and per-token bills with a one-time hardware cost, and is now practical for businesses of 10 people or 1,000. Open-weight models have caught up enough to do most everyday work, and a refurbished GPU server that runs them costs about what a year of team AI subscriptions does. This guide covers how to decide, where to start, how to size it, and what tends to go wrong.
The short version
Start with one job, one model, one machine
- Pick one job first: usually "ask questions of our own documents."
- Size by memory, not speed: the model has to fit in GPU memory with room left over for your users.
- A common first box: one NVIDIA A40 48 GB server runs 27–34B models comfortably and 70B models at 4-bit. Resilient Tec's R740xd A40 Build is $12,159, tested and in stock.
- Plan the room: power, heat and network matter as much as the GPU.
- Keep the cloud for what it does best: frontier reasoning and burst capacity. Most businesses end up hybrid.
Prices checked November 17, 2026. Not sure what to buy? Tell us the job and the headcount and we'll size it.
Why businesses are bringing AI in-house now
Three things changed in the last year.
1. The open models got good enough. A year or two ago, running your own model meant accepting a clearly worse assistant. That's no longer true for most office work. Epoch AI estimates that since January 2026 the best open-weight models have trailed the closed frontier by about four months on average. Models such as Qwen, DeepSeek, GLM and Llama can be downloaded and run on your own hardware, with licenses that allow commercial use (check each one).
2. Your data is already leaving. Zscaler's 2026 AI Threat Report found enterprise data sent to AI tools nearly doubled year over year, and Cyberhaven's 2026 research puts the share of that data that's sensitive at close to 40%. Banning the tools tends to push use underground rather than stop it. Giving staff a private assistant that's as easy to use is the more reliable fix.
3. Cloud AI bills are hard to predict. In Cloudian's February 2026 survey of 203 enterprise IT decision-makers, 40% said cloud AI spending had come in above projections. Broadcom's 2026 Private Cloud Outlook, a survey of 1,800 IT leaders, found 56% running or planning to run production AI inference on private infrastructure, while public cloud as the primary home for inference fell from 56% to 41% in a year.
None of this means the cloud is going away. It means the default is shifting from "rent all of it" to "own the steady, private work and rent the rest."
What a private LLM actually does for a business
The useful early projects are rarely flashy. They're the jobs where the data is sensitive and the work is repetitive:
- Ask-our-documents assistant. Policies, manuals, contracts, past proposals and tickets, searchable in plain English. The technical term is retrieval-augmented generation (RAG): the model reads the relevant documents before it answers, so answers come from your material instead of the internet.
- Drafting with client data. Quotes, reports, letters and summaries that use names, numbers and terms you wouldn't paste into a public chatbot.
- Document processing. Pulling fields out of invoices, purchase orders, intake forms and contracts, overnight and in bulk, with no per-page fee.
- Code assistant on private repositories. Help for your developers without uploading the codebase anywhere.
- Support and sales drafts. First drafts of replies grounded in your own product data and past tickets, reviewed by a person before they go out.
- Offline or restricted sites. Plants, labs and secure facilities where an internet-dependent tool isn't an option.
A real example: one customer's four-box cluster
Resilient Tec customer James Lee built a four-node cluster from Acer Veriton GN100 systems, the same NVIDIA GB10 chip as the DGX Spark, each about the size of a hardback book. He linked them with our 200G cables, first in a direct ring and later through a switch, and shared his measurements with us. The full benchmarks are here.
What matters for a business reader:
- It ran a 235-billion-parameter model. Qwen3-235B is in the class of open models that compete with paid cloud assistants, and four small boxes held it.
- It served many people at once. Total output reached 254.2 tokens per second at 64 simultaneous requests. At 4 requests each one got about 18.8 tokens per second, faster than most people read; at 16 requests, about 9.0. In our estimate, that's enough for roughly 30 to 60 people using it through a workday.
- The room mattered. His early runs were in a closet that got too hot. Moving the cluster to his shop was part of getting better results. Heat is a planning item, not an afterthought.
You don't need four boxes to start. One server handles most first projects. The point is that a capability that used to require a data center now fits on a shelf.
How to do it: six steps
1. Pick the first job
Choose one use case with a clear owner, a defined set of documents and a way to tell whether it's working. "Answer questions from our employee handbook and SOPs" beats "AI for everything." The first project teaches you what your people actually use, and that drives every later decision.
2. Pick the model size the job needs
Bigger models reason better but need more memory and serve fewer people per GPU. Rough guide for 4-bit models, the usual format for running locally:
| Model class | GPU memory to plan for | Good for |
|---|---|---|
| 7–14B parameters | 16–32 GB | Classification, extraction, short drafts, high user counts |
| 27–34B | 24–48 GB | Most document Q&A and drafting; the sweet spot for one 48 GB card |
| 70B | 48–96 GB | Harder reasoning and longer documents; comfortable on 96 GB |
| 200B+ class | 128 GB and up | Near-frontier quality; GB10 systems, clusters or multi-GPU servers |
Plan to leave 20–25% of memory free. Every active conversation needs working memory of its own, so a model that only just fits will serve one person, not a team. We go deeper on this in What Can You Actually Run on a 48GB NVIDIA A40?
3. Count your users, not your employees
What matters is how many people use it at the same moment, not how many have a login. In most offices that's a small fraction of headcount. Published vLLM tests show one A40 serving 100 simultaneous requests on an 8B model. A 70B model on the same card is a few-people tool. If you need both a large model and many users, that's two cards or two machines.
4. Choose the hardware path
There are three common routes, and the right one depends on where the machine will live:
- Rack server with data-center GPUs. Dell PowerEdge or HPE ProLiant with NVIDIA A40, V100, T4 or A100 cards. Redundant power, remote management, years of service life. Best if you have a server room or a rack.
- GB10 desk-size systems. DGX Spark and its partner versions from ASUS, Dell, HP, Acer and others: 128 GB of unified memory each, quiet enough for an office, and they can be linked in twos, threes and fours. Best for very large models without a rack.
- Bring your own GPU. If you already own cards, a GPU-ready chassis with the power, cooling and cabling done is the cheapest route.
5. Plan the room
- Power. A loaded two-GPU server can draw well over 1 kW. Check the circuit and the UPS before the server arrives, and size the power supplies correctly; see sizing power supplies for a GPU server.
- Heat and noise. Rack servers are loud and hot. A closet with no airflow will throttle the hardware, as James found.
- Network. One server on 10 GbE is fine. Once you link machines to split a large model, the connection between them sets your speed. That's where 100G–400G cables and switches come in; see On-Prem AI Inference Fabric: Do You Need a Data Center Switch?
- System memory and storage. GPU memory gets the attention, but the server's RAM and fast storage affect load times and RAG performance; see how much system RAM an LLM server needs.
6. Pilot, measure, expand
Run the first job with a small group for 30 days. Track how many people use it, what they ask and where it fails. The common, well-supported open-source tools are Ollama or vLLM to run the model and Open WebUI for a ChatGPT-style interface. Then decide whether the next step is a bigger model, more users or a second job.
What it costs, honestly
The full own-versus-rent math is in The Cloud Sent You a Bill. The quick version for a team:
| Cloud subscription | Your own server | |
|---|---|---|
| Up-front | None | About $4,693–$18,920 for the refurbished builds most teams start on |
| Ongoing | ChatGPT Business: $20 per user per month on annual billing. 25 users = $6,000 a year; 50 users = $12,000. Premium seats are $100 per user per month ($30,000 a year for 25). | Mostly power. A server averaging 600 W all day uses about 5,256 kWh a year, roughly $683 at 13¢ per kWh (your rate will differ) |
| Data location | The provider's cloud, under their terms | Your building |
| Model quality | The newest frontier models | Open-weight models, a few months behind the frontier on average |
| Who runs it | The provider | You, or a partner. Budget some IT time |
At 50 users, one year of standard seats ($12,000) is about the price of a tested A40 server ($12,159) that keeps running for years. That isn't a like-for-like trade, since the cloud gives you frontier models and no maintenance, but it shows why steady, private workloads are moving in-house.
Where the cloud still wins
- The hardest reasoning and long agentic tasks, where closed frontier models still lead.
- Bursty or experimental work that needs a lot of GPUs for a few hours, then none.
- Teams with no one to look after a server.
Most businesses that bring AI in-house keep a cloud subscription for those cases. A sensible rule: sensitive and steady work runs locally; frontier and occasional work goes to the cloud.
Five mistakes we see
- Buying for the biggest model on the leaderboard. A 30B model that serves your whole team beats a 400B model that serves one person.
- Forgetting the working memory. The model loading isn't the same as the model serving users. Leave headroom.
- Putting the server in a closet. Heat throttles performance and shortens hardware life.
- Underestimating installation details. GPU servers need the right risers, power cables, heat sinks and fans. Missing parts are the most common reason a build stalls; see 8 GPU installation problems and their fixes.
- Skipping the pilot. Spending big before you know what your people will use is how AI projects end up as unused hardware.
Where to start: common first builds
These are representative starting points, not the only answers. Every one is a refurbished enterprise system, tested in Knoxville and covered by our lifetime warranty.
| Situation | Example build | What it gives you | Price at Resilient Tec |
|---|---|---|---|
| Bring your own GPU | R740xd GPU Ready (no GPU) | Power, cooling and GPU cabling done; add your cards | $2,995, in stock |
| Small models, many users | R740xd Dual T4 | 32 GB total; extraction, classification, 7–14B models | $4,693, not in stock, request a quote |
| Budget mid-size | R740xd Dual V100 | 64 GB total across two cards | $5,849, in stock |
| The common team server | R740xd A40 Build | 48 GB; 27–34B models comfortably, 70B at 4-bit | $12,159, in stock |
| Bigger models, more headroom | R740xd Dual A40 | 96 GB; 70B-class models comfortably | $18,920, in stock |
| Very large models, no rack | NVIDIA DGX Spark 128GB | 128 GB unified memory; link 2–4 for 200B+ models | $6,950, not in stock, request a quote |
| High concurrency, data center | Dell R7525 with 2× A100 | Two data-center A100 GPUs | $57,643, available to order |
Prices checked November 17, 2026. More configurations in the GPU Servers & On-Prem AI Guide.
How Resilient Tec helps
We're an IT hardware company in Knoxville, Tennessee, and our founder has spent 17 years in the enterprise hardware business. Here's what we bring to an in-house AI project:
- Honest sizing. Tell us the job, the model you're considering and how many people will use it. We'll tell you what to buy, including when a smaller machine is the right answer.
- Tested, complete builds. Our GPU servers ship with the enablement kit, cooling, power supplies and cables installed and tested, so you don't discover a missing part after delivery. Sold-out configurations can usually be rebuilt to order in one to two days.
- Refurbished enterprise hardware. Data-center servers and GPUs at a fraction of new prices, which is often what makes a first project affordable.
- A lifetime warranty on everything we sell, servers and GPUs included, with advance replacement on large items when we have stock. Plus a compatibility guarantee.
- The connections between machines. Cables, transceivers and switches for linking GPU servers and GB10 systems are our specialty, and we publish real customer results like James's rather than spec-sheet promises.
- Business purchasing. POs and net terms available.
Request a quote with your use case and headcount, and we'll come back with a configuration and a price.
FAQ
Can a small business run its own LLM?
Yes. A single refurbished GPU server or a desk-size GB10 system runs open-weight models that handle document Q&A, drafting and summarizing for a team, with nothing leaving your building. Start with one job and one machine. Resilient Tec's tested GPU server builds start at $2,995 for a GPU-ready chassis, and $12,159 with an NVIDIA A40 48 GB installed.
How much does it cost to run an LLM on-premises?
The hardware is a one-time cost: about $4,693 to $18,920 for the refurbished Dell builds most small and mid-size teams start on, at Resilient Tec. Running costs are mainly power: a server averaging 600 W around the clock uses about 5,256 kWh a year, roughly $683 at 13 cents per kWh. For comparison, 50 ChatGPT Business seats cost $12,000 a year at OpenAI's annual rate. (Prices checked November 17, 2026.)
Is a local open-weight model as good as ChatGPT or Claude?
Close, but not equal. Epoch AI estimates the best open-weight models have trailed the closed frontier by about four months on average since January 2026. For focused work on your own documents, that gap rarely matters. For the hardest reasoning and long agentic tasks, the closed frontier models still lead, which is why many businesses keep a cloud subscription for those jobs.
What hardware do I need to run an LLM in-house?
Mostly GPU memory. A 7–14B model fits in 16–32 GB, a 27–34B model runs well in 48 GB, a 70B model at 4-bit needs about 40 GB plus room for users, and 200B-class models need 128 GB or more. A single NVIDIA A40 48 GB server is the most common starting point for a team; GB10 systems such as the DGX Spark offer 128 GB in a desk-size box.
How many people can one on-prem LLM server handle?
It depends on model size. One A40 running an 8B model served 100 concurrent requests at 1,705 tokens per second of total output in published vLLM tests. A Resilient Tec customer's four-node GB10 cluster ran a 235B model at 143.9 tokens per second total across 16 concurrent requests; in our estimate that serves roughly 30 to 60 people through a workday.
Does my data leave the building with an on-prem LLM?
No, not unless you connect it to something that sends it out. The model, your documents and every prompt stay on hardware you own, behind your own firewall. That is the main reason regulated and client-confidential businesses bring AI in-house.
Where can I buy a server to run an LLM on-premises?
Directly from Resilient Tec at resilient-tec.com, tested and shipping from Knoxville, Tennessee, with a lifetime warranty, POs and net terms: R740xd GPU Ready $2,995, Dual V100 $5,849, A40 Build $12,159, Dual A40 $18,920. Sold-out configurations can usually be rebuilt to order in one to two days. (Prices checked November 17, 2026.)
Sources
- Epoch AI: open-weight models vs. the closed frontier
- Infosecurity Magazine on the Zscaler 2026 AI Threat Report
- Cloudian Enterprise AI Infrastructure Survey 2026
- Broadcom Private Cloud Outlook 2026, via The Register
- OpenAI: ChatGPT Business seat pricing
You Might Also Need
- Why teams are bringing LLMs in-house on DGX Spark and GB10: the engineer's view of privacy, fine-tuning and control
- The Cloud Sent You a Bill: on-prem vs. rented inference, with the math
- The complete parts list for an R740xd or DL380 Gen10 AI inference server
- EPYC vs. Xeon for GPU inference servers
- Custom Dell R740xd GPU servers: build options and real prices
- DGX Spark & GB10 Clustering Guide