Local LLM for Law Firms: Privacy-First AI in 2026

Every law firm I have talked to in the last six months wants AI and cannot use the AI they want. The barrier is not technology. It is that "send a client matter to OpenAI" reads in the ethics opinion the way "send a client matter to a non-lawyer third party" reads in every state bar rule. The answer the hosted vendors give — SOC 2, BAA-on-request, encryption-at-rest — does not clear the trust threshold for work-product privilege or for the matter file your senior partner does not want in someone else's data center.

Local LLMs are the answer that actually clears the bar. The model runs on your hardware. The matter file never leaves your office. The vendor relationship is "we sold you a server," not "we hold your client list." This post is the practical guide to setting that up in mid-2026 — what hardware to buy, what model to run, what legal tasks each model is actually good at, and where local still falls short of Claude or GPT.

Why local is finally viable in 2026

Two things changed in the last 18 months. First, the open-weight model gap closed. Qwen 2.5, Llama 3.3, and the DeepSeek-R1 distills are within 10-15% of frontier models on the legal tasks lawyers actually want — drafting, summarization, citation checking, discovery review, deposition prep. That gap was 40% in 2024.

Second, the hardware got cheap. A single H100 is still $25-30K, but you do not need an H100 for most legal work. A workstation with 2x RTX 3090s (used, $700 each) and 128GB system RAM runs a quantized 70B model at 12-25 tokens/sec — fast enough for any drafting or analysis task. Total build: under $5,000.

The hardware decision tree

Three brackets, by firm size and budget.

Solo / 2-attorney firm: $3,000-5,000

A workstation with a single RTX 4090 (24GB VRAM) or a Mac Studio M3 Ultra with 96-192GB unified memory. The Mac runs the bigger models slower but with no fan noise and trivial setup. The 4090 box is faster but you are running Linux and managing CUDA. For a solo, the Mac Studio is usually the right call — you turn it on, run Ollama, and forget it.

Small firm, 5-15 attorneys: $8,000-15,000

A dedicated workstation with 2x RTX 3090s or a single RTX 6000 Ada (48GB VRAM). 128GB+ system RAM. NVMe storage for the model cache. Ubuntu Server with Ollama or vLLM. Network-accessible so the whole firm hits it from their laptops. The 2x 3090 build is the budget winner; the RTX 6000 Ada is the no-headaches version.

Mid-sized firm, 20+ attorneys: $30,000-70,000

Single H100 or H200 in a rack server, or 2-4 RTX 6000 Ada cards in a high-end workstation. At this size you also need MFA, role-based access, audit logging, and probably a managed deployment partner who is on the hook when the box stops responding at 4pm on a Friday. The hardware is the cheap part of this bracket; the operational layer is the rest.

Model picks by legal task

No single model is best at everything. The right move is to run two or three and route by task. Ollama makes this trivial — `ollama pull qwen2.5:72b` and `ollama pull llama3.3:70b` give you both, and you switch with an API parameter.

Drafting and summarization: Qwen 2.5 72B Instruct

The strongest open model on legal prose in our testing. Handles tone-matching to firm style, holds long context (128K tokens), follows formatting instructions reliably. Better than Llama 3.3 on briefs, motions, and client letters. Roughly 80-85% of the quality of Claude Opus on the same prompts — the gap shows up on the most complex synthesis, not on routine drafting.

Discovery review and long-document analysis: Llama 3.3 70B

Slightly weaker prose, slightly better at extraction and structured analysis. The Llama context window is workable and the model is unusually fast on inference. Pair it with a vector store (Chroma or Qdrant) and you have a local discovery review pipeline that processes 10,000 pages in an evening.

Reasoning and case analysis: DeepSeek-R1 Distill 70B

The reasoning models — DeepSeek-R1 and its distills — think longer and produce better analysis on multi-step legal questions. Use this for "compare these three contract clauses against state law" tasks, not for "draft me a settlement letter." Slower per response (the chain-of-thought traces are visible and long), but the answer quality on hard reasoning lands within 10% of GPT-5 with reasoning enabled.

Fast intake and triage: Qwen 2.5 32B

The smaller Qwen runs on a single 4090 and responds near-instantly. Use it for client intake summarization, conflict-check pre-screening, and the high-volume short tasks where latency matters more than synthesis depth.

Multilingual matters: Mistral Large 2 (quantized)

For cross-border matters, EU clients, or firms with non-English first-language clients, Mistral is the strongest open model on European languages by a meaningful margin. The 123B parameter model quantized to 4-bit runs on a single 48GB GPU.

Where local genuinely falls short

Honest list. Local models are weaker than Claude Opus 4.7 or GPT-5 on three things that matter for legal work.

First, citation accuracy. Frontier hosted models still hallucinate cases — but they hallucinate less than open models. If you are running a brief through a local LLM and asking it to add supporting citations, every citation needs a human verifying pass against Westlaw or Lexis. (Frankly, every citation from any LLM needs that pass — but the rate is worse on local.)

Second, instruction-following on complex multi-part prompts. A 70B local model will sometimes drop one of seven instructions in a complex prompt. Claude rarely does. The fix is to break complex prompts into chained calls — slower, but reliable.

Third, the very long synthesis tasks. Asking a local model to read 800 pages of deposition transcripts and produce a 20-page strategic memo is technically possible and practically disappointing. For that specific use case, the right move is a hosted model with a strong DPA and a narrow scope, or a hybrid where local does the extraction and a hosted model does the final synthesis on the structured intermediate output.

The hybrid pattern that actually works

Most law firms we work with end up at the same architecture. Local for anything touching client matter content — intake summaries, discovery review, drafting against matter files, deposition prep. Hosted (with a DPA and matter-name scrubbing) for the heavy synthesis work where the local model is visibly worse. The matter file itself never leaves; only stripped, anonymized intermediate outputs do.

This is the same pattern that drove the OpenHuman local-first agent to the top of Product Hunt this month and the same reason DeepSeek V4 going open matters more for regulated verticals than for anyone else. The hosted-agent vendors will spend 2026 negotiating BAAs they should have had two years ago. The local-first stack already cleared the bar.

The setup, end-to-end

On the workstation: install Ollama, pull two or three models, expose the API on the firm LAN. On every attorney laptop: a chat client (Open WebUI is the cleanest free option) pointed at the workstation. Add a vector store and a document loader for matter file retrieval. Set up MFA and audit logging on the workstation. Test on a single attorney for two weeks before rolling firm-wide.

Total elapsed time for a competent IT consultant: one to two weeks. Ongoing maintenance: a few hours a month, mostly model updates and the occasional OS patch. The hardware will last 3-5 years. The amortized cost runs $150-400 per attorney per year — versus the $200-800 per attorney per month that the hosted legal AI vendors quote.

What to do this week

Three steps. First, run an inventory: which AI tools are attorneys using right now, what data is going through them, what does your engagement letter say about it. Second, pilot a local setup on one workstation with one attorney for two weeks — Ollama plus Qwen 2.5 72B is a $0 software install on a $5,000 workstation. Third, if the pilot clears, scope a firm-wide rollout against your malpractice carrier's requirements before you sign the next year of any hosted AI subscription.

The hosted vendors are not the enemy. They are the right call for general research, marketing, and any non-privileged work. The point is that the privileged work — matter files, client communications, settlement strategy, anything that would make your malpractice underwriter wince if it leaked — has a local-first answer in 2026 that did not exist in 2024. Use it.

"The question is not "is AI smart enough for legal work." The question is "where does my matter file end up after I click send." Local-first answers that with one word."