GPU Prices Up 48% in Two Months. I Run LLMs in My Garage.
Source: Dev.to
The cloud GPU crisis
Nvidia Blackwell rental: $4.08/hr (up from $2.75 two months ago, a 48% increase).
CoreWeave: raised prices 20% and extended minimum contracts from 1 year to 3 years.
Anthropic: restricted their newest model to roughly 40 organizations.
OpenAI’s CFO Sarah Friar: “We’re making some very tough trades at the moment on things we’re not pursuing because we don’t have enough compute.”
Tom Tunguz identifies five dynamics hurting smaller players:
- Relationship‑based access (you need to know someone)
- Price barriers
- Speed uncertainty
- Rising commodity costs
- Forced migration to alternatives
The last one is the interesting part.
Forced alternatives means local inference
When cloud GPU prices jump 48% in two months, when minimum contracts stretch to three years, and when model providers limit access, the market pushes people toward two alternatives: smaller models and on‑premise infrastructure.
I’m already there.
My setup
I run local LLM inference on consumer hardware in my garage in Nashville:
- RTX 5090 (primary, 32 GB VRAM)
- RTX 5070 Ti (secondary, 16 GB VRAM)
- RTX 3070 (legacy, 8 GB VRAM)
Llama 3.1 8B runs on the 5070 Ti via llama.cpp. Inference costs are just electricity—no API calls, no rate limits, no three‑year contracts, no relationship‑based access.
The 5090 handles bigger models when needed. Its 32 GB VRAM fits quantized versions of models that would cost $4.08/hr to rent on Blackwell hardware.
The math
Assume you need 4 hours of GPU time per day for inference.
- Cloud (Blackwell): $4.08 /hr × 4 hrs × 30 days = $489/month (and you still need access).
- Local (RTX 5090): Card cost ≈ $2,000. Electricity for 4 hrs/day is about $15/month. After ~4 months you break even; thereafter it’s $15/month forever.
Result: $489/month vs $15/month, and you own the hardware.
Note: This ignores the capability gap—Blackwell is faster and can handle larger models. For the 8 B–70 B parameter range that most production inference needs, consumer GPUs are more than enough.
When local doesn’t work
Local inference isn’t a universal answer. You still need cloud GPUs when:
- Training models (not just inference)
- Supporting 100+ concurrent users with low latency
- Running 400 B+ parameter models that don’t fit in consumer VRAM
- Requiring enterprise SLAs and uptime guarantees
Most builders aren’t doing those things; they need inference for agents, internal tools, or prototypes, which consumer hardware can handle.
The real moat
Compute scarcity is a moat for whoever controls the GPUs—and also for whoever figures out how to not need them. Running an AI agent on a $2,000 GPU in your office instead of a $4.08/hr cloud rental gives a cost advantage that widens each month as cloud bills rise and consumer GPU cost per capability falls.
What to do
- Audit your inference costs. Identify API spend that could be moved locally.
- Start with one model locally. Llama 3.1 8B on an RTX 3070 is a good entry point—$0/month after hardware purchase.
- Keep cloud for what needs cloud. Training, high‑concurrency production, frontier models.
- Guard your cloud budget. Set runtime limits so agents can’t burn money unattended.
pip install agentguard47AgentGuard works with any provider, cloud or local. Budget caps, loop detection, and timeout guards protect your agent runs regardless of where inference happens.