Why I Decided to Test Llama 3.3 70B on a Raspberry Pi 5
I knew this was going to be a disaster before I even started.
Running a 70-billion parameter model on a Raspberry Pi 5 sounds absurd, and it is. But I wanted to understand exactly where the system would break, and more importantly, whether the bottleneck would be thermal throttling or memory bandwidth. I’ve been running smaller models on various ARM devices, and the Pi 5 seemed like an interesting edge case to explore.
My goal wasn’t to make this practical. It was to measure failure modes in a controlled way and see what happens when you push consumer ARM hardware past any reasonable limit.
My Actual Setup
Here’s what I worked with:
- Raspberry Pi 5 (8GB RAM model)
- Active cooling with a small heatsink fan
- Samsung 980 PRO 1TB via USB 3.0 adapter (NVMe over USB)
- Quantized Llama 3.3 70B model (Q4_K_M format from llama.cpp)
- Ollama running directly on the Pi (no containers, no virtualization)
- Thermal monitoring via vcgencmd
- Memory stats from /proc/meminfo
I didn’t use Docker because I wanted direct access to system metrics without any abstraction layer getting in the way. The USB NVMe was necessary because even a quantized 70B model won’t fit in 8GB of RAM, so constant swapping to storage was inevitable.
What Happened During Model Loading
The first problem hit immediately during model load. The Pi started pulling the quantized model from storage, and memory filled up within seconds. Swap kicked in almost instantly, and the system became unresponsive for about 90 seconds.
Temperature climbed from 45°C to 68°C during this phase. The CPU was pegged at 100% across all cores, but thermal throttling hadn’t started yet. The Pi 5’s thermal design can handle brief bursts up to 80°C before throttling kicks in.
What surprised me was that the bottleneck here wasn’t thermal—it was memory bandwidth. The system was thrashing between RAM and swap, and the USB 3.0 interface (even with NVMe) couldn’t keep up with the memory access patterns llama.cpp was trying to execute.
Memory Bandwidth Reality Check
The Pi 5 has LPDDR4X-4267 RAM, which theoretically provides around 17 GB/s of bandwidth. That sounds decent until you realize that Llama 3.3 70B, even quantized to 4-bit, still requires moving massive amounts of data for every token generated.
I monitored this with:
watch -n 1 'cat /proc/meminfo | grep -E "MemTotal|MemAvailable|SwapTotal|SwapFree"'
Swap usage climbed to 6.2GB almost immediately. The system was constantly paging memory in and out, and the USB 3.0 link to the NVMe drive became the real chokepoint. Even though the 980 PRO can theoretically handle 7000 MB/s reads, USB 3.0 caps out around 400-500 MB/s in practice.
Thermal Throttling vs Memory Bandwidth: What Actually Broke First
Once the model finished loading (which took about 4 minutes), I ran a simple inference test with a short prompt. The system took 47 seconds to generate a single token.
During this time:
- CPU temperature stayed between 72-76°C
- No thermal throttling was logged in dmesg
- Swap usage remained constant at around 6GB
- USB storage I/O wait time spiked to 89% according to iostat
The answer was clear: memory bandwidth, not thermal throttling, was the primary bottleneck. The Pi never got hot enough to throttle because it was spending most of its time waiting on storage I/O. The CPU couldn’t even stay busy enough to overheat.
I verified this by checking throttling events:
vcgencmd get_throttled
Output was 0x0, meaning no throttling had occurred. If thermal throttling were active, this would show a non-zero value.
What Happens When You Force Thermal Load
To isolate thermal behavior, I disabled swap entirely and forced the model to stay in RAM by using a smaller quantized version (7B model instead of 70B). This let the CPU actually work continuously without waiting on I/O.
With the 7B model:
- Token generation took 2.3 seconds per token
- CPU temperature climbed to 82°C
- Thermal throttling kicked in after about 30 seconds
- Clock speed dropped from 2.4 GHz to 1.8 GHz
This confirmed that thermal throttling is possible on the Pi 5, but only when the CPU can stay busy. With the 70B model, memory bandwidth starvation prevented the CPU from generating enough heat to throttle in the first place.
Key Observations
Memory bandwidth is the real limit. The Pi 5’s RAM and storage I/O can’t keep up with the access patterns required by large language models. Thermal throttling never became a factor because the CPU spent most of its time idle, waiting for data.
Quantization helps, but not enough. Even at Q4_K_M (4-bit quantization), the model was too large to fit in RAM, forcing constant swapping. The performance hit from storage I/O completely overwhelmed any benefit from reduced model size.
USB 3.0 is a bottleneck. Even with a fast NVMe drive, the USB 3.0 interface limited throughput to around 400 MB/s. This made swapping unbearably slow.
Active cooling didn’t matter. I ran the same test with and without the heatsink fan. Temperature differences were negligible because the CPU wasn’t generating enough sustained load to overheat.
What I Learned
This experiment confirmed what I suspected: running a 70B model on a Pi 5 is a memory bandwidth problem, not a thermal one. The system never got hot enough to throttle because it was too busy waiting on storage.
If I were trying to run smaller models (7B or 13B), thermal throttling would become relevant, especially during sustained inference. But for anything 30B or larger, memory and I/O are the hard limits.
The Pi 5 is a capable device for many tasks, but hosting large language models isn’t one of them. The architecture simply isn’t designed for the memory access patterns these models require. Even with aggressive quantization and fast storage, the system spends more time waiting than computing.
For anyone considering this setup: don’t. If you want to run Llama 3.3 70B, use a system with at least 64GB of RAM and real PCIe storage. The Pi 5 is great for smaller models or edge inference tasks, but it’s not built for this workload.
