When Containers Stop Talking: My Journey Through Docker MTU Hell
I run a mix of self-hosted services on Docker—monitoring tools, automation workflows, databases, and web apps. For months, everything worked fine. Then one day, I started seeing random timeouts. Not everywhere. Not consistently. Just enough to make me question my sanity.
A container would fail to reach another service. Five minutes later, it worked. Then it failed again. Logs showed nothing useful—just connection timeouts. No errors. No pattern I could see.
It took me two frustrating weeks to track down the real cause: MTU mismatch on Docker’s bridge network.
What Actually Happened
My setup at the time:
- Proxmox host running multiple VMs
- One VM dedicated to Docker containers
- PPPoE connection to my ISP (this matters)
- Default Docker bridge network with standard 1500 MTU
The symptoms were maddening:
- Small HTTP requests worked fine
- Larger API responses timed out
- Database queries failed unpredictably
- DNS lookups worked, but actual data transfer didn’t
What made it worse: I could ping containers from each other. I could curl small endpoints. Everything looked fine at first glance.
How I Found the Problem
I started with the usual checks. Verified containers were on the same network. Checked DNS resolution. Looked at firewall rules. All good.
Then I tried something different. I ran a ping test with varying packet sizes:
docker exec my-container ping -M do -s 1400 target-container
This worked fine. But when I increased the size:
docker exec my-container ping -M do -s 1472 target-container
Timeouts. Every single time.
That’s when it clicked. The -M do flag prevents packet fragmentation. The 1472-byte payload, plus 28 bytes of IP and ICMP headers, equals exactly 1500 bytes—the standard MTU.
My packets were being silently dropped somewhere in the network path.
Why This Happened
My ISP uses PPPoE, which adds an 8-byte header to every packet. That means the effective MTU on my WAN connection is 1492 bytes, not 1500.
Docker’s default bridge network uses 1500 MTU. When a container tried to send a packet larger than 1492 bytes to an external service (or even to another container routing through certain network paths), the packet got dropped because fragmentation wasn’t allowed.
The reason it was intermittent: small packets fit fine. Only larger transfers—like JSON responses from APIs or database result sets—hit the MTU ceiling.
How I Fixed It
I needed to lower Docker’s bridge MTU to match my actual network capacity. Here’s what I did:
First, I checked the current MTU on Docker’s bridge:
docker network inspect bridge | grep -i mtu
It showed 1500, as expected.
I created a new bridge network with a lower MTU:
docker network create --driver bridge --opt com.docker.network.driver.mtu=1492 lowmtu
Then I moved my containers to this new network. For containers defined in Docker Compose, I updated the compose file:
version: '3.8'
services:
my-service:
image: my-image
networks:
- lowmtu
networks:
lowmtu:
driver: bridge
driver_opts:
com.docker.network.driver.mtu: "1492"
After restarting the containers, the timeouts stopped. Completely.
What I Learned About MTU
MTU (Maximum Transmission Unit) is the largest packet size a network can transmit without fragmentation. If a packet exceeds the MTU at any point in its path, it either gets fragmented or dropped.
In my case, the path was:
- Container (MTU 1500)
- Docker bridge (MTU 1500)
- Host network interface (MTU 1500)
- PPPoE connection (MTU 1492)
The mismatch at step 4 caused the problem.
What’s tricky: many protocols handle this gracefully through Path MTU Discovery. But when firewalls block ICMP messages (which signal “packet too big”), PMTUD fails silently. That’s exactly what was happening in my setup.
How to Debug This Yourself
If you’re seeing similar intermittent timeouts, here’s how I’d approach it:
1. Test with different packet sizes
docker exec container-name ping -M do -s 1400 target docker exec container-name ping -M do -s 1472 target
If smaller packets work but larger ones fail, you likely have an MTU issue.
2. Check your actual network MTU
ip link show | grep mtu
Look at your WAN interface. If you’re on PPPoE, DSL, or certain VPN setups, it’s probably not 1500.
3. Verify Docker’s bridge MTU
docker network inspect bridge | grep -i mtu
Compare this to your actual network MTU. If Docker’s is higher, you have a mismatch.
4. Test with tcpdump
docker exec container-name tcpdump -i eth0 -n 'icmp'
Run this while testing connectivity. Look for “frag needed” ICMP messages. If you see them, it confirms MTU issues.
Other Scenarios I Encountered
After fixing my main setup, I ran into MTU problems in two other places:
VPN tunnels: When I set up WireGuard for remote access, I had to lower the MTU on both the VPN interface and Docker networks. WireGuard adds its own overhead, so I ended up using 1420 MTU for containers that needed to work over VPN.
Nested virtualization: Running Docker inside a Proxmox VM that itself runs on a virtualized network added multiple layers of encapsulation. I had to drop the MTU even further—down to 1450—to account for all the overhead.
What Didn’t Work
Before I figured out the MTU issue, I tried a lot of things that seemed reasonable but didn’t help:
- Adjusting Docker’s DNS settings
- Changing the bridge network subnet
- Disabling IPv6 on containers
- Tweaking TCP keepalive settings
- Rebuilding containers from scratch
None of these addressed the real problem. They just wasted time.
I also tried setting MTU on individual containers using --network-opt, but that doesn’t actually work. MTU must be set at the network level, not per container.
When to Suspect MTU Issues
Based on my experience, consider MTU problems if you see:
- Timeouts only on larger data transfers
- Small requests work, large ones fail
- Inconsistent behavior that seems random
- Problems that appear after network changes (new ISP, VPN, etc.)
- Issues that affect some protocols but not others
If basic connectivity tests (ping, DNS) work fine but actual application traffic fails, MTU is a strong candidate.
My Current Setup
I now run all my Docker containers on a custom bridge network with MTU set to 1492. This matches my PPPoE connection and leaves a small buffer for any additional overhead.
For containers that need to work over WireGuard VPN, I use a separate network with MTU 1420.
I also added MTU testing to my monitoring setup. A simple script pings key services with large packets every few minutes. If it starts failing, I get an alert before users notice problems.
Key Takeaways
MTU issues are frustrating because they’re invisible until they’re not. You can’t see them in logs. They don’t show up in basic connectivity tests. They just cause random failures that make you question everything.
The lesson I learned: when debugging network problems, test with different packet sizes early. Don’t assume MTU 1500 works everywhere. It often doesn’t.
If you’re running Docker on PPPoE, DSL, or through VPN tunnels, check your MTU settings before you waste weeks troubleshooting phantom problems like I did.
