Setting up Portainer edge agents with mTLS authentication to manage remote Docker hosts across split-tunnel VPN connections

Why I Set This Up

I run multiple Docker hosts spread across different locations — some on my home network, others at remote sites connected through split-tunnel VPNs. Managing these hosts individually became tedious, especially when I needed to deploy containers, check logs, or restart services while away from home.

I already used Portainer to manage my local Docker environments, but the standard agent setup didn't feel secure enough for remote hosts exposed over VPN connections. I wanted cryptographic authentication between the agents and the server — not just network-level isolation. That's when I decided to implement mutual TLS (mTLS) for Portainer's edge agents.

What mTLS Actually Does

In a typical TLS setup, only the server proves its identity to the client. With mutual TLS, both sides authenticate each other using certificates signed by a trusted certificate authority (CA). If either side presents an invalid or unsigned certificate, the connection is rejected.

For my use case, this meant:

• The Portainer server validates that incoming edge agents are legitimate
• The edge agents verify they're connecting to my actual Portainer instance
• Even if someone intercepts the VPN traffic, they can't impersonate either side without the correct certificates

This isn't just theoretical security theater. I've seen VPN configurations get misconfigured, and I didn't want a compromised endpoint to give someone control over my Docker hosts.

My Certificate Setup

I used my existing internal certificate authority — a simple OpenSSL-based CA I maintain for internal services. I didn't want to rely on Let's Encrypt or public CAs for infrastructure that never touches the public internet.

I generated three sets of files:

• A CA certificate (mtlsca.crt) that both the server and agents trust
• A server certificate and key (mtlsserver.crt, mtlsserver.key) for the Portainer server, with serverAuth in the extended key usage field
• A client certificate and key (client.crt, client.key) for each edge agent, with clientAuth in the extended key usage

I made sure the server certificate had the correct Subject Alternative Name (SAN) for the subdomain I use to access Portainer (portainer.internal.vipinpg.com). This is important — if the hostname doesn't match the certificate, the connection will fail.

Generating the Certificates

I won't pretend this was elegant. OpenSSL configuration files are a mess, and I spent more time than I'd like to admit getting the key usage extensions right. Here's the rough process I followed:

1. Created a new CA if you don't have one: openssl req -new -x509 -days 3650 -keyout ca.key -out mtlsca.crt
2. Generated a server key and certificate signing request (CSR) with the correct SAN
3. Signed the server CSR with the CA, specifying serverAuth in the config
4. Repeated the process for the client certificate, using clientAuth instead

The key detail: I had to manually edit the OpenSSL config to include extendedKeyUsage = serverAuth for the server cert and extendedKeyUsage = clientAuth for the client cert. Without these, Portainer rejected the certificates.

Configuring the Portainer Server

I run Portainer on a Proxmox VM using Docker. I stored the CA certificate, server certificate, and server key in /opt/portainer/certs on the host.

My original Portainer deployment command looked like this:

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    portainer/portainer-ee:latest

I modified it to mount the certificate directory and pass the mTLS flags:

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    -v /opt/portainer/certs:/certs \
    portainer/portainer-ee:latest \
    --mtlscacert /certs/mtlsca.crt \
    --mtlscert /certs/mtlsserver.crt \
    --mtlskey /certs/mtlsserver.key

After restarting the container, I checked the logs to confirm mTLS was enabled. There's no explicit "mTLS active" message, but I didn't see any certificate errors, which was a good sign.

What Broke the First Time

The first attempt failed because I forgot to update my internal DNS to point portainer.internal.vipinpg.com to the server's IP. The edge agents couldn't resolve the hostname, so they never even attempted a connection.

The second attempt failed because I used the wrong certificate format. Portainer expects PEM format, but I initially generated the server key in DER format by mistake. Converting it with openssl rsa -in server.key -outform PEM -out mtlsserver.key fixed it.

Deploying Edge Agents with mTLS

For each remote Docker host, I needed to deploy an edge agent with the client certificate and key. I used Docker Standalone for most of my setups.

I copied the CA certificate, client certificate, and client key to /opt/portainer/certs on the remote host. Then I went through the Portainer UI to add a new edge environment, making sure to use portainer.internal.vipinpg.com as the API server URL.

Portainer gave me a deployment command, which I modified to include the mTLS certificates:

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  -v /:/host \
  -v /opt/portainer/certs:/certs \
  -v portainer_agent_data:/data \
  --restart always \
  -e EDGE=1 \
  -e EDGE_ID=<generated-id> \
  -e EDGE_KEY=<generated-key> \
  -e EDGE_INSECURE_POLL=0 \
  --name portainer_edge_agent \
  portainer/agent:latest \
  --mtlscacert /certs/mtlsca.crt \
  --mtlscert /certs/client.crt \
  --mtlskey /certs/client.key

The critical part is setting EDGE_INSECURE_POLL=0. By default, edge agents skip certificate validation. Setting this to 0 enforces mTLS.

Split-Tunnel VPN Considerations

My remote hosts connect through a WireGuard VPN with split tunneling — only traffic destined for my home network goes through the tunnel. Everything else routes directly to the internet.

This setup works well for Portainer because:

• The edge agents only need to reach the Portainer server, which is on the VPN subnet
• Docker pulls and other internet traffic don't go through the VPN, keeping bandwidth usage low
• mTLS ensures that even if the VPN is compromised, the agents won't accept commands from an imposter server

I did have to make sure my WireGuard AllowedIPs configuration included the subnet where Portainer runs. On one host, I forgot to do this, and the agent couldn't connect. The logs showed "connection refused" errors, which was confusing until I realized the routing table was wrong.

What Didn't Work

The biggest issue was certificate validation failures. Even after getting the key usage extensions right, I kept seeing errors like "x509: certificate signed by unknown authority."

The problem was that I had multiple CA certificates in my mtlsca.crt file from previous experiments. Portainer expects a single CA certificate, not a chain. I had to clean up the file to include only the root CA.

Another mistake: I initially tried to use the same client certificate for all edge agents. This worked, but it defeated the purpose of mTLS. If one host is compromised, the attacker could use the same certificate to impersonate other agents. I went back and generated unique client certificates for each host.

Finally, I underestimated how finicky hostname matching is. The edge agent's EDGE_ID and EDGE_KEY are tied to the API server URL you specify during setup. If you change the hostname later (even from portainer.internal.vipinpg.com to its IP address), the agent won't connect. You have to regenerate the edge configuration in the Portainer UI.

How It's Held Up

I've been running this setup for several months now across four remote Docker hosts. The mTLS authentication has been completely transparent — I don't think about it until I need to add a new host.

Performance is identical to the non-mTLS setup. The certificate validation adds negligible overhead, and the edge agents still poll the server every few seconds without issues.

The main maintenance task is certificate renewal. My CA certificates are valid for 10 years, but I set the server and client certificates to expire after 2 years. When they do, I'll need to regenerate them and redeploy the agents. I'm planning to automate this with a script, but I haven't done it yet.

Key Takeaways

• mTLS adds real security for edge agents, especially over VPN connections where network-level trust isn't enough
• Getting the certificate key usage extensions right (serverAuth and clientAuth) is critical — Portainer won't accept certificates without them
• Use a separate subdomain for mTLS and make sure it matches the server certificate's SAN
• Generate unique client certificates for each edge agent, not one shared certificate
• Set EDGE_INSECURE_POLL=0 to enforce certificate validation
• Double-check your VPN routing if agents can't connect — split-tunnel configurations can silently block traffic

This setup isn't for everyone. If you're managing a handful of Docker hosts on a trusted internal network, the extra complexity probably isn't worth it. But if you're running remote infrastructure over VPNs or untrusted networks, mTLS gives you cryptographic proof that your agents are talking to the right server.