Building Zero-Trust Container Networks with mTLS: Automating Certificate Rotation for Service-to-Service Authentication in Portainer

Why I worked on this

After deploying a handful of containers in Portainer for various services, I noticed they communicated over plain HTTP. This wasn't just a theoretical security concern - I saw actual unencrypted traffic in my logs. While Portainer offers basic network segmentation, I needed a stronger guarantee that containers could only talk to services they were explicitly authorized to access. Mutual TLS (mTLS) seemed like the right solution, but the manual certificate management was daunting. This is how I implemented automated mTLS certificate rotation for service-to-service authentication in my Portainer setups.

My real setup or context

Here's what I started with:

• Portainer (v2.24.0) running on a Proxmox VM
• Several containers (Nextcloud, Grafana, n8n, Postgres) on different Portainer stacks
• Traefik v2 as my reverse proxy
• Let's Encrypt certificates for external access, but no internal encryption
• Basic Portainer networks separating services by function

I wanted to:

• Require mTLS for all container-to-container communication
• Automate certificate issuance and rotation
• Keep this simple enough to maintain myself
• Avoid bloated enterprise solutions

What worked (and why)

I settled on using smallstep/step-ca for my certificate authority and smallstep/step-certificates for certificate issuance. Here's how I configured it:

1. Setting up the CA

First, I deployed a container running the Smallstep CA:

version: '3'
services:
  step-ca:
    image: smallstep/step-ca
    environment:
      - STEP_DEBUG=true
      - STEP_CA_INIT_CA_NAME=My Internal CA
      - STEP_CA_INIT_CA_EXPIRY=87600h
      - STEP_CA_INIT_KID_EXPIRY=87600h
    volumes:
      - /mnt/ca-data:/data
    ports:
      - "443:443"
    networks:
      - internal-ca

The key here was creating a dedicated network for just the CA to isolate it from other services.

2. Automating certificate issuance

For each service that needed a certificate, I created a script that:

• Generated a CSR
• Requested a certificate from the CA
• Renewed certificates every 30 days

Here's an example for my n8n service:

#!/bin/bash

# Generate private key and CSR
openssl req -new -newkey rsa:2048 -nodes -keyout n8n.key -subj "/CN=n8n.mydomain.local" -out n8n.csr

# Request certificate from CA
step certificate --ca http://step-ca.mydomain.local n8n.csr n8n.crt --profile server --key n8n.key

# Bundle with CA cert
cat /mnt/ca-data/certs/ca.crt >> n8n.crt

# Move to container
docker cp n8n.crt n8n.key n8n:/etc/ssl/

I scheduled this to run monthly via a Cron job in the host VM.

3. Configuring services for mTLS

For Traefik, I added:

[entryPoints]
  [entryPoints.https]
    address = ":8443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
        certFile = "/etc/ssl/traefik.crt"
        keyFile = "/etc/ssl/traefik.key"

And for client verification:

[http]
  [http.middlewares]
    [http.middlewares.tlsauth]
      function = "verifyClientCert"
      clientCerts = ["/etc/ssl/ca.crt"]

In n8n, I configured the outgoing requests to use the client certificate:

const https = require('https');
const fs = require('fs');

const options = {
  key: fs.readFileSync('/etc/ssl/n8n.key'),
  cert: fs.readFileSync('/etc/ssl/n8n.crt'),
  ca: [fs.readFileSync('/etc/ssl/ca.crt')],
  rejectUnauthorized: true
};

https.request('https://grafana.mydomain.local:8443', options, (res) => { ... });

4. Monitoring and rotation

I set up a small monitoring script that:

• Checked certificate expiry dates
• Alerted me when certs were within 7 days of expiry
• Logged all renewal attempts

What didn't work

A few approaches I tried that didn't pan out:

• Using Let's Encrypt for internal certs - Their domain validation doesn't work well with internal-only services.
• Automating with Portainer hooks - The hook system wasn't reliable enough for certificate rotation timing.
• Container-based renewal - Renewing certificates from within containers was messy with mounted volumes.
• SPIFFE/SPIRE - While elegant, it was overkill for my 10-container setup.

I also found that some services (like Postgres) didn't support mTLS in their container images out of the box, requiring custom builds.

Key takeaways

• For small setups, a simple CA like Smallstep is perfect - don't over-engineer.
• Automate the rotation before you need it - don't wait for certs to expire.
• Test the renewal process multiple times - it's the hardest part.
• Not all services support mTLS equally - check your container images first.
• Isolate your CA - treat it like any other security-critical service.

The system has been running reliably for 6 months now, with no certificate-related outages. The most important lesson was to keep it simple - I could have spent weeks building a perfect system, but this practical approach gets the job done with minimal maintenance.