Configuring server monitoring alerts for detecting data exfiltration patterns after breaches using prometheus and custom bash scripts

Why I Started Watching for Data Exfiltration

I run a mix of self-hosted services on Proxmox VMs and Docker containers—file storage, automation tools, a few web apps. Most of it sits behind a firewall with limited external exposure, but I'm not naive enough to think that makes it safe.

A few months back, I noticed unusual outbound traffic from one of my containers during a routine check of my router logs. Nothing catastrophic happened, but it was a wake-up call. If something gets compromised—whether through a misconfigured service, a weak password, or an unpatched vulnerability—I need to know if someone's trying to pull data out.

That's when I started thinking about detection, not just prevention. I already had Prometheus running for basic system metrics, so I decided to extend it with custom monitoring specifically for exfiltration patterns.

My Setup and What I Actually Monitor

I'm not running an enterprise SOC. This is a homelab-scale solution built around tools I already use:

• Prometheus for metric collection and alerting
• Node Exporter on each VM and host for system-level metrics
• Bash scripts to parse logs and generate custom metrics
• Alertmanager to send notifications via email and Discord webhooks

The core idea is simple: track behaviors that look like data leaving the network in suspicious ways. I'm not trying to catch every possible attack vector—just the obvious ones that would indicate someone's already inside and moving files out.

What I Actually Watch For

Based on what I've read and what makes sense for my environment, I focus on:

• Large outbound transfers over HTTP/HTTPS to unfamiliar domains
• Repeated SSH or SCP connections to external IPs I don't recognize
• Sudden spikes in outbound bandwidth from specific containers or VMs
• File compression or archiving commands followed by network activity
• DNS queries to file-sharing or cloud storage domains I don't use

I don't have access to Windows event logs because I'm running Linux everywhere. That limits what I can detect, but it also simplifies the implementation.

How I Built the Monitoring

Tracking Outbound Bandwidth Per Container

Prometheus already collects network metrics via Node Exporter, but I needed more granular data per Docker container. I wrote a bash script that runs every minute via cron and exports container-level network stats in a format Prometheus can scrape.

#!/bin/bash
METRIC_FILE="/var/lib/node_exporter/container_net.prom"
> $METRIC_FILE

for container in $(docker ps --format '{{.Names}}'); do
  rx_bytes=$(docker stats --no-stream --format "{{.NetIO}}" $container | awk '{print $1}' | sed 's/[^0-9.]//g')
  tx_bytes=$(docker stats --no-stream --format "{{.NetIO}}" $container | awk '{print $3}' | sed 's/[^0-9.]//g')
  
  echo "container_net_tx_bytes{container=\"$container\"} $tx_bytes" >> $METRIC_FILE
  echo "container_net_rx_bytes{container=\"$container\"} $rx_bytes" >> $METRIC_FILE
done

This isn't perfect—Docker's stats output is inconsistent, and I had to add some sed filtering to clean up the data. But it works well enough for my purposes.

In Prometheus, I set up an alert that fires if any container's outbound traffic exceeds 500MB in a 5-minute window:

- alert: HighOutboundTraffic
  expr: rate(container_net_tx_bytes[5m]) > 524288000
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High outbound traffic from {{ $labels.container }}"

Watching for Suspicious Commands

I run auditd on my main Proxmox host to log shell commands. I wrote another script that tails the audit log and looks for patterns like:

• curl or wget with -T or --upload-file
• scp or rsync to external IPs
• tar, zip, or 7z followed by network activity within 60 seconds

The script increments a Prometheus counter whenever it sees a match:

#!/bin/bash
METRIC_FILE="/var/lib/node_exporter/suspicious_commands.prom"
tail -F /var/log/audit/audit.log | while read line; do
  if echo "$line" | grep -qE 'curl.*-T|wget.*--upload-file|scp.*@'; then
    echo "suspicious_command_count 1" > $METRIC_FILE
  fi
done

This is crude. It generates false positives when I'm manually transferring files, and it doesn't catch everything. But it's better than nothing, and I can tune the patterns over time.

DNS Query Monitoring

I run Pi-hole for DNS filtering, and it logs every query. I wrote a script that checks for queries to known file-sharing domains—things like transfer.sh, wetransfer.com, dropbox.com—that I don't use personally.

#!/bin/bash
METRIC_FILE="/var/lib/node_exporter/suspicious_dns.prom"
DOMAINS="transfer.sh|wetransfer.com|mega.nz"

grep -E "$DOMAINS" /var/log/pihole.log | wc -l > $METRIC_FILE

If the count goes above zero, Prometheus fires an alert. This has caught a few cases where a compromised script was trying to phone home.

What Actually Worked

The outbound traffic alerts have been the most useful. I caught a runaway backup script that was uploading gigabytes of logs to a misconfigured S3 bucket. Not malicious, but definitely something I needed to know about.

The DNS monitoring also flagged a container that was trying to reach a file-sharing site after I accidentally left a test API key exposed in a config file. I shut it down before anything sensitive left the network.

The command-based detection is hit-or-miss. It generates noise, and I've had to whitelist certain patterns (like my own backup scripts). But when it does trigger, it's usually worth investigating.

What Didn't Work

My first attempt at monitoring compressed files was a disaster. I tried to track every time tar or zip ran, but that fired constantly—backups, log rotation, package installations. I had to narrow it down to only flag when compression happens in directories that shouldn't be touched (like /home or /var/www).

I also tried to correlate file access logs with network activity, but the timing was too unreliable. By the time the script processed the logs, the connection was already closed. I'd need something more real-time, which I don't have the resources to build.

Another issue: I can't detect encrypted exfiltration. If someone's using HTTPS to upload data, I see the bandwidth spike but not the content. I've thought about running a TLS-intercepting proxy, but that introduces complexity and privacy concerns I'm not ready to deal with.

Key Takeaways

Monitoring for data exfiltration in a homelab is messy and imperfect. You're not going to catch everything, and you'll definitely get false positives. But having something in place is better than assuming your firewall will save you.

Start simple. Track outbound bandwidth per service. Watch for DNS queries to domains you don't use. Log suspicious commands. Use the tools you already have—Prometheus, bash, cron—and build from there.

Don't expect this to replace proper security practices. Patch your services. Use strong credentials. Limit external exposure. But if something does get compromised, at least you'll have a chance to notice before the damage spreads.

This setup has already saved me from a few self-inflicted problems, and it's given me more confidence that I'd spot something truly malicious if it happened. That alone makes it worth the effort.