TLDR: Docker port forwarding uses iptables which can override your firewall rules
The problem
The other day, I noticed unusual activity on a VPS server, it was suddenly using 100% of the CPU. It appeared that one of my docker containers had been compromised and was running a piece of malware called "kdevtmpfsi"
Someone had run some sort of rootkit on a postgres container that I had running, and was now mining crypto! So after the machine was rebuilt and the threat removed, I was left with a question: how did someone get in?
Cautionary tale
Ok the first issue of access was easy to solve, it appeared that I had left the default user and password on this container, which obviously you shouldn't do. And they probably found the container by port scanning for open ports on my public ip. But I had firewalled the port!
First a little background, so a couple days before I decided that I wanted to share a database from this machine to another machine over via point-to-point wireguard tunnel. I have ufw setup on this machine so I researched how to limit access to a particular host IP.
Something like:
ufw allow from 10.0.1.0/24 to 10.0.1.100 port 5432 proto tcp
Which looks correct when you check the firewall:
$ sudo ufw status Status: active To Action From -- ------ ---- 22/tcp (OpenSSH) LIMIT IN Anywhere 80 ALLOW IN Anywhere 443 ALLOW IN Anywhere 10.0.1.100 5432/tcp ALLOW IN 10.0.1.0/24 22/tcp (OpenSSH (v6)) LIMIT IN Anywhere (v6) 80 (v6) ALLOW IN Anywhere (v6) 443 (v6) ALLOW IN Anywhere (v6)
However, when I was testing later after cleaning everything up, I noticed I could still connect to the postgres instance from the public ip.
Now this postgres instance is launched by a simpler docker-compose.yml like this:
services:
db:
image: postgres:14.1
restart: always
env_file: .env
command: postgres -c shared_preload_libraries=pg_stat_statements -c pg_stat_statements.track=all -c max_connections=20
volumes:
- pg_data:/var/lib/postgresql/data
And I recently added a posts section to the service like this:
services:
db:
...
ports:
- "5432:5432"
After some playing around, I noticed that I could still connect to the postgres container via the public ip even without my fancy ufw rule. And I double checked, the default firewall rule:
$ sudo ufw status verbose Status: active Logging: on (low) Default: deny (incoming), allow (outgoing), deny (routed) New profiles: skip To Action From -- ------ ---- 22/tcp (OpenSSH) LIMIT IN Anywhere 80 ALLOW IN Anywhere 443 ALLOW IN Anywhere 22/tcp (OpenSSH (v6)) LIMIT IN Anywhere (v6) 80 (v6) ALLOW IN Anywhere (v6) 443 (v6) ALLOW IN Anywhere (v6)
In case you didn't know, ufw is just a wrapper around iptables that is more user friendly. So I decided to look at the actual iptables rules to see what I was setting up incorrectly using iptables -S.
As expected we can see a line generated by ufw:
-A ufw-user-input -s 10.0.1.0/24 -d 10.0.1.100/32 -p tcp -m tcp --dport 5432 -j ACCEPT
However in the first ten lines, I see my first hint. There is mention of "docker.":
$ sudo iptables -S | head -P INPUT DROP -P FORWARD DROP -P OUTPUT ACCEPT -N DOCKER -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 -N DOCKER-USER -N ufw-after-forward -N ufw-after-input -N ufw-after-logging-forward
Additionally, if we grep for port 5432, we'll see another rule slightly above our expected rule:
$ sudo iptables -S | grep -n 5432 68:-A DOCKER -d 172.22.0.6/32 ! -i br-ac9ec4f755f9 -o br-ac9ec4f755f9 -p tcp -m tcp --dport 5432 -j ACCEPT 130:-A ufw-user-input -s 10.0.1.0/24 -d 10.0.1.100/32 -p tcp -m tcp --dport 5432 -j ACCEPT
So it appears on line 68 of my firewall rules I was allowing all host ips to connect to the docker container, this rule was allowing traffic before my rule to only allow traffic from a certain ip could be checked.
The solution
You can specify a host ip to bind the port to:
ports:
- "10.0.1.100:5432:5432"
This means that, the local ip will connect:
$ psql -h 10.0.1.100 Password for user paul:
While the public ip will not:
$ psql -h 350.350.32.350
psql: error: could not connect to server: Connection refused
Is the server running on host "350.350.32.350" and accepting
TCP/IP connections on port 5432?
Take-aways
- Don't use default passwords
- Don't make assumptions about your work, verify it
- Take backups and practice restoring from those backups
- The internet is very quick to find and take advantage of your mistakes