trial and stderr

All my sites on a particular server went down at 00:13 last night. Nothing in any logs. The docker containers running the websites themselves were responsive to curls from the server. So that left the https-portal container, which reverse-proxies all the sites, as the culprit. Not that I think this is https-portal's fault, actually -- this seems like docker being more generally unstable (I just found out about that article in diagnosing this problem -- holy hell). This was corroborated by some weird errors on trying to kill/restart the container in question.

I really love https-portal. I don't want to give it up. But it certainly is a critical point of failure.

(The name of this article is a little misleading -- I fixed the problem in 10 minutes, it just took me 24 hours to get around to doing so. Nothing big impacted, just a couple personal projects. Including this blog.)

I had tried to use the sebp/lighttpd container for something before realizing it did this, making it so it can't be used without using volumes, which introduce portability issues and so should only be used when actually needed.

Don't define volumes in your Dockerfiles. Don't do it. That's a client decision.

See bistenes/lighttpd for my volume-free fork.

The real pain came when I found out that docker-compose persists volumes between builds, which is apparently not a bug, it's a feature! despite causing a great deal of wondering why on earth my containers had volumes mounted despite their not being defined anywhere at all.

I thought I'd written down the solution before, but it turned out I'd just asked about it on StackOverflow and then "solved" it by starting a new server and failing to tell docker not to wipe my iptables rules. The problem was I could either

1. use docker with my own iptables rules and "iptables": false and have those containers not able to access the internet or
2. allow docker to wreck my iptables rules and allow the whole big nasty internet access to all of my containers with exposed ports.

Apparently there's an easy way to address this now, using the DOCKER-USER chain.

I set iptables=true and append to my iptables configuration

iptables -A DOCKER-USER -i eth0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A DOCKER-USER -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -j DROP

which says "accept established, accept input HTTP(S), drop everything else."

Containers can connect out, can't be connected into, except on 80 and 443. 👍

Blood and thunder, I've been hacked!

> use Warning
switched to db Warning
> show collections
Readme
> db.Readme.find()
{ "_id" : ObjectId("59d52f735e716205267adea9"), "BitCoin" : "1Jqw2tHBkUAGY32YzettJiDAwe8A9mUzok", "eMail" : "cru3lty@safe-mail.net", "Exchange" : "https://localbitcoins.com", "Solution" : "Your DataBase is downloaded and backed up on our secured servers. To recover your lost data: Send 0.2 BTC to our BitCoin Address and Contact us by eMail with your MongoDB server IP Address and a Proof of Payment. Any eMail without your MongoDB server IP Address and a Proof of Payment together will be ignored. You are welcome!" }
>

Dastards. Idiosyncratic capitalization, to boot. Fortunately I keep backups (cron gsutil /data gs://my-backups) and there was actually nothing in this database. But what the heck happened to my iptables rules?

Looks like docker has been starting without the iptables=false flag.

I thought the solution was to echo '{ "iptables": false }' | sudo tee /etc/docker/daemon.json and restart the daemon (sudo service docker restart), to tell docker not to mess with iptables rules, but then Docker containers can't access the internet.

The true path is above, using the DOCKER-USER chain.

ADDENDUM: I'm going to try and prevent this from happening in the future using Uptime Robot, a free service that will let me know if my site goes down or ports unexpectedly become open.

Check out this crazy pernicious and largely ignored Docker issue from 2014. All the time, I'm trying to docker build something on my big Ubuntu box and I get no space left on device.

And then I SSH into the machine, run df -h -i and see that yes, indeed, I have like 20 free inodes.

First attempt at solving is docker system prune [-a][-f]

Second is

sudo service docker stop && \
 sudo rm -r /var/lib/docker && \
 sudo services docker start

I'm using AUFS. People have the same issue with Overlay and Overlay2. Some report that BTRFS doesn't have this issue. I'll try that out and report back.