Tag Archives: docker

Finish Writing Me Plz Nerd

Apache Bench

For years, my tool for simple load tests of HTTP sites has been ApacheBench.

For years, my reference for how to visualize ApacheBench results has been Gnuplot

For years, my reference for how to use Gnuplot has been http://www.bradlanders.com/2013/04/15/apache-bench-and-gnuplot-youre-probably-doing-it-wrong/

But do you really want to be writing Gnuplot syntax? It turns out that Pandas will give you great graphs pretty much for free:

df = pd.read_table('../gaussian.tsv')
# The raw data as a scatterplot
df.plot(x='seconds', y='wait', kind='scatter')
scatter
# The traditional Gnuplot plot
df.plot(y='wait')
wait
# Histogram
df.wait.hist(bins=20)

distribution

 

You can see the full source code at tsv_processing.ipynb

And re-recreate these yourself by checking out the parent repo: github/crccheck/abba

So now you might be thinking: How do you get a web server that outputs a normal distribution of lag? Well, I wrote one! I made a tiny Express.js server that just waits a random amount, packaged it in a Docker image, and and you can see exactly how I ran these tests by checking out my Makefile.

Finish Writing Me Plz

Prometheus Contained

After becoming smitten with Graphite last year, I’m sorry to say I’ve become entranced by the new hotness: Prometheus. For a rundown between the two, Prometheus’s docs do a good job. The docs aren’t bad, but there are a lot of gaps I had to fill. So I present my hello world guide for using Prometheus to get metrics on a host the Docker way. In addition to the official docs, I found Monitor Docker Containers with Prometheus to be useful. For the node explorer, discordianfish’s Prometheus Demo was valuable.

Start a Container Exporter container

This container creates an exporter that Prometheus can talk to to get data about all the containers running on the host. It needs access to cgroups to get the data, and the docker socket to know what containers are running.

docker run -d --name container_explorer \
  -v /sys/fs/cgroup:/cgroup:ro \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  prom/container-exporter

Start a Node Exporter container

This container uses the --net=host option so it can get metrics about the host network interface.

docker run -d --name node_exporter --net=host \
  prom/node-exporter

I was afraid that this would result in distorted stats because it’s in a container instead of the host, but after testing against a Node Exporter installed on the bare metal, it looks like it’s accurate.

Start a Prometheus container

This is the container that actually collects data. I’ve mounted my local prometheus.conf so Prometheus uses my configuration, and mounted a data volume so Prometheus data can persist between containers. There’s a link to the container explorer so Prometheus can collect metrics about containers. There’s a add-host so this container can access the node explorer’s metrics. Port 9090 is exposed because Prometheus needs to be publicly accessible from the Dashboard app. I’m not sure how to lock it down for security. I may add a referrer check since I don’t want to do IP rules or a VPN.

docker run -d --name $@_1 \
  -v ${PWD}/prom/prometheus.conf:/prometheus.conf:ro \
  -v ${PWD}/prom/data:/prometheus \
  --link container_explorer:container_explorer \
  --add-host=dockerhost:$(ip route | awk '/docker0/ { print $NF }') \
  -p 9090:9090 \
  prom/prometheus

Setup a Prometheus Dashboard database

Here, I’m running the rake db:setup task to set up a sqlite database. The image is my own pending pull request 386.

docker run \
  -e DATABASE_URL=sqlite3:/data/dashboard.sqlite3 \
  -v ${PWD}/prom/dashboard:/data:rw \
  crccheck/promdash ./bin/rake db:setup

Start a Prometheus Dashboard

Now that the dashboard has a database, you can start the Dashboard web app. You can access it on port 3000. You’ll need to create a server, then a site, then a dashboard, and finally, your first graph. Don’t be disappointed when your first graph doesn’t load. You may need to tweak the server settings until the graph preview from the Dashboard opens the right data in Prometheus.

docker run-d --name promdash \
  -e DATABASE_URL=sqlite3:/data/dashboard.sqlite3 \
  -v ${PWD}/prom/dashboard:/data:rw \
  -p 3000:3000 \
  crccheck/promdash

My almost unabridged setup

What’s missing in this gist is the config for my nginx layer in front of the dashboard, which is why there’s no exposed port in this gist. To get started, all you have to is put prometheus.conf in a prom sub-directory and run make promdash

Best Practices Case Study

Optimizing Docker Image Size For Real

I’ve come across tips on how to keep Docker images small and Dockerfiles with strange lines that seem to exist only to optimize image size. Well, it turns out they’re all wrong.

They may have an effect with flat Docker images, but everything else (i.e. 99% of what people do), cleanup steps are just extra steps. When Docker builds an image from a Dockerfile, every step is a checkpoint, and every step is saved. If you add 100 MB in one step, then delete it the next, that 100 MB still needs to be saved so other Dockerfiles with the same step can reuse it.

Results

REPOSITORY               TAG             IMAGE ID            CREATED             VIRTUAL SIZE
test/baseline            latest          7b590dec9b43        7 hours ago         272.6 MB
test/baseline_lines      latest          e165025980f7        9 minutes ago       272.6 MB
test/baseline_lists      latest          b40f9e108a93        About an hour ago   272.6 MB
test/combo               latest          744b502e0052        2 seconds ago       269.8 MB
test/combo2              latest          be8f1c1de02e        About an hour ago   249.8 MB
test/combo3              latest          da948e2838d9        About an hour ago   249.8 MB
test/install             latest          e7cadcbb5a05        12 hours ago        269.8 MB
test/install_clean       latest          dd1383285e85        12 hours ago        269.8 MB
test/install_lists       latest          e55f6f8ebac8        12 hours ago        269.8 MB
test/purge               latest          ef8c2aa7400b        About an hour ago   273.5 MB
test/remove              latest          75e3e5c4e246        About an hour ago   273.5 MB

Hypothesis: Docker’s base Ubuntu image does not need `apt-get clean`

I did an experiment around Docker 0.6. I think my conclusion was that `apt-get install … && apt-get clean` saved a few megabytes. But I head that you didn’t need to do that. If you compare the “test/install” and “test/install_clean” size, you’ll see there is no difference. So you don’t need `apt-get clean`.

Hypothesis: `rm -rf /var/lib/apt/lists/*` saves some space

I’ve been seeing a lot of Dockerfiles lately with this line. Including lots of official Docker images. If those guys are all doing it, surely it must have some effect. Nope.

Hypothesis: Combining similar lines saves space

There’s some overhead for each line in a Dockerfile. How significant is it? Well, it turns out it’s not. What I did find out though, is that it does save a significant amount of time and saves a lot of disk thrashing. So combining lines does not save space, but saves time.

Hypothesis: Combining multiple steps saves space

This makes sense. If you skip making checkpoints, you’re not storing intermediate states. And it turns out this is the only way to get a Docker image made from a Dockerfile smaller. But this is at the cost of readability, and more importantly, at the cost of reduced redundancy between images.

Hypothesis: `apt-get purge` saves some space

Well this hypothesis seems silly now. But I see it used now and then. Deletions do not save space.

Conclusion

Write your Dockerfiles the same way you run commands. Don’t prematurely optimize by adding extra cruft you saw someone else do. If you’re actually worried about image size, use some sort of automation to rebuild Docker images behind the scenes. Just keep that logic out of the Dockerfile. And always keep on measuring. Know your bottlenecks.