The mechanics of Docker logs are, by nature, quite different from VM logs and managing Docker logs can be complicated due to its dynamic and ephemeral nature. In the Docker universe, there are different types of logs to be mindful of: logs from the containerized applications and logs from the Docker daemon. In order to gain insight into the health of your containers, you need to understand the structure of Docker logs and what it tells you. It can give you a full picture of what is happening at every layer of the stack. Once all the logs have been collected and we have access to them, how do we make use of them and interpret the results? This post aims to give an overview of Docker logs, it’s structure, and what exactly it tells us.

Basics

Docker logs are the console output of running containers. We get the stdout and stderr streams of processes running in a container. Most official Docker images log to console.

Anything written by a container on stdout and stderr is collected by the docker engine and sent to the logging driver. This may contain startup messages if something is not correctly configured and access-log typed messages that are structured with IP addresses, timestamps, and status code.

To view a container’s logs, identify the container ID or the image name and then type:

docker logs [OPTIONS] CONTAINER

An example output:

172.17.0.1 - - [26/Dec/2017:07:25:23 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36" "-"
2017/12/26 07:25:23 [error] 7#7: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "172.17.0.2", referrer: "http://172.17.0.2/"

If the default logging driver is used, Docker logs can be found under:

/var/lib/docker/containers/<container id>/<container id>-json.log

To find the specific location of Docker logs, type: docker inspect CONTAINER | grep LogPath

An example output:

"LogPath": "/var/lib/docker/containers/61967306d3ee8855ec7ee45ae6cc4ec08ebf7e61fcacedfc096740bdc637ed92/61967306d3ee8855ec7ee45ae6cc4ec08ebf7e61fcacedfc096740bdc637ed92-json.log"	

Logging drivers

The Docker engine provides various logging drivers that determine where logs are sent or written to. The default driver for Docker logs is “json-file”, which writes the logs to local files on the Docker host in json format, and allows them to be viewed on the console with the CLI tool. If the default driver is used, the logs are removed upon removal of the container. There are other logging drivers that allow Docker logs to be sent via the driver (without parsing) to a remote destination. Some examples are syslog, gelf, fluentd, journald. A log driver can be set up for each container, which specifies where logs should go for that particular container. A global daemon-wide setting for Docker can also be created.

To set a logging driver for a container, this would be done with the --log-driver option:

docker run --log-driver=DRIVERNAME

To set extra options, use --log-opt. For example, when using the syslog driver, the address of the remote syslog server can be set:

docker run --log-driver=syslog --log-opt=syslog-address=ADDRESS

When one of these drivers is in use, the Docker logs are shipped via the driver to a remote destination and the logs are not on the local Docker host. Thus, the CLI tool to display logs will not work anymore since Docker does not know how to retrieve the logs from these systems.

For example, the syslog protocol only defines how to send logs to a remote logger, but not how to access them. With other drivers, once the logs are sent to a particular logging system, they are outside of Docker’s reach.

Structure of Docker container logs and its contents

Anything written by a container on stdout and stderr is collected by the Docker engine and sent to the logging driver. The logs contain the actual message output plus a bunch of metadata (which becomes very important at large scale)

Each line of the log consists of a timestamp (with date, full year, and down to milliseconds), a string, and perhaps an origin (stdout vs stderr).

An example output:

172.17.0.1 - - [24/Apr/2018:07:25:23 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36" "-"
2018/04/24 07:25:23 [error] 7#7: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "172.17.0.2", referrer: "http://172.17.0.2/"

Another example output:

2018-04-24T06:40:35.359+0000 I CONTROL  [initandlisten] MongoDB starting : port=27017 dbpath=/data/db 64-bit
 2018-04-24T06:40:35.359+0000 I CONTROL  [initandlisten] db version v3.6.3

 2018-04-24T06:40:35.360+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine

The log output may be different depending on what logging driver is in use. For example, priority (i.e. debug, warning, error, info) and the PID (process name or process ID) are informations that exist for syslog, but not for container output. If you use an advanced logging driver like gelf, a lot of container information will be added and every log message is a dict.

The --details flag will add on extra attributes, such as environment variables and labels, provided to --log-opt when creating the container.

An example:

docker run -d --log-opt tag=hi=what,day=wednesday alpine echo hello world

c93bf3bcedc9a87082123bbd0d17f21d7e876f8c593da129277e931b8890a4d0

docker logs c93b

hello world

docker logs c93b --details

tag=hi%3Dwhat%2Cday%3Dwednesday hello world

Docker daemon logs

Docker daemon logs offers insight into the state of the Docker service as a whole. It logs two types of events: commands sent to the daemon through Docker’s Remote API and events that occur as part of the daemon’s normal operation.

Viewing Docker daemon logs is quite different from viewing logs from the containerized applications. It depends on your operating system.