Programming

How to use Docker Compose to run a multi-container application

Markus

Jan 10, 2024 — 12 min read

Docker Compose is a tool that allows you to define and run multi-container applications using a YAML file. Docker Compose simplifies the process of creating, managing, and deploying complex applications that consist of multiple services that communicate with each other.

Benefits of using Docker Compose

Some of the benefits of using Docker Compose are:

Simplicity: You can specify all the configuration details of your application, such as the image, ports, volumes, networks, environment variables, etc., in a single file, and then use a single command to start or stop your application.
Consistency: You can ensure that your application runs the same way in different environments, such as development, testing, and production, by using the same Compose file and Docker images.
Scalability: You can easily scale up or down your application by changing the number of replicas of each service, and Docker Compose will automatically create or remove the containers accordingly.
Modularity: You can modularize your application into separate services that have their own responsibilities and dependencies, and Docker Compose will handle the communication and coordination between them.

How to get started with Docker Compose

To get started with Docker Compose, you need to install Docker on your machine, which includes the Docker Engine, the Docker CLI, the Compose plugin and other tools.

Once you have installed Docker, you can create a Compose file that defines your application. A Compose file is a YAML file that has the following structure:

version: "3.9" # specify the version of the Compose file format
services: # define the services that make up your application
  service1: # name of the first service
    image: image1 # specify the image to use for this service
    ports: # specify the ports to expose for this service
      - "port1:port2" # map port1 on the host to port2 on the container
    volumes: # specify the volumes to mount for this service
      - "volume1:volume2" # mount volume1 on the host to volume2 on the container
    environment: # specify the environment variables for this service
      - "variable1=value1" # set variable1 to value1
    depends_on: # specify the dependencies for this service
      - service2 # this service depends on service2
  service2: # name of the second service
    image: image2 # specify the image to use for this service
    # ... other configuration options for this service
networks: # define the networks to use for your application
  network1: # name of the first network
    driver: bridge # specify the driver to use for this network
  network2: # name of the second network
    driver: overlay # specify the driver to use for this network
volumes: # define the volumes to use for your application
  volume1: # name of the first volume
    driver: local # specify the driver to use for this volume
  volume2: # name of the second volume
    driver: nfs # specify the driver to use for this volume

You can save this file as docker-compose.yml in your project directory. You can also name it differently, but then you need to specify the file name when you run Docker Compose commands.

To run your application, you can use the following command in your terminal:

docker-compose up

This will pull the images, create the networks and volumes, and start the containers for your application. You can also use the -d flag to run the application in the background.

To stop your application, you can use the following command in your terminal:

docker-compose down

This will stop and remove the containers, networks, and volumes for your application.

How to build custom images with Docker Compose

Sometimes, you may want to use your own images for your services, instead of using the ones available on Docker Hub or other registries. You can do this by using the build option in your Compose file, instead of the image option. For example, suppose you have a Python web application that you want to run in a container. You can create a Dockerfile that specifies how to build the image for your application, such as:

# Use the official Python image as the base
FROM python:3.9

# Set the working directory in the container
WORKDIR /app

# Copy the requirements file and install the dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy the source code of the application
COPY . .

# Expose the port that the application listens on
EXPOSE 5000

# Run the application
CMD ["python", "app.py"]

You can save this file as Dockerfile in your project directory. Then, you can use the build option in your Compose file to tell Docker Compose to use this file to build the image for your service, such as:

version: "3.9"
services:
  web: # name of the web service
    build: . # specify the path to the Dockerfile
    ports:
      - "5000:5000" # map port 5000 on the host to port 5000 on the container

You can also specify additional options for the build option, such as the context, dockerfile, args, cache_from, etc. You can find more details about these options in the official documentation.

When you run docker-compose up, Docker Compose will automatically build the image for your service, using the Dockerfile and the options that you specified, and then run it in a container.

How to link services with Docker Compose

One of the main advantages of using Docker Compose is that it allows you to link your services together, so that they can communicate with each other. By default, Docker Compose creates a single network for your application, and attaches all the containers to that network. This means that your containers can reach each other by using their service names as hostnames. For example, suppose you have a web service and a database service in your Compose file, such as:

version: "3.9"
services:
  web: # name of the web service
    build: . # build the image from the Dockerfile
    ports:
      - "5000:5000" # map port 5000 on the host to port 5000 on the container
    depends_on: # specify the dependencies for this service
      - db # this service depends on the db service
  db: # name of the database service
    image: postgres # use the official postgres image
    environment: # specify the environment variables for this service
      - POSTGRES_PASSWORD=secret # set the password for the postgres user

In this case, your web service can connect to your database service by using the hostname db and the port 5432, which is the default port for postgres. For example, if you are using SQLAlchemy as your ORM, you can use the following connection string:

Python

engine = create_engine("postgresql://postgres:secret@db:5432")

You can also use the links option in your Compose file to create aliases for your services, or to link to services in another network.

You can also create and use your own networks in your Compose file, by using the networks option. This allows you to have more control over the network configuration and isolation of your services.

How to scale and update your application with Docker Compose

Docker Compose also allows you to scale and update your application, by changing the number of replicas of each service, or by changing the image or configuration of a service. You can do this by using the docker-compose scale and docker-compose up commands, respectively.

To scale your application, you can use the docker-compose scale command, followed by the service name and the number of replicas that you want. For example, suppose you want to scale your web service to 3 replicas, you can use the following command:

docker-compose scale web=3

This will create or remove the containers for your web service, according to the number of replicas that you specified. You can also use the --detach flag to run the command in the background.

To update your application, you can use the docker-compose up command, followed by the service name and the options that you want to change. For example, suppose you want to update your web service to use a new image, you can use the following command:

docker-compose up web --image new-image

This will pull the new image, stop and remove the old containers, and create and start the new containers for your web service.

How to use environment variables and secrets with Docker Compose

Docker Compose also allows you to use environment variables and secrets to pass configuration values to your services. Environment variables are key-value pairs that are set in the environment of the container, and can be accessed by the application. Secrets are files that contain sensitive data, such as passwords, tokens, certificates, etc., and are mounted in the container, but are not stored in the image or the Compose file.

To use environment variables, you can use the environment option in your Compose file, to specify the environment variables for each service. For example, suppose you want to set the DEBUG environment variable for your web service, you can use the following option:

version: "3.9"
services:
  web: # name of the web service
    build: . # build the image from the Dockerfile
    ports:
      - "5000:5000" # map port 5000 on the host to port 5000 on the container
    environment: # specify the environment variables for this service
      - DEBUG=True # set the DEBUG environment variable to True

You can also use the env_file option, to specify a file that contains the environment variables for each service. For example, suppose you have a file named .env that contains the following environment variables:

DEBUG=True
SECRET_KEY=abc123
DATABASE_URL=postgresql://postgres:secret@db:5432

You can use the following option to load these environment variables for your web service:

version: "3.9"
services:
  web: # name of the web service
    build: . # build the image from the Dockerfile
    ports:
      - "5000:5000" # map port 5000 on the host to port 5000 on the container
    env_file: # specify the file that contains the environment variables for this service
      - .env # load the environment variables from the .env file

You can also use the ${VARIABLE} syntax in your Compose file, to substitute the values of environment variables that are set in your shell or in a .env file at the root of your project. For example, suppose you have the following environment variable set in your shell or in a .env file:

IMAGE_TAG=latest

You can use the following option to use this environment variable in your Compose file:

version: "3.9"
services:
  web: # name of the web service
    image: my-image:${IMAGE_TAG} # use the value of the IMAGE_TAG environment variable as the image tag
    ports:
      - "5000:5000" # map port 5000 on the host to port 5000 on the container

You can find more details about environment variables here.

How to run commands with Docker Compose

Docker Compose also allows you to run commands in your containers, either as part of the service definition, or as a one-off command. You can use the command option in your Compose file, to specify the command to run when the container starts. For example, suppose you want to run a custom script in your web service, you can use the following option:

version: "3.9"
services:
  web: # name of the web service
    build: . # build the image from the Dockerfile
    ports:
      - "5000:5000" # map port 5000 on the host to port 5000 on the container
    command: # specify the command to run when the container starts
      - ./script.sh # run the script.sh file

docker compose run (run in new container)

You can also use the docker-compose run command, to run a one-off command in a new container. For example, suppose you want to run a shell in your web service, you can use the following command:

docker-compose run web bash

This will create and start a new container for your web service, and run the bash command in it. You can then interact with the container using the shell. You can also use the --rm flag to remove the container after the command exits.

docker compose exec (run in existing container)

You can also use the docker-compose exec command, to run a command in an existing container. For example, suppose you want to run a shell in your web service, but you already have a container running for it, you can use the following command:

docker-compose exec web bash

This will run the bash command in the existing container for your web service, and attach to it. You can then interact with the container using the shell.

How to attach to containers with Docker Compose

Docker Compose also allows you to attach to the standard input, output, and error streams of your containers, either individually or collectively. You can use the docker-compose attach command, to attach to the streams of one or more services. For example, suppose you want to attach to the streams of your web service, you can use the following command:

docker-compose attach web

This will attach to the streams of the web service, and display the output of the container. You can also use the --no-stdin flag to disable the standard input stream, or the --no-color flag to disable the color codes.

You can also use the docker-compose logs command, to view the logs of one or more services. For example, suppose you want to view the logs of your web service, you can use the following command:

docker-compose logs web

This will display the logs of the web service, from the beginning. You can also use the -f flag to follow the logs, or the -t flag to show the timestamps.

How to view stats and events with Docker Compose

Docker Compose also allows you to view the statistics and events of your containers, such as the CPU, memory, network, and disk usage, and the actions that occur in your application. You can use the docker-compose stats command, to view the statistics of your containers. For example, suppose you want to view the stats of your web service, you can use the following command:

docker-compose stats web

This will display the stats of the web service, such as the name, ID, CPU%, MEM USAGE, MEM LIMIT, NET I/O, and BLOCK I/O. You can also use the --no-stream flag to show only the current stats, or the --no-color flag to disable the color codes.

You can also use the docker-compose events command, to view the events of your containers, such as the creation, start, stop, kill, etc. For example, suppose you want to view the events of your web service, you can use the following command:

docker-compose events web

This will display the events of the web service, such as the timestamp, service, action, and attributes. You can also use the --json flag to show the events in JSON format, or the --no-color flag to disable the color codes.

You can find more details about these commands in the official documentation about stats and events.

How to troubleshoot your application with Docker Compose

Docker Compose also provides some tools and options to help you troubleshoot your application, such as the docker-compose ps command, the docker-compose config command, and the --verbose flag.

You can use the docker-compose ps command, to list the status of your containers. For example, suppose you want to list the status of your web service, you can use the following command:

docker-compose ps web

This will display the status of the web service, such as the name, command, state, and ports. You can also use the -q flag to show only the IDs, or the --services flag to show only the service names.

You can use the docker-compose config command, to validate and view the configuration of your Compose file. For example, suppose you want to validate and view the configuration of your Compose file, you can use the following command:

docker-compose config

This will validate the syntax and the references of your Compose file, and display the configuration as a YAML file. You can also use the --services flag to show only the service names, or the --volumes flag to show only the volume names.

You can use the --verbose flag, to show more details about the execution of the Docker Compose commands. For example, suppose you want to show more details about the execution of the docker-compose up command, you can use the following command:

docker-compose --verbose up

This will show more details about the execution of the docker-compose up command, such as the API calls, the responses, the errors, etc.

Conclusion

That summarizes some of the important bits, but there's a whole lot more! I hope you learned something new from this blog post about Docker Compose ❤️.