Setup a Gitlab CI pipeline for your Elixir project

Recently I spent some time configuring a pipeline on Gitlab CI for an Elixir project. While mix is basically all you need, there are several principles I wanted to apply for this configuration that made the setup more complex than expected, especially when trying to also speed up the build. Of course, everything in this post is opinable and I am really open to suggestions, if you have any hint please add a comment at the end of this post.

The final configured flow will be the following:

Gitlab CI will compile the project with MIX_ENV set to test in a first, separated step. As soon as this job is done it will fire 3 other tasks in parallel:

• test to run our project tests.
• format to check that our source code is compliant with elixir-format.
• credo to statically analize our source code for refactoring opportunities.

If everything goes well the pipeline will be green and a deploy could eventually be triggered.

You might ask yourself why should the pipeline be this complex when a single CI job, configured to run all the above commands in sequence, would have been enough to solve the problem. Well, mainly for three reasons:

• some of these operations can be parallelized, saving some build time.
• I think it’s important to receive a clear indication about which command failed during the build. With a single sequential job you would have to read the full log to understand what happened. If instead each job is run separately you can immediately receive a feedback about which task is green and which one is broken.
• I prefer all jobs to be isolated as much as possible, to avoid the output of one command to impact the following one. Unfortunately I had to accept some compromises about this to increase the overall speed of the build.

The first task to configure is the compilation step (beware that this runs in a runner of type docker in Gitalb CI, you can’t use the exact same configuration with other runners like the shell runner).

stages:
  - build
  - quality
  
build:
  stage: build
  image: elixir:1.9.4
  variables:
    MIX_ENV: "test"
  script:
    - mix deps.get
    - mix compile
  artifacts:
    paths:
    - _build/
    - deps/
    expire_in: 10 minutes

This job uses a predefined docker image that has Elixir already installed, fetches the dependencies and compiles the application. Before ending this task some artifacts are uploaded on Gitlab CI and stored for 10 minutes, the _build/ folder and the deps/ folder. These two folders are reused in the next steps to avoid a new compilation and speed up the build. As you can see on the next tasks, there is no need to redeclare the artifacts when you want to use them, the subsequent tasks will use them right away.

This is the setup for test, format and credo.

test:
  stage: quality
  image: elixir:1.9.4
  variables:
    MIX_ENV: "test"
    POSTGRES_DB: "app_db"
    POSTGRES_USER: "postgres"
    POSTGRES_PASSWORD: "postgres"
    POSTGRES_ENV_HOSTNAME: "postgres"
    POSTGRES_ENV_PORT: "5432"
  services:
  - name: postgres:latest
  script:
    - mix deps.get
    - mix ecto.create
    - mix ecto.migrate
    - mix test
    
format:
  stage: quality
  image: elixir:1.9.4
  variables:
    MIX_ENV: "test"
  script:
    - mix deps.get
    - mix format --check-formatted --dry-run

credo:
  stage: quality
  image: elixir:1.9.4
  variables:
    MIX_ENV: "test"
  script:
    - mix deps.get
    - mix credo suggest --min-priority 10

You may wonder why the Gitlab CI cache functionality has not been used in this example, as it is the recommended way to store project dependencies. While it works, for some reasons mix triggers a new compilation in the test, format and credo steps when using the cache, and correctly skips the compilation when using artifacts.

With this configuration each task is run in its own CI job, some jobs are executed in parallel and there is an acceptable level (at least for me) of isolation between jobs, while skipping a recompilation on each build step. In terms of execution time, most of it is spent on the compilation step, which unfortunately always starts from scratch. It would be better if subsequent commits on the same branch could be detected (which is possible with Gitlab CI caching) and if mix could just recompile the changes instead of retriggering the pipeline from scratch, but that’s an optimization I haven’t tried yet.

I hope that’s useful, if you have any feedback I am all ears.