Usage of the GitLab Steps with `.gitlab-ci.yml`

This document describes how GitLab Steps are integrated into the .gitlab-ci.yml.

GitLab Steps will be integrated using a three-stage execution cycle and replace before_script:, script: and after_script:.

setup:: Execution stage responsible for provisioning the environment, including cloning the repository, restoring artifacts, or installing all dependencies. This stage will replace implicitly cloning, restoring artifacts, and cache download.
run:: Execution stage responsible for running a test, build, or any other main command required by that job.
teardown:: Execution stage responsible for cleaning the environment, uploading artifacts, or storing cache. This stage will replace implicit artifacts and cache uploads.

Before we can achieve three-stage execution we will ship minimal initial support that does not require any prior GitLab integration.

Phase 1: Initial support

Initially the Step Runner will be used externally, without any prior dependencies to GitLab:

The step-runner will be provided as part of a container image.
The step-runner will be explicitly run in the script: section.
The $STEPS environment variable will be executed as type: steps.

hello-world:
  image: registry.gitlab.com/gitlab-org/step-runner
  variables:
    STEPS: |
      - step: gitlab.com/josephburnett/component-hello-steppy@master
        inputs:
          greeting: "hello world"
  script:
    - /step-runner ci

Phase 2: The addition of `run:` to `.gitlab-ci.yml`

In Phase 2 we will add run: as a first class way to use GitLab Steps:

run: will use a type: steps syntax.
run: will replace usage of before_script, script and after_script.
All existing functions to support Git cloning, artifacts, and cache would continue to be supported.
It is yet to be defined how we would support after_script, which is executed unconditionally or when the job is canceled.
run: will not be allowed to be combined with before_script:, script: or after_script:.
GitLab Rails would not parse run:, instead it would only perform static validation with a JSON schema provided by the Step Runner.

hello-world:
  image: registry.gitlab.com/gitlab-org/step-runner
  run:
    - step: gitlab.com/josephburnett/component-hello-steppy@master
      inputs:
        greeting: "hello world"

The following example would fail syntax validation:

hello-world:
  image: registry.gitlab.com/gitlab-org/step-runner
  run:
    - step: gitlab.com/josephburnett/component-hello-steppy@master
      inputs:
        greeting: "hello world"
  script: echo "This is ambiguous and invalid example"

Transitioning from `before_script:`, `script:` and `after_script:`

GitLab Rails would automatically convert the *script: syntax into relevant run: specification:

Today before_script: and script: are joined together as a single script for execution.
The after_script: section is always executed in a separate context, representing a separate step to be executed.
It is yet to be defined how we would retain the existing behavior of after_script, which is always executed regardless of the job status or timeout, and uses a separate timeout.
We would retain all implicit behavior which defines all environment variables when translating script: into step-based execution.

For example, this CI/CD configuration:

hello-world:
  before_script:
    - echo "Run before_script"
  script:
    - echo "Run script"
  after_script:
    - echo "Run after_script"

Could be translated into this equivalent specification:

hello-world:
  run:
    - step: gitlab.com/gitlab-org/components/steps/legacy/script@v1.0
      inputs:
        script:
          - echo "Run before_script"
          - echo "Run script"
    - step: gitlab.com/gitlab-org/components/steps/legacy/script@v1.0
      inputs:
        script:
          - echo "Run after_script"
      when: always

Phase 3: The addition of `setup:` and `teardown:` to `.gitlab-ci.yml`

The addition of setup: and teardown: will replace the implicit functions provided by GitLab Runner: Git clone, artifacts and cache handling:

The usage of setup: would stop GitLab Runner from implicitly cloning the repository.
artifacts: and cache:, when specified, would be translated and appended to setup: and teardown: to provide backward compatibility for the old syntax.
release:, when specified, would be translated and appended to teardown: to provide backward compatibility for the old syntax.
setup: and teardown: could be used in default: to simplify support of common workflows like where the repository is cloned, or how the artifacts are handled.
The split into 3-stage execution additionally improves composability of steps with extends:.
The hooks:pre_get_sources_script would be implemented similar to script: and be prepended to setup:.

For example, this CI/CD configuration:

rspec:
  script:
    - echo "This job uses a cache."
  artifacts:
    paths: [binaries/, .config]
  cache:
    key: binaries-cache
    paths: [binaries/*.apk, .config]

Could be translated into this equivalent specification executed by a step runner:

rspec:
  setup:
    - step: gitlab.com/gitlab-org/components/git/clone@v1.0
    - step: gitlab.com/gitlab-org/components/artifacts/download@v1.0
    - step: gitlab.com/gitlab-org/components/cache/restore@v1.0
      inputs:
        key: binaries-cache
  run:
    - step: gitlab.com/gitlab-org/components/steps/legacy/script@v1.0
      inputs:
        script:
          - echo "This job uses a cache."
  teardown:
    - step: gitlab.com/gitlab-org/components/artifacts/upload@v1.0
      inputs:
        paths: [binaries/, .config]
    - step: gitlab.com/gitlab-org/components/cache/restore@v1.0
      inputs:
        key: binaries-cache
        paths: [binaries/*.apk, .config]

Inheriting common operations with `default:`

setup: and teardown: are likely to become very verbose over time. One way to simplify them is to allow inheriting the common setup: and teardown: operations with default:.

The previous example could be simplified to:

default:
  setup:
    - step: gitlab.com/gitlab-org/components/git/clone@v1.0
    - step: gitlab.com/gitlab-org/components/artifacts/download@v1.0
    - step: gitlab.com/gitlab-org/components/cache/restore@v1.0
      inputs:
        key: binaries-cache
  teardown:
    - step: gitlab.com/gitlab-org/components/artifacts/upload@v1.0
      inputs:
        paths: [binaries/, .config]
    - step: gitlab.com/gitlab-org/components/cache/restore@v1.0
      inputs:
        key: binaries-cache
        paths: [binaries/*.apk, .config]

rspec:
  run:
    - step: gitlab.com/gitlab-org/components/steps/legacy/script@v1.0
      inputs:
        script:
          - echo "This job uses a cache."

linter:
  run:
    - step: gitlab.com/gitlab-org/components/steps/legacy/script@v1.0
      inputs:
        script:
          - echo "Run linting"

Parallel jobs and `setup:`

With the introduction of setup: at some point in the future we will introduce an efficient way to parallelize the jobs:

setup: would define all steps required to provision the environment.
The result of setup: would be snapshot and distributed as the base for all parallel jobs, if parallel: N is used.
The run: and teardown: would be run on top of cloned job, and all its services.
The runner would control and intelligently distribute all parallel jobs, significantly cutting the resource requirements for fixed parts of the job (Git clone, artifacts, installing dependencies.)

rspec-parallel:
  image: ruby:3.2
  services: [postgres, redis]
  parallel: 10
  setup:
    - step: gitlab.com/gitlab-org/components/git/clone@v1.0
    - step: gitlab.com/gitlab-org/components/artifacts/download@v1.0
      inputs:
        jobs: [setup-all]
    - script: bundle install --without production
  run:
    - script: bundle exec knapsack

Potential GitLab Runner flow:

Runner receives the rspec-parallel job with setup: and parallel: configured.
Runner executes a job on top of Kubernetes cluster using block volumes up to the setup.
Runner then runs 10 parallel jobs in Kubernetes, overlaying the block volume from 2 and continue execution of run: and teardown:.

Usage of the GitLab Steps with .gitlab-ci.yml

Phase 1: Initial support

Phase 2: The addition of run: to .gitlab-ci.yml

Transitioning from before_script:, script: and after_script:

Phase 3: The addition of setup: and teardown: to .gitlab-ci.yml

Inheriting common operations with default: