Pipelines can be viewed in Gitlab by clicking CI / CD → Pipelines. Each pipeline is based on a specific a commit, and the commit ID (the SHA) and branch for each pipeline is displayed on the pipeline list, as well as on each pipeline page.

The pipeline related to a commit can be run at any time—this means you can deploy any commit, on any branch, to any environment as required.

Stages

Pipelines run from left to right in a series of stages. The stages in the M.D.G. IT Magento pipeline are:

• Build: this contains the build job, which isn't specific to an environment, as well as a system job called Enabling Utilities which enables the Downstream utilities in the rightmost stage.
• Dev (optional), Staging, Production: these stages contain the deploy jobs that push code to the respective environment (i.e. server).
• Downstream: this is an expandable set of utility stages, which display when clicked.

Jobs

Each pipeline is composed of a series of jobs. Jobs can run manually, or automatically given certain conditions—for commits on a specific branch, for example. This can be customised for each project, so please let us know if you have a preferred workflow, or if you would like suggestions about how to best set this up for your use case. Manual jobs can be started by clicking the play icon, or by clicking elsewhere on the button and then clicking Trigger this manual action. Clicking on the button displays the job page, where you can see the console output of the commands that are being run. Jobs run in the background, so you can close the page while they are running, and return to the page to see the output later.

Build

The build job, runs the three steps required by Magento 2 every time a change is made to the source code: Build, Compile and Static Content Deploy. These steps dynamically generate files required by Magento to run, including the vendor directory (using composer) and the static view files, like CSS and javascript.

The output of the build job is the build artifact, which is the full set of files required by Magento to run. You can view the artifact of the build job after it successfully completes on the right of the screen, as well as download this as a tarball. These are kept for a week after the build is completed, however if they've expired you can simply rerun the build job to regenerate them. Rerunning the build job on a specific commit will always generate the same artifact.

Changing Build Step Behaviour

You can change default behaviour of each step, or add functionality to be run after a step. For each step, the pipeline will check the replacement script location to see if there is a script there, otherwise it will execute the default behaviour for that step. It will then run the script in the post-run script location if it exists. Being part of the build job, these scripts will execute in the build container, not on the server.

The repository file locations to replace or add post step jobs are:

Deploy

The deploy job downloads the artifact created by the build job, and deploys this to the respective environment. After files are deployed, this job also runs setup:upgrade and flushes the cache.

If you need to perform any actions after the application is deployed, you can add a script at mdg-src/post-deploy.sh. This will execute on the server after the application is deployed.

Shared directory and symlinked files

Specific files and directories in a Magento 2 application do not belong in the Git repository, including:

• app/etc/env.php  (configuration details that are different for each environment, database and other sensitive credentials)
• var/varnish.vcl (Varnish configuration file)
• pub/media (image files slow down a git repository if checked in. Further these are specific to an environment—you don't want product images that have just been uploaded overwritten by an old version of pub/media from git)
• var/composer_home
• var/session
• var/log (log entries for a specific environment)
• var/report (environment-specific error reports)
• var/import 
• var/import_history
• var/export
• var/importexport
• var/backups
• var/tmp

Rather than being checked in to git, separate versions of these files and directories exist in each environment. These files and directories are located in a shared directory, which is on the same level as the codebase directory which contains the Magento installation. Symbolic links are created when the application is deployed so that the application finds them where it expects to. As an example:

/var/www/html/mysite.com/codebase/app/etc/env.php is actually a symlink that points to 
/var/www/html/mysite.com/shared/app/etc/env.php

Changing symlinks

The above symlinks will be deployed to an environment, rather than any identically named files or folders that may be checked into Git. There are situations where you may need to change this behaviour. An example is where you want to check in theme files to e.g. pub/media/mytheme and have these version controlled and included in the deployment.

Normally, these files wouldn't be deployed to the server—even if they're in Git—because the entire pub/media is replaced with a symlink to ../shared/pub/media . Without modifying the symlinks, you would have to move the files in the mytheme directory to each environment manually.

Because we don't want to check the whole of the media directory into Git, the pub/media symlink is replaced with pub/media/catalog. This is done in the mdg-src/deploy_vars file. If this does not exist, please raise a support ticket with M.D.G IT.

The file defines the following symlinks, which by default should look something like the following:

MEDIA_DIRS="pub/media"
SYMLINKS="app/etc/env.php $MEDIA_DIRS var/varnish.vcl var/composer_home var/log var/export var/report var/import var/import_history var/session var/importexport var/backups var/tmp"

MEDIA_DIRS is a separate variable that also allows for the synchronization of media files between environments.

If, for example, you would like to check some files in pub/media into source control, whilst keeping larger catalog files out of source control. you would change the mdg-src/deploy_vars file with the following line:

MEDIA_DIRS="pub/media/catalog"

Downstream Utilities

These utilities run the following jobs against the respective environment:

• Cache flush flushes the Magento cache
• Check in build files, database state (development only) runs app:config:dump on the development environment and checks the resulting config.php file, along with composer.json and composer.lock into git. This results in a new commit, so if you're running a full build that needs updated versions of these files, (to install a new extension on staging or production, for example) you'd need to run the staging build on the pipeline corresponding to the new commit. Please note also that if these files haven't changed this job will fail.
• Custom commands copies src/custom_commands.sh to the environment in question and executes the file in the Magento webroot.
• DB Sync copies the database from production to the environment that the job appears under to keep the staging / dev environments up to date with database changes made in production (e.g. new products, CMS content).
• Hotfix quickly pushes changed files to an environment without running a build. Please note that this does not delete files on the server where Magento is running—if you have deleted files in git, please run a build and a deployment.
• Images sync will sync the media directories from production to staging, or from production to development. This can take a while if transferring images for the first time, but will only transfer new or changed images on subsequent syncs. You can change which directories are synced by this action by setting the MEDIA_DIRS variable in mdg-src/deploy_vars. Please note that this will not delete images on staging that don't exist on production.
• Maintenance enable turns on maintenance mode.
• Maintenance disable turns off maintenance mode.
• Reindex runs a full reindex.