“Upgrading is seeking death, not upgrading is waiting for death”, the painful upgrade path of GitLab CE

Editor’s note: This article is reprinted from the public account Operation and Maintenance Shitang. The author has been contacted to obtain reprint authorization.

In the more than ten years of development of GitLab, it has accumulated a large number of CE users in China. However, many CE users do not follow GitLab’s release rhythm (monthly release) for version upgrades. Now that GitLab has released version 16.4, There are also a large number of users using versions below 10. The author of the following article describes the process of upgrading GitLab 9.x version compiled and installed using source code, which involves database switching (from Mysql to Postgres), data backup and recovery, container operation, etc. Although the process is ” “It seems simple”, but it requires deep technical reserves and strong confidence to complete these operations, and it took two months from testing and verification to production. For many operation and maintenance personnel, the following upgrade process from the test environment to the production environment is an extremely big challenge. If a problem occurs, it may cause the company’s internal source code hosting platform to be unavailable, affecting the company’s software research and development.

From another perspective, this is also the technical debt caused by using the CE version. ManyCE users are worried about problems during the upgrade process. Adhering to the principle of “if you don’t move, there will be no problems”, they have been using Old versions, which can easily result in the product not only being unable to obtain the latest features, but also potentially exposing GitLab to external attacks due to security vulnerabilities in the old versions. The dilemma of upgrading is to wait for death.

In fact, enterprise users should try to use the Enterprise Edition (EE). In addition to getting the latest features in a timely manner, they can also enjoy enterprise-level service support. The original service can easily handle product upgrades, security problem repairs, etc., which can not only save operation and maintenance personnel from upgrading. Free from phobias, you can also use the latest features to improve research and development efficiency.

Before you begin

Starting in 2019, SonarQube and GitLab have successively announced that they will no longer provide support for MySQL data storage (Consider removing support for MySQL). Since Oracle and SQL Server are both paid commercial databases, while PostgreSQL is open source, coupled with the continuous innovation for PostgreSQL and the popularity of ChatGPT in recent years, Postgres has entered my TODO list again. From the cloud factory RDS Postgres plug-in list and several open source projects (such as: Neon: Serverless Postgres: serverless Postgres, timescaledb: time series database extension based on Postgres, FerretDB: convert MongoDB protocol into SQL using Postgres as storage backend) and Like this AI application scenario of building a dedicated LLM-driven ChatBot (pgvector used in it) based on PostgreSQL. It can be seen that Postgres is very popular, so I want to find out.

Because Postgres will definitely need to be understood and used in the future, because it has become the first choice or default database for many open source software, so it has reached the point where it must be learned. After reading the PostgreSQL tutorial, I immediately remembered the painful experience of the previous GitLab migration failure (because of the large amount of data, a test lasted for 3 hours, the time cost was too high, and the pressure caused by repeated failures was frustrating). Finally, after many failures, I continued to summarize the reasons and finally found the problem and possible solutions.

Crack the mystery

Because the code base initially built was deployed based on GitLab 9.0.6 source code and built using external RDS MySQL 5.6, daily maintenance consisted of expanding the disk or upgrading the CPU and memory. GitLab has never been upgraded in 6 years (mainly due to fear of the impact of the upgrade) It’s not the daily work of nearly a thousand people, so why bother asking for trouble if you can’t use it?) I tried to do an upgrade test a year ago, and also consulted manufacturers and the community (there are really few people who can do this), but ultimately failed (I am short of words). Looking back now, I still didn’t read the official documents carefully, and the upgrade idea was wrong from the beginning (the correct solution is given below).

Because the MySQL database is no longer supported starting from GitLab 12.1, the migration from MySQL to Postgres must be completed. Since the official all-in-one package (Omnibus GitLab Packages) uses the Postgres database by default, in order to facilitate future version upgrades, it is necessary to prioritize the migration of the current version of the database to Postgres. The following mainly shares the core process of MySQL migration to Postgres.

Back up the current environment

Backup method of source code installation
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
# Note: Pay attention to the disk space when backing up. After the backup, a time-stamped tar package file will be generated in the backup directory

Install a new Linux environment

? Install Docker

? Prepare 3 containers (mysql/postgres/pgloader) and migration configuration files

file: /data/gitlab/migrate/docker-compose.yml

version: '3'

services:
  pgloader:
    image: seanly/toolset:pgloader
    tty: true
    volumes:
      - ./:/ws
    restart: always
    security_opt:
      - seccomp:unconfined
  mysql:
    image: seanly/dbextra:mysql-5.7.35
    restart: unless-stopped
    volumes:
      - ./mysql-data:/var/lib/mysql
      - ./:/ws
    environment:
      TZ: Asia/Shanghai
      MYSQL_ROOT_PASSWORD: root123
      MYSQL_DATABASE: 'gitlabhq_production'
    healthcheck:
      test: mysql --user=root --password=$$MYSQL_ROOT_PASSWORD -e "SHOW DATABASES;"
      interval: 3s
      timeout: 1s
      retries: 5
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_general_ci
      - --max_allowed_packet=512M

  postgres:
    image: postgres:11-alpine
    restart: always
    environment:
      - POSTGRES_USER=gitlab
      - POSTGRES_PASSWORD=gitlab123
      - POSTGRES_DB=gitlabhq_production
    volumes:
      - ./postgres-data:/var/lib/postgresql/data
      - ./:/ws
    healthcheck:
      test: pg_isready -U gitlab -h 127.0.0.1
      interval: 5s

file: /data/gitlab/migrate/gitlab.loader

LOAD DATABASE
  FROM mysql://root:root123@mysql:3306/gitlabhq_production
  INTO pgsql://gitlab:gitlab123@postgres:5432/gitlabhq_production
  
WITH include no drop, truncate, disable triggers, create no tables,
     create no indexes, preserve index names, no foreign keys,
     data only,
     workers = 8, concurrency = 16,
     batch rows = 100000, batch size = 512MB, multiple readers per thread, rows per range = 500000
     
     SET MySQL PARAMETERS
     net_read_timeout = '90',
     net_write_timeout = '180'
     
     ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public'
     
     ;

Execute the command docker-compose up -d to start the container.

Get a backup of MySQL

# 1. Copy the backup file to the new host, assuming it is stored in the /data/gitlab/backup directory
# 2. Get db/database.sql.gz from TIMESTAMP_gitlab_backup.tar package

cd /data/gitlab/backup
tar xf TIMESTAMP_gitlab_backup.tar db
gunzip db/database.sql.gz

# /data/gitlab/backup/db/database.sql is the backup of MySQL
mv /data/gitlab/backup/db/database.sql /data/gitlab/migrate/mysql_database.sql

Get a backup of Postgres

This step is critical. In order to ensure the success of subsequent upgrades, a complete empty database structure is required, and the data can be migrated through the MySQL library. The way to handle it is to initialize a version through the container, and then back up the Postgres inside.

? Start a GitLab CE 9.0.6 instance

file: /data/gitalab/migrate/gitlab/docker-compose.yml

version: '3'
services:
  gitlab:
    image: 'gitlab/gitlab-ce:9.0.6-ce.0'
    restart: always
    volumes:
      - './:/ws'
    shm_size: '256m'

Start the container and backup the database

cd /data/gitalab/migrate/gitlab
docker-compose up -d
docker-compose exec gitlab bash
# Inside the container
su-gitlab-psql
pg_dump -U gitlab-psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -f /ws/gitlab_backup.sql
exit
# Outside the container
docker-compose down
mv gitlab_backup.sql /data/gitalab/migrate/

Migrate MySQL to Postgres

? Restore MySQL

cd /data/gitalab/migrate/
docker-compose exec mysql bash

# Enter msyql container
mysql -u root -p gitlabhq_production < /ws/mysql_database.sql
# Enter password root123

? Restore Postgres

cd /data/gitalab/migrate/
docker-compose exec postgres bash

# Enter the postgres container

psql -U gitlab -d gitlabhq_production -f /ws/gitlab_backup.sql

? Migrate MySQL to Postgres

docker-compose exec pgloader pgloader /ws/gitlab.loader

? Export Postgres

cd /data/gitalab/migrate/
docker-compose exec postgres bash

# Enter the container
pg_dump -U gitlab -d gitlabhq_production -f /ws/database.sql
exit
#Exit container

? Restore Postgres to backup package

mv /data/gitalab/migrate/database.sql /data/gitlab/backup/db/database.sql
cd /data/gitlab/backup/
#Package sql into sql.gz
gzip db/database.sql
# Update db/database.sql.gz of the backup package
tar rf TIMESTAMP_gitlab_backup.tar db/database.sql.gz

Restore backup and upgrade

? Install a GitLab CE 9.0.6 container

file: /data/gitlab/test-upgrade/docker-compose.yml

version: '3'
services:
  gitlab:
  image: 'gitlab/gitlab-ce:9.0.6-ce.0'
  restart: always
  ports:
    - '80:80'
    - '443:443'
    - '2222:22'
  volumes:
    - './config:/etc/gitlab'
    - './data:/var/opt/gitlab'
    - '/data/backup:/var/opt/gitlab/backups'
  shm_size: '256m'

? Start container and resume

mkdir -p /data/gitlab/test-upgrade/{data/backup,config}
cp -r /data/gitlab/backup/TIMESTAMP_gitlab_backup.tar /data/gitlab/test-upgrade/backup/

docker-compose up -d
docker-compose exec gitlab bash

# Enter the container
gitlab-rake gitlab:backup:restore RAILS_ENV=production BACKUP=${TIMESTAMP}

#Exit the container
docker-compose logs -f

? Upgraded version

The version upgrade method is very simple. Please refer to the official upgrade path for version selection. Modify the image version in docker-compose.yml and then execute docker-compose up -d to perform the version upgrade. upgrade.

Finally

Although the upgrade verification process was painful, I learned a lot. Looking back now, the most critical part of the whole process is that you only need to migrate the MySQL data to the GitLab Postgres database through pgloader. The rest is a standard upgrade process, and then use the container shielding tool to shield the dependence on the environment.

About Jihu GitLab

Jihu GitLab is an integrated DevOps platform launched by GitLab for domestic users. GitLab is the upstream of GitLab, which means that GitLab has the same functions as GitLab. Moreover, GitLab has also developed features specifically for domestic enterprises for domestic users, such as integration with WeChat, DingTalk, and Feishu. wait.

Jihu GitLab’s local operations team can provide professional enterprise-level service support for domestic enterprise users, and has currently helped hundreds of companies implement DevOps practices. JiHu GitLab’s professional service capabilities can help enterprises achieve smooth upgrades of GitLab/JiHu GitLab.