Since the Gitlab version used is relatively old, we plan to upgrade and migrate to a new server.
Upgraded from 11.7.0 to version 16.4.1.
1. Upgrade
1.1 Determine the upgrade path
Gitlab upgrade cannot span major versions, and should be operated according to the official upgrade path.
I installed Gitlab through docker, so only the upgrade method of docker is listed.
Official link: https://docs.gitlab.com/ee/update/index.html#upgrade-paths
After entering, configure the current version and target version, and the specific upgrade path will be given. Here is my upgrade path
1.2 Check before and after upgrade
Checks that need to be done before and after each upgrade to ensure the normal operation of the system:
# Status check gitlab-rake gitlab:check # Data encryption check gitlab-rake gitlab:doctor:secrets
Check out the Gitlab UI:
Whether you can log in, whether you can display the project list, whether you can pull and submit code, etc.
Version check:
1.3 Backup
GitLab 12.2 or higher:
docker exec -t <container name> gitlab-backup create
It should be noted that this command only backs up data. Configuration files, SSL certificates, etc. are not backed up and need to be backed up manually.
The backup file name is generally timestamp + _gitlab_backup.tar
The path is in backups
]# ls /home/git/data/backups/ 1697632970_2023_10_18_16.1.5_gitlab_backup.tar
1.4 upgrade
# Upgrade in order, remember to check after upgrading. docker run gitlab/gitlab-ce:11.11.8-ce.0 docker run gitlab/gitlab-ce:12.0.12-ce.0 docker run gitlab/gitlab-ce:12.1.17-ce.0 ... docker run gitlab/gitlab-ce:16.4.1-ce.0 docker run gitlab/gitlab-ce:15.4.6-ce.0
Log view
docker logs -f --tail 100 git_web docker compose logs --tail=100 -f # Need to be executed in the same directory as docker-compose.yml
2. Data migration
You can only restore a backup to the exact same GitLab version and type (CE/EE) as when the backup was created.
After installing Gitlab on the new server, execute the commands in the following order
# Stop the processes that are connected to the database # docker exec -it <name of container> gitlab-ctl stop puma docker exec -it git_web gitlab-ctl stop puma #docker exec -it <name of container> gitlab-ctl stop sidekiq docker exec -it git_web gitlab-ctl stop sidekiq # Verify that the processes are all down before continuing # docker exec -it <name of container> gitlab-ctl status docker exec -it git_web gitlab-ctl status # Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name # docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce docker exec -it git_web gitlab-backup restore BACKUP=1697632970_2023_10_18_16.1.5 # Restart the GitLab container # docker restart <name of container> docker restart git_web # Check GitLab # docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true docker exec -it git_web gitlab-rake gitlab:check SANITIZE=true
After the data was restored, I accessed git and reported an error. The website had HSTS enabled and could not be accessed. After updating the SSL certificate, the website could be accessed normally.
3. Troubleshooting
- An error occurred when upgrading from 15.4.6 to 15.11.13:
[2023-10-17T04:35:35 + 00:00] ERROR: Running exception handlers There was an error running gitlab-ctl reconfigure: gitlab_rails['smtp_tls'] and gitlab_rails['smtp_enable_starttls_auto'] are mutually exclusive. Set one of them to false. SMTP providers usually use port 465 for TLS and port 587 for STARTTLS. Running handlers complete [2023-10-17T04:35:35 + 00:00] ERROR: Exception handlers complete Infra Phase failed. 0 resources updated in 06 seconds [2023-10-17T04:35:35 + 00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out [2023-10-17T04:35:35 + 00:00] FATAL: ---------------------------------- -------------------------------------------------- ---- [2023-10-17T04:35:35 + 00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT [2023-10-17T04:35:35 + 00:00] FATAL: ---------------------------------- -------------------------------------------------- ---- [2023-10-17T04:35:35 + 00:00] FATAL: RuntimeError: gitlab_rails['smtp_tls'] and gitlab_rails['smtp_enable_starttls_auto'] are mutually exclusive. Set one of them to false. SMTP providers usually use port 465 for TLS and port 587 for STARTTLS.
It probably means that in the SMTP configuration of gitlab.rb, the two options smtp_tls and smtp_enable_starttls_auto cannot be enabled at the same time.
smtp_tls: Whether to use TLS to encrypt SMTP connections.
smtp_enable_starttls_auto: Whether to automatically use the STARTTLS command to enable TLS encryption.
Solution:
Just change the value of one of them to false and redeploy it.
- Deploy in progress
It will take some time after deployment, wait a few minutes and try again.
- can’t create Thread: Operation not permitted
Detailed error message:
[2023-10-17T09:01:03 + 00:00] INFO: Loading cookbooks [[email protected], [email protected], [email protected], [email protected], [email protected] .0, [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], gitlab-pages @0.1.0, [email protected], [email protected], [email protected], [email protected], [email protected]] Synchronizing cookbooks: ================================================== ============================== Error Syncing Cookbooks: ================================================== ============================== Unexpected Error: ------------------ ThreadError: can't create Thread: Operation not permitted System Info: ---------- chef_version=17.10.0 platform=ubuntu platform_version=22.04 ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client Running handlers: [2023-10-17T09:01:03 + 00:00] ERROR: Running exception handlers There was an error running gitlab-ctl reconfigure: can't create Thread: Operation not permitted [2023-10-17T09:01:03 + 00:00] ERROR: Report handler GitLabHandler::Exception raised #<NameError: uninitialized constant GitLabHandler::Exception::OmnibusHelper> [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/cookbooks/package/libraries/handlers/gitlab.rb:30:in `report' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb: 296:in `run_report_unsafe' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb: 284:in `run_report_safely' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb: 155:in `block in run_exception_handlers' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb: 153:in `each' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb: 153:in `run_exception_handlers' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb: 164:in `block in <class:Handler>' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb: 443:in `block in run_failed' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb: 442:in `each' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb: 442:in `run_failed' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb: 306:in `rescue in run' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb: 299:in `run' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb: 305:in `run_with_graceful_exit_option' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb: 281:in `block in run_chef_client' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb: 42:in `with_server_connectivity' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb: 264:in `run_chef_client' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base. rb:352:in `run_application' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb: 67:in `run' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client: 25:in `<top (required)>' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `load' [2023-10-17T09:01:03 + 00:00] ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `<main>' Running handlers complete [2023-10-17T09:01:03 + 00:00] ERROR: Exception handlers complete Infra Phase failed. 0 resources updated in 01 seconds [2023-10-17T09:01:03 + 00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out [2023-10-17T09:01:03 + 00:00] FATAL: ---------------------------------- -------------------------------------------------- ---- [2023-10-17T09:01:03 + 00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT [2023-10-17T09:01:03 + 00:00] FATAL: ---------------------------------- -------------------------------------------------- ---- [2023-10-17T09:01:03 + 00:00] FATAL: ThreadError: can't create Thread: Operation not permitted
Official reply:
# Link: https://docs.gitlab.com/ee/install/docker.html#threaderror-cant-create-thread-operation-not-permitted This error occurs when running a container built with newer versions on a host that doesn’t have support for the new clone3 function. In GitLab 16.0 and later, the container image includes the Ubuntu 22.04 Linux package which is built with this newer .glibcglibc This problem is fixed with newer container runtime tools like Docker 20.10.10. To resolve this issue, update Docker to version 20.10.10 or later.
Solution: Upgrade docker to 20.10.10 or higher.
- An error occurs when recovering data
2023-10-19 15:47:50 + 0800 -- Unpacking backup ... tar: 1697701149_2023_10_19_16.1.5_gitlab_backup.tar: Cannot open: Permission denied tar: Error is not recoverable: exiting now 2023-10-19 15:47:50 + 0800 -- Unpacking backup failed 2023-10-19 15:47:50 + 0800 -- Deleting backup and restore PID file ... done
Solution: Caused by insufficient permissions, just enter the container and modify the permissions.
$ docker exec -it git_web bash # cd /var/opt/gitlab/backups #ll-h -rw------- 1 root root 1.7G Oct 19 07:45 1697701149_2023_10_19_16.1.5_gitlab_backup.tar # chown git:git 1697701149_2023_10_19_16.1.5_gitlab_backup.tar #ll -rw------- 1 git git 1743431680 Oct 19 07:45 1697701149_2023_10_19_16.1.5_gitlab_backup.tar