refer to
Gitlab rapid deployment and daily maintenance (community version RPM package installation)_51CTO blog_gitlab community version usage tutorial
This article mainly records the operation process of troubleshooting the Gitlab service stop on Centos7 on 2023/11/9, as well as the handling of problems in the process.
When connecting to the server, I found that only half a gigabyte of memory was used. Did the gitlab service automatically shut down after exiting the terminal?
Check the status, the service has stopped
[root@gitlab gitlab]# gitlab-ctl status fail: alertmanager: runsv not running fail: gitaly: runsv not running fail: gitlab-exporter: runsv not running fail: gitlab-kas: runsv not running fail: gitlab-workhorse: runsv not running fail: logrotate: runsv not running fail: nginx: runsv not running fail: node-exporter: runsv not running fail: postgres-exporter: runsv not running fail: postgresql: runsv not running fail: prometheus: runsv not running fail: puma: runsv not running fail: redis: runsv not running fail: redis-exporter: runsv not running fail: sidekiq: runsv not running [root@gitlab gitlab]#
Let’s read the log first
Gitlab’s commonly used default installation directory
Gitlab component log path:/var/log/gitlab
Gitlab configuration path: /etc/gitlab/ There is a gitlab.rb configuration file under the path
Application code and component dependencies:/opt/gitlab
Storage path of each component: /var/opt/gitlab/
The default storage path of the warehouse is /var/opt/gitlab/git-data/repositories
Version file backup path: /var/opt/gitlab/backups/
nginx installation path:/var/opt/gitlab/nginx/
redis installation path:/var/opt/gitlab/redis
[root@gitlab etc]# cd /var/log/gitlab/ [root@gitlab gitlab]#ll Total usage 4 drwx------. 2 gitlab-prometheus root 47 November 8 16:50 alertmanager drwx------. 2 git root 71 Nov 8 17:14 gitaly drwx------. 2 git root 47 Nov 8 16:50 gitlab-exporter drwx------. 2 git root 47 Nov 8 16:48 gitlab-kas drwx------. 2 git root 4096 November 8 18:59 gitlab-rails drwx------. 2 git root 6 November 8 16:45 gitlab-shell drwx------. 2 git root 47 Nov 8 16:50 gitlab-workhorse drwx------. 2 root root 47 Nov 8 16:47 logrotate drwxr-x---. 2 root gitlab-www 131 November 8 16:50 nginx drwx------. 2 gitlab-prometheus root 47 Nov 8 16:50 node-exporter drwx------. 2 gitlab-psql root 47 Nov 8 16:51 postgres-exporter drwx------. 2 gitlab-psql root 47 Nov 8 16:48 postgresql drwx------. 2 gitlab-prometheus root 47 November 8 16:50 prometheus drwx------. 2 git root 93 Nov 8 16:50 puma drwxr-xr-x. 2 root root 50 Nov 8 16:47 reconfigure drwx------. 2 gitlab-redis root 47 November 8 16:48 redis drwx------. 2 gitlab-redis root 47 November 8 16:50 redis-exporter drwx------. 2 git root 47 Nov 8 16:50 sidekiq [root@gitlab gitlab]# sudo cat ./gitlab-rails/production.log Raven 3.1.2 configured not to capture errors: DSN not set Raven 3.1.2 configured not to capture errors: DSN not set Raven 3.1.2 configured not to capture errors: DSN not set Raven 3.1.2 configured not to capture errors: DSN not set Rendered layout layouts/mailer/design.html.haml (Duration: 58.6ms | Allocations: 26286) Delivered mail [email protected] (359.6ms)
Execute the command to view all logs: sudo gitlab-ctl tail
It is judged that the gitlab service stops approximately 90 minutes after exiting the terminal.
==> /var/log/gitlab/gitlab-exporter/current <== 2023-11-08_10:58:44.53934 - -> /ruby 2023-11-08_10:58:52.13681 ::1 - - [08/Nov/2023:18:58:52 CST] "GET /sidekiq HTTP/1.1" 200 119348 2023-11-08_10:58:52.13684 - -> /sidekiq 2023-11-08_10:58:59.21303 ::1 - - [08/Nov/2023:18:58:59 CST] "GET /database HTTP/1.1" 200 1968 2023-11-08_10:58:59.21307 - -> /database 2023-11-08_10:58:59.53605 ::1 - - [08/Nov/2023:18:58:59 CST] "GET /ruby HTTP/1.1" 200 1073 2023-11-08_10:58:59.53607 - -> /ruby 2023-11-08_10:59:05.53635 == Sinatra has ended his set (crowd applauds) 2023-11-08_10:59:05.61880 [2023-11-08 18:59:05] INFO going to shutdown ... 2023-11-08_10:59:05.61918 [2023-11-08 18:59:05] INFO WEBrick::HTTPServer#start done.
Because I am still in the learning stage, I don’t want to ask for a detailed explanation. I want to run through CI/CD of gitlab first. Start the gitlab service first.
[root@gitlab gitlab]# systemctl enable gitlab-runsvdir.service [root@gitlab gitlab]# gitlab-ctl restart fail: alertmanager: runsv not running fail: gitaly: runsv not running fail: gitlab-exporter: runsv not running fail: gitlab-kas: runsv not running fail: gitlab-workhorse: runsv not running fail: logrotate: runsv not running fail: nginx: runsv not running fail: node-exporter: runsv not running fail: postgres-exporter: runsv not running fail: postgresql: runsv not running fail: prometheus: runsv not running fail: puma: runsv not running fail: redis: runsv not running fail: redis-exporter: runsv not running fail: sidekiq: runsv not running
The startup fails. The first message you see is fail: alertmanager: runsv not running. You need to start the gitlab running service directory (runsvdir).
[root@gitlab gitlab]# systemctl start gitlab-runsvdir [root@gitlab gitlab]# gitlab-ctl start ok: run: alertmanager: (pid 5352) 7s ok: run: gitaly: (pid 5338) 7s ok: run: gitlab-exporter: (pid 5356) 7s ok: run: gitlab-kas: (pid 5350) 7s ok: run: gitlab-workhorse: (pid 5343) 7s ok: run: logrotate: (pid 5336) 7s ok: run: nginx: (pid 5339) 7s ok: run: node-exporter: (pid 5335) 7s ok: run: postgres-exporter: (pid 5353) 7s ok: run: postgresql: (pid 5351) 7s ok: run: prometheus: (pid 5329) 7s ok: run: puma: (pid 5341) 7s ok: run: redis: (pid 5337) 7s ok: run: redis-exporter: (pid 5328) 7s ok: run: sidekiq: (pid 5345) 7s
Check the status, it works again
[root@gitlab gitlab]# gitlab-ctl status run: alertmanager: (pid 5352) 53s; run: log: (pid 5347) 53s run: gitaly: (pid 5338) 53s; run: log: (pid 5332) 53s run: gitlab-exporter: (pid 5356) 53s; run: log: (pid 5348) 53s run: gitlab-kas: (pid 5350) 53s; run: log: (pid 5331) 53s run: gitlab-workhorse: (pid 5343) 53s; run: log: (pid 5342) 53s run: logrotate: (pid 5336) 53s; run: log: (pid 5325) 53s run: nginx: (pid 5339) 53s; run: log: (pid 5333) 53s run: node-exporter: (pid 5335) 53s; run: log: (pid 5334) 53s run: postgres-exporter: (pid 5353) 53s; run: log: (pid 5349) 53s run: postgresql: (pid 5351) 53s; run: log: (pid 5346) 53s run: prometheus: (pid 5329) 53s; run: log: (pid 5327) 53s run: puma: (pid 5341) 53s; run: log: (pid 5340) 53s run: redis: (pid 5337) 53s; run: log: (pid 5330) 53s run: redis-exporter: (pid 5328) 53s; run: log: (pid 5326) 53s run: sidekiq: (pid 5345) 53s; run: log: (pid 5344) 53s
I have a question: why the gitlab-ctl restart command cannot be started, but the systemctl start gitlab-runsvdir command can be started.
Start and stop management of Gitlab services
Start the service: gitlab-ctl start
Stop the service: gitlab-ctl stop
Restart the service: gitlab-ctl restart
View status: gitlab-ctl status
Gitlab supervisor method starts the service
Service start command: systemctl start gitlab-runsvdir.service
Service stop command: systemctl stop gitlab-runsvdir.service
Service restart command: systemctl restart gitlab-runsvdir.service
Service startup command: systemctl enable gitlab-runsvdir.service
Cancel the boot command: systemctl disable gitlab-runsvdir.service
Service view command: systemctl list-unit-files
View Gitlab service log:/usr/bin/gitlab-ctl tail #You can view the logs of all gitlab plug-ins
—-END LINE