Recover GitLab after PostgreSQL 9.6.2 failed upgrade
Recently I described, how to keep in sync with GitLab latest releases on the Raspberry Pi. And of course it lead me to a problem after unsuccessful update to version 8.17-rc3. This release has the PostgreSQL 9.6.2 embedded and apparently everything got broken in the middle of the upgrade process. After all my tries to revert to previous GitLab version, lots of time wasted (yes, each call to apt-get upgrade or install specified GitLab version was taking several hours!), I still stayed with totally unresponsive instance.
Then again I came back to basics.
I rebooted the device
sudo shutdown –r now
Installed latest GitLab-CE
sudo apt-get upgrade
Asked it to revert PostgreSQL to previous version
sudo gitlab-ctl revert-pg-upgrade
At this stage, it was showing some strange errors about inability to connect to psql service via TCP/IP, that was pretty strange. So I checked the status.
sudo gitlab-ctl status
run: gitlab-workhorse: (pid 3099) 7362s; run: log: (pid 604) 1487541047s
run: logrotate: (pid 11405) 160s; run: log: (pid 602) 1487541047s
run: nginx: (pid 1917) 7686s; run: log: (pid 608) 1487541047s
**down: postgresql: 1s**
run: redis: (pid 1930) 7685s; run: log: (pid 603) 1487541047s
run: sidekiq: (pid 3079) 7364s; run: log: (pid 607) 1487541047s
run: unicorn: (pid 3257) 7282s; run: log: (pid 606) 1487541047s
And then logs!
sudo gitlab-ctl tail postgresql
And it was continuously repeating an error, while parsing line 210 of the configuration file, what was all the time stopping the service.
sudo vi /var/opt/gitlab/postgresql/data/postgresql.con
Since it was about replication, I commented it out and service started normally.
Then manually upgraded PostgreSQL
sudo gitlab-ctl pg-upgrade
This time all went OK.