Restoring My Software Forge

A log of all that I needed to do to restore/migrate my forgejo instance onto a freshly-installed debian VPS.

I recently needed to completely reinstall the VPS I rent from OVH; it would never progress past loading the initramfs and I never managed to figure out why, even when rebooting into OVH's "Rescue Mode" through their web console.

What follows is a recount of the different actions I took to have my old data show up in my "new" forgejo instance. Namely, the (few) user accounts and (many) git repositories that my instance managed.

1. Backup the forge's data

Following the upgrade guide, I ran the forgejo dump command. This produced a 1.3 GiB zip file:

noah@my-computer$ file forgejo-dump-1765276696.zip
forgejo-dump-1765276696.zip: Zip archive data, at least v2.0 to extract, compression method=store

My instance is configured to use Sqlite for it's database needs (the default), so I didn't need to do anything else to have everything backed up.

2. Set up (rootless) podman with systemd

This is where I had the most to learn. As I mentioned in the introduction, my VPS had been running Arch Linux, and I had installed forgejo via the system package manager (pacman). This process had created a forgejo user, set up the necessary folders, and a forgejo.service file that allowed me to simply run systemctl start forgejo to have it working.

Not only is the server now running Debian, but I wanted to have forgejo running inside a container for various reasons (including decoupling the forgejo version I use/run from the version proposed by the system package manager (in this case, apt)).

Following forgejo's "Installation with Docker" page, along with reading podman's online docs and Redhat's "How to run pods as systemd services with Podman" blog post got me most of the way to my goal.

Forgejo's "Installation from binary" page was useful for creating the git user account on my VPS exactly how forgejo expects it to be. It also provided some insight into what the arch linux package had done.

Lastly, the following issues on the forgejo repositories hosted at codeberg were illuminating:

Keeping docs up-to-date, clear, and meaningful is hard - even for awesome projects like forgejo (especially for volunteer-run projects like forgejo!).

Finally, I had the following 3 files, placed at/in /etc/containers/systemd/users:

forgejo.container

[Container]
ContainerName=forgejo
Environment=USER_UID=103 #git
Environment=USER_GID=104 #git
Image=codeberg.org/forgejo/forgejo:13-rootless
Network=forgejo.network
PublishPort=3000:3000
PublishPort=2222:2222
Volume=forgejo-data:/var/lib/gitea
Volume=/etc/localtime:/etc/localtime:ro

[Service]
Restart=always
RestartSec=3

[Install]
WantedBy=default.target

forgejo.network

[Network]
NetworkName=forgejo

forgejo-data.volume

[Volume]
VolumeName=forgejo-data

Then, I could run systemctl --user daemon-reload to have sytemd generate the proper .service files it would need to run the container as a service. Finally, I could run systemctl --user start forgejo to be greeted with forgejo's installation/setup page at <my-server-host>:3000/!

I could also check that the podman volume has been created:

debian@hostname$ podman volume ls
DRIVER  VOLUME NAME
local   forgejo-data

3. Restore the data

Although I had a lot less to learn than in the previous part, this was maybe the scariest part of the process. There were so many little things to be done that I started to think it wouldn't be possible to recover the data through using the actual dump, and that I'd need to recreate and re-import each one of my repos.

a. unzip dump.zip

To not lose track of things, I created a new folder to unzip into:

debian@hostname:~$ mkdir forgejo-restore
debian@hostname:~$ unzip forgejo-dump-1765276696.zip -d forgejo-restore
...
debian@hostname:~$ ls forgejo-restore
app.ini data forgejo-db.sql repos

b. Tweak app.ini

I ended up adjusting the generated app.ini config file instead of attempting to bring over the previous one. The most important things were copying over the value for the JWT_SECRET in the [oauth2] section, and the INTERNAL_TOKEN value in the [security] section. I also set the INSTALL_LOCK value to true in the [security] section.

c. Rename the database file

The new app.ini config file now expects the database file name to be different:

[database]
PATH = /var/lib/gitea/data/gitea.db

So I renamed the database file:

debian@hostname:forgejo-restore$ mv data/forgejo.db data/gitea.db

d. Place the data folder into the podman data volume

I needed to run podman unshare beforehand to gain access to the volume's contents.

debian@hostname:~$ cd ~/.local/share/containers/storage/volumes/forgejo-data/
debian@hostname:forgejo-data$ ls -lah
total 12K
drwx------  3 debian debian 4.0K Jan 18 .
drwx------  4 debian debian 4.0K Jan 18 ..
drwxr-xr-x 15 100999 100999 4.0K Jan 18 _data
debian@hostname:forgejo-data$ podman unshare
root@hostname:forgejo-data$ ls -lah
drwx------  3 root   root   4.0K Jan 18 .
drwx------  4 root   root   4.0K Jan 18 ..
drwxr-xr-x 15 debian debian 4.0K Jan 18 _data
root@hostname:forgejo-data$ cp -R /home/debian/forgejo-restore/data/* ./_data/
root@hostname:forgejo-data$ chown -R debian:debian ./_data/
root@hostname:forgejo-data$ exit
debian@hostname:forgejo-data$

f. Place the git repository data into the podman data volume

root@hostname:forgejo-data$ cp -R /home/debian/forgejo-restore/repos ./_data/git/repositories

Note that the target directory is the one specified in the config file for forgejo, in my case:

[repository]
ROOT = /var/lib/gitea/git/repositories

g. Update the git hooks on the server

When I tried to push a new commit to a repo on my instance, I got the following error:

noah@my-computer:some-cloned-repo-of-mine$ git push
Enumerating objects: ..., done.
Counting objects: ..., done.
Compressing objects: ..., done.
Total ..., reused ... (delta ...), pack-reused ... (from ...)
remote: ./hooks/pre-receive.d/gitea: line 3: /usr/bin/forgejo: No such file or directory
To https://<instance-domain>/<username>/<repo-name>.git
 ! [remote rejected] main -> main (pre-recieve hook declined)
 error: failed to push some refs to 'https://<instance-domain>/<username>/<repo-name>.git'

I found the following hooks present, server-side, in each repository managed by forgejo:

They all take roughly the same shape. In each of them, I needed to make the following change:

- /usr/bin/forgejo     hook --config=/etc/forgejo/app.ini               <hook-name> <args>
+ /usr/local/bin/gitea hook --config=/var/lib/gitea/custom/conf/app.ini <hook-name> <args>

This can be accomplished with 2 separate sed commands:

sed -i 's/bin\/forgejo/local\/bin\/gitea/' <hook-file>

and

sed -i 's/etc\/forgejo/var\/lib\/gitea\/custom\/conf/' <hook-file>

I combined them with find into a one-liner (still inside the shell launched by podman unshare):

root@hostname:forgejo-data$ find . \
    -type f -name gitea -o -name proc-receive \
    -exec sed -i 's/bin\/forgejo/local\/bin\/gitea/' {} \; \
    -exec sed -i 's/etc\/forgejo/var\/lib\/gitea\/custom\/conf/' {} \;

The next day, while looking for a way to cleanup some old Forgejo Actions logs, I discovered the forgejo admin regenerate hooks [sub-]command in the cli docs. Maybe that does the trick instead of the bash oneliner above, but I would want to look at the source code for this command to be sure.

4. TODO:

a. Upgrade to Forgejo v14

The Forgejo team recently released version 14, along with several minor updates since:

Just for the bugfixes, it would be good to upgrade to the latest version.

b. Try again, from a fresh start

There was a lot of back-and-forth involved just to get to this point, leaving a decent amount of clutter. For example, I initialy copied the data folder from the backup to the wrong level in the data volume's hierarchy. This means there currently are some extra folders cluttering up the data volume. For this to serve as a good reference for future reinstalls, I would like to try again from scratch/a fresh data volume. In theory, the following plan should work out:

  1. introduce a new podman data volume:
    1. create a new-forgejo-data.volume file
    2. change forgejo.container to use it instead
    3. handoff to systemd by running systemctl --user reload-daemon
  2. generate a "proper" install of forgejo: go through the forgejo install process using the web interface (shouldn't be much more than navigating to :3000/, scrolling down, and clicking Install.
  3. Inspect and document the locations of generated folders inside the new podman volume.

Then, I should be able to place the data from my previous install exactly where it needs to be, in exactly as few steps as necessary. We'll see.

This would be a great time/place to try out the forgejo admin regenerate hooks CLI command I mentioned previously.

c. Restore the CI/CD runner

I was running the forgejo-runner docker image on my desktop (a much more powerful machine than the cheap VPS that hosts my forgejo instance). This was really nice for auto-building projects whenever I pushed some work to the instance's repo. In theory I should be able to simply re-read the installation guide to check that it all works, but as this instance migration/re-installation shows, in practice things are rarely as simple as the theory would suggest.