1# Shale migration
2
3Shale is the sole Git server on Zenith. Personal Forgejo is retired. Retained Forgejo repositories and database exports remain migration sources; copying the existing Shale directory does not import repositories that only existed in Forgejo. The legacy multi-server SSH router is not part of the target setup.
4
5The 2026-10-04 personal inventory found 12 Shale owned repositories, no Shale mirrors, and 19 retained Forgejo bare repositories: 18 under `clo` and one under `nix`. Forgejo's retained LFS directory contains no files. Complete both imports before Shale's first production activation.
6
7The production data migration on 2026-10-04 completed before activating either production Shale or Postgres. The retained Shale copy passed checksum and SQLite checks, then all 19 Forgejo histories imported with verified object-file hashes and ref mappings. The result contains 21 repository records. All 11 private Forgejo sources retain private access; existing Shale identities, tokens, and sessions survive. The original session secret is preserved in the Nomad variable and its private recovery copy. Production import evidence is `/var/lib/studio/shale-forgejo-import-ldfl9zuy/repositories.json`; the earlier service dataset state is `globe/prod/shale@before-shale-import-1791160403-42496`.
8
9After copying the retained Shale state into `/srv/prod/shale`, [import-forgejo-to-shale.py](import-forgejo-to-shale.py) imports the additional histories while Shale stays stopped. Its metadata JSON comes only from the separately restored **personal** Forgejo PostgreSQL database:
10
11```sql
12SELECT coalesce(json_agg(x), '[]') FROM (
13 SELECT r.id, r.owner_id, u.lower_name AS owner_name,
14 r.lower_name, r.name, r.is_private, r.default_branch,
15 r.is_empty, r.is_mirror
16 FROM repository r JOIN "user" u ON u.id = r.owner_id
17 ORDER BY u.lower_name, r.lower_name
18) x;
19```
20
21```sh
22python3 tools/import-forgejo-to-shale.py \
23 --target /srv/prod/shale \
24 --metadata /run/personal-forgejo-repositories.json
25```
26
27The importer preserves existing Shale identities and repository records. Clover's repositories retain their names; the other namespace becomes `nix/config`. It copies and hashes Git objects without copying Forgejo hooks or configuration. Missing branches, tags, pull refs, and notes retain their names. A divergent branch or tag is retained under `refs/heads/forgejo/<owner>/...` or `refs/tags/forgejo/<owner>/...`; other conflicting refs use `refs/forgejo/<owner>/...`. Existing Shale refs retain their values, including newer histories already migrated from old mirrors. Former Forgejo mirrors become owned histories, with their original mirror status recorded in the private evidence. That directory contains every original ref, its destination, all copied object hashes, and the previous SQLite database.
28
29New repository access follows verified Forgejo visibility, with pushes restricted to the owner and issue submission disabled. When private Forgejo history joins an existing repository, public or unlisted permissions are tightened to private **before objects are copied**; existing `off` permissions remain off. An import failure must leave Shale stopped. Restoring only the previous database can re-expose imported private objects through its old public permissions, so recovery must preserve the tightened access or restore the complete service dataset.
30
31The existing `snow` account's original OIDC provider and subject must survive the cutover. Keep the canonical `auth.paperclover.net/realms/master` issuer. Shale caches repository metadata and access at startup, so finish SQLite changes before starting the application. An isolated pinned-image test proved direct registration of a new repository: private anonymous web access returned 404 and Git discovery returned 401; after public permissions and a restart, its page and Git advertisement returned 200 with the original branch object ID. The test used copied data, `--network none`, and no published ports.
32
33A full-data test imported all 19 retained Forgejo histories into a disposable copy of the existing Shale state, yielding 21 repository records. Its conservative fixture metadata marked every incoming repository private. Every copied object file and every source ref passed verification. The pinned application then denied anonymous access to a new private repository and a formerly public collision, while preserving an unaffected public repository. On that isolated copy, public test permissions proved HTTP pages and Git advertisements for `bgds`, `home-infra`, and `nix/config`; `home-infra` advertised the original Forgejo `main` and `vllm`. An actual smart HTTP fetch returned a 53-object pack with a verified pack checksum. Production visibility must come from the restored Forgejo metadata, not this test fixture.
34
35After migration and authenticated browser checks, push the new infra `main` to `https://shale.paperclover.net/home-infra.git` with a Shale personal access token. The existing `home-infra` record and UUID are retained; the imported Forgejo branches remain available alongside the tested final `main`.
36
37The October 4 transport check inspected the image pinned in [service.pkl](../service/shale/service.pkl) in disposable containers without mounting real app data. Its embedded Git endpoint and account settings use HTTP and personal access tokens; no SSH listener, authorized-key interface, or forced-command handler was found. The [official installation](https://astheno.software/shale/installation/) and [configuration reference](https://astheno.software/shale/reference/environment/) also expose HTTP serving and OAuth login without SSH configuration. A `git` account must either use a Shale-aware SSH bridge or await native SSH support. Direct filesystem Git commands would bypass Shale's authorization.
38
39Zenith's Shale app directory contains a small SQLite database and 419 MB of owned repositories. `bash tools/import-shale.sh shale-preview-4eea0e3b` copied `data`, `repositories_owned`, and `repositories_mirrors` opaquely from the read-only `storage1/apps@hourly-2026-09-26_05-00` snapshot. It verified checksums and SQLite integrity, then restarted the preview. Both sides had 11 top-level owned repository directories; the preview had one healthy Nomad allocation and returned HTTPS 200. Repository contents were not inspected.
40
41For the production copy, stop Zenith's Shale container and the Snow Globe Shale job, set `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` for the new host, then run `bash tools/import-shale.sh shale`. The importer checks both jobs remain stopped, snapshots the destination dataset, verifies all three copied directories, and leaves Snow Globe stopped. Start the new job after the copy, check the SQLite state and a known login through the new Keycloak client, then switch the public route. The destination snapshot printed by the importer remains available for recovery.
42
43After a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory as well. The importer then reads the retained Shale directory from the new host's mounted old apps dataset, with no old Docker dependency.