fix(portainer): repair same-node Git routing with recoverable network migration

This commit is contained in:
archipelago
2026-09-30 09:57:25 -04:00
parent 6ac26f637c
commit eda28c4cd6
16 changed files with 856 additions and 193 deletions
+12
View File
@@ -290,3 +290,15 @@ app:
Validate with `scripts/validate-app-manifest.sh` and regenerate the catalog
with `scripts/generate-app-catalog.py` (drift-checked in CI by
`scripts/check-app-catalog-drift.py`).
### Persistent-state backup for network migrations
`app.backup_on_network_change: true` opts an app into a stopped-state snapshot
before an explicitly selected rootless network mode is migrated. The orchestrator
archives writable persistent bind mounts under the node data directory, collapses
nested mounts, excludes the runtime Podman socket, and preserves the previous
Quadlet definition for rollback. Named volumes, outside-data-root state and
symlinked mount roots fail closed rather than silently producing an incomplete
backup. A failed snapshot resumes the original service and leaves migration
pending. Private archives are retained under `migration-backups/`; fresh installs
and unchanged network configurations do not create migration snapshots.
+104
View File
@@ -0,0 +1,104 @@
# Same-node Gitea sources in Portainer
Status: root cause reproduced and network repair verified in disposable Portainer
instances; final migration integration and release acceptance remain in progress.
This change belongs to the next signed catalog, OTA and ISO. It does not modify
published 1.8.21 artifacts.
## Confirmed cause
On the affected X250, Gitea 1.27.3 and Portainer 2.45.0 run in rootless Podman
5.4.2, managed by user Quadlet services. Gitea publishes HTTP on loopback and the
Archipelago app gate serves its public port. Gitea's public ROOT_URL already
matches that gate URL.
Portainer had no explicit network selection and Podman selected pasta. Its
network namespace contained the host's LAN address. A Git request to that same
LAN address therefore reached Portainer's namespace rather than the host gate:
connection refused before authentication. The exact smart-HTTP request from the
host returned 200 with `application/x-git-upload-pack-advertisement`. From
Portainer's actual namespace the LAN request was refused, while its host mapping
returned a Git advertisement and the expected branch tip. Direct container-IP
requests timed out. Container health and host-only HTTP checks missed the defect.
A disposable Portainer using `slirp4netns` successfully created a Source through
Portainer's own API, using the original LAN clone URL. Returning that fixture to
pasta reproduced the refusal; recreating with slirp repaired it while preserving
its account and saved Source. Restart also passed. The requested branch tip and
Compose file were read from that actual Portainer network namespace. No user
stack was deployed. Deployment addresses and repository details are kept outside
this public record.
## Source changes
- Declare Portainer's rootless `slirp4netns` mode in its manifest. No shared static
container IP, host networking, all-interface backend publication or auth bypass.
- Keep Gitea's loopback HTTP backend and gate port; machine Git uses Gitea's
authentication. Remove obsolete port-3000 nginx metadata/template and the old
best-effort installer commands which silently rewrote app.ini and falsely
claimed success. Gitea owns first-run setup and operator configuration.
- Existing Quadlet reconciliation applies Network= drift. Record a durable
pending restart before updating the unit and clear it only after a successful
restart, so failed reloads/restarts and management interruptions retry.
- Detect explicit rootless network-mode drift in the older Podman runtime too.
Unspecified networks do not trigger inferred changes to unrelated apps.
- Portainer opts into `backup_on_network_change`. Before recreation, gracefully
stop the app and archive its writable persistent bind mounts, including nested
Compose state, once each. Runtime sockets are excluded. Save the previous
Quadlet definition, where present. Archives live under the node data directory's
private `migration-backups/<id>/` directory; state is never deleted. Backup
failures resume the original service and fail the migration visibly.
- Keep Podman API and Quadlet bind/network behavior covered by actual-manifest
tests. Docker remains a development fallback: it now preserves bind/protocol
declarations and rejects Podman-only networking instead of silently changing it.
## Operator use and diagnostics
Use Gitea's advertised HTTP(S) clone URL in Portainer Sources, with the Gitea
username and token in the credential fields. On first-run Gitea setup, the public
base URL must match the origin opened through Archipelago (including its port).
Keep a deliberately configured HTTPS/domain origin when one exists. Do not use a
container IP or put a token into the URL. A private repository requires repository
read permission. A successful Source check fetches Git refs; it does not deploy
a stack or establish that a Compose build uses a desired application revision.
`scripts/check-portainer-git-source.py` calls Portainer's own read-only Source
connection test. Supply a private mode-600 JSON credential file containing
`api_key` or `jwt`, and optionally `git: {username, password}`. Pass
`--portainer-url`, `--repository-url` and `--credentials-file`. It does not create
Sources or stacks and prints no credentials or raw server errors. It distinguishes
Portainer login/API failures from Git connection refusal, timeout, DNS/TLS
failure, HTML/login interception and repository authentication failure. TLS
verification stays enabled and API redirects are refused.
## Upgrade and rollback
The signed catalog embeds manifests and overrides installed disk copies. A disk
edit alone cannot deliver this fix. Publish the matching catalog with the tested
runtime, then verify the generated unit, actual network mode and Source API.
Expect a Portainer interruption while the snapshot and recreation run; duration
depends on its saved state size.
Gitea does not need recreation or an app.ini rewrite for this repair.
Keep the previous trusted catalog/runtime for rollback. Restore that catalog
before restoring the saved `previous.container`, reloading user systemd and
starting Portainer; otherwise reconciliation will correctly reapply the new
manifest. The archive is a stopped-state emergency backup, not an instruction to
roll back a live database automatically. Restore it only with Portainer stopped
and after preserving any newer state. Do not replace Gitea data/config, keys,
repositories or the production Portainer database with disposable test data.
## Validation and remaining gates
- Disposable X250 Portainer Source API: old mode refuses; repaired mode succeeds;
saved account/Source survive recreation; restart succeeds.
- Invalid Git credentials produce a repository-authentication error, distinct
from TCP refusal. Requested branch and Compose file read from Portainer context.
- Final expanded backend suite: 1,575 passed, zero failed, four existing ignored
tests, including stopped-state archive round trips and failure preservation. Container runtime suite: 78 passed.
Five diagnostic regression tests passed. Combined tests with the merged
paid-download PRs remain pending.
- Still required before release: live automatic migration with the new runtime,
snapshot/rollback verification, private-repository and install-order acceptance,
lifecycle/reboot convergence, and signed-catalog delivery to the existing app.
Record LFS/registry/SSH/browser checks and actual hardware/runtime coverage.