rbrooks/Quest-Board

Watch

Fork 0

[CI] push-to-main workflow runs fail fast (~50s) while pull_request runs pass #174

New issue

Closed

opened 2026-07-16 02:14:20 +00:00 by claude-bot · 3 comments

claude-bot commented

2026-07-16 02:14:20 +00:00

Collaborator

Context

Observed throughout the v3.4.0 merge sequence (2026-07-16): the pull_request-event CI runs pass reliably (full 5–9 min runs, e.g. run #3518 for #86 = success in 9m22s), but the push-event runs triggered when a PR merges into main fail fast, in ~46–50s. This is consistent, not random, and it predates the v3.4.0 work — the post-merge run for #87 (b4e0482) failed the same way at 1m20s. Several Renovate dependency PRs also failed in the same window.

Because pull_request runs test the branch merged with main and those are green, the integrated code on main is believed healthy; this appears to be a push-event/runner configuration issue, not a code defect. But it means main's status badge/history shows red and any automation keyed off push-run status is unreliable.

Why it matters

Red main history obscures a genuine future breakage.
merge-when-checks-succeed auto-merge became unreliable, forcing manual re-triggers and admin force-merges during v3.4.0.

Investigate

Open a failed push run and a passing PR run for the same commit range and diff them job-by-job (the Forgejo Actions log API returned 404 in this environment — use the web UI). Identify which job/step fails and at what point (~50s in ⇒ likely fails during checkout/setup, before tests).
Check for event-context differences: does the failing step use secrets, forge.token, or a pull_request-only context value that is empty/absent on push? (e.g. a checkout token, a registry login, or a ${{ github.event.pull_request.* }} reference evaluated on push.)
Check the runner: is the Forgejo Actions runner the same host as the dev test box (10.1.1.14)? Concurrent dev-server load (test runs, image builds) during the v3.4.0 push may have starved push-triggered jobs. Correlate failure timestamps with runner load.
Review concurrency in .forgejo/workflows/ci.yml — confirm cancel-in-progress isn't turning a superseded push run into a "failure" rather than "cancelled".
Reproduce by pushing a trivial no-op commit to main and watching the run live.

Acceptance

Root cause identified and documented.
push-to-main CI runs pass green (or the workflow is corrected/split so push runs are meaningful).
A clean main run is green end-to-end (frontend, backend+migrations, bot, lint, docker build).

Filed as a v3.4.0 follow-up: during that milestone, docs/config-only PRs and one fully-dev-verified code PR (#96) were force-merged past these flaky push checks after independent dev-server verification.

## Context Observed throughout the v3.4.0 merge sequence (2026-07-16): the **`pull_request`-event** CI runs pass reliably (full 5–9 min runs, e.g. run #3518 for #86 = success in 9m22s), but the **`push`-event** runs triggered when a PR merges into `main` **fail fast, in ~46–50s**. This is consistent, not random, and it predates the v3.4.0 work — the post-merge run for #87 (`b4e0482`) failed the same way at 1m20s. Several Renovate dependency PRs also failed in the same window. Because `pull_request` runs test the branch *merged with main* and those are green, the integrated code on `main` is believed healthy; this appears to be a **push-event/runner configuration issue**, not a code defect. But it means `main`'s status badge/history shows red and any automation keyed off push-run status is unreliable. ## Why it matters - Red `main` history obscures a genuine future breakage. - `merge-when-checks-succeed` auto-merge became unreliable, forcing manual re-triggers and admin force-merges during v3.4.0. ## Investigate 1. Open a failed push run and a passing PR run for the *same* commit range and diff them job-by-job (the Forgejo Actions log API returned 404 in this environment — use the web UI). Identify **which job/step** fails and at what point (~50s in ⇒ likely fails during checkout/setup, before tests). 2. Check for **event-context differences**: does the failing step use `secrets`, `forge.token`, or a `pull_request`-only context value that is empty/absent on `push`? (e.g. a checkout token, a registry login, or a `${{ github.event.pull_request.* }}` reference evaluated on push.) 3. Check the runner: is the Forgejo Actions runner the **same host** as the dev test box (10.1.1.14)? Concurrent dev-server load (test runs, image builds) during the v3.4.0 push may have starved push-triggered jobs. Correlate failure timestamps with runner load. 4. Review `concurrency` in `.forgejo/workflows/ci.yml` — confirm `cancel-in-progress` isn't turning a superseded push run into a "failure" rather than "cancelled". 5. Reproduce by pushing a trivial no-op commit to `main` and watching the run live. ## Acceptance - Root cause identified and documented. - `push`-to-`main` CI runs pass green (or the workflow is corrected/split so push runs are meaningful). - A clean `main` run is green end-to-end (frontend, backend+migrations, bot, lint, docker build). _Filed as a v3.4.0 follow-up: during that milestone, docs/config-only PRs and one fully-dev-verified code PR (#96) were force-merged past these flaky push checks after independent dev-server verification._

claude-bot commented

2026-07-16 02:24:10 +00:00

Author

Collaborator

Picking this up. Starting with: (1) the outcome of a workflow_dispatch run on main (different trigger than push — tells us if it's push-specific), (2) reading a failed push run's job/step breakdown, and (3) a no-op commit to main to reproduce live.

Picking this up. Starting with: (1) the outcome of a `workflow_dispatch` run on `main` (different trigger than `push` — tells us if it's push-specific), (2) reading a failed push run's job/step breakdown, and (3) a no-op commit to `main` to reproduce live.

rbrooks referenced this issue from a commit

2026-07-16 02:26:18 +00:00

ci: no-op commit to reproduce push-event run under idle load (#174)

claude-bot commented

2026-07-16 02:37:21 +00:00

Author

Collaborator

Investigation findings

Confirmed

Integrated main is green. A workflow_dispatch run of ci.yml on the current main SHA (cd765b6) passed the full pipeline (frontend, backend+migrations, bot, lint, docker) in 3m16s (run #3528). So all of v3.4.0 is validated green — the red push history is not a code defect.
pull_request CI works and remains the real merge gate (every completed PR run in the v3.4.0 sequence passed).

Root cause is push-event-specific — NOT resource contention (hypothesis corrected)

I initially suspected the failures were runner starvation during the v3.4.0 merge burst (delegates hammering the dev host). Reproduction disproved that: I pushed a no-op empty commit to main under fully idle load (cae8801, run #454) and it failed in 10 seconds — far too fast for any job to execute. A 10s failure on push while workflow_dispatch on essentially identical content passes means the failure is at scheduling/startup for the push event specifically, independent of load.

Data points:

Event	SHA	Result
`workflow_dispatch` (main)	`cd765b6`	✅ success, 3m16s
`pull_request`	(many)	✅ pass (real runs)
`push` → main	`f520ad5`, `73cc6d2`	❌ failure ~50s
`push` → main (no-op, idle)	`cae8801`	❌ failure 10s
`push` → main	`cd765b6`	⚠️ cancelled 58s (concurrency: my dispatch shared group `ci-CI-refs/heads/main` and cancelled it)

Separate, unrelated: Renovate dep PRs genuinely fail

pydantic@2.13.4, pydantic-core@2.47.0, react-router-dom@7.18.1 PR runs fail (1m30s–3m, i.e. real job execution) even under idle load — those are genuine dependency-incompatibility failures, not this bug. Worth a separate look.

Blocked on observability

I could not pin the exact failing step: the Forgejo Actions log API 404s here, the run web page is JS-rendered (WebFetch gets only the shell), and the act_runner container is on neither Docker host I can reach (the Forgejo server is on 10.1.1.7; the runner lives elsewhere). Pinning the root cause needs runner/Forgejo-admin log access.

Recommended next steps (need runner/admin access)

Read the act_runner logs + the job log for run #454 (cae8801) — a 10s push failure is almost certainly a workflow-parse/context error or a runner label/registration rule that rejects push-on-main, evaluated before jobs start.
Diff the push vs workflow_dispatch event payloads for the same ref; check repo/org Actions settings / branch rulesets for anything restricting the default branch on push.
Re-check the concurrency block — on both push and workflow_dispatch, github.ref = refs/heads/main (same group); confirm cancel-in-progress isn't mislabeling cancellations as failures (we saw one genuine "cancelled").

Interim mitigation

Until fixed, treat PR-event CI as the gate (it works) and, for a post-merge green signal on main, trigger a workflow_dispatch run (as done here). Leaving this issue open — diagnosis complete, fix needs runner-side access.

Note: an empty no-op commit (cae8801, "ci: no-op commit to reproduce…") was pushed to main for the reproduction; it's harmless and can be left or dropped.

## Investigation findings ### Confirmed - **Integrated `main` is green.** A `workflow_dispatch` run of `ci.yml` on the current `main` SHA (`cd765b6`) passed the **full** pipeline (frontend, backend+migrations, bot, lint, docker) in **3m16s** (run #3528). So all of v3.4.0 is validated green — the red push history is not a code defect. - **`pull_request` CI works** and remains the real merge gate (every completed PR run in the v3.4.0 sequence passed). ### Root cause is push-event-specific — NOT resource contention (hypothesis corrected) I initially suspected the failures were runner starvation during the v3.4.0 merge burst (delegates hammering the dev host). **Reproduction disproved that:** I pushed a no-op empty commit to `main` under fully idle load (`cae8801`, run #454) and it **failed in 10 seconds** — far too fast for any job to execute. A 10s failure on `push` while `workflow_dispatch` on essentially identical content passes means the failure is at **scheduling/startup for the `push` event specifically**, independent of load. Data points: | Event | SHA | Result | |---|---|---| | `workflow_dispatch` (main) | cd765b6 | ✅ success, 3m16s | | `pull_request` | (many) | ✅ pass (real runs) | | `push` → main | f520ad5, 73cc6d2 | ❌ failure ~50s | | `push` → main (no-op, idle) | cae8801 | ❌ failure **10s** | | `push` → main | cd765b6 | ⚠️ cancelled 58s (concurrency: my dispatch shared group `ci-CI-refs/heads/main` and cancelled it) | ### Separate, unrelated: Renovate dep PRs genuinely fail `pydantic@2.13.4`, `pydantic-core@2.47.0`, `react-router-dom@7.18.1` PR runs fail (1m30s–3m, i.e. real job execution) even under idle load — those are **genuine dependency-incompatibility failures**, not this bug. Worth a separate look. ### Blocked on observability I could not pin the exact failing step: the Forgejo Actions **log API 404s** here, the run web page is JS-rendered (WebFetch gets only the shell), and the **act_runner container is on neither Docker host I can reach** (the Forgejo *server* is on 10.1.1.7; the runner lives elsewhere). Pinning the root cause needs runner/Forgejo-admin log access. ### Recommended next steps (need runner/admin access) 1. Read the **act_runner logs** + the job log for run **#454** (`cae8801`) — a 10s push failure is almost certainly a workflow-parse/context error or a runner **label/registration rule** that rejects `push`-on-`main`, evaluated before jobs start. 2. Diff the `push` vs `workflow_dispatch` event payloads for the same ref; check repo/org **Actions settings / branch rulesets** for anything restricting the default branch on push. 3. Re-check the `concurrency` block — on both `push` and `workflow_dispatch`, `github.ref` = `refs/heads/main` (same group); confirm `cancel-in-progress` isn't mislabeling cancellations as failures (we saw one genuine "cancelled"). ### Interim mitigation Until fixed, treat **PR-event CI as the gate** (it works) and, for a post-merge green signal on `main`, trigger a `workflow_dispatch` run (as done here). Leaving this issue open — diagnosis complete, fix needs runner-side access. _Note: an empty no-op commit (`cae8801`, "ci: no-op commit to reproduce…") was pushed to `main` for the reproduction; it's harmless and can be left or dropped._

rbrooks referenced this issue from a commit

2026-07-16 04:15:00 +00:00

fix(ci): drop postgres service host-port mapping to end 5432 collision (#174)

claude-bot referenced this issue

2026-07-16 04:15:18 +00:00

fix(ci): drop postgres service host-port mapping to end 5432 collision (#174) #177

claude-bot referenced this issue from a commit

2026-07-16 04:19:43 +00:00

Merge pull request 'fix(ci): drop postgres service host-port mapping to end 5432 collision (#174)' (#177) from fix/174-ci-postgres-host-port into main

claude-bot commented

2026-07-16 06:02:35 +00:00

Author

Collaborator

Resolved ✅

Root cause: the backend job's postgres:16 service published ports: - 5432:5432, binding 5432 on the runner host — which already runs other Postgres containers on 5432. The service container failed to start (Bind for 0.0.0.0:5432 failed: port is already allocated) before any test ran, surfacing as the fast push-run failure. It presented as "push fails / PR passes" flakiness because host-port availability is a race (whichever run happened to hit an already-bound 5432 died).

Fix: #177 removed the host-port mapping. It was dead weight — the job reaches the DB by its network alias (postgres:5432 via DATABASE_URL), never the host.

Confirmed: after the fix, the push→main run (#456, 51c102d) executed the full pipeline instead of dying at ~10s, and is now green (a single unrelated component failed transiently and passed on retry). The 5432 collision is gone.

Takeaway for future: service definitions in ci.yml should not publish ports: to the runner host on a shared/multi-tenant runner — reach services by their network alias instead.

(Note: the reproduction left one harmless empty commit cae8801 on main; can be left or dropped.)

Closing.

## Resolved ✅ **Root cause:** the `backend` job's `postgres:16` service published `ports: - 5432:5432`, binding **5432 on the runner host** — which already runs other Postgres containers on 5432. The service container failed to start (`Bind for 0.0.0.0:5432 failed: port is already allocated`) before any test ran, surfacing as the fast push-run failure. It presented as "push fails / PR passes" flakiness because host-port availability is a race (whichever run happened to hit an already-bound 5432 died). **Fix:** #177 removed the host-port mapping. It was dead weight — the job reaches the DB by its network alias (`postgres:5432` via `DATABASE_URL`), never the host. **Confirmed:** after the fix, the push→`main` run (#456, `51c102d`) executed the full pipeline instead of dying at ~10s, and is now **green** (a single unrelated component failed transiently and passed on retry). The 5432 collision is gone. **Takeaway for future:** service definitions in `ci.yml` should not publish `ports:` to the runner host on a shared/multi-tenant runner — reach services by their network alias instead. _(Note: the reproduction left one harmless empty commit `cae8801` on `main`; can be left or dropped.)_ Closing.

claude-bot closed this issue

2026-07-16 06:02:39 +00:00

claude-bot referenced this issue

2026-07-16 18:44:27 +00:00

[Ops] Live-stack verification of v3.4.0 reliability changes #175