Skip to content

Delete stale primary HPAs when autoscalerRef is removed - #1965

Open
nedal87 wants to merge 1 commit into
fluxcd:mainfrom
nedal87:fix/stale-primary-hpa-cleanup
Open

Delete stale primary HPAs when autoscalerRef is removed#1965
nedal87 wants to merge 1 commit into
fluxcd:mainfrom
nedal87:fix/stale-primary-hpa-cleanup

Conversation

@nedal87

@nedal87 nedal87 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Problem

Removing spec.autoscalerRef from a Canary leaves the generated primary HPA
behind. It continues scaling the primary Deployment after autoscaling was
intended to be disabled.

Fixes #1654. Related: #259.

Solution

When autoscalerRef is absent, Flagger searches the Canary namespace for a
stale primary HPA and deletes it only when all of these checks pass:

  • the Canary has a UID;
  • the HPA name ends in -primary;
  • it targets <targetRef.name>-primary;
  • its controller owner reference matches the exact Canary UID.

No HPA is deleted based on its name alone. This addresses the ownership-safety
concern raised in #1030.

Successful cleanup is memoized per Canary UID, so Flagger performs one
namespace-scoped HPA list after startup or after autoscalerRef is removed,
rather than on every analysis tick. Failures are reported and retried without
blocking normal Canary reconciliation.

A pure Canary UpdateFunc cleanup would miss references removed while Flagger
was unavailable and HPAs that were already stale before upgrading.

No RBAC changes are required; the upstream chart and kustomize manifests
already grant Flagger HPA list and delete permissions. This change is limited
to autoscaling/v2 HorizontalPodAutoscalers.

Testing

Added tests for:

  • cleanup after removing autoscalerRef;
  • cleanup of an already-stale HPA after controller startup;
  • preservation of unmanaged and differently owned HPAs;
  • LIST and DELETE failure retries;
  • disable, re-enable, and disable lifecycle;
  • primary workload health.

Validated with:

  • go test ./...
  • race tests for pkg/canary and pkg/controller
  • go vet and go build ./...
  • live kind tests covering cleanup, re-enablement, ownership safety, and
    Canary deletion garbage collection
Delete Flagger-managed primary HPAs after autoscalerRef is removed.

Require the Canary controller owner and expected primary target.

Keep discovery failures from blocking Canary reconciliation.

Signed-off-by: Nedal Eskaf <nedaleskaif87@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant