Skip to content

fix(job): check Artifact Registry and Docker credentials only for Crane builds - #6488

Merged
agrawalkhushi18 merged 1 commit into
GoogleCloudPlatform:developfrom
Neelabh94:fix/job-docker-creds-prebuilt-image
Oct 17, 2026
Merged

agrawalkhushi18 merged 1 commit into
GoogleCloudPlatform:developfrom
Neelabh94:fix/job-docker-creds-prebuilt-image

Conversation

@Neelabh94

Copy link
Copy Markdown
Contributor

Problem

gcluster job submit refused to start until the workstation had Docker credentials and the Artifact Registry API was enabled in the cluster's project, even when --image named an image that was already pushed. The Docker check was also keyed on the cluster's region: a cluster in us-west4 running an image from us-south1-docker.pkg.dev was told to run gcloud auth configure-docker us-west4-docker.pkg.dev,gcr.io.

Neither check has anything to do with a pre-built image, which the cluster nodes pull. They only matter when gcluster builds the image itself with Crane (--base-image + --build-context) and pushes it to <cluster-region>-docker.pkg.dev in the cluster's project (imagebuilder.GenerateImageName).

Changes

  • Build-only checks. ensurePrerequisites is now ensureBuildPrerequisites and runs only when --build-context is set. The basic checks (gcloud, auth, ADC, kubectl, gke-gcloud-auth-plugin) still run for every job from JobCmd.PersistentPreRunE.
  • Regional Artifact Registry host only. isDockerCredsConfigured no longer requires a gcr.io credential helper; a build never touches gcr.io. Crane authenticates through authn.DefaultKeychain, so the credHelpers["<region>-docker.pkg.dev"] = "gcloud" entry (or credsStore: gcloud) is the one that matters. The remediation names exactly that host.
  • No caching of the Docker check. The result was cached per project for 24h in ~/.gcluster/job_prereq_state.json, but the required host is per region: a build in us-central1 followed within 24h by a build in us-west4 skipped the check and failed at push time with a raw UNAUTHORIZED. Reading the Docker config is a local, sub-millisecond operation, so it is now evaluated on every build; only the network-backed Artifact Registry API lookup stays cached. docker_creds_configured is removed from the state file (older files still parse; unknown JSON fields are ignored).
  • Docs updated accordingly.

User-visible behaviour change

A user whose Docker config only has gcr.io configured and who runs a Crane build is now asked to configure <region>-docker.pkg.dev instead — the only host the push touches.

Testing

  • go test ./cmd/job/ ./pkg/imagebuilder/, all pre-commit hooks.
  • New/updated tests:
    • TestSubmitCmd_BuildPrerequisitesOnlyForCraneBuild: end-to-end through cobra — --image reaches the orchestrator without Docker credentials and without a gcloud services list call; --base-image/--build-context runs both checks and asks for us-west4-docker.pkg.dev only.
    • TestEnsureBuildPrerequisites_FreshState: with a fresh state, credentials for the cluster's region skip the checks; credentials for another region only, or none, do not.
    • TestEnsureBuildPrerequisites_MissingDockerCreds, TestIsDockerCredsConfigured updated for the regional-host semantics.
    • Tests use a verbatim recordingPrereqStore; the existing MockPrereqStore force-sets every flag once a timestamp exists, which would have hidden this regression.
  • Mutation-checked: reverting each of the three behaviours (call-site gating, gcr.io requirement, per-project cache) fails exactly the test written for it.

Submission Checklist

  • Fork your PR branch from the Toolkit "develop" branch (not main)
  • Test all changes with pre-commit in a local branch #
  • Confirm that "make tests" passes all tests
  • Add or modify unit tests to cover code changes
  • Ensure that unit test coverage remains above 80%
  • Update all applicable documentation
  • Follow Cluster Toolkit Contribution guidelines #
@Neelabh94 Neelabh94 added bug Something isn't working go Pull requests that update Go code release-bugfix Added to release notes under the "Bug fixes" heading. labels Oct 15, 2026
@Neelabh94 Neelabh94 self-assigned this Oct 15, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the prerequisite validation logic for 'gcluster job submit'. It ensures that build-specific checks, such as Artifact Registry API enablement and Docker credential configuration, are only executed when a build is actually triggered. By decoupling these checks from standard job submissions and making the Docker credential validation region-aware, the changes improve reliability and reduce friction for users deploying pre-built images.

Highlights

  • Build-only Prerequisite Gating: Artifact Registry and Docker credential checks are now gated to run only when a Crane build is performed (using --build-context), preventing unnecessary failures for jobs using pre-built images.
  • Regional Docker Credential Validation: Docker credential checks now specifically target the regional Artifact Registry host required for the build, removing the incorrect dependency on gcr.io.
  • Removal of Docker Check Caching: The Docker credential check is no longer cached in the state file, ensuring accurate validation across different regions and preventing stale authorization errors.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the prerequisite checks for gcluster job submit so that Docker credential and Artifact Registry API checks are only executed during Crane builds (when --build-context is specified). It simplifies the registry credential verification by checking the specific regional registry directly, removes the caching of this check since it is region-dependent, and updates the associated unit tests and documentation. I have no feedback to provide as the changes are clean, well-tested, and properly documented.

@Neelabh94
Neelabh94 force-pushed the fix/job-docker-creds-prebuilt-image branch from 45e90cf to 6751497 Compare October 15, 2026 04:54
@Neelabh94
Neelabh94 marked this pull request as ready for review October 15, 2026 04:54
@Neelabh94
Neelabh94 requested a review from a team as a code owner October 15, 2026 04:54
@Neelabh94
Neelabh94 force-pushed the fix/job-docker-creds-prebuilt-image branch from 6751497 to adeef83 Compare October 16, 2026 13:57
…ne builds

A pre-built --image is pulled by the cluster nodes, yet `gcluster job submit`
refused to start until the workstation had Docker credentials for the
cluster's region and the Artifact Registry API was enabled in the cluster's
project. Both only matter when gcluster builds the image itself with Crane
(--base-image/--build-context) and pushes it to
<cluster-region>-docker.pkg.dev in that project.

- Run the Artifact Registry API and Docker credential checks only for a
  Crane build (ensureBuildPrerequisites). The basic checks are unchanged.
- Require a gcloud credential helper for the regional Artifact Registry host
  only; a build never touches gcr.io. The remediation names that host.
- Read the Docker config on every build instead of caching the result per
  project for 24h: the registry depends on the cluster's region, so a result
  cached in one region let a build in another region fail at push time.
  docker_creds_configured is dropped from the state file; older files parse.
@Neelabh94
Neelabh94 force-pushed the fix/job-docker-creds-prebuilt-image branch from e19a264 to 86e5e61 Compare October 17, 2026 01:11
@agrawalkhushi18
agrawalkhushi18 enabled auto-merge (squash) October 17, 2026 06:01
@agrawalkhushi18
agrawalkhushi18 merged commit a05eddd into GoogleCloudPlatform:develop Oct 17, 2026
23 of 81 checks passed
@Neelabh94
Neelabh94 deleted the fix/job-docker-creds-prebuilt-image branch October 28, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working go Pull requests that update Go code release-bugfix Added to release notes under the "Bug fixes" heading.

4 participants