[Neelabh94]

Summary

Prevents gcluster job submit from re-applying manifests.yaml over an existing or partial JobSet installation when the webhook temporarily has 0 ready endpoints (for example during a controller pod restart or node preemption), and guards the fresh-install path against leftover or concurrently created controller/Service objects in jobset-system.

Changes Included

• 4-State JobSet Classification & Fast Path (pkg/orchestrator/gke/infra_manager.go):

  • Classifies cluster JobSet state using CRD jobsets.jobset.x-k8s.io and Service jobset-system/jobset-webhook-service (checkAndInstallJobSetCRD, checkResourceExists). Treats 403 Forbidden as existing so restricted users on shared clusters never trigger an install.
  • Healthy cluster fast path (verifyOrWaitForJobSetWithCRD): When the JobSet CRD is confirmed present, checks EndpointSlice readiness of jobset-system/jobset-webhook-service first so healthy clusters require only 2 kubectl reads (CRD + EndpointSlice) and skip the separate Service existence read.
  • Existing installation (crdExists && serviceExists): Never re-applies or modifies an existing JobSet installation. Polls EndpointSlice readiness (waitForJobSetWebhookEndpoints / pollJobSetWebhookEndpoints) for up to 5 minutes (jobSetWebhookWaitTimeout, 3s poll interval, 100-attempt cap) using --request-timeout=10s per read and logging periodic progress every 30s.
  • Partial or ambiguous installation (partialJobSetInstallError): When only one of the CRD or webhook Service exists (or one returns 403 Forbidden while the other is NotFound), fails fast without modifying the cluster and prints --context-pinned investigation and recovery commands (kubectl delete namespace jobset-system, Helm reinstall / gcluster deploy, or kubectl apply --server-side --force-conflicts with version guidance).

• Fresh-Install Guards & Per-Context Manifests (pkg/orchestrator/gke/infra_manager.go):

  • Cluster-scoped RBAC preflight (installJobSetOnFreshCluster, missingPermissions): Verifies create permissions on namespaces, customresourcedefinitions.apiextensions.k8s.io, clusterroles.rbac.authorization.k8s.io, clusterrolebindings.rbac.authorization.k8s.io, mutatingwebhookconfigurations.admissionregistration.k8s.io, and validatingwebhookconfigurations.admissionregistration.k8s.io via kubectl auth can-i <verb> <resource> --all-namespaces before attempting installation.
  • Pre-apply live object check (checkNoLiveJobSetObjects): Queries jobset-system via g.kubeClient.GetResource for all four selector-bearing JobSet objects (Deployment/jobset-controller-manager, Deployment/jobset-controller, Service/jobset-webhook-service, and Service/jobset-controller-manager-metrics-service) and requires every object to return NotFound before downloading, writing, or applying manifests. Aborts fail-closed with remediation instructions if any object already exists or if any read/client initialization fails.
  • Context-scoped manifest files (jobSetManifestFile, writeGeneratedManifest, applyManifestFile): Writes cleaned JobSet manifests to ~/.gcluster/generated/jobset-<context>.yaml (or jobset.yaml when context is empty) so runs against other clusters do not overwrite the manifest file referenced in recovery instructions if kubectl apply or rollout waiting fails.

• Webhook Timeout & Controller Pod Diagnostics (pkg/orchestrator/gke/infra_manager.go):

  • Distinguishes persistent endpoint read failures (!everRead) from an unready webhook (jobSetWebhookTimeoutError).
  • Inspects Service/jobset-webhook-service .spec.selector and pods in jobset-system (diagnoseJobSetWebhook, describeJobSetPods, podProblem, waitingContainerProblem), reporting selector match counts, pod phases/labels, the first waiting container in InitContainerStatuses or ContainerStatuses (Reason, container name, image, message), or PodScheduled=False (Unschedulable) reasons, plus --context-pinned kubectl diagnostic commands.

• Shared EndpointSlice Readiness & Kueue Integration (pkg/orchestrator/gke/types.go, pkg/orchestrator/gke/kueue.go):

  • Updates k8sEndpointSliceList.HasReadyEndpoint() in types.go so Conditions.Ready is a *bool, treating nil (omitted) as ready alongside true when Addresses is non-empty per the discovery.k8s.io/v1 EndpointConditions specification.
  • Adds serviceGVR (v1/services) and permissionCheck to types.go.
  • Updates checkKueueInstallPermissions in kueue.go to use g.missingPermissions (--all-namespaces) and updates checkKueueEndpoints to reuse g.checkServiceEndpointsReady("kueue-system", "kueue-webhook-service") when useEndpointSlice is enabled.

• Context-Pinned kubectl Command Formatting & Docs (pkg/orchestrator/gke/kube_auth.go, docs/gcluster_job_guide.md):

  • Adds kubectlCommandLine(kubeContext, args...) in kube_auth.go to render copy-pasteable kubectl commands pinned with --context and quoting whitespace-containing arguments.
  • Updates docs/gcluster_job_guide.md to document the JobSet readiness wait and non-destructive install behavior during gcluster job submit.

Testing

• Unit tests in pkg/orchestrator/gke/infra_manager_test.go, pkg/orchestrator/gke/kube_auth_test.go, and pkg/orchestrator/gke/kueue_test.go (KUBECONFIG=/dev/null go test -race -count=1 ./pkg/orchestrator/gke/...) covering:
  • 4-state JobSet CRD + webhook Service classification and 2-read healthy fast path
  • Webhook EndpointSlice polling, Ready == nil handling, timeout diagnostics, and InitContainerStatuses / ContainerStatuses / Unschedulable reporting
  • checkNoLiveJobSetObjects across all four controller Deployment and Service resources (present, 403 Forbidden, timeout, transport error, dynamic client init error)
  • --all-namespaces RBAC permission checks for both JobSet and Kueue
  • Per-context generated manifest filenames and kubectlCommandLine argument quoting
• Verified end-to-end on a local kind cluster across fresh admin/non-admin clusters, v0.10.1 controller pod restarts, Helm 0.12.0 scaled-to-zero deployments, and partial uninstall states.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the reliability of the JobSet installation process within the GKE orchestrator. By implementing a wait-for-ready strategy instead of re-applying manifests, it prevents potential conflicts and service disruptions caused by re-installing over existing deployments. The changes include improved diagnostic reporting for partial installations, better handling of RBAC permissions, and more robust endpoint readiness checks, ensuring a smoother experience when managing JobSet on GKE clusters.

Highlights

• JobSet Installation Logic: Replaced the blind re-application of JobSet manifests with a robust wait-and-verify mechanism that respects existing installations.
• Partial Installation Recovery: Added diagnostic logic to detect partial or ambiguous JobSet states and provide users with actionable recovery commands.
• RBAC Verification: Introduced cluster-scoped permission checks to verify required privileges before attempting to install JobSet.
• Endpoint Readiness: Improved EndpointSlice readiness detection to handle transient webhook unavailability during controller restarts.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly refactors and hardens the JobSet installation and verification process in the GKE orchestrator. It introduces robust checks to prevent modifying existing or partial JobSet installations, implements detailed webhook endpoint polling with a 5-minute timeout, and adds comprehensive diagnostic reporting (including pod status and container waiting reasons) when timeouts occur. Additionally, it consolidates permission and endpoint checks across both JobSet and Kueue, and introduces a helper to format copy-pasteable kubectl commands. Feedback on the changes suggests extending the podProblem diagnostic helper to also inspect p.Status.InitContainerStatuses so that failures in init containers are not missed.

[agrawalkhushi18]

LGTM