fix: convert one-time jobs to Helm hooks to prevent ArgoCD OutOfSync

[pinkneyj]

Why

When the following Jobs complete and are deleted, ArgoCD marks the application as OutOfSync because it expects them to exist as tracked resources:

• cloudzero-backfill
• cloudzero-confload (config-loader-job.yaml)
• cloudzero-helmless
• cloudzero-init-cert

Additionally, the current approach of embedding a checksum/config in job names to trigger re-runs on config changes creates resource churn and leaves stale Job objects in the cluster.

What

Convert the one-time Jobs to Helm hooks by adding the following annotations:

"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-delete-policy": before-hook-creation

This means:

• Helm runs the jobs before install/upgrade but does not track them as managed resources, so ArgoCD no longer flags their deletion as OutOfSync
• before-hook-creation deletes the previous job run before creating a new one, replacing the need for a checksum/config-based name

The CronJob in backfill-job.yaml is left unchanged as it manages its own lifecycle.

Note on previous hook rejection

A comment in backfill-job.yaml notes that Helm hooks were previously rejected due to incompatibility with helm template, ArgoCD, and Flux. This is addressed as follows:

• ArgoCD: Helm hooks are supported when the ArgoCD server is configured with --enable-helm-hooks
• helm template: Hooks are excluded from helm template output by design. Users relying on helm template | kubectl apply will need to handle these jobs separately or use helm install/upgrade directly
• Flux: Flux's Helm controller supports hooks natively from v0.19+

Files changed

• charts/cloudzero-agent/templates/backfill-job.yaml — Job only (not CronJob)
• charts/cloudzero-agent/templates/config-loader-job.yaml
• charts/cloudzero-agent/templates/helmless-job.yaml
• charts/cloudzero-agent/templates/init-cert-job.yaml

How Tested

Validated the following against a live cluster:

• Deleted the hook jobs post-deployment and confirmed ArgoCD remained Synced with no OutOfSync errors
• Triggered a helm upgrade and confirmed hook jobs were recreated as expected via the before-hook-creation delete policy
• Confirmed the cloudzero-backfill CronJob was untouched and continued to run on its configured schedule

[greptile-apps]

Greptile Summary

This PR converts four one-time Kubernetes Jobs to stable-name Helm pre-install and pre-upgrade hooks to avoid ArgoCD tracking completed Jobs.

• Removes configuration checksums from one-time Job names.
• Adds before-hook-creation cleanup to the backfill, config-loader, helmless, and certificate initialization Jobs.
• Leaves the scheduled backfill CronJob as an ordinary managed resource.
• The selected hook phases run before resources required by these Jobs are created or updated.

Confidence Score: 4/5

This PR should not merge until the hook ordering is changed so fresh installations have the required resources and upgrades process the new revision's configuration.

The new pre-install hooks depend on ordinary resources that do not yet exist, while the pre-upgrade config-loader consumes ordinary ConfigMaps before their new revision is applied.

Files Needing Attention: helm/templates/config-loader-job.yaml, helm/templates/backfill-job.yaml, helm/templates/helmless-job.yaml, helm/templates/init-cert-job.yaml

Important Files Changed

Filename	Overview
helm/templates/_helpers.tpl	Removes checksum suffixes from one-time Job names, making hook resource names stable across revisions.
helm/templates/backfill-job.yaml	Converts the immediate backfill Job to a pre-install/pre-upgrade hook despite its dependencies remaining ordinary resources; the CronJob remains unchanged.
helm/templates/config-loader-job.yaml	Converts config-loader to a pre-hook, causing fresh-install dependency failures and processing of previous-revision ConfigMaps during upgrades.
helm/templates/helmless-job.yaml	Converts the Job to a pre-hook although its required ConfigMap is not created until ordinary manifests are applied.
helm/templates/init-cert-job.yaml	Converts certificate initialization to a pre-hook while its service account and RBAC remain ordinary release resources.

Prompt To Fix All With AI

### Issue 1
helm/templates/config-loader-job.yaml:12-13
**Pre-install hooks lack dependencies**

On a fresh installation, Helm runs these pre-install Jobs before creating their ordinary ConfigMaps, service accounts, Secrets, and RBAC resources. The config-loader and helmless hooks are always rendered, so their pods cannot start and the installation blocks before the required release resources are created.

### Issue 2
helm/templates/config-loader-job.yaml:12-13
**Pre-upgrade hook reads stale configuration**

When an upgrade changes values represented in the agent, validator, helmless, webhook, or aggregator ConfigMaps, this pre-upgrade hook runs before Helm applies the new ordinary manifests. It therefore loads and validates the previous revision's configuration, causing the deployed revision and the configuration sent to CloudZero to disagree until a later successful run.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "remove checksum naming" | Re-trigger Greptile}

[greptile-apps]

Pre-install hooks lack dependencies

On a fresh installation, Helm runs these pre-install Jobs before creating their ordinary ConfigMaps, service accounts, Secrets, and RBAC resources. The config-loader and helmless hooks are always rendered, so their pods cannot start and the installation blocks before the required release resources are created.

Prompt To Fix With AI

This is a comment left during a code review.
Path: helm/templates/config-loader-job.yaml
Line: 12-13

Comment:
**Pre-install hooks lack dependencies**

On a fresh installation, Helm runs these pre-install Jobs before creating their ordinary ConfigMaps, service accounts, Secrets, and RBAC resources. The config-loader and helmless hooks are always rendered, so their pods cannot start and the installation blocks before the required release resources are created.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Pre-upgrade hook reads stale configuration

When an upgrade changes values represented in the agent, validator, helmless, webhook, or aggregator ConfigMaps, this pre-upgrade hook runs before Helm applies the new ordinary manifests. It therefore loads and validates the previous revision's configuration, causing the deployed revision and the configuration sent to CloudZero to disagree until a later successful run.

Prompt To Fix With AI

This is a comment left during a code review.
Path: helm/templates/config-loader-job.yaml
Line: 12-13

Comment:
**Pre-upgrade hook reads stale configuration**

When an upgrade changes values represented in the agent, validator, helmless, webhook, or aggregator ConfigMaps, this pre-upgrade hook runs before Helm applies the new ordinary manifests. It therefore loads and validates the previous revision's configuration, causing the deployed revision and the configuration sent to CloudZero to disagree until a later successful run.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[dmepham]

Hi @pinkneyj , thank you so much for the contribution! We really appreciate you taking the time to do so, and it goes a long way in improving the usability and function of the chart.

Regarding your change; this is certainly solving a legitimate problem. However, while we want this chart to work well with standard deployment tools, we try not to design the basic functions of the chart around features from a particular deployment tool. Even though ArgoCD handles helm hooks by mapping them to their own lifecycle annotations, we cannot be sure that all users will have a deployment strategy that does so. ArgoCD itself ignored helm hooks in previous versions.

Some other thoughts/questions:

1. Is there some other mechanism that is deleting Jobs? By default, the Jobs in this chart do not have a TTL set, and so a completed Jobs sits in completed state and would not normally trigger OutOfSync. Is it possible to exclude the completed Jobs from whatever cleanup you have running?
2. The Jobs support custom annotations. I am wondering if using the ArgoCD Resource Hooks in these custom annotations would solve your problem? Or even the helm hooks that you've added here, for that matter. The important distinction there is that custom annotations are owned and maintained by the user, based on their own deployment ecosystem. This is in contrast to annotations that are owned and maintained by the chart itself, which cannot know about the deployment ecosystem.
3. Regarding the config-hash clutter problem - I am unable to reproduce this behavior. The helm release diff removes the old config-hash Jobs in my testing. Can you show me how I might reproduce this issue?

Thank you again for contributing. This is a helpful change, and I want to make sure we get the benefits while also keeping the chart flexible. We might be able to get to the same place with more instructive documentation.