[doc-healer] Healer improvement: artifact-constant check should verify the artifact is actually produced before flagging a docs [Content truncated due to length]

[github-actions]

What class of gap the check mis-handles

The Daily Documentation Healer's Step 2 artifact constant check currently says:

For each constant found, verify that the artifact name value is listed in docs/src/content/docs/reference/artifacts.md. If a constant is missing from the reference page, treat it as a documentation gap and add it.

This treats any ArtifactName = "..." constant that is absent from artifacts.md as a documentation gap. That heuristic produces a false positive for artifact constants that are declared ahead of their feature implementation.

Concrete false positive found this run

EvalsArtifactName = "evals" (pkg/constants/job_constants.go:103) is not listed in artifacts.md. A naive application of the rule would add an evals section. But the feature is not implemented:

// pkg/workflow/evals_job.go
// buildEvalsJob will build the evals job for BinEval-style evaluations.
// TODO: Implement evals job ...
// Returns nil (no-op) until the implementation is complete.
func (c *Compiler) buildEvalsJob(data *WorkflowData) (*Job, error) {
	return nil, nil
}

Because buildEvalsJob returns nil, no compiled workflow uploads an evals artifact (verified: no evals artifact upload in pkg/workflow/js/*.cjs or generated .lock.yml files). Documenting it now would describe a feature that does not exist — exactly the kind of speculative doc the healer is supposed to avoid.

Which step failed

Step 2, artifact constant check. The rule keys off the existence of a constant, not off whether the artifact is actually produced by a compiled workflow.

Proposed refinement

Before treating a missing artifact constant as a gap, confirm the artifact is really emitted. Concretely, amend the Step 2 wording to add a gate:

For each ArtifactName constant found, first confirm the artifact is actually produced before treating a missing artifacts.md entry as a gap. An artifact is "produced" if its name appears in an actions/upload-artifact step in a generated .github/workflows/*.lock.yml (or in the JS upload helpers under pkg/workflow/js/), and the constant's job builder is implemented (not a return nil no-op stub such as buildEvalsJob). If the constant exists but is not yet produced by any workflow, skip it — it is a forward-declared constant for an unimplemented feature, not a documentation gap.

This keeps the check catching real drift (a shipped artifact missing from the reference) while suppressing forward-declared constants like evals.

Note

This is a refinement to the healer's own logic. If EvalsArtifactName is intended to remain unused for a while, no docs action is needed until buildEvalsJob is implemented and a workflow uploads the evals artifact.

Related

• Constant: pkg/constants/job_constants.go:102-110
• Stub: pkg/workflow/evals_job.go
• Reference page: docs/src/content/docs/reference/artifacts.md

Generated by 📝 Daily Documentation Healer · 155.1 AIC · ⌖ 33 AIC · ⊞ 7.6K · ◷

• expires on Jul 10, 2026, 4:08 PM UTC-08:00