Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 41 additions & 41 deletions docs/user/reference/config/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,71 +302,71 @@ rpm-channel = "none"

## Source File References

The `[[components.<name>.source-files]]` array defines additional source files that azldev should download before building. These are files not available in the dist-git repository or lookaside cache — typically binaries, pre-built artifacts, or files from custom hosting.
The `[[components.<name>.source-files]]` array defines additional source files to fetch or generate before buildingbinaries, pre-built artifacts, or archives generated on-the-fly by a script.

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Filename | `filename` | string | **Yes** | Name of the file as it will appear in the sources directory |
| Hash | `hash` | string | Conditional | Expected hash of the downloaded file for integrity verification. Required for the `prep-sources` command unless `--allow-no-hashes` is used, in which case the hash is computed automatically from the downloaded file. |
| Hash type | `hash-type` | string | Conditional | Hash algorithm used (examples: `"SHA256"`, `"SHA512"`). Required when `hash` is specified. When omitted alongside `hash` for the `prep-sources` command and `--allow-no-hashes` is used, defaults to `"SHA512"`. |
| Origin | `origin` | [Origin](#origin) | **Yes** | Where to download the file from |
| Replace upstream | `replace-upstream` | bool | No | When `true`, intentionally replaces an entry with the same `filename` in the upstream dist-git `sources` file. The matching upstream entry **must exist**, otherwise source preparation fails (this guards against stale configs and filename typos). When `false` (default) a filename collision with an upstream entry is also an error. Requires `replace-reason`. |
| Replace reason | `replace-reason` | string | Conditional | Human-readable explanation for the replacement. **Required** when `replace-upstream = true`; **must be empty** otherwise. Captured in the `prep-sources` warning log so the override is auditable. |
| Filename | `filename` | string | **Yes** | Name of the file in the sources directory |
| Hash | `hash` | string | Conditional | Expected hash. Required unless `--allow-no-hashes` is passed to `prep-sources` (which computes and prints the hash). |
| Hash type | `hash-type` | string | Conditional | Hash algorithm (`"SHA256"`, `"SHA512"`). Required with `hash`; defaults to `"SHA512"` when auto-computed. |
| Origin | `origin` | [Origin](#origin) | **Yes** | How to obtain the file |
| Replace upstream | `replace-upstream` | bool | No | Replace the same-named entry in the upstream `sources` file. The upstream entry must exist. Requires `replace-reason`. |
| Replace reason | `replace-reason` | string | Conditional | Required when `replace-upstream = true`. Logged by `prep-sources` for auditability. |

### Origin

The `origin` field specifies how to obtain the source file.
Two origin types are supported.

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Type | `type` | string | **Yes** | Origin type. Currently only `"download"` is supported. |
| URI | `uri` | string | No | URI to download the file from (required when type is `"download"`) |
#### `"download"` — fetch from a URI

### Example
`origin = { type = "download", uri = "https://..." }`. The `uri` field is required.

```toml
[[components.shim.source-files]]
filename = "shimx64.efi"
hash = "7741013d9a24ce554bf6a9df6b776a57b114055e..."
filename = "shimx64.efi"
hash = "7741013d9a24ce554bf6a9df6b776a57b114055e..."
hash-type = "SHA512"
origin = { type = "download", uri = "https://example.com/repo/pkgs/shim/shimx64.efi/sha512/.../shimx64.efi" }
origin = { type = "download", uri = "https://example.com/repo/.../shimx64.efi" }
```

[[components.shim.source-files]]
filename = "shimaa64.efi"
hash = "57aa116d1c91a9ec36ab8b46c9164ae19af192b..."
#### `"custom"` — generate via a mock script

Use `origin.type = "custom"` when a source archive must be assembled or modified (e.g. stripping sensitive test fixtures from an upstream tarball). azldev runs the script inside a fresh mock chroot — the script **must write all output to `/azldev-gen/output/`**, which azldev packages into the archive named by `filename`. Network access is always enabled; the mock config comes from the project's default distro.

The `script` and `mock-packages` fields are nested under `[origin]`:

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Script | `origin.script` | string | **Yes** | Script filename (relative to the component's spec dir) to run in mock. Required for `origin.type = "custom"`. |
| Mock packages | `origin.mock-packages` | array of string | No | Extra RPM packages to install in the mock chroot before the script runs. |

On first use, omit `hash` and run `prep-sources --allow-no-hashes` to generate the archive and print its hash, then copy it into the TOML.

```toml
[[components.yara.source-files]]
filename = "yara-4.5.4-azl-stripped.tar.gz"
hash-type = "SHA512"
origin = { type = "download", uri = "https://example.com/repo/pkgs/shim/shimaa64.efi/sha512/.../shimaa64.efi" }
hash = "abc123..." # from: prep-sources --allow-no-hashes
origin.type = "custom"
origin.script = "gen-yara-stripped.sh" # relative to the component's spec directory
origin.mock-packages = ["cmake"] # omit if not needed
```

### Replacing an upstream `sources` entry

By default, declaring a `source-files` entry whose `filename` matches one already
listed in the upstream dist-git `sources` file is an **error** — this protects
against accidental shadowing of the upstream tarball.

To intentionally substitute a different artifact for an upstream entry (e.g. to
ship a locally-patched tarball), set `replace-upstream = true` together with a
non-empty `replace-reason`:
A `source-files` entry whose `filename` collides with an upstream `sources` entry is an error by default. Set `replace-upstream = true` (with a non-empty `replace-reason`) to intentionally substitute it:
Comment thread
Tonisal-byte marked this conversation as resolved.

```toml
[[components.example.source-files]]
filename = "example-1.0.tar.gz" # same filename as the upstream 'sources' entry
hash = "deadbeef..."
hash-type = "SHA512"
origin = { type = "download", uri = "https://internal.example.com/example-1.0-patched.tar.gz" }
filename = "example-1.0.tar.gz"
hash = "deadbeef..."
hash-type = "SHA512"
origin = { type = "download", uri = "https://internal.example.com/example-1.0-patched.tar.gz" }
replace-upstream = true
replace-reason = "patched to fix CVE-2026-0001 before upstream"
replace-reason = "patched to fix CVE-2026-0001 before upstream"
```

When `prep-sources` runs:

- The matching upstream entry is removed from the generated `sources` file and
the user-defined entry takes its place.
- A `WARN`-level event is logged citing the `replace-reason`, both upstream and
new hashes, so the override is auditable.
- If **no upstream entry** exists with that `filename`, `prep-sources` fails —
this is almost always a stale config or filename typo. Drop
`replace-upstream` (and `replace-reason`) if you intended a brand-new
`prep-sources` removes the matching upstream entry, inserts the new one in its place, and logs a `WARN` with both hashes and the reason. If no upstream entry with that filename exists, `prep-sources` fails — this is almost always a stale config or filename typo. Drop `replace-upstream` if you intended a brand-new
artifact instead.

`replace-upstream` and `replace-reason` are per-entry switches, not a
Expand Down
20 changes: 17 additions & 3 deletions internal/app/azldev/cmds/component/history_customizations.go
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,9 @@ func appendPackageItems(
}

// appendSourceFileItems emits one item per declared source-file reference,
// plus a distinct item for the high-signal ReplaceUpstream toggle (which
// actively masks a same-named upstream source and would otherwise be hidden
// behind the plain filename entry).
// plus distinct items for high-signal toggles: the ReplaceUpstream flag (which
// actively masks a same-named upstream source), the Script field (custom
// generation script), and MockPackages (chroot deps for that script).
func appendSourceFileItems(
items []CustomizationItem, sourceFiles []projectconfig.SourceFileReference,
) []CustomizationItem {
Expand All @@ -222,6 +222,20 @@ func appendSourceFileItems(
Value: sourceFile.Filename,
})
}

if sourceFile.Origin.Script != "" {
items = append(items, CustomizationItem{
Kind: "source-files.script",
Value: sourceFile.Origin.Script,
})
}

for _, pkg := range sourceFile.Origin.MockPackages {
items = append(items, CustomizationItem{
Kind: "source-files.mock-packages",
Value: pkg,
})
}
}

return items
Expand Down
9 changes: 9 additions & 0 deletions internal/app/azldev/cmds/component/history_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ func TestCustomizationCollectorsCoverEveryFingerprintableField(t *testing.T) {
reflect.TypeFor[projectconfig.ReleaseConfig](),
reflect.TypeFor[projectconfig.ComponentRenderConfig](),
reflect.TypeFor[projectconfig.SourceFileReference](),
reflect.TypeFor[projectconfig.Origin](),
}

// Maps "StructName.FieldName" -> short note describing how the field
Expand Down Expand Up @@ -133,10 +134,18 @@ func TestCustomizationCollectorsCoverEveryFingerprintableField(t *testing.T) {
// their own Kind. Hash/HashType are deliberately NOT emitted as output:
// the file's *presence* is the customization signal, and a checksum-only
// change is still caught by toml-commits / fingerprint-changes.
// Origin is walked as its own struct (see below); its Script and
// MockPackages fields are emitted when set (custom-origin source files).
"SourceFileReference.Filename": "source-files",
"SourceFileReference.Hash": "not emitted (checksum change caught via toml-commits/fingerprint)",
"SourceFileReference.HashType": "not emitted (ditto Hash)",
"SourceFileReference.ReplaceUpstream": "source-files.replace-upstream",
"SourceFileReference.Origin": "delegates to Origin walk",

// Origin -- Script and MockPackages are fingerprint-relevant; Type and Uri
// are tagged fingerprint:"-" (download location, not build input).
"Origin.Script": "source-files.script",
"Origin.MockPackages": "source-files.mock-packages",
}

actualFields := make(map[string]bool)
Expand Down
43 changes: 28 additions & 15 deletions internal/app/azldev/cmds/component/preparesources.go
Original file line number Diff line number Diff line change
Expand Up @@ -130,21 +130,7 @@ func PrepareComponentSources(env *azldev.Env, options *PrepareSourcesOptions) er
"synthetic history requires overlays to be applied")
}

var preparerOpts []sources.PreparerOption
if !options.WithoutGitRepo && !options.SkipOverlays {
preparerOpts = append(preparerOpts,
sources.WithGitRepo(env, env.LockReader(), distro.Version.ReleaseVer),
sources.WithDirtyDetection(),
)
}

if options.AllowNoHashes {
preparerOpts = append(preparerOpts, sources.WithAllowNoHashes())
}

if options.SkipSources {
preparerOpts = append(preparerOpts, sources.WithSkipLookaside())
}
preparerOpts := buildPreparerOptions(env, distro, options)

preparer, err := sources.NewPreparer(sourceManager, env.FS(), env, env, preparerOpts...)
if err != nil {
Expand All @@ -159,6 +145,33 @@ func PrepareComponentSources(env *azldev.Env, options *PrepareSourcesOptions) er
return nil
}

// buildPreparerOptions assembles [sources.PreparerOption] values based on the resolved
// distro and command-line options.
func buildPreparerOptions(
env *azldev.Env,
distro sourceproviders.ResolvedDistro,
options *PrepareSourcesOptions,
) []sources.PreparerOption {
var opts []sources.PreparerOption

if !options.WithoutGitRepo && !options.SkipOverlays {
opts = append(opts,
sources.WithGitRepo(env, env.LockReader(), distro.Version.ReleaseVer),
sources.WithDirtyDetection(),
)
}

if options.AllowNoHashes {
opts = append(opts, sources.WithAllowNoHashes())
}

if options.SkipSources {
opts = append(opts, sources.WithSkipLookaside())
}

return opts
}

// CheckOutputDir verifies the output directory state before source preparation.
// If the directory exists and is non-empty, it either removes it (when Force is set)
// or returns an actionable error suggesting --force.
Expand Down
45 changes: 42 additions & 3 deletions internal/projectconfig/component.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,43 @@ type OriginType string
const (
// OriginTypeURI indicates that the source file is fetched from a URI.
OriginTypeURI OriginType = "download"

// OriginTypeCustom indicates that the source file is generated by running a local
// shell script inside a mock chroot. The script is expected to populate a specific
// output directory; azldev then packages that directory into a deterministic archive.
OriginTypeCustom OriginType = "custom"
)
Comment thread
Tonisal-byte marked this conversation as resolved.

// Origin describes where a source file comes from and how to retrieve it.
// When omitted from a source file reference, the file will be resolved via the lookaside cache.
type Origin struct {
// Type indicates how the source file should be acquired.
Type OriginType `toml:"type" json:"type" jsonschema:"required,enum=download,title=Origin type,description=Type of origin for this source file"`
Type OriginType `toml:"type" json:"type" jsonschema:"required,enum=download,enum=custom,title=Origin type,description=Type of origin for this source file" fingerprint:"-"`
// Uri to download the source file from if origin type is 'download'. Ignored for other origin types.
Uri string `toml:"uri,omitempty" json:"uri,omitempty" jsonschema:"title=URI,description=URI to download the source file from if origin type is 'download',example=https://example.com/source.tar.gz"`
Uri string `toml:"uri,omitempty" json:"uri,omitempty" jsonschema:"title=URI,description=URI to download the source file from if origin type is 'download',example=https://example.com/source.tar.gz" fingerprint:"-"`

// Script is the filename of a shell script, relative to the component's spec directory,
// that is run inside a mock chroot to generate this source file.
// Required when [Origin.Type] is 'custom'; must be empty otherwise.
Script string `toml:"script,omitempty" json:"script,omitempty" jsonschema:"title=Script,description=Shell script filename (relative to the component spec directory) to run in mock to generate this source file. Required when origin type is 'custom'."`

// MockPackages is a list of RPM package names to install in the mock chroot before
// running [Origin.Script]. Only valid when [Origin.Type] is 'custom'.
MockPackages []string `toml:"mock-packages,omitempty" json:"mockPackages,omitempty" jsonschema:"title=Mock packages,description=RPM packages to install in the mock chroot before running the generation script. Only valid when origin type is 'custom'."`
}

// HashInclude implements the hashstructure [Includable] interface so that
// [Origin.Script] and [Origin.MockPackages] are omitted from the component
// fingerprint when they hold their zero values.
func (o Origin) HashInclude(field string, _ any) (bool, error) {
switch field {
case "Script":
return o.Script != "", nil
case "MockPackages":
return len(o.MockPackages) > 0, nil
}

return true, nil
}

// SourceFileReference encapsulates a reference to a specific source file artifact.
Expand All @@ -69,7 +97,7 @@ type SourceFileReference struct {
HashType fileutils.HashType `toml:"hash-type,omitempty" json:"hashType,omitempty" jsonschema:"enum=SHA256,enum=SHA512,title=Hash type,description=Hash algorithm used for the hash value"`

// Origin for this source file. When omitted, the file is resolved via the lookaside cache.
Origin Origin `toml:"origin,omitempty" json:"origin,omitempty" fingerprint:"-"`
Origin Origin `toml:"origin,omitempty" json:"origin,omitempty"`
Comment thread
Tonisal-byte marked this conversation as resolved.

// ReplaceUpstream, when true, intentionally replaces an upstream entry with the same
// [SourceFileReference.Filename] in the dist-git 'sources' file. The matching upstream
Expand All @@ -84,6 +112,17 @@ type SourceFileReference struct {
ReplaceReason string `toml:"replace-reason,omitempty" json:"replaceReason,omitempty" jsonschema:"title=Replace reason,description=Required when 'replace-upstream' is true. Human-readable explanation for the replacement." fingerprint:"-"`
}

// HashInclude implements the hashstructure [Includable] interface so that
// [SourceFileReference.Origin] is omitted from the component fingerprint when
// neither [Origin.Script] nor [Origin.MockPackages] are set.
func (r SourceFileReference) HashInclude(field string, _ any) (bool, error) {
if field == "Origin" {
return r.Origin.Script != "" || len(r.Origin.MockPackages) > 0, nil
}

return true, nil
}

// ComponentPublishConfig holds publish channel settings for a component's packages.
// The zero value means all channels are inherited from a higher-priority config layer.
type ComponentPublishConfig struct {
Expand Down
55 changes: 55 additions & 0 deletions internal/projectconfig/configfile.go
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,10 @@ func validateSourceFiles(sourceFiles []SourceFileReference, componentName string
return err
}

if err := validateCustomSourceRef(ref, componentName); err != nil {
return err
}

if err := validateOrigin(ref.Origin, ref.Filename, componentName); err != nil {
return err
}
Expand Down Expand Up @@ -223,6 +227,46 @@ func validateReplaceUpstream(ref SourceFileReference, componentName string) erro
return nil
}

// validateCustomSourceRef enforces the pairing rules for the 'script' and 'mock-packages'
// fields on a [SourceFileReference]:
// - 'script' is required when 'origin.type' is 'custom'.
// - 'script' must be empty when 'origin.type' is not 'custom'.
// - 'mock-packages' must be empty when 'origin.type' is not 'custom'.
func validateCustomSourceRef(ref SourceFileReference, componentName string) error {
if ref.Origin.Type == OriginTypeCustom {
if ref.Origin.Script == "" {
return fmt.Errorf(
"source file %#q in component %#q has 'custom' origin but no 'script'; "+
"a non-empty 'script' filename is required for 'custom' origin",
ref.Filename, componentName)
}

if err := fileutils.ValidateFilename(ref.Origin.Script); err != nil {
return fmt.Errorf(
"invalid 'script' value %#q for source file %#q in component %#q:\n%w",
ref.Origin.Script, ref.Filename, componentName, err)
}

return nil
}

if ref.Origin.Script != "" {
return fmt.Errorf(
"source file %#q in component %#q has 'script' set but origin type is %#q; "+
"'script' is only valid when origin type is 'custom'",
ref.Filename, componentName, string(ref.Origin.Type))
}

if len(ref.Origin.MockPackages) > 0 {
return fmt.Errorf(
"source file %#q in component %#q has 'mock-packages' set but origin type is %#q; "+
"'mock-packages' is only valid when origin type is 'custom'",
ref.Filename, componentName, string(ref.Origin.Type))
}

return nil
}

// validateOrigin checks that a source file [Origin] is present and valid for its type.
// For [OriginTypeURI] ('download'), the [Origin.Uri] field must be a valid URI with a scheme.
func validateOrigin(origin Origin, filename string, componentName string) error {
Expand Down Expand Up @@ -255,6 +299,17 @@ func validateOrigin(origin Origin, filename string, componentName string) error
"URI %#q is missing a scheme (e.g. 'https://')",
filename, componentName, origin.Uri)
}

case OriginTypeCustom:
// Script validation is handled by validateCustomSourceRef on SourceFileReference.
// Reject 'uri' since it is meaningless for custom-generated sources.
if origin.Uri != "" {
return fmt.Errorf(
"source file %#q in component %#q has 'uri' set but origin type is 'custom'; "+
"'uri' is only valid when origin type is 'download'",
filename, componentName)
}

default:
return fmt.Errorf(
"unsupported 'origin' type %#q for source file %#q, component %#q",
Expand Down
Loading
Loading