Add coreos-metadata, or don't

[cgwalters]

We dropped cloud-init.

Today we don't include coreos-metadata.

[arithx]

I'd be in favor of adding something similar to coreos-metadata. Having something extract metadata from disparate cloud metadata systems allows a common workflow on different clouds.

[sdemos]

really, all that is needed for coreos-metadata to work in an rhcos context is the parameterize the prefix of the attributes it emits so it can say something like RHCOS_<whatever> instead of COREOS_<whatever>. at least, that's the only thing off the top of my head, I would have to do a quick scan to make sure that's actually true.

At some point we discussed renaming coreos-metadata to the ignition agent. I would be in favor of doing that before we include it here, but I don't think it's really a blocker if we want to bikeshed that more.

[arithx]

And changing the output location.

[cgwalters]

What I find confusing is trying to explain the story of "Ignition is the only-run-once system provisioning path, except for this small subset of metadata keys which we do handle dynamically".

On the other hand...it seems clear that we're going to break a lot of workflows if we don't support the ssh-keys endpoint transparently.

The translation of the provider metadata...I see how it's useful, but OTOH it's also something one can easily grab from a containerized tool as well, right?

[cgwalters]

it seems clear that we're going to break a lot of workflows if we don't support the ssh-keys endpoint transparently.

My vote is we include for this reason alone. @csssuf mind if I move the systemd units from coreos-overlay into the repo so it's easier to package as an RPM? And add a basic Makefile that does an install?

really, all that is needed for coreos-metadata to work in an rhcos context is the parameterize the prefix of the attributes it emits so it can say something like RHCOS_ instead of COREOS_.

Is there value in renaming the variables? Particularly given we'll have both RHCOS and FCOS...(or I guess we could prefix with FCOS_ on both rhcos/fcos, but help me understand: is there a specific reason to rename or is it just "let's rename things since it's not quite the same OS?"

[cgwalters]

coreos/afterburn#90

[jlebon]

Can someone fill in the blanks and clarify why coreos-metadata exists at all vs. integrating this directly into Ignition? The README.md there isn't clear on this. We can already fetch e.g. user-data at that time, so supposedly we could fetch other endpoints too, no? Or are there details it makes available that can only be discovered post-pivot?

[arithx]

@jlebon: I believe it's because coreos-metadata runs on every boot and generates potentially changed metadata between boots.

Someone else please jump in if I'm incorrect or there's additional reasons.

[bgilbert]

The translation of the provider metadata...I see how it's useful, but OTOH it's also something one can easily grab from a containerized tool as well, right?

The usual use for coreos-metadata is to configure services running directly in the host.

Is there value in renaming the variables?

I'd love to rename coreos-metadata, /run/metadata/coreos, and the COREOS_* variables to not use the distro name at all.

Can someone fill in the blanks and clarify why coreos-metadata exists at all vs. integrating this directly into Ignition?

Ignition runs in the initramfs, which is not really the right place to be running routine metadata fetches.

[cgwalters]

The usual use for coreos-metadata is to configure services running directly in the host.

Like the embedded etcd for locksmith? But we're no longer going to be doing that one right? Are there any other cases?

@jlebon: I believe it's because coreos-metadata runs on every boot and generates potentially changed metadata between boots.

Right...in AWS for example if you change the ssh key for an instance after it's booted, that works with cloud-init. GCE is even more aggressive about having SSH keys for a "project" dynamically propagate across instances.

This is one of those "just how immutable is it" type things - if someone leaves the organization and I want to drop their SSH key, (or conversely, a new admin joins) - do I have to tear down my whole cluster node by node because Ignition only runs once?

Of course though, one could use a privileged Kube daemonset to do more dynamic ssh key handling after a cluster is set up...

But IMO we need something that ingests the ssh keys, and eh...yeah we could add it to Ignition but since we already have coreos-metadata...

[bgilbert]

Like the embedded etcd for locksmith? But we're no longer going to be doing that one right? Are there any other cases?

Well, we'll need something like locksmith. flannel is the other common one right now.

[cgwalters]

Moved to https://github.com/coreos/fedora-coreos-tracker/

[lucab]

Status summary for both RHCOS and FCOS.

• RHCOS
  • will carry coreos-metadata binary
  • will carry coreos-metadata.service systemd unit
  • will NOT carry coreos-metadata-sshkeys@.service systemd unit

This follows openshift-installer unified ssh-keys handling via ignition: openshift/installer#127.

• FCOS
  • will carry coreos-metadata binary
  • will carry coreos-metadata.service systemd unit
  • will carry coreos-metadata-sshkeys@.service systemd unit

This follows CL approach, and is tracked at coreos/fedora-coreos-tracker#4.