formats.kdl: init (usage ergonomics)

[sodiboo]

Like #412029, but different somehow? Idk, this isn't necessarily separate, it doesn't "supersede" that PR, but i implemented some code changes in my latest comment, and i'd like to get feedback on it, so it's nice to have a separate code review thing.
(i guess it does supersede it now. and closes #198655)

Mainly, in this comment, i wrote a bunch about what i thought could be a more ergonomic way to define a KDL document. In this PR, i've also updated the documentation for how to use the KDL format with my changes. However, it is notably quite long! Previously, i was quite adamant that this KDL format belongs in pkgs.formats, but because the documentation i wrote is relatively much longer and i think that the KDL format in particular kinda needs a usage example, it probably no longer belongs in the same section as all the other formats, but i do think that documentation does belong somewhere. So, i'm open for suggestions for alternate namespaces like nezia first suggested here.

Scope

The commit formatting is currently a bit "unpolished": before this is ever merged (if it is merged), they should all be squashed into a formats.kdl: init (or whatever other namespace we decide on). The initial commit that gets merged does not need all the history of our discussions.

The particular way i've implemented some helper types (uniqFlat{List,Attrs}Of) by placing them in the format is probably not idiomatic? Like, i recognize this needs to change? But i'm not sure how exactly.

The main thing i want feedback on first is not the specifics of how i've written my library code, but rather, do we think the way i suggest the KDL format be used, is reasonable? At least, for the purposes of niri which is the program i maintain a Nix module for, this is exactly what i need and want. Please comment on how using this is like when writing a configuration in the style of the four configs mentioned at the end of this comment. Please comment on the documentation i've written, if it's fundamentally wrong, or my explanations are weird. Please comment on all the user-facing aspects of this.

I also realize (like i've said in the other PR) that long-term, there needs to be a better "extensible" KDL settings-like format. Something where you can define rules for how sections are merged, and then define the config in multiple locations and have them merged in an application-specific way. I recognize that this should exist. This PR isn't that. This PR is a foundation for that by making the KDL document format representable in Nix. That's all. But i want to make that representation ergonomic to write.

Call for module authors

Also, are there any other modules that currently exist outside of Nixpkgs with a KDL configuration? I genuinely don't know! And I don't know what those applications need. I'm approaching this mainly from the perspective of niri (and the needs of my niri-flake), so the "style" of flattening the document and having optional nodes is taken from my existing libraries in that repo. I think what i've put together here is sufficiently simple and generic that it's easy to grasp, and useful to most module authors.

namespacing

Once the KDL formats are merged and available in stable Nixpkgs, i plan to make use of them in my niri-flake. What i want to do, then, is to have my settings module be a standalone thing to import, which you can place manually into your config. It will have a "main" output like .rendered which should have a type of "KDL document". I don't really want to have to thread through a pkgs instance to make use of the KDL type. This is a primary reason why i'd love to have it in a different namespace.

My settings module shouldn't be inherently dependent on any packages: it's just Nix library code, and you're (eventually) supposed to plug in the .rendered part from my comprehensive settings mapping, into the config = part of the upstream modules (which exist in home-manager? perhaps?). But i do want to make use of the KDL type from Nixpkgs; i don't want to have to reimplement the "splatting" logic in niri-flake just to avoid importing pkgs (although i am currently reimplementing it, so?? i can just keep that? it's not a Huge Problem)

recommended usage patterns

As i said above, i think it's actually really important to have a usage example. Mainly, i want to encourage module authors making use of this format to not name their options settings. That should be reserved for an "extensible kdl"-like format, with multi-definition merging.

But what should we call the options? I think for niri, it would make sense to just have a single config option, for the entire configuration. Like, the upstream modules (nixpkgs, home-manager, hjem-rum, etc), really don't need to do much: for instance, niri will already read the systemd Locale1 settings to choose the keyboard layout, so the existing keyboard options should just apply without any configuration.

For more complex situations, where a single "config" is not viable, i think rendered is a good name for the full config (or for any submodule of it, that can render a "partial" config as a KDL document). That name comes from the fact that it should only be assigned once, in a place where you "render" your Nix options into a single KDL document.

The distinction here is that config is for "user-defined" configuration inputs, and rendered is for "output" configurations from a Nix module. It's also easy to go from config to rendered by making a rename config->extraConfig.

If you have multiple large sections of config, which are mapped to Nix submodules, each dealing with their own bit, it makes sense to render those subsections independently.

So, for instance, a potential way niri-flake could look in the future is to have a top-level rendered output like so:

{
  rendered = [
    cfg.input.rendered
    cfg.layout.rendered

    (map (cfg: cfg.rendered) cfg.spawn-at-startup)

    cfg.debug.rendered
    cfg.extraConfig
  ];
}

"splat"

The above example is also making use of my "splat" functionality, which is maybe not the best name for it? It sounds like it's a language feature, when it's not. But idk what else to call it. In particular, this is not just more ergonomic than having to ++ different parts of the config, but it is also more lazy (as mentioned in my last comment on the other PR). mkMerge would also be equally as lazy, but its usage is indistinguishable from defining the config in multiple places, which is a footgun when working with raw KDL documents, because implicit concatenation across files is almost never what you want.

The splatting could perhaps be made explicit? I think my code would look uglier then, but i could see use for something like a lib.mkSplat that can "splat" multiple list elements, at a single element definition. Currently, list merging allows elements to have lib.mkIf and lib.mkMerge and if they resolve to no definitions, the element is deleted; otherwise, the definitions are merged as that elemType instead of becoming multiple list elements. But in KDL documents, we want a more "splat"-like behaviour; which is why i implemented that (and not in terms of lib.mkMerge, because it has different semantics). In regular listOf, obviously such a splat would have to be explicit with a lib.mkSplat, and if this existed already, i would just use it in my implementation; but since it doesn't, i made it implicit, which works fine here because "KDL document in KDL document" can only really mean you want to splat.

Is the splatting actually a good idea? I think it will make using the KDL format a lot nicer. I will make extensive use of the splatting in my code. Or whatever we call it.

---

cc @KiaraGrouwstra @nezia1 (the two others who are currently participating in discussion in the other PR)

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and others READMEs.

---

Add a 👍 reaction to pull requests you find important.

[KiaraGrouwstra]

nice work! these are angles i definitely had not taken into account at #412029.

it maybe does feel unfortunate aspects like extent of laziness aren't as easy to test.

on others having worked on kdl from nix and might be able to provide feedback, i can mostly find home-manager's toKDL function @h7x4 wrote for its zellij module - a function further worked on by @GaetanLepage and @quentinmit.

[sodiboo]

it maybe does feel unfortunate aspects like extent of laziness aren't as easy to test.

Basically, the only "laziness" that matters here is that the top-level document list on an option should not be strict in any other values. That is, it should not depend on any conditionals or such. This is important, because no matter what we do, the module system is always strict in option definitions (because they are always checked for _type). In practice, this means that the definition value should be just a list literal expression (which is not strict in the value of anything) to avoid "infinite recursion" issues.

All other parts of the document don't really need to be "lazy"; we're passing it directly into toJSON anyways, which does a deep eval in one go, so it doesn't matter.

[KiaraGrouwstra]

for what it's worked, this has worked for me just as well for the purpose of configuring niri, at least.
i don't have strong opinions on formats vs writers (@nezia1?), but i feel inclined to have this supersede #412029.

[sodiboo]

then i guess this one should be "closes #198655"
yay for linked list of KDL pull requests

[toastal]

Pardon, it might be outside of scope but is fromKDL a goal too? I would (metaphorically) kill to not be locked into only JSON (or worse, TOML).

[poseidon-rises]

Pardon, it might be outside of scope but is fromKDL a goal too? I would (metaphorically) kill to not be locked into only JSON (or worse, TOML).

That just seems difficult on so many levels. I'm sure it could be done, but imo it's outside the scope of this PR, multiple home-manager programs need this pulled (zellij and niri off the top of my head).

[sodiboo]

Pardon, it might be outside of scope but is fromKDL a goal too? I would (metaphorically) kill to not be locked into only JSON (or worse, TOML).

As in, using the KDL-in-Nix DSL defined here (with "documents", "splats", "nodes", etc) and returning a JSON-shaped configuration?

Something like this is potentially worth doing in the future if we want to support arbitrary KDL queries (with KQL) or validation of "KDL schema"; but it's certainly not in scope for this PR. I also don't quite get why you'd want to do this. In the context of Nix configuration in evalModules, the KDL configuration is just strictly less flexible than the existing module system, for like, no benefit.

If you mean, like JSON-in-KDL(-in-Nix), then yes you can totally do that with this PR. just plug the output of kdl.generator into another derivation that converts "JSON-in-KDL" to JSON. but Why would you do that?

[poseidon-rises]

As i said above, i think it's actually really important to have a usage example. Mainly, i want to encourage module authors making use of this format to not name their options settings.

I personally don't agree with this. settings is standard naming for nearly all modules. Just look up settings on your nixos options searcher of choice and you will find thousands of results. I don't see a reason to break this standard for specifically this format.

[sodiboo]

As i said above, i think it's actually really important to have a usage example. Mainly, i want to encourage module authors making use of this format to not name their options settings.

I personally don't agree with this. settings is standard naming for nearly all modules. Just look up settings on your nixos options searcher of choice and you will find thousands of results. I don't see a reason to break this standard for specifically this format.

As i've explained in detail, KDL is different in a way that matters a lot for Nix, and completely shatters what is expected by settings: the toplevel composition primitive is an ordered list, and evalModules just doesn't deal with that, when individual elements need to be merged. (which is why this PR, compared to the last one, explicitly prevents accidentally merging multiple disjoint definitions)

settings is standard for formats that play nicely with evalModules, which is fundamentally based on attrsets. settings is an attrset that can be inspected and you can easily extract its values, like config.services.meilisearch.settings.http_addr. KDL documents are not as easily consumable in this way: list accessing in Nix is awkward, and whether order matters or how an option is structured, is very application dependent. (like i said, it would be interesting if we could do something with KDL schema or KDL queries down the line; to make it easier to "extract" parts of a config this way and bring it to parity with attrsets; but it's way out of scope for this initial implementation)

I also wish to, at some point, support something that allows you to define how to merge KDL documents. I think of this as "extensible KDL": a more flexible, fully-fledged, fits-in-a-module-no-questions-asked, alternative to formats.kdl, which builds on and lowers into the primitives created here. I want to reserve the settings name for that.

It's also common to make settings a submodule and have some suboptions visible in the manual: That is absolutely not going to be supported by this KDL implementation (because it's not an attrset-composed type)

(and, on a more personal note, this is programs.niri.settings and i don't want that to be occupied by an objectively less flexible just-KDL config option in NixOS or home-manager)

config is a standard name for anything that doesn't play as nicely with evalModules. Just look up config on any of the same search engines, and you will find more results than settings. It's undeniable that in NixOS, config is the standard name for an "opaque" configuration value, that can usually only be defined once (including a common variation called configFile), and once defined, cannot really be inspected. KDL fits much more closely into that description, than what users expect of settings. This is also why i think it might be worth putting in a different namespace (pkgs.formats.* generally deals with "settings"-like types); i just don't know where else to put it in that case.

[KiaraGrouwstra]

i would argue (convergence in) the settings name discussion should not have to be in scope for the purpose of this PR

[nezia1]

Hello, apologies for not having checked this earlier! I'm currently jet lagged and haven't had the capacity to review this fully. However, I feel it's really important that we get the namespacing right, for a few reasons:

KDL documents are not as easily consumable in this way: list accessing in Nix is awkward, and whether order matters or how an option is structured, is very application dependent.

This is why I've been stressing that pkgs.formats isn't the right place for KDL functions. The goal of pkgs.formats, and RFC 0042 by extension, is to encode purely structural languages like JSON and TOML. You said yourself:

The KDL format is really more like extraConfig with types.lines than any "settings" shaped format. We should likewise always use the config terminology to refer to KDL-shaped configuration options.

But the entire point of pkgs.formats is to encode languages that AREN'T like extraConfig - languages that are easily mergeable, and have a singular code representation. KDL isn't one of these.

i would argue (convergence in) the settings name discussion should not have to be in scope for the purpose of this PR

I understand where you're coming from on this, but I believe it's absolutely in scope. It's difficult to get anything standardized under nixpkgs, mainly because we have to get so many people to agree with it. RFC42 is one of the few standards we have that keeps modules somewhat consistent. Diverging from this format would break a well-established design pattern, and be a big problem for subsequent PRs, as the namespace would no longer have any meaning.

[sodiboo]

i would argue (convergence in) the settings name discussion should not have to be in scope for the purpose of this PR

Like @nezia1 said, that's definitely in scope. The reason for splitting off the PR is (a) to discuss the functionality changes (restrictions in usage + "splat" ergonomics), as well as (b) introducing better documentation for the format. I included an example module with my preference of option naming, and mentioned in the PR why i think we need to diverge from settings. I think it's important we get it right.

[sodiboo]

It seems at least i and @nezia1 are in agreement that this probably needs a different namespace. But uh. Which namespace? I genuinely don't know.

[nezia1]

It seems at least i and @nezia1 are in agreement that this probably needs a different namespace. But uh. Which namespace? I genuinely don't know.

As I said, I'd be completely fine with something like pkgs.kdl. I think introducing a new namespace here would make sense, and would allow for adding new functions to it! It just sucks that KDL is in such a weird spot, between fully mergeable and a pkgs.writers programming language, but I think this is probably our best course of action.

Or perhaps we could go with pkgs.fromKDL? I don't mind either, but we'd have to get more opinions from people more experienced with the underlying namespaces under pkgs/lib.

[KiaraGrouwstra]

fromKDL

toKDL?

[toastal]

fromKDL

toKDL?

¿Por qué no los dos? (would genuinely love to see it, even if out of scope)

[eclairevoyant]

Having functions (that aren't package expressions) at the top-level would be confusing. IMO it merits going in some other namespace (there's a pkgs.lib right?)

[llakala]

Having functions (that aren't package expressions) at the top-level would be confusing.

Already happens a ton (just open up Noogle and scroll down past the subattrs). I don't like it - I wish there was a separate output for functions that rely on pkgs - but there is precedence for it.

[nezia1]

(there's a pkgs.lib right?)

Unfortunately not. pkgs functions seem to just be shoved in there arbitrarily, I'd honestly like for this to have some sort of structure, as we have a wonderful namespacing under lib but pkgs just seem to be way less strict about it.

[sodiboo]

I've opened a draft PR in home-manager, with a "real" example of a module. It's basically a subset of what i want niri-flake's settings structure to look like, ideally, in the future, at some point. Go take a look at it.

• niri: init module (playground for nixpkgs KDL format) nix-community/home-manager#7534

[sodiboo]

(rebasing because the home-manager PR depends on new maintainers entries that didn't exist in nixpkgs 3 weeks ago)

[quentinmit]

So, if my understanding is correct, home-manager's toKDL supports arbitrarily ordered/duplicate nodes, but only in children, and assumes all toplevel nodes are uniquely named and unordered (which doesn't hold for niri's output, workspace, and spawn-at-startup)

I think you're right about the top level. That's an oversight on my part - it should be pretty easy to fix, I just didn't think of it. You can see how it works if you nest things under a top-level node:

lib.hm.generators.toKDL { } {
  value._children = [
    {
      output._args = ["DP-1"];
      output.mode = "2560x1440";
    }
    {
      output._args = ["HDMI-A-1"];
      output.mode = "1920x1080";
    }
  ];
}

which generates:

value {
        output "DP-1" {
                mode "2560x1440"
        }
        output "HDMI-A-1" {
                mode "1920x1080"
        }
}

I think we could either add support for _children as a direct attr in the toKDL toplevel argument, or we could allow toKDL to take a list in addition to an attrset and that would implicitly mean a list of ordered children. I'm not sure which one would be more ergonomic - probably the former? Or we could even do both. I'll send a PR to fix this up in home-manager.

The design principle here is that "normal" nodes (e.g. nodes that only have args, or only have a single arg even) can be handled with normal-looking attrsets in Nix, and you can sprinkle in _children when you need an escape hatch. This is much more ergonomic than forcing everything to be a list at every level of the hierarchy, because then you lose a bunch of the benefits of the module system like automatic merging/overriding nested attrs and you have to make every caller aware of having to call a node constructor.

The way that I think about KDL is that it's more in the family of XML than it is in the family of JSON/YAML/etc. It represents a more highly-structured document, and it doesn't just have data types that cleanly map to Nix's list/attrset/scalars. The builtins.toXML generator, for instance, can't actually generate arbitrary XML documents. It generates a very strict structure that represents the passed Nix value and the Nix documentation recommends pairing it with XSLT to generate the final XML document.

(Incidentally, this is a problem with YAML in Nix - YAML has complex value types called "tags" that can't be generated by the built-in YAML generator because it can only generate JSON. See how e.g. the services.home-assistant.config option gets post-processed with sed in an attempt to allow generating tags.)

[sodiboo]

I think we could either add support for _children as a direct attr in the toKDL toplevel argument, or we could allow toKDL to take a list in addition to an attrset and that would implicitly mean a list of ordered children. I'm not sure which one would be more ergonomic - probably the former? Or we could even do both. I'll send a PR to fix this up in home-manager.

Yeah, probably both. The "children" of a node should have exactly the same schema as the toplevel part.

The way that I think about KDL is that it's more in the family of XML than it is in the family of JSON/YAML/etc.

Yes

It represents a more highly-structured document, and it doesn't just have data types that cleanly map to Nix's list/attrset/scalars.

I wouldn't say "more highly-structured", but it's fundamentally differently structured. The data types do map very cleanly to Nix (for instance, the scalars are one-to-one), it's just that the toplevel datatype is a list and Nix's module system sucks at dealing with complicated lists. XML and KDL have in common that they don't allow nesting attrsets: they are only part of a node's "properties", which contain leaf values, and not full nodes. They're not just prioritizing ordered lists, they force you to use them. It's not a better or higher-level structure, just a different one.

The builtins.toXML generator, for instance, can't actually generate arbitrary XML documents. It generates a very strict structure that represents the passed Nix value and the Nix documentation recommends pairing it with XSLT to generate the final XML document.

Honestly, idk wtf builtins.toXML is useful for. Yeah, it converts a Nix value to "XML" but the XML is very Nix-specific. It's similar to "JSON-in-KDL" in purpose; the name is just Wrong on its own.

The design principle here is that "normal" nodes (e.g. nodes that only have args, or only have a single arg even) can be handled with normal-looking attrsets in Nix, and you can sprinkle in _children when you need an escape hatch.

I agree that it is very nice to use attrsets for briefer configuration, but i've expressed that i think "it's an attrset unless you need an escape hatch" is not the correct way to do that. it doesn't let me do things like this^1, and it's also misleading, because that's not how KDL works. At all. It's useful for several real-world apps and services, because at the end of the day it's all deserialized to something like this (usually with a predictable schema in KDL), but it's fundamentally not how KDL is structured and i would really like if there is a "Correct" implementation of the actual structure of KDL. home-manager's toKDL can and should build a higher level interface than that; and eventually something Like That should be in nixpkgs too.

A more correct way to interact with actually KDL-shaped configurations is probably with something like KDL schema for validation/merging and KDL query for reading attributes.

^1 (note that my mapping here is also somewhat misleading, it doesn't easily expose ordering, although here's my current workaround; see intended usage in this user's workspace configuration)

This is much more ergonomic than forcing everything to be a list at every level of the hierarchy, because then you lose a bunch of the benefits of the module system like automatic merging/overriding nested attrs

Yes. That's expected. I've talked about this. The point of splitting off this PR was to specifically restrict that. Notice how the type this PR exposes is mergeUniqueOption: it's intentionally not mergeable at all in order to signal to users that they can't expect a reliable merging/overriding behaviour. This isn't a problem for "leaf" apps/services like niri, which are very much "configured by the end user". It's only a huge issue for core services that should be configured by many other modules throughout nixpkgs or home-manager. I expect users of this KDL format to do something like this to allow certain options to be overridable by e.g. the rest of home-manager. But most parts of the config are entirely specific to the user's system and it doesn't matter if we have proper overrides. If they want to override individual options for specialisations, they should just make their own options. Better yet, import something like niri-flake's settings module, for everything to be Nix-idiomatic-like. Again, this is also the reason why i want to encourage not using the name settings for KDL-configured modules like this; it should be reserved, and usable by mappings like niri-flake (which are sufficiently complex that they probably don't belong inline in nixpkgs or home-manager, unless like, systemd one day decides to use KDL for everything)

you have to make every caller aware of having to call a node constructor.

As long as we expose any general KDL configuration, callers must be aware of what KDL is, how nodes work, and what they can do with them. That named constructor is optional (you don't have to call it), but the fact that a node has attrs like name and arguments is something we want callers to be aware of. home-manager's toKDL also exposes the same complexity with its _args, _props, etc.

[krostar]

Hi everyone 👋

I’ve been following this PR with interest — KDL support in nixpkgs looks like a great addition!

I was wondering if there’s any consensus forming (or if decisions are still needed) around the remaining blocking points — for example, namespace placement (pkgs.formats vs lib.kdl vs pkgs.kdl), naming conventions, or related topics.

Could a maintainer share whether there’s a clear direction, or what the decision-making process might look like to help move this PR forward?

Also, is there anything missing in the PR that you would like help with @sodiboo? I’d be happy to contribute if there’s something I can do to help it progress.

[sodiboo]

oh shit what the hell did i just do (snapshot)

[N4CH723HR3R]

any update on this? can i help with something?

[tree-sapii]

Stale pr? I would love to have this, even it its would do IFD, but as idea, would it be better to implement this at the builtins level? As a nix function like toJSON, it would offer the same benefits as the rust implementation of this as well as be incredibly fast.

[N4CH723HR3R]

Stale pr? I would love to have this, even it its would do IFD, but as idea, would it be better to implement this at the builtins level? As a nix function like toJSON, it would offer the same benefits as the rust implementation of this as well as be incredibly fast.

no, to be a builtins function you would need to ALWAYS provide the same input/output as long as the tree stays the same. there is a reason why builtins.toTOML doesnt exist

By relying on a library, we are subjected to all changes in its output format, (...) for Nix this is not acceptable, because the output of evaluation must be byte for byte reproducible, as it affects derivation hashes.