request: `pkgs.formats.kdl` to support applications adopting KDL config language

[colemickens]

Project description

Zellij, for example, as adopted KDL as a config language/format.

It would be nice if there was similar support for KDL like we have for JSON/YAML/etc.

[AgathaSorceress]

I looked into this and this seems to unfortunately be more complicated than something like JSON/YAML, due to KDL nodes not only being key:value pairs, but also having arguments and properties.

Here's an example KDL document consisting of one node

thermal-paste "NT-H1" amount="3.5g" {
    description "it tastes nice :)"
}

It could be represented in nix like this:

{
  thermal-paste = {
    properties = { amount = "3.5g"; };
    arguments = [ "NT-H1" ];
    children = { description = { arguments = [ "it tastes nice :)" ]; }; };
  };
}

by passing this attribute set to builtins.toJSON, we get:

{
  "thermal-paste": {
    "arguments": [
      "NT-H1"
    ],
    "children": {
      "description": {
        "arguments": [
          "it tastes nice:)"
        ]
      }
    },
    "properties": {
      "amount": "3.5g"
    }
  }
}

Which is quite ugly, but could then be recursively parsed by an external program (either a custom implementation of json2kdl similar to json2yaml, or (this option might be easier) upstreaming it into remarshal).

The parser would need to make these assumptions:

• the JSON root element {} is equivalent to a KDL Document, and keys of that element are KDL Nodes.
• each key in the root element represents the Identifier of the corresponding KDL Node, and its value represents the elements that come after the Identifier.
• each JSON object corresponding to a Node can have 3 optional fields: arguments, properties and children.
• If a key that represents a node is followed by an empty JSON object, it should be treated as a KDL Node with no fields, but not skipped.
• arguments can only be a list of String, Number, Bool or Null (not all elements have to be the same type)
• properties can only be an object where the key:value pairs represent Identifiers and Values, but not Nodes
• children can only be a list of Nodes, parsed the same way as Nodes at the root of the object.

Parsing the actual values should not be a problem, as there are already quite a few KDL libraries for most common languages.
KDL's 4 data types should not be a problem either, as both Nix and JSON have them, but Null might be tricky due to some parsers making the assumption that "explicit absence of a value" and "a value not being specified" are the same.

Potential problems

• This requires JSON objects to follow a strict format, rather than being able to parse any generic JSON object into a KDL document
• JSON-in-KDL already exists, however its purposes is to embed JSON in KDL, not convert JSON to KDL. Creating a program that does the latter could make this confusing (especially in a context of an existing project not strictly related to this specific issue, such as remarshal).
• Adding json2kdl to remarshal would likely require a significant amount of work, as not only json2kdl would have to be implemented but every permutation of {cbor,json,msgpack,toml,yaml,kdl}2{cbor,json,msgpack,toml,yaml,kdl}
• I did not find a good way to represent comments and type annotations, however they could probably be left out due to not being strictly necessary for the purpose of converting Nix to KDL for configuration files.
• There does not seem to be a way to represent KDL in Nix while retaining the clean syntax. The JSON representation does not matter, as it is not exposed to the user, so I considered using nix functions (as they have a similar syntax) instead of attribute sets to represent nodes, but the function name needs to be defined somewhere to not be a syntax error. Another alternative is something like this:

node = name: arguments: properties: children: { ${name} = { inherit arguments properties children; }; }

node "thermal-paste" [ "NT-H1" ] { amount = "3.5g"; } [
  (node "description" ["it tastes nice :)"] { } [ ])
]

which has the downside of the node function having to be in scope.
It is also possible to add default values to avoid the { } [ ] at the end, but doing so would return us to the original syntax (but worse):

node = {name, arguments ? [], properties ? {}, children ? []}: { ${name} = { inherit arguments properties children; }; }

node {
  name = "thermal-paste";
  arguments = [ "NT-H1" ];
  properties = { amount = "3.5g"; };
  children = [
    (node {
      name = "description";
      arguments = [ "it tastes nice :)" ];
    })
  ];
}

[AgathaSorceress]

I have written a program that converts JSON to KDL here
In theory, the only things left are:

• adding json2kdl to nixpkgs
• adding a KDL format in pkgs-lib/formats.nix similar to the YAML one

[feathecutie]

It could be represented in nix like this:

{
  thermal-paste = {
    properties = { amount = "3.5g"; };
    arguments = [ "NT-H1" ];
    children = { description = { arguments = [ "it tastes nice :)" ]; }; };
  };
}

Note that this representation makes the assumption that each node identifier can only occur once per parent, which is not necessarily the case in KDL.
This representation also does not necessarily preserve the order of nodes (unless the order of attributes is somehow preserved consistently in Nix?), which is significant in KDL.

As far as I can tell, the proposed function representation (referenced below for clarity) would solve this issue, but I'm not sure how to properly implement this in pkgs.formats.

node = name: arguments: properties: children: { ${name} = { inherit arguments properties children; }; }

I've played around with integrating the proposed attrset representation into pkgs.formats using @AgathaSorceress's json2kdl program until I noticed this issue, and could provide this experimentation for reference.

To expand on representing KDL documents as attrsets, which seems to be the preferred/required representation for pkgs.formats, it is probably necessary to represent the document and all lists of children as actual Nix lists instead of attrsets, to allow duplicate identifiers and preserving order. The representation of nodes inside this list could be something like this:

{
  identifier = "thermal-paste";
  properties = { amount = "3.5g"; };
  arguments = [ "NT-H1" ];
  children = [
    {
      identifier = "description";
      arguments = [ "it tastes nice :)" ];
    }
  ];
}

Using this representation, it would also be very easy to provide an additional node helper function that can convert the more idiomatic function call syntax into this structure:

node = identifier: arguments: properties: children: { inherit identifier arguments properties children; }

or even:

node = {identifier, arguments ? [], properties ? {}, children ? []}: { inherit identifier arguments properties children; }

I will try to further experiment with this, and might try to upstream it if I manage to create a working implementation.

[feathecutie]

For reference, there is now a working(?) implementation of this at https://github.com/feathecutie/nixpkgs/tree/formats-kdl.

The only remaining issue is that I can't get the NixOS manual to build on this branch when I define any NixOS option with type (pkgs.formats.kdl {}).type. Instead, I keep encountering error: stack overflow (possible infinite recursion) when trying to build the manual using something like nix-build nixos/release.nix -A manual.x86_64-linux, probably because it is trying to recursively generate the type documentation even though the type documentation should be overridden.

Feel free to use this code (and feel free to open a nixpkgs PR with it if you manage to fix this, otherwise I'll probably keep trying stuff out and open a PR myself).