Package structure

[infinisil]

Issue description

It is hard for new package contributors to know where they should define their package, since the pkgs subdirectory is laid out fairly arbitrarily. And then there's the pseudo-sorted humongous all-packages.nix that's become unmanageable at this point.

Goal

When new contributors easily know where they can define their package.

[infinisil]

@fricklerhandwerk also talked to me about cleaning up the file structure in other ways in nixpkgs, in particular things like:

• Cleanly separating package definitions and nixos, mainly by moving the doc directory to something like pkgs/doc. Also suggested was renaming pkgs to nixpkgs
• Cleaning up the structure under pkgs in general

[roberth]

Cleanly separating package definitions and nixos,

Why though?

I highly agree with the kind of profound insight someone shared at NixCon 2018 (iirc) that nixpkgs is an automated knowledge system.
Reasoning from there, I'll add that knowledge does not stop at some constructed architectural boundary.
I'd favor a functional directory layout over a layout primarily dictated by implementation aspects.

We'd have for example the following files

• nixpkgs/package-set.nix <- pkgs/top-level/all-packages.nix, references all other packages
• nixos/lib/default.nix: nixosSystem and friends
• lib/lib/default.nix: lib/default like NixOS, is primarily a library
• lib/manual/*.md
• postgresql/package/default.nix
• postgresql/nixos/default.nix <- nixos/modules/services/postgresql.nix (or wherever it is)
• postgresql/nixos/manual.md
• postgresql/release-notes/22.05.md
• postgresql/nixos/test/default.nix
• postgresql/nixos/test/backup.nix
• haskell/packages/....nix <- pkgs/development/haskell-modules/....nix
• hasura/package/default.nix: references haskellPackages just like today
• ...

I believe this to be much more human-friendly, as everything postgres is in one place, instead of spread out across the tree.

(btw, personalized release notes? heck yeah! this gets us halfway there)

I'll also note that this structure is similar to a flake. Convergent evolution I'd say. Both are functional divisions as opposed to technical. We may or may not be able to exploit the similarity.

One flaw perhaps is to have a large amount of files in the root of the repo. We can improve this by moving all concepts into a /concept directory. That way the GitHub "front page" doesn't freak out, and you can ls your repos without regrets.
What's not a flaw is to have a large directory. I consider that a feature, because it removes all of the entropy added by the current hierarchy.
Another issue I didn't address is the collisions between similarly named packages. We can resolve these in a similar way we resolve them in pkgs.*.

A functional directory layout might also help us not try to lovingly cram everything into passthru, because the derivation is not always the right functional key for everything. (inspired by NixOS/nixos-search#506 (comment))

[fricklerhandwerk]

@roberth I don't disagree with your proposal (and will probably need more time to let it sink in to appreciate it fully), but want to clarify that I would simply like to remove some of the noise and reduce some of the potential for confusion without changing too much.

So to judge any such proposal I would apply the following measure: what is the smallest change that has a tangible impact on clarity and ease of navigation?

PS: I fully agree on the nixpkgs repository being an expert system for building (Nixpkgs) and running (NixOS) software. That person deserves their name attached to this insight.

[roberth]

That person deserves their name attached to this insight.

I've checked the youtube transcripts of NixCon '18, but couldn't find it with (the non-[ ]) words like knowledge [system], expert [system], [knowledge] base, database, information [system]. Might have been a lightning talk, none of which were published.
:(

[roberth]

Hi @blaggacao, and thank you for sharing your experience.

Currently we're in the process of narrowing down what to work on in the architecture team, and auto-called packages seems to be the obvious candidate. This is a lot narrower in scope, but we should be mindful of fitting it in the larger context that includes this thread.

As such, I don't think we need to come up with a conceptual framework for what goes into the package directories, how that's layered, etc. This allows us to keep the initial plan simple, so that we don't have to burden the community with unnecessary terminology when propose the first changes.

So for now I won't go into too much detail about the layers or other specifics, but instead rely exclusively on familiar "Nixpkgs" concepts. Your input is definitely worth circling around to when we continue this discussion.

At risk of being ill-recived by some

It's relevant input. If someone disagrees with the article, we can have a discussion and learn together.

All my experience so far tells me that domain centric tooling is superior for a problem domain than tooling-induced vertical slicing of that domain.

I'll note that as validation of the idea that the new directory structure should invite other files than just an callPackage-invoked package function. Thanks!

[oneingan]

I highly agree with the kind of profound insight someone shared at NixCon 2018 (iirc) that nixpkgs is an automated knowledge system.

https://discourse.nixos.org/t/nixpkgs-nixos-is-an-expert-system-database/671

[infinisil]

We can consider this addressed with RFC 140 now!