New __include__ key to compose a config from other configs

CHANGELOG.rst

@@ -40,6 +40,10 @@ Added
   ``add_method_arguments``, and links applied on instantiate can target the
   parameters of these groups (`#986
   <https://github.com/mauvilsa/jsonargparse/pull/986>`__).
+- New ``__include__`` key accepted at any level of a config, which merges into it
+  the configs that it gives, so that a config can be composed from others.
+  Enabled with the new ``config_include_enabled`` of ``set_parsing_settings``
+  (`#990 <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
 
 Fixed
 ^^^^^
@@ -108,6 +112,13 @@ Fixed
 - Subcommand parsers not logging when the logger of the parent parser is set
   after adding the subcommands (`#989
   <https://github.com/mauvilsa/jsonargparse/pull/989>`__).
+- The ``jsonschema`` completion described the init parameters of a class as
+  accepted by the first argument that has it, e.g. for other arguments that
+  ``skip`` different parameters of the class (`#990
+  <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
+- A config file given as value of a subclass type failing with ``Config file
+  loop detected`` instead of the actual error, e.g. an invalid ``class_path``
+  (`#990 <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
 
 Changed
 ^^^^^^^
@@ -176,6 +187,13 @@ Changed
   positional that accepts a variable number of values, since argparse parses
   this combination incorrectly, e.g. taking the subcommand name as one of the
   values (`#986 <https://github.com/mauvilsa/jsonargparse/pull/986>`__).
+- The ``jsonschema`` completion now has in ``$defs`` the schema of the
+  ``$schema`` key and one definition per known subclass, named by its import
+  path, so that parts of the schema can be referenced from other schemas (`#990
+  <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
+- ``from_config`` now fails with ``ArgumentError`` for an invalid config,
+  including a missing required parameter, instead of ``TypeError`` or
+  ``ImportDenied`` (`#990 <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
 
 Removed
 ^^^^^^^

DOCUMENTATION.rst

@@ -1386,6 +1386,7 @@ Adding a config file argument and parsing some arguments then gives:
 
     os.chdir(cwd)
     shutil.rmtree(tmpdir)
+    set_parsing_settings(config_include_enabled=False)
 
 .. doctest:: config
 
@@ -1416,6 +1417,72 @@ To parse a config file or a config string without parsing command line
 arguments, use :meth:`parse_path <.ArgumentParser.parse_path>` or
 :meth:`parse_string <.ArgumentParser.parse_string>`.
 
+.. _including-configs:
+
+Including configs
+-----------------
+
+A config can be composed from others with an ``__include__`` key, which is
+enabled with ``set_parsing_settings(config_include_enabled=True)``. Its value is
+the path of a config file or a list of them, relative to the config that has the
+key, or to the working directory for a config not from a file, e.g. a command
+line value or given to :meth:`parse_object <.ArgumentParser.parse_object>`. It
+is accepted at any level, also in sub-config files, and must be the first key
+where it is given, since the keys that follow override what the included configs
+set. Included configs can include others, and since paths are relative, a group
+of config files can be moved around without being modified. Unlike
+:ref:`sub-config-files`, includes do not depend on ``sub_configs``, and
+:meth:`save <.ArgumentParser.save>` with ``multifile=True`` does not keep them
+as separate files.
+
+The included configs, and then the keys that follow, are merged exactly as if
+they had been given one after the other, e.g. as several ``--config``. So the
+same rules apply, for instance a ``dict`` value is replaced, items are appended
+when the key is given with a ``+`` suffix, see :ref:`list-append`, and changing
+a ``class_path`` discards the ``init_args`` that the new class does not accept,
+also for the items of a list of classes. An ``__include__`` is not supported
+within a value that is not validated, e.g. ``Any``, since nothing merges it.
+
+Continuing with the example above:
+
+.. testcode:: config
+
+    set_parsing_settings(config_include_enabled=True)
+
+.. code-block:: yaml
+
+    # File: lev1.yaml
+    opt1: from lev1
+
+.. code-block:: yaml
+
+    # File: main.yaml
+    __include__: example.yaml
+    lev1:
+      __include__: lev1.yaml
+      opt2: from main
+
+.. testsetup:: config
+
+    pathlib.Path("lev1.yaml").write_text("opt1: from lev1\n")
+    pathlib.Path("main.yaml").write_text(
+        "__include__: example.yaml\nlev1:\n  __include__: lev1.yaml\n  opt2: from main\n"
+    )
+
+.. doctest:: config
+
+    >>> cfg = parser.parse_args(["--config", "main.yaml"])
+    >>> cfg.lev1.opt1
+    'from lev1'
+    >>> cfg.lev1.opt2
+    'from main'
+
+.. note::
+
+    Including configs is experimental. Behavior details might change in
+    non-major releases.
+
+
 Serialization
 -------------
 
@@ -2455,7 +2522,9 @@ checkpoint_hook.yaml]``, or appending one item at a time as explained in
 Relative paths inside a sub-config file are resolved with respect to the
 directory of that file, so a group of config files can be moved around without
 being modified. :meth:`save <.ArgumentParser.save>` with ``multifile=True``
-writes each sub-config back to its own file, keeping the original structure.
+writes each sub-config back to its own file, keeping the original structure. To
+override some values of a sub-config file in the same config, use
+``__include__`` instead, see :ref:`including-configs`.
 
 :ref:`subclasses-disabled` types also accept a sub-config file, whose content is
 the fields of the type, without ``class_path`` and ``init_args``. This only
@@ -3361,7 +3430,19 @@ the config can point to the generated schema with a ``$schema`` key:
 The key is accepted in any config that a parser loads, :ref:`sub-config-files`
 included, and it is removed before parsing, so it never becomes part of the
 parsed namespace. Accordingly, every object in the schema that describes a
-config accepts the key.
+config accepts the key. With ``config_include_enabled``, these objects also
+accept the ``__include__`` key, see :ref:`including-configs`, and require none
+of their keys, since a config that includes others, or is included, can be
+partial.
+
+A config that is not the root, e.g. a sub-config file or an included config,
+needs the part of the schema for where it goes. Since not all editors support a
+JSON pointer in ``$schema``, the config can point to a small schema that
+references that part:
+
+.. code-block:: json
+
+    {"$ref": "schema.json#/$defs/mymodule.MyModel/properties/init_args"}
 
 The schema is derived from the same information that the ``--help`` output is
 based on, so it includes:
@@ -3391,6 +3472,10 @@ based on, so it includes:
 
 Subclasses and types that are used in more than one place are added once to
 ``$defs`` and referenced with ``$ref``, which also makes recursive types work.
+Each known subclass has its own definition named by its import path, e.g.
+``mymodule.MyModel``. When an argument accepts different init parameters than
+the class has, e.g. due to ``skip``, its definition is a variant named after the
+first argument that uses it, e.g. ``mymodule.MyModel@model``.
 
 The schema is meant to accept what the parser accepts, but for subclass types it
 is stricter. A string is accepted, since it can be a class path or a path to a

jsonargparse/_actions.py

@@ -19,20 +19,20 @@
     parser_context,
 )
 from ._loaders_dumpers import get_loader_exceptions, load_value
-from ._namespace import Namespace, ValueSource, copy_provenance, value_source_context
+from ._namespace import Namespace, copy_provenance
 from ._optionals import _get_config_read_mode, ruamel_support
-from ._paths import path_dir_context
 from ._type_checking import ArgumentParser
 from ._util import (
+    ComposedConfig,
     Path,
     argument_error,
+    config_file_context,
     default_config_option_help,
     get_import_path,
     import_object,
     indent_text,
     iter_to_or_str,
     iter_to_set_str,
-    load_config_path_context,
     merge_config,
     parse_value_or_config,
 )
@@ -279,6 +279,10 @@ def resolve_subclass_spec(self, value):
 
         if self.basetype is None or not is_subclasses_disabled(self.basetype):
             return value
+        if isinstance(value, ComposedConfig):
+            value.own = self.resolve_subclass_spec(value.own)
+            value.includes = [(self.resolve_subclass_spec(layer), path) for layer, path in value.includes]
+            return value
 
         def resolve_class(class_path):
             try:
@@ -313,11 +317,12 @@ def _load_config(self, value, parser):
             if cfg is value:
                 cfg, cfg_path = parse_value_or_config(value)
                 cfg = self.resolve_subclass_spec(cfg)
-            if not isinstance(cfg, (dict, Namespace)):
+            if not isinstance(cfg, (dict, Namespace, ComposedConfig)):
                 raise TypeError(f'Parser key "{self.dest}": Unable to load config "{value}"')
-            source = None if cfg_path is None else ValueSource("config file", cfg_path, parser.parser_mode)
-            with load_config_path_context(cfg_path), path_dir_context(cfg_path), value_source_context(source):
+            with config_file_context(cfg_path, parser.parser_mode):
                 cfg = parser._apply_actions(cfg, parent_key=self.dest)
+            if cfg_path is not None:
+                cfg["__path__"] = cfg_path
             return cfg
         except (SubclassesDisabledError, ImportDenied) as ex:
             raise TypeError(f'Parser key "{self.dest}":\n{indent_text(str(ex))}') from ex

jsonargparse/_common.py

@@ -285,6 +285,7 @@ class ImportDenied(ImportError, ValueError):
     "add_print_completion_argument": False,
     "stubs_resolver_allow_py_files": False,
     "omegaconf_absolute_to_relative_paths": False,
+    "config_include_enabled": False,
     "unset_sentinel": None,
     "import_path_verdicts": {entry: False for entry in default_import_path_denylist},
 }
@@ -366,6 +367,7 @@ def set_parsing_settings(
     add_print_completion_argument: bool | None = None,
     stubs_resolver_allow_py_files: bool | None = None,
     omegaconf_absolute_to_relative_paths: bool | None = None,
+    config_include_enabled: bool | None = None,
     unset_sentinel: bool | None = None,
     subclasses_disabled: list[type | Callable[[type], bool]] | None = None,
     subclasses_enabled: list[type | str] | None = None,
@@ -419,6 +421,11 @@ class when a value for a type that accepts any value, i.e. ``Any``,
             with ``omegaconf+`` parser mode, absolute interpolation paths are
             converted to relative. This is only intended for backward
             compatibility with ``omegaconf`` parser mode.
+        config_include_enabled: If ``True``, configs accept an ``__include__``
+            key to be composed from other configs, see :ref:`including-configs`.
+            The ``jsonschema`` completion type then accepts the key and requires
+            no keys in configs, since these can be partial. Default is
+            ``False``.
         unset_sentinel: If ``True``, parsers will use the :obj:`.Unset` sentinel
             for arguments that have not been given a value (instead of
             ``None``). This allows distinguishing between ``None`` as an
@@ -456,6 +463,7 @@ class when a value for a type that accepts any value, i.e. ``Any``,
         "add_print_completion_argument": add_print_completion_argument,
         "stubs_resolver_allow_py_files": stubs_resolver_allow_py_files,
         "omegaconf_absolute_to_relative_paths": omegaconf_absolute_to_relative_paths,
+        "config_include_enabled": config_include_enabled,
     }
     for name, value in bool_settings.items():
         if isinstance(value, bool):