Nested populated transclusions

[RR5555]

Issue

When trying to transclude nested populated (not just containing the include statement) files, we get the following:

uv run poe test tests/test_config.py -k test_config_data_nested_transclusions -vv -rP
Poe => pytest --color=yes tests/test_config.py -k test_config_data_nested_transclusions -vv -rP
====================================================================== test session starts ======================================================================
platform linux -- Python 3.9.23, pytest-8.4.1, pluggy-1.5.0 -- /workspace/copier/.venv/bin/python
cachedir: .pytest_cache
rootdir: /workspace/copier
configfile: pyproject.toml
plugins: cov-6.2.1, gitconfig-0.7.0, xdist-3.8.0
16 workers [1 item]       
scheduling tests via LoadScheduling

tests/test_config.py::test_config_data_nested_transclusions 
[gw0] [100%] FAILED tests/test_config.py::test_config_data_nested_transclusions 

=========================================================================== FAILURES ============================================================================
_____________________________________________________________ test_config_data_nested_transclusions _____________________________________________________________
[gw0] linux -- Python 3.9.23 /workspace/copier/.venv/bin/python

    def test_config_data_nested_transclusions() -> None:
        config = Worker("tests/demo_transclude_nested/demo")
>       assert config.all_exclusions == ("exclude1", "exclude2")

tests/test_config.py:388: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/gitpod/.local/share/uv/python/cpython-3.9.23-linux-x86_64-gnu/lib/python3.9/functools.py:993: in __get__
    val = self.func(instance)
copier/_main.py:637: in all_exclusions
    return self.template.exclude + tuple(self.exclude)
/home/gitpod/.local/share/uv/python/cpython-3.9.23-linux-x86_64-gnu/lib/python3.9/functools.py:993: in __get__
    val = self.func(instance)
copier/_template.py:327: in exclude
    self.config_data.get(
/home/gitpod/.local/share/uv/python/cpython-3.9.23-linux-x86_64-gnu/lib/python3.9/functools.py:993: in __get__
    val = self.func(instance)
copier/_template.py:302: in config_data
    result = filter_config(self._raw_config)[0]
/home/gitpod/.local/share/uv/python/cpython-3.9.23-linux-x86_64-gnu/lib/python3.9/functools.py:993: in __get__
    val = self.func(instance)
copier/_template.py:265: in _raw_config
    return load_template_config(conf_paths[0])
copier/_template.py:101: in load_template_config
    flattened_result = lflatten(filter(None, yaml.load_all(f, Loader=_Loader)))
.venv/lib/python3.9/site-packages/funcy/seqs.py:193: in lflatten
    return list(flatten(seq, follow))
.venv/lib/python3.9/site-packages/funcy/seqs.py:184: in flatten
    for item in seq:
.venv/lib/python3.9/site-packages/yaml/__init__.py:93: in load_all
    yield loader.get_data()
.venv/lib/python3.9/site-packages/yaml/constructor.py:45: in get_data
    return self.construct_document(self.get_node())
.venv/lib/python3.9/site-packages/yaml/constructor.py:55: in construct_document
    data = self.construct_object(node)
.venv/lib/python3.9/site-packages/yaml/constructor.py:100: in construct_object
    data = constructor(self, node)
copier/_template.py:92: in _include
    return [
copier/_template.py:93: in <listcomp>
    yaml.load(path.read_bytes(), Loader=type(loader))
.venv/lib/python3.9/site-packages/yaml/__init__.py:81: in load
    return loader.get_single_data()
.venv/lib/python3.9/site-packages/yaml/constructor.py:49: in get_single_data
    node = self.get_single_node()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <copier._template.load_template_config.<locals>._Loader object at 0x7f77911fbd30>

    def get_single_node(self):
        # Drop the STREAM-START event.
        self.get_event()
    
        # Compose a document if the stream is not empty.
        document = None
        if not self.check_event(StreamEndEvent):
            document = self.compose_document()
    
        # Ensure that the stream contains no more documents.
        if not self.check_event(StreamEndEvent):
            event = self.get_event()
>           raise ComposerError("expected a single document in the stream",
                    document.start_mark, "but found another document",
                    event.start_mark)
E           yaml.composer.ComposerError: expected a single document in the stream
E             in "<byte string>", line 2, column 1:
E               !include ../copier_files/nested/ ... 
E               ^
E           but found another document
E             in "<byte string>", line 3, column 1:
E               ---
E               ^

.venv/lib/python3.9/site-packages/yaml/composer.py:41: ComposerError
======================================================================= 1 failed in 9.54s =======================================================================

This is due to the fact that _include in load_template_config is using yaml.load, which does not seem to accept several "documents", that is namely the use of ---, which to my understanding is necessary to both have questions and includes in the same yaml file.

---

This PR

This PR introduces a test to show that failure case in the current state of the repo, along with a correcive patch.
It also adds a note how relative path for nested transclusions.

Test

The test function is tests/test_config.py::test_config_data_nested_transclusions, and it uses demo files from directory tests/demo_transclude_nested.

Patch

The patch basically:
Replaces:

yaml.load(path.read_bytes(), Loader=type(loader))

By:

lflatten(filter(None, yaml.load_all(f, Loader=_Loader)))

Which was already used in the first place to apply the first loading:

    _Loader.add_constructor("!include", _include)

    with conf_path.open("rb") as f:
        try:
            flattened_result = lflatten(filter(None, yaml.load_all(f, Loader=_Loader)))
        except yaml.parser.ParserError as e:
            raise InvalidConfigFileError(conf_path, quiet) from e

This change passes the new test, and works in practice.

Docs

Add a note that precise that nested transclusions have to be made relative to copier.yml, not to each other.

Possible future change or desiderata

Relative path all the way

The paths in the files are relative to copier.yml and not to each other.
It might be better if the file paths are relative to each other instead.
But, at least for now, this way works fine for what I wanted to personally do with my copier template.

Avoid circular transclusion

This is actually not a problem inherent to this patch, this problem is already present in the current master branch.
As these are not directly related to this patch, I will make a different PR to propose a test that results in:

self = PosixPath('/workspace/copier/tests/demo_transclude_circular_bis/demo')

    def __fspath__(self):
>       return str(self)
E       RecursionError: maximum recursion depth exceeded

/home/gitpod/.local/share/uv/python/cpython-3.9.23-linux-x86_64-gnu/lib/python3.9/pathlib.py:671: RecursionError

It could be nice to catch a circular transclusion as soon as it happens instead of waiting for the recursion ceiling, and provides a more comprehensive message for the user such as which files collide.

[sisp]

Thanks for discovering this limitation of nested YAML includes and contributing support for nested multi-document YAML includes, @RR5555! 🙇

These are great discoveries and improvements! 👌 Just one request: The way I see it, this PR comprises 3 aspects, which I'd prefer to split into 3 PRs for better clarity and granularity:

1. Adding support for nested multi-document includes plus a test. This test happens to test also the base path of the includes, which is a different concern, so I'd limit this test to cover nested includes where all files are in the same directory.

2. Testing that nested include paths must all be relative to copier.yml. As mentioned in 1., the test in this PR tests both the base path of the includes and nested multi-document includes. I'd create a dedicated PR with a test that asserts the include path to be relative to copier.yml with only single-document includes.

3. Documenting that nested include paths must all be relative to copier.yml. I'd create a dedicated PR for this.

   Reading this added documentation in the context of the example shown in that subsection, I think the config directory and file names are more abstract. Could we make this a bit more concrete? Perhaps you have a concrete example that you could share that led to the discovery of the problem for which you're contributing a fix in this PR – although this documentation is only about the base path of the relative include paths? If not, feel free to let me know and I'll try to think of something.

[RR5555]

Thank you very much for reviewing my PR ^^

Done

I have:

• Pruned: the docs commit of this PR
• Applied: 1. limit scope of the test to nested includes in same directory
  (see test: limit the test to cover nested includes where all files are in … 1d4cf68)

---

• Created: branches for 2. & 3.
• Populated: the new branches through commit cherry-picking.

I will submit separate PRs for 2. & 3..

Please let me know if other modifications are required for this PR.

Practical case

I indeed had a more practical case leading to the PRs:
I separated my questions to benefit from modularity.

copier.yml
copier_files
|-project_details.yaml
|-Makefile
|-generated
  |-generated_classifiers_include.yaml
  |-project_framework.yaml
  |-project_natural_language.yaml
  |-project_topic.yaml
  |-project_development_status.yaml
  |-project_intended_audience.yaml
  |-project_operating_system.yaml
  |-project_typing.yaml
  |-project_environment.yaml
  |-project_license.yaml
  |-project_programming_language.yaml  

flowchart LR
A[copier.yaml] --- Alink(includes) --> B[project_details.yaml] --- Blink(includes) --> C[generated_classifiers_include.yaml] 
subgraph copier_files
B
Blink
subgraph generated
C --- Clink(includes)
Clink --> D[project_framework.yaml]
Clink --> E[project_natural_language.yaml]
Clink --> F[project_topic.yaml]
Clink --> G[project_development_status.yaml]
Clink --> H[project_intended_audience.yaml]
Clink --> I[project_operating_system.yaml]
Clink --> J[project_typing.yaml]
Clink --> K[project_environment.yaml]
Clink --> L[project_license.yaml]
Clink --> M[project_programming_language.yaml]
end
end

Loading

./copier_files/project_details.yaml asks about the more simple fields of pyproject.toml related to the project, like the repo, the homepage, ...
The generated yaml files are made from scraping https://pypi.org/classifiers/ to ask multi-choice questions on the classifiers by categories to fill-out pyproject.toml>[project]>classifiers with recognized ones.

Hence, I encountered both the nested includes problem and the relative-to-copier.yml path "problem".

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.00%. Comparing base (4831f35) to head (1d4cf68).
⚠️ Report is 107 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2251      +/-   ##
==========================================
+ Coverage   97.78%   98.00%   +0.22%     
==========================================
  Files          55       55              
  Lines        6088     6111      +23     
==========================================
+ Hits         5953     5989      +36     
+ Misses        135      122      -13     

Flag	Coverage Δ
unittests	98.00% <100.00%> (+0.22%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sisp]

LGTM! 🎉 Thanks, @RR5555! 🙏

[RR5555]

Thank you for your reviews ^^