✨ Schema validation v2

[ubmarco]

This is a follow-up on #1441.
I plan not to merge #1441 and merge this PR instead.

Differences:

• Aligning with standards in JSON schema for re-using subschemas via $defs and $ref
• Fully typed implementation including runtime checks of valid schema user input
• Auto inject the default string type if not given
• Replace trigger_schema with select which aligns with query language terminology
• Replace trigger_schema_id with the mentioned $ref mechanism
• New schemas root key validate with sub-keys local and network for the 2 validation types
• Network validation items does not allow the select key anymore as the selection happens by linking target needs.
  This cleans up an ambiguity in the other PR.
• Network validation errors now bubble up to the root json schema and are displayed to see exactly why the chain fails
• More network rule types for better control over debug schema output
• Rewrite test cases to use a declarative definition as yaml, so all pieces can be given in one place:
  • conf.py
  • ubproject.toml
  • index.rst
  • schemas.json
  • expected ontology warnings
• Simplified the code logic
• String patterns (regex) are constrained to a basic subset that works across engines
• Added docs
  • Examples and explanations
  • Comparison with needs_warnings and needs_constraints and migration path
• Many more test cases
• items with minItems and maxItems and contains with minContains and maxContains are now semantically equivalent to JSON schema spec

[codecov]

Codecov Report

❌ Patch coverage is 90.23904% with 98 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.81%. Comparing base (4e10030) to head (3ca68e0).
⚠️ Report is 186 commits behind head on master.

Files with missing lines	Patch %	Lines
sphinx_needs/schema/reporting.py	75.71%	34 Missing ⚠️
sphinx_needs/schema/config_utils.py	89.57%	27 Missing ⚠️
sphinx_needs/schema/core.py	92.77%	19 Missing ⚠️
sphinx_needs/builder.py	57.89%	8 Missing ⚠️
sphinx_needs/needs.py	80.55%	7 Missing ⚠️
sphinx_needs/schema/process.py	93.33%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1467      +/-   ##
==========================================
+ Coverage   86.87%   88.81%   +1.93%     
==========================================
  Files          56       67      +11     
  Lines        6532     8271    +1739     
==========================================
+ Hits         5675     7346    +1671     
- Misses        857      925      +68     

Flag	Coverage Δ
pytests	88.81% <90.23%> (+1.93%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[chrisjsewell]

Initial review comments

(without yet looking over the full implementation of the schema checking)

[chrisjsewell]

This is bad though, the values should be coerced to these from the start, if read from a (string) directive, or just checked if read from a needs.json.
This will definitely be the case in ubcode.
What is the path to change sphinx-needs, so that it can handle storing the data as the correct type?

[ubmarco]

While I agree with you that the internal types should change with the definition, my plan was to do that in a 2-step approach. Users should adopt the ontology in a non-breaking way (switch from warnings/constraints to this, transpile from SHACL, ...). This takes time and efforts. Then in the meantime we change the internal representation which affects at least filter functions, and data/schema in exported needs.json. My current feeling is changing the types is a bigger thing in the codebase that delays this PR. The adoption of the new interface and the schema feedback is already of good value for end users. We can switch the type system while adoption happens.

[chrisjsewell]

see comment, emit a warning

then ok thats it for now; this PR can be merged, but there will be changes that need to be made when I get into it