Performance improvements

This release is focussed predominantly on performance. It includes several algorithmic improvements that eliminate possible exponential processing and in general takes more care to avoid recomputation.

Aside from algorithmic changes, these changes also prepare for numerous further optimizations. The algorithmic changes already already give enough of an improvement (especially for some large files) to warrant another release.

Language change

This release also introduces a slight language change regarding disjunctions. Consider

x: *1 | a
a: *2 | 3

Previously, the result of x would be *1 | *2 | 3, in other words, a default value of 1 | 2. With the new semantics, the default in this case will be 1. In general, with the new semantics, if a disjunction has a term with a default, the other terms can never result in a default value.

Superficially, this makes it easier to see where defaults may come from and eliminates the need to check all terms of a disjunction. But this change also prevents some unintuitive behavior and even cases where making values more specific may lead to a less specific result.

Backwards compatibility

The changed disjunction rule may cause backwards incompatibility.

There are some minor bug fixes as a result of these changes that may change the behavior For instance, [...] | [] was previously normalized to [].

There are also some changes to error messages as a result of these changes.

Changelog

cc41853 doc/ref/spec.md: change disjunction rule
d5b4ca8 internal/core/adt: add logging utilities for debugging
0c3791e internal/core/adt: add memory management
008d37d internal/core/adt: catch errors for circular failures
b619e0c internal/core/adt: consider closed status for equality
396202c internal/core/adt: don't pre-expand disjunctions
69e0c96 internal/core/adt: introduce HasDefaults for Disjunction
514082a internal/core/adt: keep nodeContext in Vertex
a595db2 internal/core/adt: skip partial unify if unnencessary
02ff29b internal/core/convert: fix faulty label update
778719e internal/core/eval: breadth-first disjunction resolution
2be20f8 internal/core/eval: do not precompute matchers
399ba76 internal/core/eval: export fieldSet fields
0ef6caa internal/core/eval: make arcMap a list
1fa69d5 internal/core/eval: recursive disjunctions
9ca9e04 internal/core/eval: separate out dynamic fields
0e1e7b5 internal/core/eval: small cleanup of evaluation code
062b3d8 internal/core/eval: sort empty disjunction message to top of group
359685a internal/core/eval: use env from fieldSet
d4c600f internal/core/export: extract docs from nested Vertex values
c62d8a0 internal/core: merge adt with eval package
cea55b2 internal/core: reimplementation of closedness algorithm
93c79a3 tools/flow: record dependencies per value based on path

Closing the gap with the language spec

This release is characterized by closing some significant remaining gaps in with the language specification.

There are the usual bug fixes and improvement to error messages, though.

Language

The language has now been brought closer to the spec, implementing some of the remaining gaps. There have also been some simplifications that will allow further generalizing the language down the road more easily and without introducing backwards incompatibility.

Multiple comprehensions per list

As the spec already allowed, it is now possible to include multiple comprehensions per list.

For instance:

src:    [ "John", "Mary" ]
cond: true
a: [
    for x in src { x },

    if cond { "Sasha" }
]

The list operators (concatenation and multiplication) are now also implemented in terms of comprehensions. There is a great deal of confusion about list operators (is the result open, closed, when do overlapping constraints apply?). This confusion is eliminated by writing the equivalent as list of multiple comprehensions. Our intention is to remove these operators and have comprehensions (and later queries) be the only way in which lists can be concatenated or multiplied. Of course, cue fmt will help in the transition there as usual.

Embedded scalars

Just as it is possible to write

[ #a, #b ]

#a: 1
#b: 2

at the top of a CUE file, it is now also possible to have an embedded scalar along with definitions in any struct:

x: {
    [ #a, #b ]

    #a: 1
    #b: 2
}

Originally, the top-level construct was allowed to make CUE strictly compatible with JSON. Generalizing this to any struct makes the use of imported file objects consistent with normal structs. Moreover, it turns out that this construct can be quite useful overall.

Old CUE code is not affected by this change.

Semantics of [pattern]: value fields (no change after all)

The spec recently included a change that proposed to use the JSON schema semantics for pattern constraints ([k]: v), rather than the old CUE semantics. To aid in the transition, cue fmt rewrote these fields to be enclosed in curly braces, which mimics the old semantics under these planned rules.

It has now been decided to keep the old semantics and stop rewriting old uses.

As it turns out, the old semantics was used quite a bit and was actually the majority use case. The argument previously was that the JSON schema semantics was not easily expressible in terms of the old semantics, but that the old semantics is easy to simulate with the JSON schema semantics.

Instead we now intend to allow the JSON schema semantics additionally by writing [...k]: v. The nice thing here is that any pattern which is to be matched additionally, is now always preceded by a .... It also is a nicer generalization of the syntax.

But above all, it removes the need for a potentially painful transition.

Removed integer division operators

CUE allowed integer division and remainder per the div, mod, quo, and rem operators. These were relatively rarely used and complicated the CUE syntax, formatting and other parts of various subsystems.

Moreover, they made it lexically impossible to allow indexing of the form a.3 (instead of a[3]) down the road. We are not certain if we will allow this, but allowing it makes the syntax considerably more regular and consistent. With this change, we have the option to do so in a backwards compatible way.

These operators have now been replaced with builtins. As usual cue fmt will rewrite old uses to use the builtin for some time to come. It will use the __foo variant to avoid shadowing issues. These can be manually rewritten to remove the leading underscores if necessary.

Variable argument builtins

The model for builtins, which are still underspecified in the spec, is narrowed down more. It is now possible to define variable argument builtins.

The main reasons:

1. Variable argument builtins are quite useful in being able to extend APIs in a backwards compatible way,
2. It allows introducing some environment dependence in a clean, intuitive and nonetheless hermetic way.

The first target was pkg/path which now supports multiple OSes (see below).

Bug fixes

Errors in nested comprehension clauses were sometimes dropped, causing a failure in a nested if clause to be silently ignored. This has now been fixed.

Other fixes includes crashes related to let clauses and bytes interpolation.

Error messages

There are various areas where error messages have been improved.

cue fmt

There are various printing bugs that were fixed. Overall the stability of fmt should have been improved notably.

API

The API got a few fixes related to backwards compatibility and should now be more like v0.2.0 where this makes sense.

pkg/path

This package has now been extended to support multiple operating systems simultaneously. Most builtins have been extended to allow an additional optional argument to indicate the OS for which the builtin is to be applied. For instance:

path.Split("foo\bar", os)

will now return a different value depending on whether os is path.Windows or path.Unix. The os argument can also be omitted, in which case it defaults the existing behavior, which is identical to path.Unix.

Note that os will not default to the currently running OS. This would make the behavior non-hermetic. The idea is that the existing injection mechanism can be used (or a small extension of it to make this specific case easier) to inject an environment-specific value.

Backwards compatibility

The cue tool is now a bit more aggressive to panic if it encounters an internal inconsistency error. In return, it now allows the user to define a CUE_DEBUG=0 environment variable to disable this. In many cases the crashes can be ignored while still getting useful results.

Not using this option and filing bugs can help improve the stability of CUE though.

Main backwards incompatibility to note:

• removed integer division operators (now builtins). This can be fixed by running cue fmt.
• some of the bug fixes may cause CUE files that previously succeeded erroneously to fail now.

Changelog

e77ccb1 cmd/cue/cmd: don't bail for non-matching files
fecdd83 cmd/cue/cmd: ignore possible top-level tasks
bbe493a cmd/cue/cmd: more position information for field errors
fe73e8f cue/format: format let clause
110d0bf cue/format: pad floats with leading or trailing periods
fac5d2e cue/load: allow ./... to load all packages
555fb73 cue/parser: fix comment placement
1c904cc cue: allow multiple comprehensions in lists
8872b98 cue: fix PathCorrection tests
6c49cf0 cue: implement embedded scalars
d5177fd cue: map API values to more native ADT
b687b7f cue: remove aliases used for refactoring
19f6f3f doc/ref/spec.md: perpare for allowing a.4.f
f52a0ed doc/ref/spec.md: revert to the old semantics of [k]: v
76ea22c internal/core/adt: add Assert for debugging
37293f9 internal/core/adt: add methods for concreteness
41afc87 internal/core/adt: automatic renaming
2d568d3 internal/core/adt: better message for permanently non-concrete values
f62bfed internal/core/adt: clean up Builtin and Validator semantics
4db8ffb internal/core/adt: do list artihmetic with comprehensions
cbcb701 internal/core/adt: error message improvements
d0dd888 internal/core/adt: improve error message for optional fields
08a1652 internal/core/adt: introduce base value type
6de877a internal/core/adt: make progression check an assert
8b13752 internal/core/adt: move decimal logic into one place
b8c852d internal/core/adt: pass errors up in nested comprehensions
36f2051 internal/core/compile: add integer division builtins
9c6ded8 internal/core/compile: don't panic for incorrect labels
13e4af1 internal/core/compile: fix issue in and builtin
ff3e32f internal/core/convert: allow results of math to be integer
31896af internal/core/eval: add more tests for embedding
d49777f internal/core/eval: better incomplete error messages
55602d9 internal/core/eval: fix closedness issue for lists
9f1fec6 internal/core/eval: hoist vertex processing logic
cefe38b internal/core/eval: keep semantics for [K]: T
acc35f1 internal/core/eval: track fieldSets for data values
e7e27e0 internal/core/export: allow showing recursive errors.
039fb59 internal/core/export: fix bytes printing bug
2ea30a2 internal/core/export: fix crash with let comprehension export
16809ea internal/core/export: support byte interpolations
63594ef internal/core/runtime: assign first label to _.
0811662 internal/core/subsume: tests embeded scalars
a4d0928 internal/core/subsume: various bug fixes
136e51b internal/core: allow variable arg functions
e841714 internal/core: move equality checks from eval to adt
d174bc0 internal/cue/adt: remove use of Default in node
fa6308f internal/encoding: don't clobber files on crash
ec6f95d internal/filetpes: improve error messages
fb3b02e pkg/internal/builtin: use normal formatting for builtin test results
a50960c pkg/path/testdata: adapt Go implementation
a2692ef pkg/path/testdata: move files to minimize diffs
c77b9b0 pkg/path/testdata: stage filepath files from Go
85f8b46 pkg/path: activate OS-dependent version
64434c4 tools/fix: rewrite integer division
d5ced74 tools/flow: ignore faulty tasks outside the root

Bug fixes and error message improvements

This release is mainly focused on various bug fixes and error message improvements. This change also continues to factor out code from cmd/cue to make the functionality available outside of this tool.

Below is a selection of the fixes.

Although this is officially alpha, it is probably a more solid release to use than v0.2.2. For this reason the "Pre-release" tag has been left off from this release.

Language

Cycle handling

Various bugs related to cycle handing were fixed, including hangs and spurious detection of cycles.

Closing gaps with the specs

Hidden fields are now scoped per package.

Package pkg/sort

Sort now always uses stable so that reproducible outcomes between different compiles of CUE can be guaranteed.

API

cue.Path

The introduction of #-style definitions are not handled well by the old API. This is a consequence of them living in a separate namespace, which is not expressible in the old API.

The new cue.Path type is introduced to start addressing this. The first method that supports this is LookupPath.

This API is designed to be compatible with the planned query extension.

Package tools/flow

This new package allows arbitrary user-defined CUE schema to be interpreted as task definitions, automatically handling dependency analysis and scheduling tasks based on these dependencies.

This was written from scratch and makes use of a new native dependency analysis package, allowing for more precise analysis.

cue command

Go bridge

The translation of CUE to Go is now brought in line with Go’s encoding/json, fixing some bugs that caused the interpretation to differ.

cue cmd

This now uses the new tools/flow package, fixing a bunch of outstanding bugs. Most notably, it now allows dynamic task creation, for instance by using comprehension, and depending on such dynamic tasks: $after should now only be needed if there are really no dependencies at the CUE level.

Changelog

b4aa96d all: implement hidden identifiers scoped per-package
aa87887 ci: drop use of set-env in CI scripts
4cda4a3 cmd/cue/cmd: allow inlined embedding of pointer types
c715b94 cmd/cue/cmd: use tools/flow
e05eee7 cue/errors: correct handling of wrapping list errors
54b13db cue/literal: fix multiline quotes
6165736 cue: allow access to Selectors of Path
f4f2a02 cue: dig up wrapped error in valueError
74329a1 cue: doc improvements
56eb4b1 cue: generate error node rather than panic on faulty output.
03abe87 cue: remove most uses of xerrors
409dacf cue: support getting and looking up paths
fcd96de doc/ref/spec.md: define "regular"; fix typos
8661255 encoding/protobuf: fix trailing comment in oneof bug
b5821dc interanal/core/runtime: move building logic from cue package
b886b0f internal/core/compile: fix mutual dependent let clauses
3585705 internal/core/convert: fix for embedded structs are not honored
b99fd08 internal/core/cue: fix hang on cycles in embeddings in comprehensions
c60e115 internal/core/dep: first stab at precise dependency analyzer
7463d11 internal/core/dep: support dynamic dependencies
1888d65 internal/core/eval: fix bug in cycle handling
1d8c688 internal/core/eval: fix bug with nested embeded comprehensions
aee9955 internal/core/eval: fix cycle bug for comprehensions
1a2105e internal/core/eval: fix hang in cyclic in comprehension evaluation
3cfb4ab internal/core/eval: fix irregular dynamic fields
d1cbe10 internal/core/eval: fix spurious cycle detection
635fbdd internal/core/eval: fix spurious cycle for bulk constraints
7980ec5 internal/core/eval: handle disjunction failure more properly
ab0a2f3 internal/core/eval: re-allow ... with embeddings
30704a7 internal/core/eval: reduce per-node allocations
1e0faf0 internal/core/export: dedup let clauses for merged structs
187c734 internal/core/export: handle alias and lets
cd621ff internal/cue/eval: fix faulty status in tentative result
dbfa73b internal/cuetxtar: allow multiple golden files
f2a4a42 internal/filetypes: hard-wire common case
737a103 internal: hoist a single embedding in ToExpr
99d18dc pkg/list/sort: always use stable sort
00f345b tools/flow: API for CUE-based workflow engine
d3ff4a1 tools/flow: add IgnoreConcrete option

Language refinements and fixes

This is a relatively small release. But given that it contains some significant bug fixes and performance improvements, it seemed useful to get these out sooner.

This also

Closedness refinements

The rules surrounding closedness have been further clarified in the spec where this was previously ambiguous. Most notably, it was previously unclear whether B.b in

#A: {a: int}

B: {
    #A
    b: c: int
}

should be closed or not. This is now clarified to it not being closed. In other words, fields added alongside an embedded definition that were not part of the original embedding are not closed automatically.

This does not change the current behaviour.

There was also a bug fix related to closing structs that could leave struct open if they were referenced through a selector path like #A.b, where the reference to #A failed to trigger a close.

There remains one known gap in the implementations compared to the spec: embedding a closed struct will currently cause the field to be closed, whereas the spec says that the struct in which it is included should already be closed. This will be a backwards-incompatible change.

field not allowed error

As far as is known, aside from the spec deviation mentioned above (which is somewhat esoteric), there are no more known deviations that result in a spurious error of this kind.

There is, however, one known case where this error seems to appear erroneously. Consider the following:

#a: {b: int} | {}
b: #a & {b: string}

This may still result in a “field b not allowed” error. The problem is that the error reporting around disjunctions is still lacking. What happens is that only the error of one of the failed disjuncts is reported. In this case it fails on the first disjunct (as int != string) and the subsequently fails on the second, which doesn’t allow b. As only one of the errors is reported, it may appear that it fails on the first disjunct, which is just confusing.

The intention is to considerably improve disjunction errors soon. In the meantime, the added position information for conflict errors will hopefully help a hand to disambiguate such errors.

let clauses are now allowed in comprehensions

This has been a part of the spec for a long time, but now has finally been implemented.

... anywhere in struct

This was already allowed by the spec, compiler, and evaluator, but now is also allowed by the parser.

Simple string literals now allowed as selector

Another thing that has been allowed by the spec for a while is the use of string literals as selectors. So a.”foo”.bar is now allowed. Only single-line strings without interpolations are allowed at the moment (as per the spec).

Command line tool

cli.Ask

This is now supported.

Error reporting

Error reporting has been improved further. Most notably, positions for conflict errors (as resulting from 3 & 4), now have positions added and should be on par with v0.2.2. This also adds the fundamentals for adding richer position notation.

Performance

This release fixes one big performance issue, which could lead to exponential runtime for disjunctions. This can have a big impact for many configurations. For one user, it reduced the running time from 5 hours to less than 2 seconds.

As part of this fix, we now introduced a benchmark suite.

There are tons of significant performance improvements still possible, so this is not the last of it.

Notable Bug fixes

String handling

CUE used to piggyback on Gos strconv.Quote` for quoting strings. This was not ideal, as CUE and Go strings are not the same. It also resulted in some bugs. This is now fixed. This results in better string formatting overall.

Handling of files named -

The cue command line tool sometimes hung when a directory contained a file named -. This is now handled.

Yaml version

Until now, CUE would output YAML in version 1.2 and read it in version 1.1. And it seems that the “right thing to do” is to implement version 1.2 these days. This has now provisionally been fixed to read it mostly as version 1.2. A more proper implementation would be to replace internal/third_party/yaml to use the yaml.v3 package.

This means, for instance, that the value no is now treated as a string instead of the boolean value false. (And thus the demo on cuelang.org will have to be replaced.)

Message-level options in Protobuf

Handling of message options in .proto files would previously fail. They are now translated into declaration options in CUE with the format: @protobuf(option name=value).

Backwards compatibility

YAML

As YAML 1.2 is not backwards compatible with YAML 1.1, inputs to CUE that relied on a YAML 1.1 interpretation may now yield different results.

Closedness rules

The refinement of the rules of closedness are largely backwards compatible (and what is not is not yet implemented), but some of the bug fixes may give slightly different results nonetheless.

Changelog

8ad07fa Readme.md: remove Go Report Card
2ac4d85 cue/format: indent multiline string literals based on context.
c886094 cue/literal: implement CUE-specific quoting
11b507c cue/load: don't hang on files named -
6354c40 cue/parser: allow ... anywhere in struct
241c5bf cue/parser: allow let clause in comprehension
80f0bee cue/parser: better error message for old-style definitions
732b6d5 cue/parser: support string selector labels
e9286f0 doc/ref/spec.md: fix builtin func
ff306b7 doc/ref/spec.md: fix spec bug and broken link
15e8a05 doc/ref/spec.md: refinement of closedness
65468d5 doc/tutorial/kubernetes: update import
3af2683 encoding/protobuf: parse message options
d23402c internal/core/compile: fix alias resolution bug
742593f internal/core/compile: fix hidden definition bug
da69dc0 internal/core/eval: ... does not apply to defintions
0eb7cc5 internal/core/eval: add more positions to conflict errors
3914ef8 internal/core/eval: check for nil pointer
03fa5a8 internal/core/eval: fully evaluate nested unification
f0df4df internal/core/eval: performance: unify an arc only once per node
1fdc02a internal/core/eval: simplify validators upon evaluator
1c29042 internal/third_party/yaml: drop support for non-standard scalars
5bfe37d pkg/tool/cli: support ask task

v0.3.0-alpha3

This release mainly introduces per-file inclusion based on tags, bug fixes for bugs introduced in the new evaluator, and tests to close issues raised on the old evaluator that are now verified to be fixed.

Language extensions

Attribute declarations in preamble

Attribute declarations are now allowed before the package clause and import declarations. This is used to implement file-wide inclusion.

Any scalar now allowed in interpolations

This includes boolean and byte values. Boolean values are represented as their JSON value. In string interpolations, bytes are converted to valid UTF-8 by replacing the maximal subpart of ill-formed subsequences with a single replacement character (W3C encoding standard).

Whole file inclusion

CUE now allows files to be conditionally included in a build. To conditionally include a file in a CUE program, one can now write:

@if(prod)

package acmecorp

and use the -t prod flag to cause this file to be included. The expression can be any valid CUE expression using only boolean operators and boolean values.

See cue help injection for more details.

Closedness rewrite

A large number of bug fixes related to an incorrect “field not allowed” error are now in. The algorithm for closedness has been significantly simplified and streamlined based on the benefit of hindsight.

Note that one may still have a misleading “field not allowed” error if a field is disallowed in one disjunct and allowed in another, but where there is another field in conflict. Right now only one of the errors is reported.

CUE API

New evaluator

The CUE API is now moved to the new evaluator.

Package dependencies

Builtins now live in their own package and are no longer copied into the cue API. This was part of the effort to break the implementation into logical pieces, instead of having one large package.

This has the additional advantage that each builtin package now only lives in its own directory and is not copied. This makes maintenance easier and also removes some of the limitations imposed by the old implementation.

Unfortunately, this means that custom loader implementations (ones that do not use cue/load for loading packages) will have to include

import 	_ "cuelang.org/go/pkg"

somewhere in the package to trigger linking in the builtin packages.

cue tool

This release also marks the first steps of moving functionality from the cue command line tool to the CUE APIs. The intention is that the cue tool will be mostly a wrapper around core functionality present in the API. Other tooling could provide a different “view” without being second-class citizens in the ecosystem and while providing consistency between such tools.

The first step towards this is the injection mechanism. The existing parts of this were moved to the cue/load package. The new file inclusion functionality, meanwhile, was directly implemented in cue/load.

The next step will be to move non-CUE file support and packaging mechanisms to cue/load and friends. Then next up is the tooling layer, starting with an API for dependency analysis.

Next steps

There are two outstanding issues on the table to come to a beta release:

Disjunctions

Although there are no known semantic bugs related to disjunctions, the code needs some restructuring to improve error messages and to fix some performance issues. This is slated for the alpha4 release.

Structural cycles

There are still some bugs outstanding related to structural cycles. The plan is to address all of these in an alpha5 (or possibly beta) release.

Backwards compatibility

Go version

Support for go1.12 was dropped. In exchange, this release is verified to build with Go’s tip (1.16).

CUE API

The CUE API has now fully been moved over to use the new evaluator. The old evaluator code has been removed.

• cue.Merge: the purpose of Merge is to unify the underlying instances as data. It now no longer also unifies definitions. Bug fixes in closedness exposed this previously unnoticed oversight.

Custom loader implementations (ones that do not use cue/load for loading packages) will have to include

import 	_ "cuelang.org/go/pkg"

somewhere in the package to trigger linking in the builtin packages.

Language changes

CUE code that previously expected a boolean or bytes value substituted in a string interpolation to fail my now break.

cue cmd

Commands that operate on multiple instances simultaneously (e.g. cue cmd foo ./...) can no longer reference definitions in the underlying commands. For more info see cue.Merge in the API section.

Changelog

d69319c all: add missing copyright
4c10692 all: fix four previously ignored errors
6c27cef ci: drop go1.12 add go1.15
c3f30d8 ci: name dispatch workflow differently to default branch push workflow
1c54297 cmd/cue/cmd: fix get go package handling
1e8906a cmd/cue/cmd: move injection mechanism to cue/load
faf617c cmd/cue/cmd: verify issue 236 is fixed
e71970c cmd/cue/cmd: verify issue 322 is fixed
8776561 cmd/cue/cmd: verify issue 425 is fixed
ba5708b cmd/cue: get go: add --package flag
1a9b88d cmd/cue: get go: use constant.Value.ExactString to extract constant
f25c04d cmd/help: fix example in doc
089f461 cue/load: add support for build tags
bc101b3 cue/load: add tool and test files to BuildFiles if requested
20c637a cue/load: relax places where @tag is allowed
bd3dd75 cue/load: rewrite pre-resolved files after injection
c264475 cue/parser: allow attributes before package and import clauses
4cce6c4 cue: apply wrapping to all errors in list
d2fdbf0 cue: clean up handling of preamble declarations
80e70b9 cue: fix instance error handling
c76a530 cue: implement fill in terms of Value.Fill
845df05 cue: make internal cue port main API implementation
abce145 cue: move to external builtin packages
ce28b20 cue: prepare to hoist builtin code
9acd624 cue: prepare to hoist builtins, step 2
40a6bcd cue: remove dead code
6f84d11 deps: upgrade to github.com/rogpeppe/go-internal v1.6.1
edea6f3 doc/ref/spec.md: fix len count for structs
043c534 doc/ref/spec: clarification on cycles
dd0fa88 doc/ref/spec: consider definitions for closedness
c7c14fd doc/tutorial/kubernetes: remove a duplicate kind field
e3a03a1 internal/core/adt: exclude definitions in data conversion
db18b37 internal/core/adt: prevent nil interface bug for Source
f5fa009 internal/core/adt: remove all closedness IDs in ToData
30ca062 internal/core/adt: support interpolation of bytes and bool
df01042 internal/core/compile: better error message for user-defined errors
d55e2b5 internal/core/eval: allow lists resulting from dependent expressions
c72ce5d internal/core/eval: fix empty data vertex bug
0b57e43 internal/core/eval: fix interpolation of numbers
e0c19d1 internal/core/eval: fix issue 507
304b02e internal/core/eval: nested comprehension fix
62528c3 internal/core/eval: rewrite of closedness algorithm
dc2401c internal/core/eval: verify issue 258 is fixed
ddd99d5 internal/core/eval: verify issue 293 is fixed
ea5d964 internal/core/eval: verify issue 295 is fixed
480e1a1 internal/core/eval: verify issue 299 if fixed
541bf03 internal/core/eval: verify issue 306 is fixed
5905c22 internal/core/eval: verify issue 332 is fixed
121ace7 internal/core/eval: verify issue 342 is fixed
07ce2f6 internal/core/eval: verify issue 353 is fixed
4ecec98 internal/core/eval: verify issue 370 is fixed
fbf9de3 internal/core/eval: verify issue 398 is fixed
c6ade67 internal/core/export: fix bug 473
b9b2c54 internal/core/export: better handling of generated CUE
1e1fe6f internal/core/export: handle scalar values in adt.Vertex
67d1d42 internal/core/export: suport default picking
df9468e internal/core/export: verify issue 349 is fixed
60c207f internal/core/runtime: collect errors for all files in resolve
a6ce869 internal/core: finalize list in slice
148f67a internal/core: fix incorrect error handling
3d863fc internal/core: move CloseID from Environment to Conjunct
bf882b0 pkg/internal: copy cue builtin code to separate package
dda7de4 pkg/math: fix issue 418
4f4b7a3 pkg/tool/file/append: create file if none exists.
c8a4514 pkg/tool/http: fix handling of errors
248794e pkg: generate per-directory tests from builtin_test.go
1bcb73a update go-internal to pull in fix for Go 1.16

v0.3.0-alpha2

DO NOT USE THIS RELEASE

v0.3.0-alpha1

This release implements the long-anticipated move of cue to the new evaluator. This particular alpha release only switches over the cue command. A follow-up release will switch over the API as well. This allows people to upgrade to the tool without having to worry about API breakage just yet.

Note that this is an alpha release. This release includes an almost complete rewrite of CUE core. Although it will see many benefits, it can be expected to need some more ironing out. This includes new bugs in semantics and performance. But it also means dealing with bugs in the old evaluator that are now uncovered.

Overview of new capabilities

This release of the new evaluator mostly aims to address old bugs and doesn’t introduce many new things. However, this still means it addresses some important things:

Considerable performance improvements. More on that later.
Many bug fixes: the new evaluator is designed with embeddings, definitions, and optional constraints in mind, instead of them being an afterthought.
Structural cycle detection: this should put an end to stack overflows (but see caveats below).
Non-monotonic constraints: this allows things like structs.MinFields(3) to work. This also enables the implementation of things like not (for JSON Schema support), association lists, and all sorts of other goodness.

A more official ticking off the box of existing issues is still forthcoming.

Under the hood it has an almost complete implementation of the following features:

JSON Schema semantics for pattern constraints (bulk optional fields) and additional constraints (allowing ...T in structs as well).
Embedded scalars: embedding scalars is allowed at the top-level, but now would be allowed anywhere: with the same rules that apply at the top-level. For instance, { "foo", #field: 2 } would be allowed.

These features will be enabled in a later release (in most cases by simply relaxing the parser), once they are better tested.

Under the hood, the new evaluator has been broken up into about 10 components, instead of being one large monolith, separating out clear responsibilities. The largest package is now about 1/15th the size of the original implementation. This clarifies dependencies and responsibilities.

Things that are known to be broken or incompatible

Tooling overlay

Previously, *_tool.cue files were interpreted in a different namespace that lived as an overlay over the other configurations. This was hard to explain for users and also results in many awkward situations, like the inability to print _tool.cue files.

This functionality is not implemented in the new evaluator. This means the command section is merged in with the other files. This may cause clashes.

The idea behind not implementing it is that the new #-style definitions, which live in a different namespace, allow for another, better approach: commands can simply be declared as definitions to avoid clashes with regular fields.

For cases where the user also has no control over the definition names, the idea is to allow for a convention where a tooling logic can be in a separate pkg_tool package to accompany a regular package, not unlike a foo_test package in Go. To CUE this would then just be a regular package. Feedback appreciated.

if comprehension fix

The old evaluator used to silently ignore an error of the condition of an if clause, making it the same meaning as false. This was incorrect and led to several bugs. The new evaluator now treats errors as errors.

If the old behavior was desired, one can achieve the same result using disjunctions. One such case was:

foo: string
if len(foo) > 0 {
}

The old (faulty) behavior can be simulated writing this as follows:

foo: string
if *(len(foo) > 0) | false {
}

Disjunction bug fix

In the old evaluator, sometimes allowed a selector to index into an unresolved disjunction, causing (null | { c: 2 }).c to resolve. This is against the spec and is always illegal in the new evaluator. So one may find cases where one will have to write (d&{}).c to first resolve the disjunction.

Error messages

Please don’t send bug reports on faulty error messages for this release.

The new evaluator implements a different approach to error message generation. This allows the creation of very tailored and detailed error messages. It is just not done yet for the most part. Net effect:
a lot of the line number information is missing,
the path is sometimes missing (but also improved in some cases, as the path information is now more reliable),
some of the error messages are still sloppy.

Reentrancy limitations are now enforced

This means CUE should no longer crash on structural cycles.

Please submit bug reports if CUE hangs in an infinite loop or has a stack overflow

The spec disallows structural cycles. This includes reentrancy for the form

foo: { n: int, out: foo&{n:2}.out+1 }

The new evaluator also has some performance stats built in. This showed that CUE really isn’t great at doing reentrancy and will quickly venture into unexpected exponential behavior.

CUE’s strength is to combine results of computation, not do the computation itself. So instead, the way forward for CUE is to make it easier to shell out to other languages like Go, Python, Javascript and/or WASM.

That said, at the moment shelling out to other languages can only be done by creating custom invocations in the tooling layer. A convenient way to do this is not yet supported.

For the time being, we could consider supporting a structural cycle allowance e.g. as a command line flag or environment variable that allows a certain number of structural cycles to be ignored by CUE. This could help people transition off the use of reentrancy. Feedback appreciated.

Printing

Printing of CUE values has been completely rewritten and as a result will result in the inevitable differences. CUE now has a stricter distinction between values and expressions. This may cause printed values to be rendered differently. Also, printing of definitions still needs some work and can be expected to have some bugs.

There will always be the inevitable cosmetic changes, but:

please report where changes of printing results in bad incompatibilities.

Field order

CUE used to print fields in the order that a field name was declared anywhere. This often resulted in fields being sorted in schema order, but this was clearly not always the case.

The new evaluator does a topologic sort on fields before printing them. This means that fields will observe the order of fields of all structs that combine into a result. Insofar there remains ambiguity, the fields will be combined arbitrarily. In case different structs give conflicting orderings, the result is undefined.

The algorithm does not yet take the relative ordering of embedded structs into account.

tools/trim

Trim relied on very intricate behavior of the API as well as various deprecated API features. It made assumptions that no longer held as the language progressed even in the old evaluator.

The code has been adjusted to use the new APIs. It now often does not remove the top-level element that was inserted by a comprehension. Otherwise it should largely give the same results. The new evaluator also fixes some inconsistencies that fixes known bugs in the old evaluator that failed to remove some fields in some cases.

Other things to watch out for

Cycle handling

One of the benefits of the “always evaluate everything” approach is that CUE can annotate a result of evaluation with cycle points. This means it is possible to rely on a single point where cycles are detected. This is great, as cycle detection is complex! Previously, stack overflow detection was implemented in many components, each with their own peculiarities.

This means that cycle detection has been removed from many components. As a result, there may be bugs in the assumptions in components like the printer, for instance, that will still result in cycles. For instance, printing debug strings of values that are being evaluated may be treacherous. In general, though, the idea is that the evaluator clearly marks cyclic points in the graph, forcing an API to make a conscious decision about descending into a cycle.

Please submit bug reports if CUE hangs in an infinite loop or has a stack overflow

Note that although cycle detection is complex, it is not expensive. The implementation is based on unification algorithms that have cycle detection as a nearly free side-effect. The cost is currently minimal and can probably be improved still.

Performance characteristics

CUE now always evaluates an entire configuration value. This is required by the spec to be fully correct. It also simplifies things greatly, and is paramount in supporting certain new capabilities. It does mean that the API may have very different performance characteristics. cue.Unify may do more than expected, while cue.Validate now just collects errors from an evaluated tree so may run a lot faster.

An aim of the new evaluator was to remove any gratuitous exponential computation. The old evaluator could run into exponential behavior for aliases, for instance. The new algorithm is a non-copying unification algorithm: it therefore no longer needs to descend the graph to make copies. This was a performance quagmire in the old evaluator.

The new evaluator has been designed to allow for O(n) execution under certain circumstances (not using comprehensions or disjunctions without discriminators). It has not been implemented that way though! Currently the performance characteristics can be expected to be strictly better than the old implementation, though, barring having a worse constant in some cases.

_That said, there may still be performance bu...

v0.2.2

This is the last release before commencing the move to the new evaluator.

Backwards incompatible changes

yaml.Validate

This now uses Unification again. This brings it in line with the json package.

This change was done to facilitate moving to the new evaluator. This seems no longer necessary. Either way, using subsumption for validation seems wrong. It is better if the desired validation can be specified in the language itself.

Language feature support override

This is the last version that supports :: under the hood and the last release that will allow moving from :: to new-style definitions using cue fix. The cue tool now supports the CUE_SYNTAX_OVERRIDE environment variable to specify the features to allow at a specific version. So even though this feature disallows :: by default, it can be allowed again by setting this environment variable to v0.1.0, for instance.

This will not re-enable features that have been permanently removed.

OpenAPI fixes

OpenAPI is not always compatible with JSON Schema. For instance, it may use an older version. CUE used the new interpretation of exclusive ranges, whereas OpenAPI v3.0.0 and before required the old interpretation. Both interpretations are now supported and are switched based on the selected version (an API feature).

cue completion

This release exposes the Cobra command completion feature. This is currently only tested for bash and is known to not work, for instance, for Fish.

cue get go

This will now generate unexported constants using the _# prefix.

API

Fill

Now allows a cue.Value to be set anywhere in a Go value.

Builtins

list.SliceRunes

Allow slicing of strings per rune.

Embedding in NewStruct

Added ast.Embed helper to specify embeddings in ast.NewStruct.

Testing

Most txtar-based tests now support the CUE_UPDATE environment variable for updating tests. This allows running

CUE_UPDATE=1 go test ./...

to update all these tests at once. It is not possible to use the --update flag for a bulk update, because not all tests implement it, causing such tests to fail.

Changelog

29fa119 cmd/cue/cmd: allow overriding syntax limitation
ab43a15 cmd/cue/cmd: include tool and test files in fix
94db2eb cmd/cue/cmd: include unexported constants for get go
f0f62b8 cue/ast: allow use of Embed in NewStruct
10fbc21 cue/ast: fix typo
a20e1d4 cue/errors: don't equate error messages without line info
0ccdee2 cue/gen.go: fix bug in generation.
1784fca cue: add test cases
097cf16 cue: allow Value to be set at any path in Fill
8f07f84 cue: print hex numbers correctly
ed61df2 cue: return empty struct for disallowed unification
21ca371 doc/ref/spec.md: alternative definitions for closedness and friends
ecfbd72 encoding/openapi: date-time & string format
616d9e8 encoding/openapi: fix generation bug in CRD path
d3c0096 encoding/openapi: update tests
c615c91 encoding/yaml: fix bug where an empty document is not treated as null
9ebfa80 feat: add completion command
16a2b89 internal/cuetxtar: add Load function
97b215f internal/cuetxtar: allow CUE_UPDATE env variable.
8a0ac3e internal: add ToStruct helper.
57f8242 pkg/encoding/yaml: validate concreteness instead of instance of for Validate
f01cfc5 pkg/strings: add SliceRunes

Prep for new evaluator and JSON Schema/OpenAPI fixes

Disabling of previously deprecated language features

The new evaluator will no longer support some of the legacy syntax. Also, it will adopt JSON Schema semantics for bulk optional fields (to be called pattern constraints) and will have more flexible semantics for ellipsis to allow encoding “additionalConstraints” more easily.

To allow for a smooth transition, this release disables the older deprecated semantics and syntax to remind people to run cue fix to upgrade their configurations, as cue fix won’t work for these deprecated features once the new evaluator comes around.

API changes

To facilitate moving the API as well, LookupDef now automatically adds a # prefix if a field cannot be found and is not a valid definition in the new syntax. This is an unambiguous addition.

JSON Schema and OpenAPI fixes

Exclusive minimum and maximum legacy support

The current OpenAPI implementation used the current JSON Schema semantics for representing exclusive minima and maxima. OpenAPI v3.0.0, however, mandates the use of the deprecated encoding. The OpenAPI generator now generates such constraints using the legacy syntax. The OpenAPI generation API now has a Version field that allows enabling the new behavior if set to 3.1.0.

As for now, the JSON schema reader will accept the legacy encoding unconditionally in addition to accepting the standard encoding.

$ref changes

OpenAPI defines that a$ref replaces all other fields in a schema. The JSON Schema RFC seems to allow at least some fields to be alongside $ref and there are some side comment notes that seem to imply that this is explicitly allowed. In order to avoid any confusion, however, $ref is now wrapped in an allOf if it may otherwise cause a conflict.

JSON Schema model fixing for mixed type fields

The original JSON Schema made the erroneous assumption that a constraint for a certain type implies that the value must be of this type. This is not the case. So

type: [string, integer]
minimum: 3
maxLengh: 4

will now be encoded as

strings.MaxRunes(4) | (int & >=3)

cue get go

The cue get go tool now also converts Go constraints encoding the field tags. This bring is in line with the API converters.

The tool now also has a --local flag to allow generating files in the Go package directory, rather than the corresponding cue.mod/gen subdirectory. This is not set to the default as to not break existing behavior.

Changelog

ddc8eb4 .github: disable generation of schema
083afe7 all: update most tests to not use ::
78cc785 ci: fix vendor path of GitHub workflow definitions
c47316f ci: only run gorelease check on latest versions of Go
2f69bfa ci: re-add Go 1.12.x to the build matrix
019060d cmd/cue/cmd: extract cue constraints in tags in get go
4d31efd cmd/cue/cmd: get go: import files to cue module if in main module
8bfdfe3 cue/ast/astutil: add Expr to File conversion (ToFile)
f5315d4 cue/ast: don't require ellipsis to be at end of struct.
197f4c8 cue/ast: introduce NewNull()
53d18cc cue: allow omitting leading "#" for LookupDef
7efeb70 cue: deprecate :: and combining bulk optional fields with other fields
cb21170 encoding/gocodec: Validate helper function
7aa2eb7 encoding/jsonschema: add support for OpenAPI nullable
a91b869 encoding/jsonschema: constraints don't imply types
eb7706b encoding/jsonschema: fix printing of error message
91d267a encoding/jsonschema: record more line information
50b9067 encoding/openapi: wrap $ref in allOf when needed.
2a8b4ed encoding/openapi|jsonschema: allow bool for exclusiveNum
3e62450 go.mod: use github.com/rogpeppe/go-internal/testscript
519db58 internal/encoding: don't simplify interpreted encodings
1d213a3 pkg/list: remove use of :: (Incompatible!!!)
5515ee9 tools/fix: move fix functionality to separate package

Language Changes and Streamlined CLI semantics

This release makes several minor and bigger language changes. The changes are a result of analyzing CUE’s interoperability with other languages and what is needed for this. It is expected that these are the last major changes and that this fixes the look and feel of the language. In addition to the usual cue fmt-based rewrites, we now also provide a cue fix command to aid in the transition. Also, the old formats are still supported for now. Minor language changes and additions to the language are still planned on the path to stability.

The new language changes allow for a simpler JSON Schema mapping. This, in turn, allows for a more straightforward combination of schema and data on the command line.

New syntax for definitions

Before, definitions were indicated with a double colon ::. They lived in the same namespace as regular fields. This property complicated defining automated conversions of some other data formats to CUE. For instance, JSON Schema keeps separate sections for schema and definitions. As there is no enforced convention, as in Go, for naming the two kinds differently, one could not map these to the same struct. This forced schemas to be moved to another location, which turned out to be cumbersome and unnatural.

Another issue with the old syntax is that one could not determine from a reference a.b whether b would refer to a definition or regular field. Not a big problem per se, but it lacked clarity.

As of v0.2.0, definitions are denoted with special identifier starting with a #. As this was not a legal identifier before, such identifiers can safely be used exclusively for definitions. Effectively, definitions now have their own namespace. Other than that, definitions work as before. They still close a struct, they are not output during export, and you simply use their identifier to refer to them.

Before:

A :: b: int
b: c :: string

D: A
E: b.c

After

#A: b: int
b: #c: string
D: #A
E: b.#c

The #c notation is just an identifier: no special syntax is needed to handle them. As before, "#foo" still denotes a regular field with a name starting with #.

Using an initial # to distinguish between definitions and regular fields was inspired by Go's use of initial casing to distinguish between exported and non-exported identifiers. Indeed because CUE is a JSON superset and interop focused, there is little control over casing. As we explain later in the Hidden Fields section, we also resurrect _ as a means of excluding fields and definitions from output.

The new notation may take a bit of getting used to, but the increased readability and the flexibility from the separate namespaces are big wins. So far the feedback has been overwhelmingly positive.

The Go, Protobuf, JSON Schema, and OpenAPI mappings, as well as the tutorials, have been ported to use #-style definitions. Please beware that if one has other templates based on this they need to be rewritten as well.

Limitations

The new definition syntax is more restricted than the old syntax. In general, it is no longer allowed to have dynamic definition names. For instance, the syntax disallows using interpolations for creating names. Also bulk optional fields only apply to regular fields. We think this actually benefits static analysis and is overall a good change. If need be, though, the old semantics can be simulated by containing a struct within a definition used as a map. In case this proves to be too limiting, we have a possible language change up our sleeves that would allow dynamic definitions again.

Old style definitions will keep being supported up to the next minor release. Use cue fix to rewrite current files (see below).

API changes

The current API assumed definitions and fields lived in the same namespace. The current API is therefore inherently broken. The Struct.FieldByName has been deliberately broken to force users to disambiguate. To evaluate a reference, it is recommended to use Dereference instead of passing the result of Reference to Lookup.

There are some good ideas to make the API considerably simpler and more powerful. This is contemplated to be built on top of the new adt package, which is developed as part of the evaluator rewrite, which is planned for the next minor release.

cue fix

To aid in the conversion to #-style definitions, the cue fix command is introduced. This command does a module-wide update of packages by default, but allows specifying individual files and packages.

This is no longer piggybacked on cue fmt. Unlike previous rewrites, the definition rewrite requires evaluation. In addition, not all current definitions are legal in the new syntax. The converter will not remap those. Also, unfortunately, due to API limitations, the converter can also not handle all legal cases. We hope that such cases will be rare, though.

Cases that cannot be handled are clearly marked with locations that may reference the definition included. Definitions with issues have are marked with a deliberately obscure @tmpNoExportNewDef attribute, indicating they need manual fixing

There is also a possibility there will be undetected breakage. This may happen if a selection is made in a disjunction where the field may be either a definition or regular field. The fact that this is even possible indicates that the old model was probably too flexible and possibly more bug prone. With new-style definitions such shenanigans are no longer possible.

Unfortunately, cue fix currently does not handle updating multiple packages within the same directory at once unless manually specified. We intend to allow this at some point.

Tool semantics changes

Previously, the cue tool handled schemas and data differently. In practice, the distinction is not all that clear, making this distinction somewhat “forced”. The cue tool now treats data and schemas as the same thing and always unifies them without distinction by default. This greatly simplifies its use and makes it a very powerful tool for operating directly on non-CUE schema.

There are two exceptions to the new “always unify all files” rules: 1) with cue vet multiple data files are still individually verified against a single schema. 2) the -d option still separates schema from data files. One could argue that the latter isn’t really necessary anymore, as we will see next.

JSON Schema mapping

The following command

cue export schema.json data.yaml

converts the schema.json (which has a proper value for $schema to detect the format) to a schema and data.yaml to data, and unifies the results. This will naturally fail if the contents of data doesn’t correspond to the schema. There is no special logic needed to detect these cases, other than knowing when to interpret a data file as JSON Schema.

Note that #-style definitions were key to make this possible. Without them, there would be no obvious way to map JSON Schema to the root of a config as merging fields and schema to the same namespace could result in conflicts.

OpenAPI merging

Another example of the new capabilities is that one can now merge schema as one would merge data. For instance, the following command merges two OpenAPI files and then outputs it again as OpenAPI:

cue def openapi: file1.json file2.json --out openapi

The user won’t even see the intermediate CUE.

Hidden fields

This release also officially resurrects hidden fields (identifiers written as, for example, _foo). They were removed from the spec as with the introduction of definitions they were believed to be an incongruent feature. They fit, however, quite nicely with the new-style definitions. They were still in use and proved to be more useful than expected. As they were never really removed from the implementation nothing changes. However, their comeback is now official.

For clarity, both hidden fields and definitions are not shown in exported output. They serve different functions, though.

A definition defines a new composite type, the most general specification of something. They are at the other end of the spectrum to concrete instances. As definitions are supposed to fully define a schema, CUE can use them to catch typos in field names by detecting fields that are not supposed to be there.

A hidden field can be any value and is not subjected to such scrutiny. They are used to define fields that are not converted to output without making them a definition.

Although this is not yet enforced, hidden fields are local to a package. To make the matrix complete identifiers starting with _# denote hidden definitions, or definitions local to a package. These can be used today, although this restriction is also not yet enforced.

Streamlined syntax for list comprehension

List comprehensions are now written as

[ for x in src { x + 1 } ]

This brings them in line with field comprehensions, where the value also comes after the comprehension clauses. Overall, this greatly increases readability.

This also harmonized the syntax. It simplifies both spec and implementation, and prepares for some of the constructs mentioned in the query proposal.

The one thing that may require some getting used to is the curly braces alluding to the value being a struct. If the curly braces contain a single scalar value, however, it is promoted to be the result of an iteration. CUE users may already be familiar with this construct at the top-level scope, where “embedded scalars” are the final result of evaluation. This construct was originally invented to allow omitting curly braces, while still being a superset of JSON (JSonnet uses the same trick). It turns out, however, that this trick can be quite useful when generalized to apply to any struct, not just the top-level one. We continue to investigate this possibility. Con...