cat-file: add %(objectmode) and submodule message to batch commands #1929
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update the 'run_tests' test wrapper so that the first argument may refer to
any specifier that uniquely identifies an object (e.g. a ref name,
'<OID>:<path>', '<OID>^{<type>}', etc.), rather than only a full object ID.
Also add tests that use non-OID identifiers, ensuring appropriate parsing in
'cat-file'. The identifiers used in some of the added tests include a space,
which is incompatible with the '%(rest)' atom. To accommodate that without
removing the test case, use 'test_expect_failure' when 'object_name'
includes a space.
Signed-off-by: Johannes Schindelin <[email protected]>
Signed-off-by: Victoria Dye <[email protected]>
Add a formatting atom, used with the --batch-check/--batch-command options, that prints the octal representation of the object mode if a given revision includes that information, e.g. one that follows the format <tree-ish>:<path>. If the mode information does not exist, an empty string is printed instead. Signed-off-by: Johannes Schindelin <[email protected]> Signed-off-by: Victoria Dye <[email protected]>
When an object specification is passed to 'cat-file --batch[-check]' referring to a submodule (e.g. 'HEAD:path/to/my/submodule'), the current behavior of the command is to print the "missing" error message. However, it is often valuable for callers to distinguish between paths that are actually missing and "the submodule tree entry exists, but the object does not exist in the repository". To disambiguate without needing to invoke a separate Git process (e.g. 'ls-tree'), print the message "<oid> submodule" for such objects instead of "<object> missing". In addition to the change from "missing" to "submodule", the new message differs from the old in that it always prints the resolved tree entry's OID, rather than the input object specification. Note that this implementation maintains a distinction between submodules where the commit OID is not present in the repo, and submodules where the commit OID *is* present; the former will now print "<object> submodule", but the latter will still print the full object content. Signed-off-by: Victoria Dye <[email protected]>
|
/submit |
|
Submitted as [email protected] To fetch this version into To fetch this version to local tag |
|
This patch series was integrated into seen via git@84f8508. |
|
This branch is now known as |
|
This patch series was integrated into seen via git@0757446. |
|
On the Git mailing list, Junio C Hamano wrote (reply to this): "Victoria Dye via GitGitGadget" <[email protected]> writes:
> This series re-attempts the changes proposed last year [1] for extending the
> information about tree entries available from the 'cat-file' batch format
> commands. It also (hopefully) addresses the initial round of feedback that
> series received.
>
> The first patch updates 't1006-cat-file.sh' to test non-OID object
> specifications. In response to the feedback in [2], I added more careful
> quoting and a couple tests using paths with spaces. This change revealed a
> (likely known) limitation of the '%(rest)' atom when processing object names
> with spaces. To make that limitation explicit, I marked the relevant test as
> expected to fail.
>
> The second patch adds "mode" support. This is essentially unchanged from its
> initial submission, save for some conflict resolution in the test script.
>
> The final patch takes a different approach to submodule resolution than the
> initial submission; rather than treat the entry as a "regular" commit object
> with empty content, we now print an error message similar to the "missing",
> "ambiguous", etc. cases, but with the tree entry's OID rather than the input
> object name.
I did not send any line-by-line reviews, but after reading these
patches I didn't see anything questionable. Unless we see others
comments that need to be addressed, let's merge it to 'next' in
preparation for the next cycle.
Thanks. |
![gitgitgadget[bot]](https://avatars.githubusercontent.com/in/12836?s=60&v=4){kind=small}
| @@ -307,6 +307,11 @@ newline. The available atoms are: | |||
| `objecttype`:: | |||
There was a problem hiding this comment.
On the Git mailing list, Jeff King wrote (reply to this):
On Mon, Jun 02, 2025 at 06:55:54PM +0000, Victoria Dye via GitGitGadget wrote:
> Add a formatting atom, used with the --batch-check/--batch-command options,
> that prints the octal representation of the object mode if a given revision
> includes that information, e.g. one that follows the format
> <tree-ish>:<path>. If the mode information does not exist, an empty string
> is printed instead.
Overall, this looks good to me. I have a few small comments below,
though I'm not sure if they merit a re-roll or not.
> @@ -345,6 +347,9 @@ static int expand_atom(struct strbuf *sb, const char *atom, int len,
> else
> strbuf_addstr(sb,
> oid_to_hex(&data->delta_base_oid));
> + } else if (is_atom("objectmode", atom, len)) {
> + if (!data->mark_query && !(S_IFINVALID == data->mode))
> + strbuf_addf(sb, "%06o", data->mode);
> } else
> return 0;
> return 1;
Looking at this hunk raised a few questions. Fortunately with answers. ;)
First, in other parts of this if/else chain, when mark_query is set we
need to perform some action (usually setting up the object_info
pointers). But we _don't_ need to do that here, since we get the mode
info "for free" from get_oid_with_context(). Good.
Second, how do we reliably get S_IFINVALID? We can see that the
expand_data struct is now initialized with it:
> +#define EXPAND_DATA_INIT { .mode = S_IFINVALID }
But that seems like it would be a bug, since we only initialize it once,
in batch_objects():
> @@ -866,7 +872,7 @@ static int batch_objects(struct batch_options *opt)
> {
> struct strbuf input = STRBUF_INIT;
> struct strbuf output = STRBUF_INIT;
> - struct expand_data data;
> + struct expand_data data = EXPAND_DATA_INIT;
> int save_warning;
> int retval = 0;
>
> @@ -875,7 +881,6 @@ static int batch_objects(struct batch_options *opt)
> * object_info to be handed to oid_object_info_extended for each
> * object.
> */
> - memset(&data, 0, sizeof(data));
> data.mark_query = 1;
> expand_format(&output,
> opt->format ? opt->format : DEFAULT_FORMAT,
>
> static int is_atom(const char *atom, const char *s, int slen)
> {
...and then call batch_one_object() over and over. So at first glance,
doing this:
(echo HEAD:Makefile; echo HEAD) |
git cat-file --batch-check='%(objectmode)'
would let the mode from the first object bleed over into the second. But
that doesn't happen, because we overwrite expand_data.mode for each
object unconditionally, here:
> @@ -613,6 +618,7 @@ static void batch_one_object(const char *obj_name,
> goto out;
> }
>
> + data->mode = ctx.mode;
> batch_object_write(obj_name, scratch, opt, data, NULL, 0);
>
> out:
And there we are relying on ctx.mode, which we get from
get_oid_with_context(), which always falls back to S_IFINVALID if no
mode is available. Good.
But I think that means that the value set in EXPAND_DATA_INIT is never
used, and we could continue to zero-initialize the struct with memset?
That said, it's probably OK to err on the side of over-initializing. The
worst case is probably somebody later reading the code being confused
about the importance of the line. And at best it may prevent a future
code path from unexpectedly reading a funny value.
And on to the third question. In the non-batch code path of
cat_one_file(), we do:
if (obj_context.mode == S_IFINVALID)
obj_context.mode = 0100644;
which made me wonder if we should be harmonizing our behavior. But that
mode is used only for passing to filter_object() and textconv_object().
Neither of which really care about the mode, and this is mostly just
saying "eh, do your regular thing as if it were a blob we found at
--path". I suspect we could get the same effect by just passing a
hard-coded 100644 to those functions, but probably not worth changing
now (and certainly very orthogonal to your patch). But the important
thing is we do not really need to worry about being consistent with this
line. Good.
-Peff
| @@ -368,6 +373,14 @@ If a name is specified that might refer to more than one object (an ambiguous sh | |||
| <object> SP ambiguous LF | |||
There was a problem hiding this comment.
On the Git mailing list, Jeff King wrote (reply to this):
On Mon, Jun 02, 2025 at 06:55:55PM +0000, Victoria Dye via GitGitGadget wrote:
> To disambiguate without needing to invoke a separate Git process (e.g.
> 'ls-tree'), print the message "<oid> submodule" for such objects instead of
> "<object> missing". In addition to the change from "missing" to "submodule",
> the new message differs from the old in that it always prints the resolved
> tree entry's OID, rather than the input object specification.
OK. I read over the discussion from last year, which I think mostly
centered around this patch. I do still think in the long run it would be
nice for cat-file to produce what output it _can_ for a missing object
(e.g., the oid and mode).
But I think it is OK to punt on that for now. Because "<oid> missing"
lines already exist, we'd probably need to put such behavior behind a
new command-line option. So while "<oid> submodule" lines would be
unnecessary in that hypothetical future world, we are not digging the
hole any deeper, from a backwards-compatibility standpoint.
Although speaking of backwards compatibility, I guess older readers may
be surprised that the old "missing" message becomes a "submodule" one.
They may need to be updated if they were written carefully to bail on
unknown input (and were happy seeing "missing" messages for submodules).
So there may be some fallout, but it's not like the existing messages
were particularly useful in the first place.
> Note that this implementation maintains a distinction between submodules
> where the commit OID is not present in the repo, and submodules where the
> commit OID *is* present; the former will now print "<object> submodule", but
> the latter will still print the full object content.
Hmm, that is an interesting point. It feels kind of arbitrary, but I'm
having trouble making a strong argument for one direction or the other.
The way you've written it means that readers need to be prepared to
parse _both_ the mode and "<oid> submodule" lines to find submodules.
But maybe there's some value in finding out more information about
submodule commits you do have in-repo.
The implementations are similar. Replacing this hunk:
> diff --git a/builtin/cat-file.c b/builtin/cat-file.c
> index b11576756bcc..4b23fcecbd8e 100644
> --- a/builtin/cat-file.c
> +++ b/builtin/cat-file.c
> @@ -496,7 +496,10 @@ static void batch_object_write(const char *obj_name,
> &data->oid, &data->info,
> OBJECT_INFO_LOOKUP_REPLACE);
> if (ret < 0) {
> - report_object_status(opt, obj_name, &data->oid, "missing");
> + if (data->mode == S_IFGITLINK)
> + report_object_status(opt, oid_to_hex(&data->oid), &data->oid, "submodule");
> + else
> + report_object_status(opt, obj_name, &data->oid, "missing");
> return;
> }
>
with:
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 4b23fcecbd..1b200e1607 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -488,6 +488,11 @@ static void batch_object_write(const char *obj_name,
if (opt->objects_filter.choice == LOFC_BLOB_LIMIT)
data->info.sizep = &data->size;
+ if (data->mode == S_IFGITLINK) {
+ report_object_status(opt, oid_to_hex(&data->oid), &data->oid, "submodule");
+ return;
+ }
+
if (pack)
ret = packed_object_info(the_repository, pack, offset,
&data->info);
so I think the decision is really about what people will find most
useful. So I dunno. It is mostly a coin-flip, leading me to say that
what you picked just came up "heads" and is good enough. ;)
-PeffThere was a problem hiding this comment.
On the Git mailing list, Victoria Dye wrote (reply to this):
Jeff King wrote:
> On Mon, Jun 02, 2025 at 06:55:55PM +0000, Victoria Dye via GitGitGadget wrote:
>
>> To disambiguate without needing to invoke a separate Git process (e.g.
>> 'ls-tree'), print the message "<oid> submodule" for such objects instead of
>> "<object> missing". In addition to the change from "missing" to "submodule",
>> the new message differs from the old in that it always prints the resolved
>> tree entry's OID, rather than the input object specification.
>
> OK. I read over the discussion from last year, which I think mostly
> centered around this patch. I do still think in the long run it would be
> nice for cat-file to produce what output it _can_ for a missing object
> (e.g., the oid and mode).
One way to handle that could be changing the message to something like:
submodule SP <mode> SP <oid>
but...
>
> But I think it is OK to punt on that for now. Because "<oid> missing"
> lines already exist, we'd probably need to put such behavior behind a
> new command-line option. So while "<oid> submodule" lines would be
> unnecessary in that hypothetical future world, we are not digging the
> hole any deeper, from a backwards-compatibility standpoint.
>
> Although speaking of backwards compatibility, I guess older readers may
> be surprised that the old "missing" message becomes a "submodule" one.
> They may need to be updated if they were written carefully to bail on
> unknown input (and were happy seeing "missing" messages for submodules).
> So there may be some fallout, but it's not like the existing messages
> were particularly useful in the first place.
...I suspect that'd be even less compatible with existing automation around
'cat-file' than just swapping out "submodule" for "missing", and users can
theoretically infer that the mode is 160000 (S_IFGITLINK). That said, if at
some point in the future we support submodules with a different mode, then
an explicit value would be fairly useful.
Happy to change it or keep it the same, I have no strong preference either
way.
>
>> Note that this implementation maintains a distinction between submodules
>> where the commit OID is not present in the repo, and submodules where the
>> commit OID *is* present; the former will now print "<object> submodule", but
>> the latter will still print the full object content.
>
> Hmm, that is an interesting point. It feels kind of arbitrary, but I'm
> having trouble making a strong argument for one direction or the other.
> The way you've written it means that readers need to be prepared to
> parse _both_ the mode and "<oid> submodule" lines to find submodules.
> But maybe there's some value in finding out more information about
> submodule commits you do have in-repo.
This was pretty much my thought process on it. It was a somewhat arbitrary
choice, but what tipped me towards distinguishing the cases is that I'd
rather have information like size, content, etc. about a commit and not need
to use it, than need it but not have it available. That, and it does
maintain the existing treatment of self-referential submodules.There was a problem hiding this comment.
On the Git mailing list, Jeff King wrote (reply to this):
On Wed, Jun 04, 2025 at 05:12:54PM -0700, Victoria Dye wrote:
> > OK. I read over the discussion from last year, which I think mostly
> > centered around this patch. I do still think in the long run it would be
> > nice for cat-file to produce what output it _can_ for a missing object
> > (e.g., the oid and mode).
>
> One way to handle that could be changing the message to something like:
>
> submodule SP <mode> SP <oid>
Hmm, yeah. That seemed weird to me at first because it doesn't
necessarily match what the caller asked for via batch-check. But really,
mode and oid are the only reasonable things we could report anyway[1].
And the mode is implicit in the word "submodule", so really there is
only the oid to report.
[1] For now, at least. If we ever finally unify all of the various
formatting code, then one might in theory be able to feed a refname
to cat-file and print information about the ref, or perhaps other
meta-information. But let's not worry about that hypothetical for
now.
> ...I suspect that'd be even less compatible with existing automation around
> 'cat-file' than just swapping out "submodule" for "missing", and users can
> theoretically infer that the mode is 160000 (S_IFGITLINK). That said, if at
> some point in the future we support submodules with a different mode, then
> an explicit value would be fairly useful.
>
> Happy to change it or keep it the same, I have no strong preference either
> way.
Right, that makes sense. I do wonder if:
<oid> missing submodule
might be friendlier to readers who are matching on /^[0-9a-f]+ missing/,
but now I am just guessing at a hypothetical program. So it may not be
worth going down that rabbit hole, and we can just go with what you
posted.
We can always worry about extending it later with an option to say "turn
placeholders for missing objects into empty strings" or similar.
I did come across one other interesting case while thinking about this,
though. When running:
git cat-file --batch-check='%(objectname) %(objectmode)'
we do not need to access the object at all! So why does a submodule
entry cause us to complain? The answer is that cat-file will (mostly for
historical reasons) confirm the existence of the object name that is fed
to it by calling oid_object_info(). The only exception is when we are
doing --batch-all-objects, since there we know we have the object,
because we found it by iterating the odb. And we optimize out the extra
call for that case (which makes a big difference if you're just printing
the object names).
But since we don't expect submodule entries to exist in the first place,
it might be reasonable to loosen that check. Something like this, though
I think it could benefit from some refactoring:
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 4b23fcecbd..bb52d9b673 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -304,8 +304,20 @@ struct expand_data {
* This flag will be true if the requested batch format and options
* don't require us to call oid_object_info, which can then be
* optimized out.
+ *
+ * The "submodule" variant is true if the format doesn't require it,
+ * but other options mean we'd usually continue to do so to check
+ * object existence. We can still omit the call for submodules in that
+ * case.
+ *
+ * This might be less confusing if we break skip_object_info down into
+ * two parts:
+ * - does the format require oid_object_info?
+ * - do the other options require checking existence?
*/
unsigned skip_object_info : 1;
+ unsigned skip_submodule_info : 1;
+
};
#define EXPAND_DATA_INIT { .mode = S_IFINVALID }
@@ -477,7 +489,8 @@ static void batch_object_write(const char *obj_name,
struct packed_git *pack,
off_t offset)
{
- if (!data->skip_object_info) {
+ if (!(data->skip_object_info ||
+ (data->skip_submodule_info && data->mode == S_IFGITLINK))) {
int ret;
if (use_mailmap ||
@@ -939,6 +952,12 @@ static int batch_objects(struct batch_options *opt)
strbuf_release(&output);
return 0;
+ } else {
+ struct object_info empty = OBJECT_INFO_INIT;
+
+ if (!memcmp(&data.info, &empty, sizeof(empty)) &&
+ opt->objects_filter.choice == LOFC_DISABLED)
+ data.skip_submodule_info = 1;
}
/*
I don't think that needs to be part of your series, though. We'd still
potentially need to handle the missing-submodule case for format
requests that actually look at the object, which would hit the "<oid>
submodule" case you're adding. So it could come later (or not at all),
and it's probably only worth pursuing if it would make life easier for
your intended caller.
-Peff
|
On the Git mailing list, Jeff King wrote (reply to this): On Wed, Jun 04, 2025 at 07:43:21AM -0700, Junio C Hamano wrote:
> > The final patch takes a different approach to submodule resolution than the
> > initial submission; rather than treat the entry as a "regular" commit object
> > with empty content, we now print an error message similar to the "missing",
> > "ambiguous", etc. cases, but with the tree entry's OID rather than the input
> > object name.
>
> I did not send any line-by-line reviews, but after reading these
> patches I didn't see anything questionable. Unless we see others
> comments that need to be addressed, let's merge it to 'next' in
> preparation for the next cycle.
I left a more detailed review. But it's mostly musing and self-answering
questions. I'd be OK to see this progress as-is.
-Peff |
|
This patch series was integrated into seen via git@519bb38. |
|
This patch series was integrated into seen via git@faf6c3b. |
|
This patch series was integrated into next via git@fb997e4. |
|
There was a status update in the "New Topics" section about the branch "git cat-file --batch" learns to understand %(objectmode) atom to allow the caller to tell missing objects (due to repository corruption) and submodules (whose commit objects are OK to be missing) apart. Will cook in 'next'. source: <[email protected]> |
|
This patch series was integrated into seen via git@f13eb6a. |
|
This patch series was integrated into seen via git@34c613a. |
|
There was a status update in the "Cooking" section about the branch "git cat-file --batch" learns to understand %(objectmode) atom to allow the caller to tell missing objects (due to repository corruption) and submodules (whose commit objects are OK to be missing) apart. Will cook in 'next'. source: <[email protected]> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This series re-attempts the changes proposed last year [1] for extending the information about tree entries available from the 'cat-file' batch format commands. It also (hopefully) addresses the initial round of feedback that series received.
The first patch updates 't1006-cat-file.sh' to test non-OID object specifications. In response to the feedback in [2], I added more careful quoting and a couple tests using paths with spaces. This change revealed a (likely known) limitation of the '%(rest)' atom when processing object names with spaces. To make that limitation explicit, I marked the relevant test as expected to fail.
The second patch adds "mode" support. This is essentially unchanged from its initial submission, save for some conflict resolution in the test script.
The final patch takes a different approach to submodule resolution than the initial submission; rather than treat the entry as a "regular" commit object with empty content, we now print an error message similar to the "missing", "ambiguous", etc. cases, but with the tree entry's OID rather than the input object name.
As for the motivation behind the change (re: [3]), the goal of this series is to be able to get more of the information available internally about an object in '
cat-file --batch[*]' -- in the case of a tree entry, the main things missing were the file mode and the presence (and OID) of submodule pointers. As Junio mentioned in [4], using a single long-running process to resolve objects is far more performant than spawning multiple processes to resolve tree entries with something like 'ls-tree', especially when resolving entries across multiple trees or resolving a mix of tree entries and OIDs, refnames, etc. The object resolution logic in 'cat-file' meant that the mode & submodule OID information were already (mostly) available, but we didn't have a way to output it.The intent of this series is to make the new format options/outputs to get those fields as unobtrusive as possible, but I'm happy to do something more like the previous series if that would be preferable.
[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/[email protected]/
[3] https://lore.kernel.org/git/[email protected]/
[4] https://lore.kernel.org/git/[email protected]/
CC: [email protected]
CC: [email protected]