A new, modular, vibe code friendly architecture for tlparse

[ezyang]

Redux of #78

I've been thinking about how to make tlparse more vibe coding friendly. Specifically, I want to make it easier for people to add custom, one-off frontends to view the data in tlparse, without having to do a full deploy process. My vision is to have a GitHub-like ecosystem of tlparse viewers which are maintained by individual people for their own use cases. People should be able to easily fork and customize viewers for their needs, and we can have a set of popular viewers that we can recommend to people by default (and maybe, eventually, a set of "official" viewers maintained by a core team).

To do this, we must define what the abstraction boundary that vibe code apps will interact with. A logical choice that requires no work is for vibe code viewers to directly interact with the raw TORCH_TRACE files that are generated. However, I think this is unwise: for example, a raw trace from a minified version of one of our production models is 150 MB! While this is not unmanageable, at this size it will take nontrivial time to load the trace in thef irst place.

The next most obvious fix is to separate out artifacts from the log stream, so that you have to do a separate HTTP call to fetch actual intermediate build products like Inductor code dumps. On the log I'm looking at, this reduces the trace size to 22M, with 40k rows. Discussion from https://x.com/pavpanchekha/status/1756070960837660949 indicates that at this size, rendering and not raw processing is the bottleneck. Frontends should either further filter this down, or make sure to implement some sort of virtual rendering functionality (I'm not clear if this can be conveniently done with vibe coding.)

Beyond this, my only other ideas is that you can do some further preprocessing in Rust itself, but it's not clear to me how we'd negotiate that.

For the actual architecture for vibe coded frontends, there are a still constraints to mindful of:

• The most convenient place for the postprocessed JSON files to live is on Manifold. This cannot be directly accessed by a client side JS application as there are no CORS headers, so you need to host a proxy that can access it. The proxy serve can/should be shared by everyone.
• Plain HTML/JS is relatively easy to scaffold, but if people prefer writing/vibe coding things in React we need to make some decisions about how the actual hosting works (there's an interaction with the proxy as well). A template for React is a lot more pressing.
• Should have web testing, similar to ScubaDuck. Public traces that we can test against are a must, ideally hosted in some centralized location so they can be reused. (Plain separate Git repo is probably OK, LFS probably not necessary?) Maybe also want a private store as well for internal only testing.
• Ideally, vibe coding can be directly on laptop or in public cloud, so external SOTA tools (claude code, codex can be used).
• Need some sort of "GitHub" fork/clone functionality, as well as app hosting. Actually, it wouldn't be crazy to do the hosting entirely with GitHub pages, with you running the proxy locally as needed. The proxy would need some specific CORS settings for safety. Ideally some sort of Metamate-based whitelist? If the ecosystem goes public, you need to audit deployments because they can exfiltrate sensitive data. Meta employees would be generally safe. (Also, you want to be able to run things locally!)
• We will need a comprehensive documentation about the JSON format which can be fed in as context to the LLM.

[jamesjwu]

I think some sort of diagram may make it a bit more clear to me, but are you suggesting that tlparse only return a set of post processed JSON streams of different artifacts, that a frontend viewer will then stitch together into a page?

If so, do we still plan on having a default viewer?

The devil is in the details of what the JSON format will look like as the generalized output of all tlparse artifact streams. The problem with it is, the more work we do, the less customization the vibe coded viewers will be able to handle, and the less work we do, the less useful tlparse is as a tool (we might as well just have 100 different vibe coded tlparses).

My hot take is that the "parser" integration architecture can be expanded to fit this structure. The vibe code interface is a function which takes as input some subset of structured logs and returns a list of HTML outputs. The function could take in JSON as an abstraction. When running tlparse, you choose which parsers you want to run, and the main index.html page just renders each parser in its own tab. There is a default set of parsers that are chosen, but you can configure this with some config.json maybe. tlparse is responsible for calling the custom viewers on the correct data, and the custom viewer is responsible for telling tlparse which structured logs its interested in consuming.

In MVC speak, the model is the structured logs, the view is implemented by custom parsers, and the controller is tlparse.

Some details that I think I'd like:

• Lazy vs. Eager views. Eager views are rendered at tlparse run time, and become raw HTML. Lazy views are postprocessed into JSON, and the view itself does the work of composing that JSON on the spot. Chromium events/perfetto is a lazy view today, whereas output code is an eager view. Maybe a lot of vibe coded views would be lazy views, so that they can host their own viewer, and tlparse just links to the page. (It would be optimal if you didn't have to leave the tlparse page at all though)

• Viewers should be extendable from each other. The internal fb perfetto view is different from the external one but they share the same underlying chromium traces. I should be able to just extend the class and make some edits internally, and have internal tlparse behave differently because of it. Every internal feature in tlparse that appears on index.html should just be a custom viewer.

• Don't have a strong opinion on separating artifacts from log stream, but it would theoretically work with this paradigm: you just query for the artifacts in the separate location if your custom parser/viewer needs it.

[ezyang]

I think some sort of diagram may make it a bit more clear to me, but are you suggesting that tlparse only return a set of post processed JSON streams of different artifacts, that a frontend viewer will then stitch together into a page?

In the JSONL PR #114 I simply added an extra raw.jsonl output to the tlparse output which is what frontend viewers can ingest.

If so, do we still plan on having a default viewer?

I don't see any reason to delete it. But I would think maybe eventually it would be less useful, or merely a landing page to get to custom viewers.

The devil is in the details of what the JSON format will look like as the generalized output of all tlparse artifact streams. The problem with it is, the more work we do, the less customization the vibe coded viewers will be able to handle, and the less work we do, the less useful tlparse is as a tool (we might as well just have 100 different vibe coded tlparses).

The JSONL format is exactly the format that PyTorch produces. There isn't any format shifting. (Actually, I did make on change, which was to preprocess the string table, but that's very special and I don't expect to have to keep doing it.)

My hot take is that the "parser" integration architecture can be expanded to fit this structure. The vibe code interface is a function which takes as input some subset of structured logs and returns a list of HTML outputs. The function could take in JSON as an abstraction. When running tlparse, you choose which parsers you want to run, and the main index.html page just renders each parser in its own tab. There is a default set of parsers that are chosen, but you can configure this with some config.json maybe. tlparse is responsible for calling the custom viewers on the correct data, and the custom viewer is responsible for telling tlparse which structured logs its interested in consuming.

I intentionally want the viewers to live external to the main tlparse repo, so that is easier for people to make one off viewers and publish them without gooping things up, to give people more flexibility in what tech stacks they want to implement them with (e.g., JS vs TS vs React vs whatever). I don't want people to have to deal with Rust build step, or really to have to write Rust at all. I don't want them to have to go through code review on the main tlparse repo.

If you want the main tlparse landing page to show the custom viewers, I would probably just propose we have some JavaScript that loads a list of links that let you load up the jsonl to an external viewer which is hosted somewhere else.

Maybe a lot of vibe coded views would be lazy views, so that they can host their own viewer, and tlparse just links to the page.

Yes exactly

It would be optimal if you didn't have to leave the tlparse page at all though

It is good to have viewers in isolated browser contexts, so crappily written viewers that use too many resources don't crash everything.

Viewers should be extendable from each other. The internal fb perfetto view is different from the external one but they share the same underlying chromium traces. I should be able to just extend the class and make some edits internally, and have internal tlparse behave differently because of it. Every internal feature in tlparse that appears on index.html should just be a custom viewer.

My model for how this works is you'll publish your viewer as a Git repo, and people can fork and remix viewers. This is especially feasible because you're using an LLM to do the coding, so you don't actually have to understand the code, you just ask the LLM to make the UX change you wahnt.

Don't have a strong opinion on separating artifacts from log stream, but it would theoretically work with this paradigm: you just query for the artifacts in the separate location if your custom parser/viewer needs it.

Yes. The main difficulty is managing CORS. I think I want some sort of proxy service that can fetch the JSONL and let your client side application view them.