How to view rendered manifests in CI

[pvickery-ParamountCommerce]

I am trying to find a way to view the rendered manifests diffs for each environment in a PR to my main branch. I was trying to use Kargo Render to generate PRs to the target environment branches but since it is (getting) archived, I am curious if there is another option to be able to do this?

I was thinking to create the PR to the target branches with Kargo Render using GIthub Actions, and then, after merging the PR to main, Kargo would promote each stage by merging the PRs but @krancour mentioned this has some problems and is not recommended.
#3720 (comment)

Ideally, I would also be able to pass the target PR diffs to a log pattern analyzer tool like klp to get a high level view of the changes to all environments.

[krancour]

@pvickery-ParamountCommerce thank you for moving the conversation to a dedicated thread. 😄

How to view rendered manifests in CI

First, I want to suggest we strike "CI" from the conversation. Kargo was built around the notion that CI platforms were being over-leveraged to do "CD" things because CD platforms weren't offering any solution to the common problem of "how do I move changes from environment to environment in a controlled fashion?" CI platforms tend to be a bit more general-purpose / open-ended than CD platforms in terms of capability, so it was natural for people to fall back on them to roll their own solutions. But no matter how hard one tries, CI is bad at these things.

CI: Good at synchronous processes. Example: Lint my code, run my unit tests, build a container image, push the image to a repository -- and do it all as fast as possible.

CD: Tends to be much more asynchronous. Software gets deployed to some environment and tends to sit there for some time while automated or manual tests are performed, stakeholders review and sign off, etc.

Kargo was predicated on making a crisp distinction between CI and CD and filling the capability vacuum in the CD space that led to over-leveraging CI and putting square pegs (CD problems) in round holes (CI platforms).

Repositories are the rendezvous point between CI and CD. CI should produce artifacts and push them somewhere. CD should pick them up and take it from there. And configuration (e.g. Kubernetes manifests) is a kind of artifact.

I was trying to use Kargo Render to generate PRs to the target environment branches but since it is (getting) archived

A bit of history: Kargo came first. In the early days of the project, we knew we really loved the "rendered branches" pattern -- the idea of taking Helm charts or manifests + kustomization.yaml and using those tools to expand that input into environment-specific plain YAML and then stashing that plain YAML in an environment-specific branch. The advantages of doing so are, I think, exactly what you're after -- the ability to do a comprehensive review of the actual changes headed for some environment. This is easier and less error prone than mentally "running" configuration management tools like Helm or Kustomize in your head to guess at what the changes actually look like when Argo CD gets done applying them. We knew we loved this way of doing things and while Kargo itself was under heavy development, changing constantly, unproven, we decided to separate the bits that did this rendering into its own separate tool so we, ourselves could take advantage of this pattern more easily sooner than if we waited for Kargo itself to be feature complete.

When we used Kargo Render ourselves, our use looked pretty similar to what you've been describing. Someone would commit something to main, and Kargo Render (through a GitHub Action) would apply configuration management tools to that input from main to produce environment-specific plain YAML that would then get PR'ed to environment-specific branches that Argo CD was continuously syncing Applications to. The large weakness in this model is that it doesn't advance changes from environment to environment in a highly controlled fashion. The moment a commit was merged to main, we'd have PRs open against a test branch, a staging branch, and production branches in multiple regions. We could look at those PRs and see the exact changes pending to each environment, but those PRs were opened all at once. If we didn't like the changes to test and decided to close the PR unmerged, we had to also go and close the PRs to staging and prod. If we like the changes to test, we had to merge them, then merge the changes to staging (if we liked them, also), and then prod. i.e. A whole lot of tentative changes were pending to downstream environments from the moment we committed to main, but without us having decided yet if we like the result in test.

This is very close to what you're doing or wanting to do. You'd proposed Kargo Render opening all these PRs for you "up front" and then Kargo carries out promotions by merging these existing PRs. tbh, this feels like chaos.

To round out the history, as Kargo became more capable and less opinionated over time, and we came to trust it more (we dogfood it heavily), Kargo became capable of doing the good things Kargo Render had been doing, but in a far more controlled and less chaotic fashion and that was the point where we decided Kargo Render had no more life left in it as a separate piece of tech.

The canonical way of addressing your use case in Kargo, without Kargo Render, is this:

1. "Artifacts" are updated. This could mean that CI has built a new image and pushed it to a repository. It could equally mean that a developer has pushed a configuration change to main.

2. Kargo notices new artifact(s) and creates a piece of Freight, which is a reference to the latest version(s) of your artifact(s). Example: The most recent version of some image + the most recent commit to a Git repo containing configuration.

3. Depending on "promotion policy" Freight is automatically or manually promoted to some "zero stage" (an informal term we haven't used in quite some time, but it denotes a stage on the "far left" of your pipeline). The typical promotion process would look like this:

   1. Clone the Git repo containing your configurations. Check out the exact commit referenced by the Freight.

   2. Make any necessary modifications. e.g. Update some YAML to reference the version of some image that was referenced by the Freight.

   3. Run config management tools. e.g. Run helm template or kustomize build.

   4. Open a PR (optional, but I know this is what you want) that writes the output of the previous step to an environment-specific directory or branch in a Git repo. (We like branches.)

   5. _Now you have a PR open that shows you the diffs between the environment's current plain YAML and the proposed plain YAML. Review it to your heart's content. If you don't like the changes, close them. The Promotion ends there. Important: No PR will ever have been opened to "downstream" environments like staging or prod if you decided you didn't like the PR to test! This is why I say this is less chaotic than what was going on with Kargo Render. If you do like what you see in the PR, merge it. The Promotion resumes and typically completes by "nudging" an Argo CD application to make it sync to the recently-merged, environment-specific, plain YAML.

   6. If you have defined any automated verification processes, they now run in the test environment. If those are satisfied, the Freight is now verified in test and eligible to move downstream.

   7. Once again, depending on policy, this Freight, now eligible to move downstream, will advance to staging (and later prod) through new Promotions that are repetitions of the above steps that advance the same changes into the corresponding environment-specific branches.

Recap: You got to review changes every step of the way. No unnecessary PRs were ever opened. (i.e. If you didn't like the changes to test, you didn't have to go close PRs with the corresponding changes to staging and prod). And Kargo was in-control of the whole business of moving changes from environment to environment. Once Kargo saw the artifacts that had been published by CI, it took things the rest of the way, without leaning on CI to automate any parts of the process.

Here's a slightly simplified version of the above (without the PRs) straight from the docs:

https://docs.kargo.io/user-guide/how-to-guides/working-with-stages#promotion-templates

Ideally, I would also be able to pass the target PR diffs to a log pattern analyzer tool like klp to get a high level view of the changes to all environments.

When the PR is open, go crazy. Use whatever automation, tools, or processes you like to decided whether you like what you see and want to merge so the Promotion proceeds, or close, to halt the Promotion.

[pvickery-ParamountCommerce]

Thanks for your detailed response, I really appreciate the help!

I think me and my team just need to update our mindset to be OK with viewing the diffs after merging to our main branch instead of viewing them beforehand in the initial PR. Kargo has caused us to reevaluate our requirements for GitOps and focus less on the "rules" of traditional GitOps and more on an ideal workflow.

I may have more questions next week but I am signing off early for the long weekend, thanks again!

[krancour]

I think it wasn't clear that you wanted PRs opened to all environment-specific branches before merging to main and not immediately after merging to main. So you want PRs open against main and all the environment-specific branches at once.

That is an interesting notion, and I can see its advantages, but I can also see a very high likelihood of that being the source of problems that won't otherwise occur. I believe you are much, much more likely to encounter merge conflicts in your environment-specific branches this way.

That being said, what you're looking for absolutely can be modeled with Kargo, but I still wouldn't recommend it.

[pvickery-ParamountCommerce]

So you want PRs open against main and all the environment-specific branches at once.

Ya, that is what I was thinking at first. We would require the feature branch to be up to date with our main branch before running Kargo Render or merging the PR to prevent merge conflicts in the target branches. We are a small team so this is doable, but I understand this wouldn't scale well for a larger team with a busy repo. The other issue, as you pointed out, is that we would need to close PRs for multiple target branches if we don't want continue with the change. Closing the PRs could be automated, but they cannot be not deleted, which would leave a lot of noise in the repo.

My main goals are to

1. View the all downstream diffs in one location
2. View each diff before merging to main.

Since Kargo Render is archived and Kargo doesn't focus on CI, I should look for other solutions for #2 just use Kargo for the CD.

My plan is to create tags after merging PRs which save the merged PR number. Then I use that value to reference the initial PR in the PRs opened against the target branches. This way the initial PR will have references to all the target PRs so all the changes can be viewed from one page. This solves our problem as we can still view/review each PR before merging, but also have a single page to view all the changes afterwards.

When the git-merge-pr step is available, we can batch our PRs instead of opening one at a time, and wait for the 'leftmost' target PR to be merged, then automatically promote the remaining environments by merging the PRs.

Thanks again for all the help, this really helped us figure out how to start using Kargo 😄