feat(azure_logs_ingestion sink): Initial azure_logs_ingestion sink

[jlaundry]

Summary

The current azure_monitor_logs sink uses the Data Collector API, which has been deprecated and will be removed in September 2026.

This sink uses the replacement Logs Ingestion API.

While I did consider making this a drop-in replacement for the existing sink, users need to make numerous breaking infrastructure changes, including:

• Creating new Data Collection Endpoint and Data Collection Rule resources
• Moving from a workspace-based secret key to an OAuth credential (App Registration, Managed Identity, etc.)
• (optionally) Re-configuring logs to use the built-in tables, instead of _CL custom tables.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

1. Following the Tutorial steps, create a Log Analytics workspace, App Registration, Data Collection Endpoint, and Data Collection Rule
2. Set the AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET environment variables from the App Registration
3. Use the following vector.yaml:

sources:
  stdin:
    type: stdin

sinks:
  azure:
    type: azure_logs_ingestion
    inputs:
      - stdin
    endpoint: https://dce-e42z.westus2-1.ingest.monitor.azure.com
    dcr_immutable_id: dcr-00000000000000000000000000000000
    stream_name: Custom-vector_CL

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

References

• Closes: Azure DCR-based custom logs -> Logs ingestion API #20978
• Mentioned in: Enhance Azure Identity Integration in Vector: Ability to Utilize Passwordless Connections #20625 - while this PR doesn't resolve this issue for azure_blob, by using the Azure Identity crate, this sink supports passwordless credentials.

[bits-bot]

All committers have signed the CLA.

[pront]

Hi @jlaundry, thanks for this contribution!

It looks good. Two things:

• We will need some documentation files. See an example here (all files under website). Note that base/ is generated by make generate-component-docs.
• Is the intention to complete replace the azure_monitor_logs sink? If that's the case maybe we can mark the existing one as deprecated in favor of this new sink.

[rtrieu]

Thank you for your contribution! Approving with one very minor suggestion.

[jlaundry]

We will need some documentation files. See an example here (all files under website). Note that base/ is generated by make generate-component-docs.

Apologies, I thought this page was auto-generated as well - added now.

Is the intention to complete replace the azure_monitor_logs sink? If that's the case maybe we can mark the existing one as deprecated in favor of this new sink.

Good point, added 🙂

[pront]

What extra resources does the new sink require? Worth explaining this a bit better for users of the old sink.

[jlaundry]

Done (bb71403)

[pront]

Hi @jlaundry, we received this report #23036 and we will be reverted to older azure_* crate versions. Does this affect your PR?

[jlaundry]

Hi @jlaundry, we received this report #23036 and we will be reverted to older azure_* crate versions. Does this affect your PR?

From memory, there is a minor refactor required in https://github.com/jlaundry/vector/blob/02562be6447af36404d8b5668434e317c87a45b2/src/sinks/azure_logs_ingestion/config.rs#L139 to change it back to azure_identity::create_default_credential()?;, and possibly subsequent type changes... but reverting to 0.17 or 0.19 won't fundamentally block this PR, as thankfully I was using the raw REST API 🙂

Probably easiest if you rollback the package first, and then I'll rebase, retest, and push.

[pront]

FYI - Reverted deps #23039

[jlaundry]

FYI, I started preparing the rebase, but I've seen that the Azure Rust team have recently decided to change how authentication works with the azure_identity SDK: Azure/azure-sdk-for-rust#2283

Depending on what they decide, we may need to explicitly configure credentials, either via the vector config file or environment variables. The azure_blob sink will need a similar change (unless using connection_string config).

So instead of releasing an initial sink, and then requiring another config/environment change, I'll wait until there's stability in the SDK before moving forward with this PR.

(I'm not giving up!!!)

[yoelk]

Hi @jlaundry,
Thanks a lot for your initiative! In the company I work in we need this sink exactly.
I went over your PR and I haven't seen explicit auth fields (like connection_string for AzureBlobStorage or shared_key for AzureMonitorLogs). Are they implicit in some way? Or are client secrets not supported for authentication?
I'm asking because our use case is running vector on an AWS machine and connecting to multiple Azure sinks, so if only AAD or ENV_VAR based authentication is currently supported, we wouldn't be able to use it.
Do you consider adding the explicit possibility to use client secrets per sink? I'd be happy to contribute to that.

By the way, I see that the discussion here is closed. Does it mean you will go forward with merging your PR?

Thanks a lot!
Joel

[pront]

So instead of releasing an initial sink, and then requiring another config/environment change, I'll wait until there's stability in the SDK before moving forward with this PR.

Makes sense. I feel that these crates are unfortunately a bit unstable and each version update is risky. Thank you for your interest in contributing 👍

[Renizmy]

Hello
@jlaundry, @pront
Is the above-mentioned blocking point still applicable? I don't have a lot of experiences in rust but I would be interested in working on it

[pront]

Hi all, currently the azure_* crates are not in a good state. Coincidentally, @thomasqueirozb was looking at this issue today. He will comment on this PR if we have a solution.

[jlaundry]

For those playing along at home (hi @yoelk @Renizmy), a summary of the current issues and why we're blocked:

• Vector's azure_blob sink currently uses the azure_storage and azure_storage_blobs crates, which are deprecated/legacy/EOL. The last version released was 0.21.0, which aligns to the 0.21.0 azure_core and azure_identity crates.
• The proposed replacement azure_storage_blob crate is a ground-up re-implementation, currently in it's infancy; Microsoft have a big scary warning that there are bugs, and this crate must not be used in production.
• In addition, the azure_core and azure_identity 0.22.0 crates changed the Traits of various components, and refactored the project structure, making the updated crates incompatible with the last azure_storage out of the box.
• I've seen other projects in the same boat do things like compatibility shims to use azure_storage 0.21.0 with more recent azure_core (which is what @thomasqueirozb is working on in chore(azure_blob sink)!: Update azure (0.25) and azure storage (0.21) #23351)... which works, but adds technical debt to each project. The azure_storage crate will need to be removed eventually.
• But, the bigger issue: for those unfamiliar with Azure workloads, there are a multitude of different ways to get credentials, depending on the deployment type and usage requirements (Managed Identities, Workload Identities, Azure CLI, certificate files... all the way down to good old OAuth Client ID & Secret). Usually, these are abstracted by the language's SDK through the DefaultAzureCredential class.
  • The Go SDK and Python SDK documentation have better descriptions and examples of how this works if you're interested
• Starting with azure_identity 0.22.0, the Microsoft team decided to make DefaultAzureCredential different for the Rust SDK, and only use development credentials, for unspecified security reasons. While a ChainedTokenCredential was proposed (again, similar to the pattern established in the Go/Python/.NET/JavaScript SDKs), this was also removed.
• The net result is that Rust projects that upgrade to azure_identity >= 0.22.0 will need to explicitly add configuration for the user to specify what credential type they're intending to use, and then implement a switch to instantiate the appropriate Credential - otherwise, current production deployments that use Managed/Workload Identities or AZURE_* environment variables will just stop working.
• ... and this morning, I see that Microsoft are considering re-designing the identity library, using cross-compiled .NET code (🫤), so more turbulence is on the horizon...

What I think this means for this PR, and my thoughts/opinions for the wider project:

1. Upgrading azure_identity in #23351 will break current users of the azure_blob sink unless they are using a connection_string. Given that this is on the critical path to update Vector to use http 1.x, this is probably still worth doing - but existing users will need to migrate their config to use a connection string.
2. Once #23351 has been merged, I can then restart development on this PR, and I'll spend some time designing some reusable config options for selecting the appropriate Credential.
3. Finally, once the azure_storage_blob crate reaches production stability, that's probably the point to migrate the azure_blobs sink, and as part of that introduce the various identity config options.

Also note: The existing azure_monitor_logs sink is unaffected by all this drama, because it (only) uses a shared key credential. However, the upstream API is still going to be deprecated September 2026.

[pront]

1. Upgrading azure_identity in #23351 will break current users of the azure_blob sink unless they are using a connection_string. Given that this is on the critical path to update Vector to use http 1.x, this is probably still worth doing - but existing users will need to migrate their config to use a connection string.

Hi @jlaundry, I missed the context on this one. How does this break existing users? E.g. assume we keep the connection_string and go ahead with that PR. The old configs will still load. Are you saying that it will break in production? Making connection_string mandatory has other benefits so we will probably go ahead with making it mandatory.

[jlaundry]

I've updated the PR with a PoC implementation, which allows either explicitly configured client secret credentials in the config, or Managed/Workload identities from environment variables.

I.e., it will look like:

sinks:
  az:
    type: azure_logs_ingestion
    inputs:
      - src
    endpoint: https://example.newzealandnorth-1.ingest.monitor.azure.com
    dcr_immutable_id: dcr-00000000000000000000000000000000
    stream_name: Custom-vector_CL
    auth:
      azure_credential_kind: managedidentity

or:

    auth:
      azure_tenant_id: 00000000-0000-0000-0000-000000000000
      azure_client_id: 00000000-0000-0000-0000-000000000000
      azure_client_secret: 00-00~000000-0000000~0000000000000000000

Still to do:

• Error handling - currently this config causes panics if the credential creation fails
• azure_credential_kind: clientsecretcredential to allow reading from environment variables skipping for now... the number of use cases for this is low.
• Create some unit tests (I had some WIP stashed in https://github.com/jlaundry/vector/blob/feature-azure_logs_ingestion-tests/src/sinks/azure_logs_ingestion/tests.rs)
• Create a specific config directive for using a Managed Identity User-Assigned ID, which will be needed for the federated credentials.

Once this lands, next steps would be:

• Look at upgrading the azure_core & azure_identity crates to v0.30
• Move the AzureAuthentication struct out of this sink specifically, so that it could be used for Azure Storage in the future (maybe... noting the issue that azure_storage_blob crate doesn't support access keys / connection strings, and maybe won't ever)

[joelkoenka-catonetworks]

Thanks for the update @jlaundry .
I didn't know you can authenticate with an azure_client_secret when working with DCE.
If you have a link to the relevant documentation, I'd be glad to read about it

[jlaundry]

Good news - I think we're ready for final review and a v1 release 🙂

As mentioned above, after merge the next steps for me to look at:

• Look at upgrading the azure_core & azure_identity crates to v0.30 (now that the Azure Storage sink has been tied to azure_core v0.21)
• Move the AzureAuthentication struct out of this sink specifically, so that it could be used for Azure Storage in the future (maybe... noting the issue that newer azure_storage_blob crate doesn't support access keys / connection strings, and maybe won't ever, so there may not be any point)

Also @joelkoenka-catonetworks I'll do another blog post about using Managed Identity with Client Assertion shortly, which will be a better fit for your use-case; but this goes over one of the use-cases I've been testing with, including how to setup an App Registration with DCE/DCR: https://www.jlaundry.nz/2025/using_vector_to_send_commonsecuritylog_data_to_sentinel/