Modify EventCounter usage to support the new metric APIs as well

[noahfalk]

Tasks:

• HostingEventSource
• ConcurrencyLimiterEventSource (obsolete in .NET 8)
• KestrelEventSource
• HttpConnectionsEventSource
• Rate limiting - Add metrics to rate limiting #47758
• Move to Microsoft.Extensions.Metrics - Replace internal Microsoft.Extensions.Metrics with package #47618
• Fix counter names - The names of the metrics produced by the new meters do not seem clear #48309
• Review metrics tag types - Review types passed to metrics tags #48530
• Docs about how to export metrics to Prometheus and Grafana
• Docs about how to unit test metrics - Add metrics unit testing AspNetCore.Docs#30068
• Add protocols to Kestrel connection metric - Kestrel connection metrics - add protocol and TLS info? #47831
• Add authn/authz counters Moved a new issue: Investigate AuthN/AuthZ metrics in ASP.NET Core #47603

---

I've been experimenting with how a library could optionally support the new Meter APIs while also being back-compatible with pre-existing EventCounters. It would be nice to test this on a more real-world use case in ASP.Net to see if we like how it plays out. If it doesn't look good of course we could always change the pattern.

The rough pattern I had was to:

1. Add the new Meter API usage SxS with the EventCounters.
2. Switch code that was directly invoking into EventCounter.WriteMetric + IncrementingEventCounter.WriteMetric so that it instead calls Counter.Add() or Histogram.Record().
3. Add a forwarder so that those Counter.Add() + Histogram.Record() calls with also chain to invoking the pre-existing WriteMetric().
4. For PollingCounter + IncrementingPollingCounter, pull the data from the same API that ObservableCounter/ObservableGauge pull it from.

In total it looked like this...
New Meter code + app logic:

using System;
using System.Collections.Generic;
using System.Diagnostics.Metrics;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApp
{
    class Program
    {
        static Meter s_meter = new Meter("ContusoHatStore");
        internal static Counter<int> s_hatsSold = 
            s_meter.CreateCounter<int>("Hats-Sold", "Hats", "Number of Hats Sold");
        internal static ObservableCounter<long> s_transactionsProcessed = 
            s_meter.CreateObservableCounter<long>("TransactionsProcessed", GetTransactions, "Transactions", "Number of Transactions queued");
        internal static ObservableGauge<long> s_heapSize = 
            s_meter.CreateObservableGauge<long>("Heap-Size", GetHeapSize,
            "B", "Size of GC heap");
        internal static Histogram<double> s_histogram = 
            s_meter.CreateHistogram<double>("Request-Latency", "ms", "Request Latencies");

        static long s_transactions;
        internal static long GetHeapSize() => GC.GetTotalMemory(false);
        internal static long GetTransactions() => s_transactions;

        static async Task Main(string[] args)
        {
            ContusoHatStoreEventSource contusoHatStoreEventSource = new ContusoHatStoreEventSource();

            CancellationTokenSource cts = new CancellationTokenSource();
            Task t = ProcessPretendTransactions(cts.Token);
            Console.WriteLine("Transactions running, hit enter to stop");
            Console.ReadLine();
            cts.Cancel();
            await t;
        }

        static async Task ProcessPretendTransactions(CancellationToken token)
        {
            while(!token.IsCancellationRequested)
            {
                PretendTransaction();
                await Task.Delay(10);
            }
        }

        static void PretendTransaction()
        {
            Random r = new Random();
            s_hatsSold.Add(r.Next(1, 5), KeyValuePair.Create<string,object>("Size", r.Next(3,10)), KeyValuePair.Create<string,object>("Color", r.Next(1000) > 500 ? "Blue" : "Red"));
            s_transactions++;
            s_histogram.Record(r.Next(100, 500));
        }
    }
}

Pre-existing EventCounter code + forwarder:

using System;
using System.Collections.Generic;
using System.Diagnostics.Metrics;
using System.Diagnostics.Tracing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp
{
    [EventSource(Name ="ContusoHatStore")]
    class ContusoHatStoreEventSource : EventSource
    {
        IncrementingEventCounter m_hatsSold;
        IncrementingPollingCounter m_transactionsProcessed;
        PollingCounter m_heapSize;
        EventCounter m_requestLatencies;
        InstrumentForwarder _forwarder = new InstrumentForwarder();

        public ContusoHatStoreEventSource() : base("ContusoHatStore")
        {
            m_hatsSold = new IncrementingEventCounter("Hats-Sold", this)
            {
                DisplayName = "Hats Sold",
                DisplayUnits = "Hats"
            };
            m_transactionsProcessed = new IncrementingPollingCounter("Transactions", this, () => Program.GetTransactions())
            {
                DisplayName = "Transactions",
                DisplayUnits = "Transactions"
            };
            m_heapSize = new PollingCounter("Heap-Size", this, () => Program.GetHeapSize())
            {
                DisplayName = "Heap Size",
                DisplayUnits = "B"
            };
            m_requestLatencies = new EventCounter("Request-Latency", this)
            {
                DisplayName = "Request Latency",
                DisplayUnits = "ms"
            };

            _forwarder.Forward(Program.s_hatsSold, value => m_hatsSold.Increment(value));
            _forwarder.Forward(Program.s_histogram, value => m_requestLatencies.WriteMetric(value));
        }
    }

    class InstrumentForwarder
    {
        MeterListener _listener = new MeterListener();
        public void Forward<T>(Counter<T> counter, Action<T> action) where T : struct
        {
            _listener.SetMeasurementEventCallback<T>((instrument, value, tags, state) => { ((Action<T>)state)(value); });
            _listener.EnableMeasurementEvents(counter, action);
        }
        public void Forward<T>(Histogram<T> histogram, Action<T> action) where T : struct
        {
            _listener.SetMeasurementEventCallback<T>((instrument, value, tags, state) => { ((Action<T>)state)(value); });
            _listener.EnableMeasurementEvents(histogram, action);
        }
    }
}

@shirhatti

[shirhatti]

I'll start with Hosting, but if this pattern works well, we have a few more metrics sources to address

• HostingEventSource
• ConcurrencyLimiterEventSource
• KestrelEventSource
• HttpConnectionsEventSource
• Rate limiting

Thanks for contacting us.

We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[shirhatti]

@davidfowl

[adityamandaleeka]

@shirhatti Can you update this issue with current state and re-milestone it as needed?

[shirhatti]

Related: https://github.com/open-telemetry/opentelemetry-dotnet/pull/2159/files

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[noahfalk]

I forgot we had this issue. The current path I want to explore is if we can have the runtime automatically create a Meter with the same name for any EventSource that creates counters. If that path was successful ASP.Net wouldn't need to do any specific work until you wanted new features not available with EventCounters (such as histograms for latency percentiles).