0

I am building a system which has high scale requirements (> 1 million requests per second). I am building this application using Azure service fabric cluster(s).

I have read and seen videos on ETW logging - https://blogs.msdn.microsoft.com/vancem/2012/08/13/windows-high-speed-logging-etw-in-c-net-using-system-diagnostics-tracing-eventsource/

http://answers.flyppdevportal.com/MVC/Post/Thread/b0302547-7ffb-4ff2-aef6-6e15ebe695b3?category=azureservicefabric

https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-event-aggregation-wad

I am still not sure what is a long term choice for logging in my system. Some of the questions I have -

  1. ETW is fast, but does it support all logging features viz. logging performance counters, log levels viz. Debug, Info, Warn, Error etc.
  2. For my scale requirements (>1million requests per sec) Is application insights good enough ? Why should I used ETW logging over App insights ?
  3. What can I not get from application insights that I can get from ETW logging ?
  4. In terms of overhead on application thread/process is ETW significantly better than application insights or they are similar ?
4

1 Answer 1

3

I think you are trying to compare apples with oranges, let me explain why.

AI

Application Insights (AI) is a sink, which means you log something using the SDK and it will send it to AI. AI will buffer written events and send them once every while (I believe the default is 30s) to the cloud using http calls.

AI is never meant to be used for massive, high scale logging. If you take a look at the pricing you will see it will cost you dearly when large volumes of data are coming in.

ETW

ETW is not really a sink. You can log anything to an eventsource but as long as no listener is attached your events are going nowhere. The listener determines where the events are logged to.

For high scale metrics logging the team has extended the EventSource with support for EventCounters (see this doc)

The good thing about ETW is you can attach a listener in the same process that also writes to the EventSource or you can create a listener in a separate process (on the same machine) and you can then configure where your logging should go to. That could be an ETL file that is later analyzed or processed or it can go to a high scale data ingestion endpoint like Azure EventHub and from there to, for example, a data lake or blob store for further analysis.

The questions

ETW is fast, but does it support all logging features viz. logging performance counters, log levels viz. Debug, Info, Warn, Error etc.

Yes it does. You can enable logging by severity level and / or keywords.

For my scale requirements (>1million requests per sec) Is application insights good enough ? Why should I used ETW logging over App insights ?

As explained I think AI is not meant for this kind of scale in terms of performance and price. Though data is collected / buffered in a separate thread it is not designed to handle this kind of load. It is limited to 32K events per second (source)

What can I not get from application insights that I can get from ETW logging ?

Performance and flexibility in where logged events end up.

In terms of overhead on application thread/process is ETW significantly better than application insights or they are similar ?

No they are not similar, ETW has much lower overhead. Its support is baked in the OS and it is just blazing fast.

You can use EventFlow as suggested in the comments, it does support in process and out of process EventSource listeners.

If you end up using other logging framework do pick one that supports structured logging as ETW does.

Final thoughts

I think you should first of all think about what you want to log and what you want to do with it. It could be perfectly acceptable to log exceptions and some metrics to AI so you can have live monitoring of your application and log other metrics or usage details to another store like azure tables for non-real-time analysis. Don't put all your money on just one sink but determine a log strategy first.

AI strength is rich visualizations but that comes at a price. Azure Data Lake analytics does not support visualization out of the box but using u-sql on Petabytes of log data can be useful for other scenario's. I hope you see where I am getting at.

1
  • Thanks @Peter Bons for detailed and insightful answer. In my first milestone of the product I am using ETW as It came integrated with my service fabric cluster when I created the application in VS. I added EventFlow to send the logs to App Insights for now. As you said with with ETW / EventFlow I can change the sink without changing the application code. Thanks again. May 21, 2018 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.