Strange ASP.NET performance results?

[MalcolmEvershed]

I just saw the Round 6 results and the results seem way lower than I expected. @bhauer, was there anything weird about the run or environment? Was everything actually updated? Just checking. Here's what looked strange to me:

1. The JSON results show 'aspnet' and 'aspnet-mvc'. What does 'aspnet' represent? It can't represent 'aspnet-stripped' because that pull request has not been accepted. So how can it be in the Round 6 results?
2. Almost all the results are slower between Round 5 and Round 6. That seems pretty odd.

Anyway, what a disappointment: we did all that work profiling and making improvements that caused real performance gains in our test setup and now those gains don't show up in the real run. Weird.

Thanks.

[bhauer]

@MalcolmEvershed Agreed. Let me speak with @pfalls1 tomorrow and see what we can figure out and we'll follow up. Thanks for pointing out the "aspnet" problem. The characterization of "aspnet" is definitely incorrect in the R6 data and I'll need to fix it up tomorrow.

[MalcolmEvershed]

Thanks!

[bhauer]

@MalcolmEvershed We don't yet have any important findings or ideas to share, but I did want to just report:

• The "aspnet" option in the filters panel was added to cover the non-MVC submissions. We have two ASP.NET contributions that, as I understand it, are not using the official ASP.NET MVC framework. I've renamed them in the charts to "aspnet-jsonnet" and "aspnet-svcstk" (Service Stack). I am not familiar enough to understand how the plain "servicestack" test differs from the test that pairs it with ASP.NET. You're right, we don't yet have the newer stripped "aspnet" tests included.
• I had incorrectly labeled several of the Mono tests as "IIS," which caused IIS to show up on the Linux results. That's been fixed.

[MalcolmEvershed]

I think aspnet-jsonnet is really more like ASP.NET MVC using the JSON.Net JSON serializer instead of the default ASP.NET MVC JSON serializer. And aspnet-svcstk is more like ASP.NET MVC using the JSON serializer that comes with ServiceStack instead of the default ASP.NET MVC JSON serializer.

This spreadsheet comparing Round 5 and Round 6 shows that almost all the Windows tests took a hit in Round 6. Note that aspnet-mvc didn't take as big a hit, so I guess the performance improvements are reflected in there.

[FransBouma]

It's also weird that ASP.NET has to go through a full framework with EF (in alpha state) and MVC and is compared against CPP code which simply writes the output of a query to a string which is sent back as result. Apples - oranges. The 'stripped down' version of the ASP.NET tests come with a readme which says "This is stripped down so much it's hard to believe anyone would use this in production", yet the top performers have code which is precisely that: stripped down to the bare bones.

Has anyone profiled the ASP.NET code on mono / linux to see why it's so dog slow? Is it the MySQL provider, mono itself, their ASP.NET implementation?

[bhauer]

@FransBouma Due to the breadth of test implementations, it has been fairly common for readers to conceive of and then criticize comparisons that can be considered unfair when evaluated in isolation. It's important to bear in mind that in this project no single framework is being compared to any other single framework in isolation, but rather a broad field of frameworks.

It's not one apple against one orange. Consider it a comparison of the entire produce section. There are a bunch of apples, a bunch of oranges. You can compare the apples alongside the oranges, or you can ignore the oranges and look only at the apples.

I think most of us would agree that it's at-best a secondary interest knowing how ASP.NET MVC stacks up against a C++ web development platform (I assume you're referring to cpoll-cppsp). Personally, I would never write web applications in C++, but I'm not one to rule it out for others. To me, it's much more interesting to know how ASP.NET MVC compares to other full-stack frameworks. If it's easier to interpret the data with platforms and micro-frameworks hidden, I recommend filtering them out from the view using the filters panel. For example here is the single query test on open-source full-stack frameworks:

http://www.techempower.com/benchmarks/#section=data-r6&hw=i7&test=db&c=1&f=zik0zi-zik0zj-35r

As to the particulars of ASP.NET MVC, @MalcolmEvershed here has spent a good deal of time working on improving matters. Unfortunately, in the most recent round, the results did not improve as much as we hoped. Part of the problem is that it has been difficult for contributors such as Malcolm to smoke-test their changes prior to us running a round of tests because the process has been frustrated by the immaturity of our benchmark scripts and we hope to improve those. In some frameworks' cases, much depends on specific configuration details (version of library X, setting of configuration value Y, etc.). In time, we'd like to avoid the current model of submit a pull request and keep your fingers crossed that the results are good.

[MalcolmEvershed]

I'm the one who wrote "This is stripped down so much it's hard to believe anyone would use this in production". Maybe I should have made it clearer by saying that "I wouldn't use this in production" since there are a lot of opinions of what is ok to run in production (for example, I probably would run most *nix app servers behind nginx which would add more overhead). I'm open to suggestions.

I think a meta-problem/area-for-improvement with the benchmarks is that people that read the results might not know why a particular framework is fast or slow. I think one may be more inclined to use a 'slow' framework if one understood why it was 'slow' or you might be more inclined to avoid a 'fast' framework if you understood why it was 'fast'. For example, the Windows ASP.NET results are highly dependent on the database drivers and some of those are surprisingly wasteful (suggesting that there are very few users and that you wouldn't actually want to use them in production).

I haven't tried anything with Mono.

As to the recent Windows ASP.NET results, I think that the overall IIS<->ASP.NET per-request pipeline has too much overhead to put it in the same ballpark as, say, Go. I look forward to seeing the next run results to quantify how much that overhead is (by comparing it to the ASP.NET Stripped results and the HttpListener results). But I think I've kind of given up on trying to make ASP.NET (a full framework, as you mention) as fast as the bare bones frameworks.

[bhauer]

@MalcolmEvershed Thanks for the thoughts.

As a wishful-thinking / long-term goal, independently I had a goal of enhancing the results site to include a details page on each framework to include elements such as:

• Framework logo
• Home page URL
• Official discussion group/mailing list URL
• Link to the source for the framework's test in the GitHub repo.
• Link to the pull request that originated the framework in our project, if applicable.
• Representative example code snippet (e.g., the multi-query test implementation code); possibly more.
• A brief stats card for the framework.
• Possibly a Disqus comment thread about the particular test/framework.

Looping back to your latest comment here, on such a page it may be interesting to add notes from the test implementer, such as yourself. As I understand the tone of your comment, you would like to be able to communicate some key points that those who review a test's results should understand. This would be a place to discuss, for example, the point about the relative immaturity of the .NET MySQL drivers.

[MalcolmEvershed]

@bhauer, yeah, those notes would be useful. For sure, I'd love to see other test implementors explain the profiler results of their frameworks (like Python for instance).

[FransBouma]

@bhauer I'm sorry, but I have to disagree. It is apples vs. oranges. On the benchmarks site, switch on 'full' only, and you'll see that ASP.NET has 1.1% of the performance of Gemini, which happens to be the framework of the host of the benchmarks. Are you really going to say that ASP.NET is that slow, that it can barely manage 1.1% of the performance of a Java stack? I truly hope not.

I write O/R mappers/modelers for a living and have been dealing with benchmarks for over a decade now, so I have a little bit of knowledge what to look for. However, most people do not, they simply look at the graphs and see: "framework X is much faster than framework Y". And not faster as in 'it can do 2-10% more queries per second', but faster as in 'Y has 1.1% of the performance of X, so that's a no-brainer'. I only have to point to the regular threads about this benchmark on reddit or HN.

Benchmark results which are listed together in a graph or table, are compared to eachother, and you know that, as you grouped various tests together, but not all in 1 graph. However, this only makes sense if the part you want to benchmark is the one which differs in the frameworks benchmarked. Take the 1-query test, the one which is the default chosen to be viewed when I visit http://www.techempower.com/benchmarks/.

The pipeline of the request handling consists of the following parts:

request to DB:
client -> webserver -> webframework -> ORM -> DB Client -> Low-level API -> DB

result to Client:
DB -> Low level API -> DB Client -> ORM -> Webframework -> JSON serializer -> Webserver -> Client.

When I look at the tests for the various languages, only the client and the DB are equal (and sometimes even that's not the case!). All other blocks/parts are different, or omitted. This means that you're really comparing apples vs. oranges.

To proof that, let's do a little thought experiment: let's say for all frameworks all parts are performing equal, except the DB Client + Low-level API: on .NET it performs horribly, and the C++ part performs excellent (and the rest is in between). This alone could cause the variation in performance as shown today, and make framework X look like a much better choice than framework Y, but under the hood, they all performed the same, as by rule of this experiment the only part which fluctuates in performance is the DB client + Low-level API to the DB.

Unless it's proven how much time is spend in the element benchmarked, it's useless to enlist the frameworks in 1 table, however because they're placed together, they're compared together, even though one can filter out some of them.

If I look at the ASP.NET code, I can't really find an area where code is likely to be slow. Perhaps a few cycles can be gained here and there, but that's minor. It's also not the area where the slow performance is located. We don't have to look at ASP.NET code to see what I mean, just look at the cpoll-cppsp code on MySQL (144,711) vs. cpoll-cppsp on PostgreSql (79,562). That's a 30% drop in performance by simply switching to another DB client, low-level API and DB. 30%! I.o.w.: the time spend in the web framework is minor compared to the time spend in the DB Client, low-level API and DB, because otherwise a switch of DB client wouldn't make such a difference. (a 10% speed gain in a part which takes 30% of the overall execution time doesn't make any difference on the overall execution time).

I'm sure ASP.NET isn't the top performer, but I'm also pretty sure it's impossible to shine in this benchmark unless the DB client and low-level API code used by the framework are top-notch. 1 glance on the MySQL C# provider code and one knows this won't be a winner. Entity Framework isn't a top performer either (see my own benchmarks of various frameworks which only differ in ORM/data access code: http://pastebin.com/AdsKitr3) but it's also not that dog slow that some javascript code is more than 10 times faster.

IMHO, if you want to benchmark the web frameworks, you should use in-memory objects and return those as JSON. That way, you test the speed of handling calls from clients and how fast objects can serialized to JSON and at least you keep more parts of the complete chain equal to one another.

I've asked the ASP.NET Team to see whether they can help (if they care, that is ;)).

[pdonald]

IMHO, if you want to benchmark the web frameworks, you should use in-memory objects and return those as JSON. That way, you test the speed of handling calls from clients and how fast objects can serialized to JSON and at least you keep more parts of the complete chain equal to one another.

We already do. There is a JSON serialization test and a Plaintext test. ASP.NET is still at least twice as slow as webgo.

[FransBouma]

We already do. There is a JSON serialization test and a Plaintext test. ASP.NET is still at least twice as slow as webgo.

There too, ASP.NET has 1.0% of the performance of Gemini (if I filter on 'full' frameworks only). Gemini 215,536 responses per second, ASP.NET a measly 2,187.

On the plaintext tests, it's even worse: Gemini scores 454,617 responses, and ASP.NET scores 1,828. I'm sorry, but are you seriously saying this massive difference is because ASP.NET is slow? I can only conclude based on these numbers, something is seriously wrong with the ASP.NET setup. I do that because e.g. the full stackexchange sites run on a full ASP.NET stack, and if the ASP.NET numbers would reflect real-life performance, they'd never manage to run even a fraction of their sites on the small server farm they're using.

ASP.NET might be slower than Gemini, but not that slow that it can barely manage to get 0.4% of Gemini's performance in the plain-text tests.

[pdonald]

What you are looking at is mono on Linux. To see Microsoft's ASP.NET results, click on Win under Hardware.

[FransBouma]

@pdonald I'm not a retard. I was referring to the main charts on the benchmark site, as that's what's linked to, that's what's discussed out there, and you know that. But if I compare EC values: JSon serialization on EC2: Gemini: Linux, 26440, and ASP.NET: Windows, ~ 3K, this means ASP.NET still has about 11% of the performance of Gemini.

Of course, they're not the same OS, but as many elements in the complete chain aren't equal, why would 1 extra element be a dealbreaker....

This thread IMHO shows the benchmark owners don't really show much concern about the ASP.NET numbers. You're not obligated to, of course, don't get me wrong, but IMHO if this benchmark is meant to have any real value, you should care and make sure it's fair so the obvious conclusions people will draw from it will indeed have value and they can base decisions on them (because people will do that).

[pdonald]

What makes you say that we don't care about ASP.NET numbers?

This is a quote from the OP:

Anyway, what a disappointment: we did all that work profiling and making improvements that caused real performance gains in our test setup and now those gains don't show up in the real run.

• Improve ASP.NET performance by up to 13%
• Heavily stripped down version of aspnet tests: twice as fast for some tests
• Improve ASP.NET CPU utilization by adjusting minWorkerThreads
• Suggestions to improve perfomrance

make sure it's fair

Can you please take a look at these files and tell us what's not fair about them?

• JSON controller
• Plaintext controller
• Web.config
• Build & deployment script
• IIS installation

Any suggestions on how to optimize the ASP.NET code and also IIS configuration are welcome.