

Unfair comparisons - bhauer
http://tiamat.tsotech.com/unfair-comparisons

======
sker
Benchmark Nazis will always complain about _any_ benchmark, no matter what.
Don't let them dissuade you from continuing work on this amazing project.

Personally, this project has prompted me to change my career path from C# to
something else, due to the abysmal performance of ASP.NET. I'm still waiting
to see more tests, especially the asp-stripped one, and hopefully some
OWIN/Katana in the near future. But for now, I'm seriously considering
JS/Node.js or Scala/JVM as my main development platform. So thanks.

Is there any date for the next round?

~~~
MalcolmEvershed
I worked on improving the aspnet (on Windows) results and I committed aspnet-
stripped (based on @pdonald's aspnet tests) and HttpListener.

As that committer, I have to ask: Why would you change your career path due to
supposed 'abysmal performance' reported on a website?

Keep in mind that the 'abysmal performance' of many of the frameworks can
still handle millions of requests per day. Plus, many of the 'slower'
frameworks have other development benefits like ease of development, community
ecosystem, static type checking, etc.

The reason I replied is because I'm considering jumping from aspnet to a
'slower' framework like django and I'm not basing my decisions solely on the
results of this collaborative benchmark where I don't even know if experts
have tuned the results, and where I don't know _why_ the results are 'poor'.

~~~
pekk
You can actually know that experts often haven't tuned the results because
many PRs are being ignored ever since the beginning of this project. There is
a pretty clear agenda.

~~~
bhauer
Say what? Which of these [1] is old and being ignored?

[1]
[https://github.com/TechEmpower/FrameworkBenchmarks/pulls](https://github.com/TechEmpower/FrameworkBenchmarks/pulls)

------
mdasen
The area that I would be the most critical of the benchmarks is that it's
unclear whether frameworks that use process based concurrency are being set up
well. With frameworks that use process based concurrency, generally one wants
to run as many processes as can fit in RAM with a nice amount of headroom.
That way, when one app process is waiting on I/O, another will be handling
another request (and the operating system handles the context switching).

Right now, the setup.py for Django specifies that it will create 3 * NCPU
worker processes. On the i7, that would probably be 24 processes and possibly
using an eighth of the RAM in the box. Similarly, for Rails, the Unicorn
configuration specifies 8 worker processes (one process per GB of RAM)
regardless of whether it's on EC2 or the i7 (while Django goes from 6 to 24
processes depending on the box it's on since it's 3 * NCPU). For all we know,
the CPU usage might be around 25% and the RAM usage even less. As such, it's
hard to determine if the speed achieved is because of the limitations in the
framework or whether the box is simply being under-utilized due to sub-optimal
configuration.

If you're going to do another around, more than any other changes, I think a
view of the CPU and RAM usage would really help - if for no other reason than
to show that the test is accurate. Right now, I have questions about whether
the process concurrency frameworks are well set-up. Why would Django benefit
from getting 4x the number of worker processes on the i7, but not Rails? Is 24
processes utilizing the i7 box with Django? CPU and RAM numbers would help
provide answers.

~~~
bhauer
To the degree feasible given the breadth of submissions we've received, we
have worked to ensure that frameworks that use process-level concurrency are
configured to in fact use multiple processes. See question #5 from our
questions section [1].

That said, across the board, inclusive of both process-concurrent and thread-
concurrent frameworks, we never rule out the possibility of a configuration
glitch. We've asked that any subject-matter experts for any framework
perceived as underperforming review the code and configuration. To date, we've
received a great deal of pull requests to deal with many such matters. On the
other hand, very few such tweaks have had a significant impact on performance
data.

We do intend to capture CPU, network, and IO utilization in the future. For
the time being, we invite fans/SMEs to review the code, run the tests on their
own multi-core machine and/or EC2 instances and determine if there is room for
improvement.

[1]
[http://www.techempower.com/benchmarks/#section=motivation](http://www.techempower.com/benchmarks/#section=motivation)

------
frik
Thanks for compiling the Web Framework Benchmarks!
[http://www.techempower.com/benchmarks/#section=data-r6](http://www.techempower.com/benchmarks/#section=data-r6)

I discovered some interesting and speedy platforms like OpenResty. And PHP
performance for multiple queries isn't too bad either - top five.

For round #7, I suggest an addition of the HipHop-PHP (hhvm) Facebook's speedy
open source PHP implementation. [https://github.com/facebook/hiphop-
php/wiki](https://github.com/facebook/hiphop-php/wiki)

~~~
tmister
AFAIK hhvm doesn't have namespace support yet. So Symfony2, laravel, ZF2 can
not be run in it. But basic PHP performance would skyrocket I assume.

~~~
frik
HHVM is moving fast - Namespace support arrived. I heard reflection
compatibility with PHP.net needs some work. They are fixing bugs that prevent
top PHP project from working.

------
morgo
I think it would be interesting if respective technology vendors could be
motivated to tune the configuration, and be bound by some rules. eWeek did a
DB shootout like this 10 years ago, and (as the rumor goes) led to the
creation of the MySQL query cache to specifically beat the benchmark.

Getting the various vendors involved would require a lot of clout though :(

Here is some commentary on the MySQL configuration:
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/config/my.cnf)

* innodb_log_flush_at_trx_commit is not specified, so it's going to default to 1 (durable). This is a little unfair, since other databases won't do this.

* innodb_buffer_pool_size isn't specified, so it's going to default to 128M. A database that uses mmap files will have an atvantage here - since it can automatically use free memory and won't require this level of tuning.

* I thought it was fair to disable query cache though :)

~~~
bhauer
Actually, mostly in an informal matter, we have in fact received many test
implementations from maintainers of frameworks directly. If you go through the
pull requests, you may recognize some folks. I don't want to name drop though.

I had in passing asked a few if they would be willing to apply an "official"
indicator next to their rows. But the response had been uniformly "not
interested." Maybe later I'll ask again. :)

Thanks for the notes on the MySQL configuration. There are a lot of gray areas
in setting up the configuration, but we do seek recommendations such as these.
Aside from concessions we've made to the matter of running a benchmark, the
overriding goal for configuration is that "production-class" configuration is
preferred.

I'm not an expert at InnoDB tuning, so you might tell me that
"innodb_log_flush_at_trx_commit = 1" is not typical for production
environments, in which case I'd want to make a change. But if it _is_ typical
for production, I'd like to keep it as-is.

If the point is that other databases do not have something comparable, that
speaks to the point of the linked blog entry. These things are not directly
comparable, feature-by-feature. We would like Postgres and MongoDB to be
deployed in a typical "production-class" configuration, but that may or may
not be feature-identical with MySQL.

~~~
rsynnott
In many production environments where a few lost writes in the event of OS
crash aren't entirely the end of the world, innodb_log_flush_at_trx_commit is
set to 2. In some it will even be set to 0 (in that instance, data is lost in
the event of MySQL crash). If you care deeply about every write definitely
hitting disk in a recoverable manner, you set it to 1, but at that point you
may have to do other stuff (is the OS lying to you? Is your RAID controller
lying to you? Is your disk lying to you?)

> If the point is that other databases do not have something comparable, that
> speaks to the point of the linked blog entry.

Other databases do have something comparable. It looks like MongoDB actually
doesn't, but that's highly unusual. Postgres does (I believe it's on by
default), Riak does, Voldemort does (one could make an argument that it's
safer to leave it off in the latter two cases than elsewhere).

~~~
bhauer
If I am parsing you right, it sounds like "innodb_log_flush_at_trx_commit = 2"
is what you would recommend for a majority of production environments.

~~~
morgo
Durability will typically be very expensive to implement unless you have a
fast SSD or RAID controller with NVRAM, so it will skew the test considerably
if you make MySQL offer it, but say MongoDB does not have to.

But I do recommend it for the majority of people. Shameless plug:
[http://www.tocker.ca/2013/06/19/deciding-whether-or-not-
to-m...](http://www.tocker.ca/2013/06/19/deciding-whether-or-not-to-make-
mysql-durable.html)

Also: [http://dom.as/2013/08/09/on-durability/](http://dom.as/2013/08/09/on-
durability/) (Domas works on MySQL@Facebook).

~~~
rsynnott
Incidentally, on the EBS durability thing, it's addressed here:
[https://forums.aws.amazon.com/thread.jspa?threadID=27590](https://forums.aws.amazon.com/thread.jspa?threadID=27590)
\- An fsync isn't a guarantee, or wasn't in 2013, but they estimate the risk
of loss after fsync to be very low.

Can't find anything on whether their newer SSD-providing instances guarantee
durability, though.

~~~
morgo
That's really cool. Thanks!

------
janerik
A moving background makes it easy to read! Thumbs up!

~~~
bhauer
Indeed! Glad you like it.

Jokes aside, if you hate it, you can turn it off at the bottom right. Maybe
some day I'll disable it by default. But for now, I kind of enjoy these
responses. Edit: very well, it's disabled by default. :)

~~~
Mister_Snuggles
It does a fantastic job of pegging one CPU!

~~~
bhauer
Yeah, I've noticed that Chrome is especially bad since it still does very
little to GPU-accelerate SVG. Frankly, one reason I had left it enabled for so
long was to make a point about Chrome's bad GPU+SVG support. Why should it be
necessary to use the CPU to move a semi-circle?

But in any case, I'd rather not distract from the points I'm trying to make in
the blog entry, so the background is now off by default.

~~~
rsynnott
Where does it actually run well? It doesn't peg a core on Safari, but
scrolling still feels laggy and unpleasant. I initially assumed you'd
implemented your own inertial scrolling, as with old Google mobile sites.

~~~
bhauer
It runs well on all my PCs, which admittedly are all Windows. It's just an
SVG+SMIL animation. My opinion is that in 2013, it should run entirely on the
GPU. But I acknowledge that's not the case as much as I think it should be.

On my box, it uses ~2.1% CPU and ~16% GPU in Aurora.

~~~
rsynnott
Ah, yep; it actually works quite acceptably in Firefox. Very unpleasant in
Chrome and Safari, though.

------
qwerta
Because benchmarks are usually just useless. It is like what has better
acceleration on free fall? Race car or truck?

------
AsymetricCom
Since we're throwing fruit onto the stand..

I'd love to see Perl Mason on here (the benchmarks)

Make the DB something CPAN like tie::Storable and load it in RAM. Lets see how
that stands up XD

