#techempower-fwbm

Link to thread about R10p1 data and a message soliciting input about SQLite: https://groups.google.com/d/msg/framework-bench...

Please weigh in with your opinions. I'd like a tiny bit of consensus on the record before acting.

[FrameworkBenchmarks] knewmanTE opened pull request #1344: reimplemented Gemini-Postgres due to Travis error (master...gemini-postgres2) http://git.io/AQOa

lwan results are pretty weird in the plaintext benchmark.

ah, it seems it's doing pipelined requests. lwan doesn't play well with those yet. is there a way to not use them?

yeah, that seems to be the problem; the json encoding benchmark doesn't seem to use the pipeline script.

That is the case. The plaintext test is the only one to use pipelining.

We do not presently have the ability to selectively disable pipelining on a per-framework basis for the plaintext test.

would it be too late to come up with a patch to do that?

sems unfair

every test should be run with the same command

or it's not the same test

if that ends up being the conclusion, I'll pull out the plaintext test from lwan, as it's testing something it doesn't support (which is also unfair).

That seems best for Round 10, acidx.

It seems perhaps I should be more clear about pipelining in the requirements for plaintext? All I have in there is this: "Server support for HTTP pipelining is strongly encouraged but not required."

I should be more clear that tests WILL be conducted with pipelining.

that's why I didn't bother writing proper pipelining support (also because pipelining isn't enabled by browsers, anyway).

acidx, the rest of us did bother adding pipelining support, specifically for this benchmark. :P

I will take the blame for a badly worded requirement.

I'll write something more clear immediately.

Smerdyakov: it said it wasn't required, only encouraged.

As far as I was aware, a server that did not support it would still accept the requests and fulfill them as it could.

Smerdyakov: but it's OK, it's too late to change that now. I'll pull out the plaintext test for this round.

acidx, the open-source benchmarking code reveals the details, but I agree that it's a feature to have that behavior summarized in prose.

brianhauer: one of the reasons browsers do not enable pipelining by default is that some servers do not play well with it.

acidx: probably a lot of servers doesn't support it properly

and that's why some are slow

for this test

acidx, do you know what accounts for the large difference in lwan's ranking for json vs. database tests?

I have just posted a revision to the requirements that says "Requests will be sent using HTTP pipelining" in the introduction paragraph and changes the enumerated requirements to say "Server support for HTTP pipelining is required."

brianhauer, so now those who don't want to implement pipelining can't even play?

Well...

That is not my intent.

I was quite happy to see lwan in the first row for the json test (which is heavier than the plaintext, obviously), but that sort of went away quickly when I saw the plaintext results :P

brianhauer, but the word "required"....

how about just make a plaintext that doesn't use pipelining

I mean "support" in a loose sense here in that the server should gracefully respond.

in addition

Smerdyakov: database access is synchronous. it wasn't tuned or anything, I wrote that for the benchmarking just for completeness sake.

acidx, you mean synchronous with just one action pending at a time, across all requests?

How about "Server support for HTTP pipelining is assumed. Servers that do not support pipelining may be included but should downgrade gracefully. Confirm your server's behavior with the wrk load generation tool" ?

Smerdyakov: basically, yeah. only one request at a time per worker thread.

" HTTP/1.1 conforming servers are required to support pipelining. This does not mean that servers are required to pipeline responses, but that they are required not to fail if a client chooses to pipeline requests"

acidx, and you don't have 40+ worker threads?

that's what wikipedia has to say on the matter

Smerdyakov: everyone else waits.

xt, I will link to that.

acidx, OK, now I understand.

brianhauer: source http://www-archive.mozilla.org/projects/netlib/...

Revised and posted: http://ft.trillian.im/8e77b44609228d593aedfcbaa...

let's see if I can at least find the reason Lwan barfs with pipelined requests.

acidx, is lwan used for dynamic web apps in the wild? If so, which languages are those apps usually written in?

Smerdyakov: there seems to be users as I get the occasional bug report, but I have no idea what they're using it for

OK. All these C and C++ entrants into TFB have me wondering if anyone is actually writing serious web apps in those languages in the wild.

I really doubt it. :)

Hahaha! I had wondered the same thing.

I personally would never do it, but I suspect there are those who would.

I actually thought of doing a discuss-like service to have comments on my blog (which is just a bunch of static files), but I'm too lazy to even start that.

Mostly I don't trust myself to not end up writing an application rife with security defects.

that can be tricky with C, yes, but security defects are not just a problem of the language.

I suspect I already have enough unknown security defects in applications I am writing on relatively "safe" platforms like the JVM.

there's support to write handlers in Lua with Lwan, for instance. it's not as complete as openresty, but I got an existing web framework running after some persuasion with a sledgehammer

Not all security issues are determined by programming language, but many can be _ruled_out_ by language choice.

Smerdyakov: This paper is awesome.

while trying to see why pipelined requests wedges Lwan, wrk says the latency is -nanus.

brianhauer, thanks. :)

yeah, the paper it's already in the queue to be read.

[FrameworkBenchmarks] kevmoo opened pull request #1345: Experimenting with 32-bit vs 64-bit server setup (master...dart_hacks) http://git.io/AQrX

ah, found a way to get pipelined request to work in Lwan.

brianhauer: still around?

[FrameworkBenchmarks] kevmoo closed pull request #1345: Experimenting with 32-bit vs 64-bit server setup (master...dart_hacks) http://git.io/AQrX

got pipelined requests working in Lwan. still need to polish things, but it's ~50% faster.

I guess I'll be able to compete afterall :)

hmmmm, looking at the preview data, I think I should put nginx back in front of my server :s

and maybe play with epoll again

the latency numbers seem WAAAYYY out of whack

jberger: huh. Note that there was a lot of contention about wrk vs wrk2 - https://github.com/TechEmpower/FrameworkBenchma...

Multple claims that wrk2 gives you trustable latency metrics while wrk doesnt

looking at the other perl frameworks, I also notice a TON of errors in theirs, is that being measured correctly?

they are still getting the measured rps ignoring the failures?

I can’t speak to how wrk actually does what it does internally - and TFB just uses the values it returns IIRC

well I also need to tune the mojo app better

I must say, I'm a little disappoint

Oh, that’s a bummer

Well at least mojo is at the top of the pack for all perlisms

Except for dancer-raw on multiple queries, but…it’s using raw

hm, I guess mojo is classified as raw there too though, so that’s just a straight up case where it’s doing better. Wonder why

FWIW, there are definitely frameworks that sets up nginx on one configuration and don’t use it on another

dancer is using mysql

which is why I want to make a pg variant at least

[FrameworkBenchmarks] methane opened pull request #1346: Fix uwsgi errorlog is written to `--ini` file. (master...fix/uwsgi-daemonize-err) http://git.io/A5T7

[FrameworkBenchmarks] bclozel opened pull request #1347: Upgrade Boot and customize Undertow (master...boot12) http://git.io/A5np

[FrameworkBenchmarks] lpereira opened pull request #1348: Update Lwan to a version that supports request pipelining. (master...lwan-pipeline) http://git.io/A5Bl

not bad

lets hope every python web framework will support asyncio in the future

acidx, I noticed that your TechEmpower-specific code is in the main GitHub repo for lwan. I don't know if this issue has ever come up before, but IMO that might be considered a violation of the rules.

how so?

Well, eventually there will be comparison of source-code stats across frameworks, which will only look in FrameworkBenchmarks.

It's also inconvenient for folks who want to compare frameworks.

ah, I see

is there rules about that?

I don't remember seeing them.

But maybe there should be such rules.

pretty sure it's done for convenience, not to lower the line number :-)

so you came of sounding a bit harsh

Yeah, I'm sure.

But that doesn't mean it isn't appropriate to move some files around.

agreed

[FrameworkBenchmarks] msmith-techempower closed pull request #1336: Gemini postgres (master...gemini-postgres) http://git.io/A3uK

[FrameworkBenchmarks] msmith-techempower pushed 2 new commits to master: http://git.io/AFOW

FrameworkBenchmarks/master 072abd0 INADA Naoki: Fix uwsgi errorlog is written to `--ini` file.

FrameworkBenchmarks/master 2a5c6c6 Mike Smith: Merge pull request #1346 from methane/fix/uwsgi-daemonize-err...

[FrameworkBenchmarks] lneves opened pull request #1349: Update netty to chase the performance enhancements made since the latest release (master...master) http://git.io/AFuH

[FrameworkBenchmarks] codylerum opened pull request #1350: update to wildfly 8.2 (master...wildfly-8.2) http://git.io/AbJH

Smerdyakov: well, I hope that happens only for the next round.

acidx, yeah, doesn't seem urgent.

[FrameworkBenchmarks] msmith-techempower deleted gemini-postgres2 at 0c91f8d: http://git.io/Ab5i

[FrameworkBenchmarks] stefanocasazza opened pull request #1351: better scalability (master...master) http://git.io/ANcD

TE-msmith: gh#1352 would be a good starter if you have employees working on learning TFB (after R10 is done, naturally)

Remove Gemini's WEB-INF lib folder: https://github.com/TechEmpower/FrameworkBenchma...