03hawkowl 10r4714214 05A(branches/optional-deps-7995-2/): Branching to optional-deps-7995-2. ...
qjb joined the channel
licensed has quit
dude-x__ joined the channel
03adiroiban 10r4714314 trunk/twisted/web 05A(8101.bugfix) 07M(http.py test_http.py): Merge http-server-invalid-header-8101-2: Fix exception in twisted.web.http.HTTPChannel.headerReceived when malformed data is received. ...
adiroiban closed <https://tm.tl/#8101>; - Malformed HTTP Header causes exception in twisted.web.http server
Hello! What would be the best time to catch someone who could help me solve a mystery on socket writing delays?
mmattice
What's the mystery?
have you strace'd the process yet?
jamesdean
hey, I have not solved it, can't really think of a reason why it is slow... I have posted a question on stackoverflow, but I am writing here, because I know no one will reply there...
I saw that this morning. I noticed you're using deferToThread() calls so that may be your issue
but haven't looked too deep into it...mostly because I'm not a pro with threads
asdf
jamesdean, how are you measuring this?
jamesdean
Well I think deferToThread is the only way to make the server non-blocking. (I think)
the speeds?
asdf
how long transport.write takes, i mean
oh
you dont mean transport.write itself, just the whole request-response cycle?
jamesdean
I mean the time it took to receive the transport.write. I am measuring it on the client side (php side)
$x=microtime(true); \n $data=fgets($this->socket,$this->buffer_length); \n $x2=microtime(true); \n echo "time taken for request #".$this_request_id." - ".($x2-$x)."\n";
It's a peace of the code from the stackoverflow q
piece*
asdf
hmm right
jamesdean
If php would support threading I would have an issue, but you know php...
wouldn't* ... I haven't slept much, can't push the right keys sometimes...
asdf
i'd suggest measuring the time on the python side though
right now you're measuring both the python server AND the delays of the sockets (kernel)
when you're measuring the whole system, it's hard to know where the bottleneck is :)
jamesdean
Well yes, I understand that, but I really do not know how I could measure it otherwise. My only thought of how I could make it faster, is to close the socket and. It is faster ~8 times that reusing the same one. BUT I know it can be solved somehow without doing so. I tried every possible combination I could think of... That is why I'm seeking help on design of it :(
There really isn't much on the php side that can be changed (I think), maybe only the "$data=fgets($this->socket,1024);" could be modified, but I already tried numerous other methods, yet they do not change the times.
I tried to use the MSG_PEAK flag, it is fast, BUT it only receives the first "get" request, so it really isn't an option
asdf
interesting that closing the connection would help
jamesdean
Exactly, it improves the times by 8-12 times... Also I tried using listenUNIX instead of listenTCP - no help there....
Closing it makes sense since it eliminates the delay somewhere there, but back to the question - where is the delay at?...
asdf
hmm, how big are the messages, btw?
jamesdean
15 bytes (for now)
dw
the synchronization in this script looks pretty shoddy
jamesdean
15-20 at max
asdf
you think it's cause of nagles/delayed acks? :p
dw
request_is_being_processed test/set is unsynchronized, so you might possibly be having 0 or >1 threads running concurrently
tomprince
It doesn't look like anything you are doing there is particularly CPU intensive, so getting rid of threads is probably wise.
dw
presumably 'store' dict is a stand-in for a real store implementation
im guessing the problem is almost certainly bad synchronization, the thread(s) are reading from a queue the main thread is mutating, and both are mutating a state variable that both are testing
brodolfo has quit
jamesdean
Yes, "store" will be replaced later. About the synchronization.. In my mind, it seems to be synchronized. When the processing of the current queue item is finished, it checks if there is anything else and runs until it is finished. And once the thread fails/finishes - "request_is_being_processed" is marked as False
There are a few bugs since the last update which I didn't see, but at this amount of calls, it should be the synchronization
I can of course eliminate threading (for now) and see how it goes
OK, I removed the threading - same results
in lineReceived replaced the threading with self.dataParseProcess() \n return None
I mean't "it shouldnt be the synchronization". Update the code on stackoverflow...
dw
sending multiple gets in a row results in self.transport.write being called in the wrong order
because there is nothing serializing calls to it. i have it generating error messages with newlines written halfway in the middle of the error
if it is not generating newlines predictably then your PHP won't be waking up predictably
martinsbalodis joined the channel
did you also remove callFromThread from dataWritePre?
jamesdean
yes I forgot about that, doing it right now
Remove it - no change (well maybe a 0.001 but that is not what I am looking for)
about the new lines - php's "fgets" waits for a new line, and if it is there - stops reading.
I am sending new lines "\r\n" with transport.write every time a "get" is made
dw
jamesdean: is PHP's underlying IO buffering block based?
fgets may return data up to \r\n from the buffer, but the process filling the buffer may be reading in much larger chunks
IIRC there is a function call to make file pointers unbuffered in PGP
jamesdean
hmm, this makes a lot of sense. But I really couldn't find anything related to controlling how the buffer is read.
also, I can't find "IIRC" on google/php documentation