#twisted

glyph: looks like it also doesn't handle pipelining HTTP requests - possibly related

vdog: Pipelined HTTP requests are explicitly disabled.

vdog: do you have that poc yet?

glyph: Using openssl s_client won't work, due to the primitave way it handles user input, so I have to write C code

ugh

vdog: feel free to take your existing crummy C example and just attach it to a ticket :). You can always clean it up later

it's a 10 file C project

not suited for a bug report

ah. gotcha.

I should note that in that header file, even the definition references DTLS - this is not a TLS feature :)

glyph: I didn't write it

glyph: Ugh... also doesn't happen with my BIO based C example

vdog: Can you make it happen with an SSLSocket-based C example?

runciter: welcome back!

glyph: I don't have one

glyph: :)

vdog: Oh. So your other C example is just "way more complex" not "not using BIOs"?

my other C example is not an example, its a big program

I'm trying to avoid all the crap of establishing a socket

can you extract the ssl heartbeat/write bits?

so I grabbed the example openssl code, but it uses BIOs

extract the bits?

I could give you a wireshark trace

vdog: oh sorry i mean the C code that writes immediately after sending a heartbeat

vdog: How are you getting the data into Twisted?

let me see if the BIO is setting TCP_NODELAY

I have a C program that opens an SSL connection

its a big program

vdog: is it something where you can't share its source code?

if it isn't, send it my way

that's really confusingly worded - what i mean is, if its license lets you share its source, i don't mind if it's a big program

i can help produce an SSCCE

vdog: the big C program is also speaking HTTPS, though, correct?

glyph: yes

I found new example code, im going to try it out

vdog: It's possible that the heartbeat is a red herring, and the problem is just the HTTP response layer

vdog: The removal of pipelining and the addition of HTTP/2 resulted in a lot of hacky flow-control stuff people were doing in earlier versions of Twisted, which was always wrong, but sort of worked by accident, suddenly just being a hang

glyph: Ok got code

its actually a little more convoluted

glyph: http://pastebin.com/HeD9UAGm

https://paste.pound-python.org/show/Q4RLaX4fAhG... (repasted for vdog)

runciter: http://pastebin.com/HeD9UAGm

vdog: thanks! can you share your Resource code too?

runciter: http://pastebin.com/eNjt5G3f

https://paste.pound-python.org/show/eniYuGyOBYn... (repasted for vdog)

vdog: and how'd you generate they key material?

oh, the certificate?

vdog: well, did you generate the certificates yourself?

let me see if I can find a quick solution for that

yeah

you can just do something like this:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

and update the server.py

er..

the server python file

to point to those files

sure but you have a client cert and stuff?

the client C code ignores the cert

vdog: what version of twisted?

pip freeze says 16.5.0

runciter: see above

k

vdog: and what platform?

ubuntu 16.04 ?

k

vdog: and the behavior you see is that twisted apparently freezes?

right, twisted appears to not read any data after the heartbeat

but it does respond to the heartbeat

vdog: so what i'm seeing is that the heartbeat causes an exception

this does not appear to depend on the version of libssl

vdog: are you up to filing a ticket?

hrm, it only occurs in this weird situation

heartbeats, in general, don't cause that

runciter: do you think this is in the python layer?

vdog: it appears to be an interaction between TLSMemoryBIOProtocol and HTTPChannel

I see

vdog: that is, the exception i'm getting

i can't reproduce the hang

are you doing a wireshark trace?

yep

runciter: are you getting the heartbeat and application record in the same packet?

yep

but twisted still processes the http request?

vdog: twisted blows up with an exception

but can you keep twisted running and have another socket connect?

oh, sure

the exception is that the HTTPChannel no longer has an associated socket

because twisted sees the socket as closed, so it calls TLSMemoryBIOProtocol's connectionLost, which tries to flush received data out of BIO and upwards into the wrapped protocol, so it calls the wrapped protocol's dataReceived, which eventually hits HTTPChannel.requestReceived, which wants to call getHost, but the socket is gone

i'm going to do a terrible hack

runciter: I see, yes my example program reacts the same. I think I know the problem

vdog: oh so it doesn't hang?

because all this means is that a given connection didn't go as planned

runciter: my example program causes the crash as well, but my big piece of code acts differently

let me dig for a bit

the rest of the event loop is OK

vdog: hacking out the offending getHost call against the transport in HTTPChannel.reqeuestReceived lets twisted receive part3

but it closes the connection on the client

I think the problem with the example program is that the receive fails, the program exits, and the socket is killed

yes

well, i mean

runciter: updated code: http://pastebin.com/8KHxeNhi

https://paste.pound-python.org/show/mph2FhZrdKQ... (repasted for vdog)

vdog: we want to print the error that SSL_read encountered on line 269

there, that makes them client/server deadlock

runciter: ok, I don't think it matters, but sure

this is a server side problem, I am almost certain

vdog: when you write C, you *always* need to check error conditions

vdog: ok, so one thing is you are using non-blocking sockets in your C code but you have no select(2) or poll(2) calls

vdog: consequently part 3 (line 269) fails with SSL_ERROR_WANT_READ

i thought this was blocking

and your program fails

i just copy pasted this

ok

this should be blocking..

vdog: it appears to be - i made a bad assumption

vdog: sorry, it's been a little while, but i believe the man page of SSL_get_error is helpful here

I'm working on it

Any TLS/SSL I/O function can lead to either of SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE. In particular, SSL_read() or SSL_peek() may want to write

data and SSL_write() may want to read data.

but, wireshark shows the packet being delivered

for me

wireshark also shows that the client sends an RST

did you use the new code?

no, but you didn't fix the problem of not handling the error condition from SSL_read in the new code

it doesn't matter

it does matter

calling SSL_read will not compel the server to send bytes

yes, it will

no, it won't

if there's a new handshake that needs to occur

the handshake is complete

once SSL_write completes, the data is sent, that is the contract

after that, it is up to the server to make a response hit the wire

sure, error checking on read is what you do in the real world, but it won't make the server put a response on the wire

at least not in this situation

so the difference is that you retry the read while the return code is less than 0?

yeah

it keeps the program from dying, which caused the RST packet

which is how you would handle most of the errors that come out of SSL_read anyways

except that you wait on epoll or whateverr

vdog: ok, so a couple of things

vdog: one thing is that it's only correct to retry the read if you got an SSL_ERROR_WANT_READ/_WRITE; it happens that this is what SSL_read returns, so it's the correct thing to do

but, in this case, the retry blocks forever