#twisted

new website task https://tm.tl/#9055 by emattiza: Typographical Error in telnet_echo.tac on website

Tickets pending review: https://tm.tl/#8566 (rodrigc), #8966 (the0id), #9006, #9028, #9040, #9042, #8748

Tickets pending review: https://tm.tl/#8566 (rodrigc), #8966 (the0id), #9006, #9028, #9040, #9042, #8748

Tickets pending review: https://tm.tl/#8566 (rodrigc), #8966 (the0id), #9006, #9028, #9040, #9042, #8748

emattiza submitted <https://tm.tl/#9055>; - Typographical Error in telnet_echo.tac on website (unassigned) for review

Tickets pending review: https://tm.tl/#8566 (rodrigc), #8966 (the0id), #9006, #9028, #9040, #9042, #8748, #9055

hi, could anyone check my 20 lines of code. Is that a valid way to handle my protocol messages in a non blocking way? http://pastebin.com/HmTQ4Unb

https://paste.pound-python.org/show/MNmBy11k0Df... (repasted for thaxy)

thaxy, depends on whether what you're doing drops GIL or not

umm sorry whats GIL :/

CPython has a lock that prevents more than one thread from executing Python code at any one time

if the first 4 bytes are length, you should likely be using twisted.protocols.basic.Int32StringReceiver and stringReceived, not just a Protocol and dataReceived

yeah I already do thatn asdf, tanks

Int32StringReceiver does all its work in dataReceived, if you override it like that, then you're not using it :p

I am new to twisted, I just want to make sure that I implement a responsive server and that my deserialization and maybe zip/unzip function don't block anything

Is this zipfile?

Anyway, it depends entire on what you're doing whether threads help or not

If what you're doing involves touching Python objects, most likely not that much

and database operations?

With database operations strong maybe though threads are typically the worst way to do that. You'd preferably want native async DB operations

so it would be fine to execute all your "logic" (if it is fast enough) in the dataReceived function?

so in the reactor thread, I guess.

Hello. I'm new to twisted. Just ran the simpleclient.py But why does it close the connection after receiving data once? How can I keep the connection open and receive further data? (I've commented out the lose connection line)

thaxy, I'd say yes. If you start having heavy processing, I'd consider worker processes instead of threading

multihunter, show the code you're running?

Maybe AMP or whatever to wrap it up

asdf: https://paste.pound-python.org/show/urYVvQw5o37...

ok thanks for the advice. I thought that the callInThread method uses already the built in thread pool and that I could consider them as workers

Threads are not the same thing as processes

multihunter, this doesn't look like it would close the connection by itself; perhaps the other side is closing it?

Threads run under global lock, processes parallelize as much as your hardware allows

asdf: I'm using a program called packet sender doing tests on the other side.

so in that case you would suggest to open a thread via callInThread and use a queue in there for the work which should be done?

multihunter, looks like it closes the connection, then; when i connect your code to a server (i use `$ socat TCP-LISTEN:6000 -`), it stays open just fine

and hey, it's a "sender" but it also listens for connections? :)

asdf, my protocol inherits from Int32StringReceiver. I thought that this would make my dataReceived to return only "full messages" not chunks

thaxy, no, implement stringReceived instead

asdf, -.-" thanks for that

thaxy, Int*StringReceiver implements dataReceived and does the chunking part there, and when it decides it received a full message, it calls your stringReceived

(in general, all protocols do that, so implementing dataReceived when you're subclassing a ready-made protocol is almost always a mistake)

thaxy, stringReceived will be called without the length prefix, so you can lose the s[4:]

asdf, ok thanks it works that saved me some painful hours in the future I guess :)

asadf, jup already got that ;)

asdf: Actually I'm not even sure if twisted can do what I want. The program is gonna connect to a game simulator and receive data (game info - data comes over n over) and store and update data in some variables and then make decisions and send commands to the game simulator

The connection should be kept open the whole time

multihunter, ah, like a bot for a game? sure, it's perfectly doable, but implementing the game protocol will probably be lots of work!

as i suppose it's unlikely you'll find it already implemented

nanonyme, could you give me some more hints where I can lookup how to implement such a thing as worker processes (with regards to twisted)

asdf: Exactly. The game server has some tanks. Sends me positions, health, angles, time and accepts commands like turn_right, turn_left, forward, backward, shoot, etc

multihunter, do you know the protocol? as in, how does the data actually look on the wire? i mean something like, for example it might look like "TANK 1 100hp\nTANK 2 200hp\n" or something

in this simple example you'd have a line-delimited protocol

they're delimited by paranthesis

like this ( self ( 0 19.6806 1500 ) ( -1.36868 0 0.355956 ) ( 100 ) ) ( opponents (

oh, ugh, that's not the easiest thing to work with :)

well there are already codes written in vb.net and c++

the problem with vb.net code is that it's not using any threads and everything is in a while loop. So every command I send will be sent over and over

multihunter, anyway, the basic idea is that dataReceived is called with each chunk of the data - it's TCP, so data can be fragmented; so usually, inside your dataReceived, you add the data to a buffer, then examine the buffer and decide if there's enough data for a full message, then parse it; then remove the parsed data from the buffer and leave any unparsed bits (those are parts of the next message)

btw, parsley is a library that could make this more structured/easier, but it comes with some learning curve http://parsley.readthedocs.io/

and is it possible to send commands while dataReceived is receiving and parsing data?

multihunter, sure, using self.transport.write like that; or schedule a timer using reactor.callLater (or t.i.task.LoopingCall if you want repeated) for something to happen later, etc

btw I can ignore the unparsed bits since it sends me a lot of data even faster than the tanks move

i mean, if you start ignoring data, how will you know when interesting data starts again? :)

I check if data contains "self" "opponents" etc

or I can increase the buffer size

the problem with TCP is, if the server sends "abc", and then sends "def", the client might well receive more packets: "a", "bcd", "ef"; or even it might just receive one packet with "abcdef"

so in this case, eventually you'd get a packet that starts with just "self" because the ( was in the previous part, and this would make parsing a lot harder, right..

right

i mean, of course you can ignore parts of the data, but you can't do it blindly, need to keep track where the ignored part ends

this is always surprising with TCP, because on LAN it behaves much nicer than on the internet...

what if I have a while loop in my decision section? somelike while health > 50: shoot

it's localhost :)

eh, what does a while loop mean here? just keep spamming 'shoot' over the network as fast as you can? :p

:)) that was an example. For example in order to turn the tank I have to use while angle != 90: turn_left

so i'd expect it'd be something like, you parse the messages and see what angle you're at, then you send a "start turning" message, and then just sometime later when you receive a message that your angle is at 90, you send a "stop turning" message