#twisted

Hi

I have a little bit of a situation

I have 5 million irc addresses and I need to connect once to each of them and store the output that they provide on connecting

I have a primitive script but it doesn't throttle and I run out of file descriptors pretty quickly

Here's my quick hacky code https://u.teknik.io/YjZbg.txt

aix, if you'd like some rate limiting, first, you need a way to know when a connection is done

so imagine eg. your "bot" function creates a `d = Deferred()` and passes that along with all the other info into the factory, so the protocol has access to it. Then when the protocol decides it's done, so eg. in the connectionLost method, it calls `d.callback(some_return_value)`

and let's have the "bot" method return that deferred

if you just do that, then you'll be able to use DeferredSemaphore with that function

so, instead of `bot(spl[0], 6667, blahblah)`, you would first prepare a semaphore `sem = DeferredSemaphore(100)`, and then just keep running `sem.run(bot, spl[0], 6667, blahblah)`

ah

alright

that will still create all 5m deferreds in memory up front

and 100 is the maximum fds at any one time?

but meh that's like what, less than 100mb probably

yep that's how many of the bot() functions can run at a time

the semaphore will examine the deferred that the function returns to know when to run the next one

this is exactly why i love twisted

btw if you used the more recent endpoints api, you could get rid of that factory entirely so the code would be a lot shorter

alright

aix, here's an example: https://bpaste.net/show/c12b9f254119

https://u.teknik.io/GMw0T.txt this one look about right?

hmm .callback() needs an argument actually, can't fire the deferred with nothing

so you could store some data on the protocol while the connection is under way and return it there

well you'll also want to actually write the __init__ with storing .nickname and all that :)

alright so

the connectionlost callback?

indentation is wrong there too btw, you're mixing tabs and spaces

yep, i mean, `self.finished.callback()` takes a mandatory argument

can just do .callback(None) if you dont want to return anything

ah okay

btw you copied the `.addCallback(all_done)` but didn't actually write an 'all_done' function

do i need an all_done method?

if you want one

you didn't actually run the reactor either btw

right now, this script kinda does nothing

oh

that'd be why

so twisted.internet.task.react is a newer way to run the reactor, so it takes a function that returns a deferred

like `@react def main(reactor): results = []; blahblahblah; return gatherResults(results)`

and it'll exit when that gatherResuls fires

but just doing plain old reactor.run() is fine too

alrighty

so is this how i should be doin it? https://u.teknik.io/5R8Pg.txt

doesn't seem like i'm getting any output

reactor.run() is blocking btw so the gatherResults will never run

and I think the reactor is either not getting called or blocking something

hmm looks fine, probably some error somewhere, let's see..

except for the indentation with mixed tabs and spaces >.>

so what output are you getting? anything at all?

the 'all_done' func needs to take an argument btw, it'll be a list with all the results

well, it prints addbot loads of times

then waits

and on ^c prints loads of these https://u.teknik.io/BFLgS.txt

definitely doing something

loads as in should be 100?

keeps printing addbot

not sure how many it actually loads

yeah, it'll run the whole loop up front preparing the calls, and then it'll be calling one by one, up to 100 concurrently

I'll try with a million

okay maybe just 10k

uh

looks like it loads all of them at once

and i don't think all_done ever gets called

it does load them all at once yes

it calls sem.run() the 10k times

and then sem will actually run the functions

with the rate limit

ah okay

oh

i screwed up the indentation

you'll want the gatherResults line before the reactor.run() btw

https://u.teknik.io/wYu8T.txt there, proper indentation

HAH

it works!

nice

very slowly

I think my first script did it faster

crank it up to 1k at a time or something i suppose

it's at 900

or maybe there's only 2 irc networks in the first 10k lines

i'll run it on a better box with a higher ulimit and see what happens

oh btw remember to call the superclass in connectionLost

otherwise it'll leak LoopingCalls because ircclient uses one for the ping

just like `irc.IRCClient.connectionLost(self)` if you're on py2 because i think that's still an old-style class

the superclass?

i mean like super()? but you can't actually use super() with old-style classes

so you need to write the method yourself

i mean, spell out the base class

okay, there

i'll leave this running overnight and see what happens

break 10 minutes after you detach, what else ;))

okay other than my box freezing up i don't think anything is happening

how's the RAM use?

1.89/1.93GB

maybe loading all the data up front wasn't a great idea after all

huh, that's a lot

it goes there right after loading the lines?

if i try something like 10k it works fine

pretty fast too

hmm ok maybe let's try the other way then

described here: http://as.ynchrono.us/2006/05/limiting-parallel...

sec i'll show an example

asdf: hold up

i'm gonna have to sleep soon

perhaps we can continue this tomorror?

tomorrow*

yeah me too but this should be fast to check

alright

how about this https://bpaste.net/show/e50194aa4d4e

at the bottom i've put the code from that article

so basically this will only go through the file as needed

as opposed to loading the whole thing up front

alright

so the DeferredSemaphore needed like O(lines) of memory, this just needs O(concurrency)

so this is all generators, right

the file loader, and the `work` inside the 'parallel' function

finished = parallel(data, 100, run_bot) what does the 100 mean in this line?

that's the concurrency

ah

alright, trying it out

oh

add the base class call to connectionLost again

because i didn't in this paste :)