#dat

thanks!

ogd: sent you a signal about _donuts_

ngoldman: great now the FBI knows because they are obviously in this IRC

BE COOL OGD

🇺🇸

mafintosh: hey can you ask andreas about his interest in rabin fingerprinting?

arghhh our job would be so much easier if OSes had `cp --reflink` implemented already

ogd: figured out the chromium/electron CPU usage

beaujingles: wow!!! how??

ogd: it’s a bug in libjingle/webrtc: https://gist.github.com/beaugunderson/aa01ab548...

beaujingles: omg

_pings_since_last_response has a length of 700,000+ in some of those connections

lol

and it allocates a string and concatenates them all each time Connection::UpdateState

two bugs really… one is that it gets into that state to begin with

the second is that it does all of that debug print preparation when VERBOSE isn’t even set, so it’s not even printing the result!

wow

beaujingles: so its just in a debug message function lol

haha yeah, 97% cpu in string concatenation for a message that’s never printed

omg

epic

(though i’m sure having 700,000+ unresponded pings tracked in std::vectors throughout a bunch of connections is bad for memory/perf too!)

wowwww

epic facepalm moment

beaujingles: i wonder why our use case noticed it and nobody else has

:D

prob cause we have lots of connections and keep it open a long time?

ogd: the mesh topology maybe? or, could be that it’s specific to simplepeer? long-lived connections?

yea maybe nobody is really doing long lived data channels yet

since i recompiled everything i did notice that it took longer to occur… whether that’s because of fewer peers or changed code i’m not sure, took a couple days to get really high though

i also sampled it a few times over 24 hours

mafintosh: i bet we could port this to a node addon pretty easily https://github.com/cschwede/python-rabin-finger...

and all the connections i saw had a normal number of unresponded pings, like 5

oh actually i bet i have a linode graph for it

ogd: that looks easy

beaujingles: so the unresponded pings are only tracked for logging purposes?

ogd: tracked for other purposes as well, but the CPU issue we encountered was primarily due to logging

ogd: did we agree on having two leveldbs? one for the graph and one for the database view?

mafintosh: leveldb per dataset folder

mafintosh: and one for the graph

mafintosh: so 1 + N

ogd: what was the reasoning for one for each dataset?

mafintosh: do make it possible to download a dataset fast

mafintosh: without getting all datasets

ogd: as a tarball?

mafintosh: yea to leave the option open

mafintosh: unless you can see a downside

ogd: thinking

ogd: used in TooManyFailures check, for example: https://code.google.com/p/webrtc/source/browse/...

ogd: i think its fine - i'd prefer to add that in a later version though (just to do fewer things and its just the view so its easily regenerated)

mafintosh: k

ogd: only place it’s ever cleared is OnConnectionRequestResponse

beaujingles: just catching up to your libjingle adventure

fun stuff!

hehe yeah

beaujingles: hmm so why "all the connections i saw had a normal number of unresponded pings, like 5"? shouldnt the number be super high even if the logging stuff was disabled?

ogd: that’s before it gets into the bad state

just saying that the bad state can maybe happen suddenly?

beaujingles: ah ok

also i just printed the next 50 breakpoints, so could have missed a connection with more

going to make this process dump a core to study further

was kind of disappointed that lldb seems to have worse STL support than gdb, it couldn’t figure out the length of the std::vector correctly

beaujingles: are you gonna open a libjingle bug for the logging issue?

when i figure out how, yes :)

beaujingles: https://code.google.com/p/webrtc/issues/list

ah sweet

beaujingles: awesome work btw, that bug is quite the deal breaker

hehe it totally nerd sniped me

haha

worth it though

yeah for sure, glad i found a root case for the CPU, still unsure on the root cause for it getting into that state though but i think i have enough to open bugs for both

beaujingles: yea agreed

mafintosh: hey so i tried out that python rabin thing

mafintosh: and it totally works

mafintosh: https://gist.github.com/maxogden/85bab277f3e11d...

mafintosh: i basically edited a random point

mafintosh: the left column is before the edit, right is after

mafintosh: another simple implementation (untested) https://github.com/namelessjon/librabin/blob/ma...

ogd: https://code.google.com/p/webrtc/issues/detail?...

beaujingles: nice

beaujingles: i let the wrtc peeps know

beaujingles: so they dont think its in their code

coo

hah https://github.com/stevegt/librabinpoly#history

mafintosh: hmm might not be relevant but check out 'fingerprinting dags' in this paper http://citeseer.ist.psu.edu/viewdoc/download?do...

mafintosh: also interesting http://delivery.acm.org/10.1145/2530000/2522721...

mafintosh: ori might be the closest thing to dat i've seen actually

random quote from the internet "This idea of analyzing history and looking for evidence about the pedigree of a change is a classic Version Control topic, the sort of thing that practitioners spend all their waking hours contemplating. It's enormously powerful, but devilishly intricate in the details."

mafintosh: also heres the rabin implementation from ori https://github.com/orifs/ori/blob/master/libori...

erikg: you might be interested in that ori paper i linked. has some pretty interesting ideas in it

beaujingles++ great sleuthing on the electron cpu bug!

ngoldman: i’d like to understand the ‘actual’ bug a bit more (the 700k+ unresponded-to pings)

but haven’t had time to dig in to that

cuz one response to the logging bug is “well, that just masks the problem"

(possible response)

ogd: the rabin fingerprinting one?

mafintosh: i'm having some trouble getting pbs to work

because the concatenation of protobuf messages is just a larger message

i guess i need to implement the key/value pairs manually on top of the zerocopystreams

mafintosh: https://github.com/ekg/protobufstream

i'm having some trouble wrapping his head around the idea of a stream of messages that itself makes a valid protobuf message

mafintosh: isn't it the case that the message encoded by the stream will only be valid if there was an entry in the schema that defined a message built up of just repeated elements in the stream?

has nextflow been passed around: http://www.nextflow.io/