#ipfs

/

      • whyrusleeping
        jbenet: after looking a bit more, im fairly certain that the msgio buffers are a large part of the problem, i *think* that ipfs holds two buffers per connection, and its unclear at the moment how quickly it cleans up after dead connections
      • i dont think it can tell that they are dead until it tries to send something to them
      • domanic_ joined the channel
      • domanic has quit
      • inconshr_ has quit
      • inconshreveable joined the channel
      • siraj joined the channel
      • siraj has quit
      • siraj joined the channel
      • patcon joined the channel
      • tilgovi joined the channel
      • ipfsbot
        [go-ipfs] whyrusleeping pushed 1 new commit to bithack: http://git.io/pcvzCA
      • go-ipfs/bithack 618a4c7 Jeromy: cleanup, use a workgroup over channels
      • inconshreveable has quit
      • inconshreveable joined the channel
      • inconshreveable has quit
      • patcon has quit
      • chriscool joined the channel
      • inconshreveable joined the channel
      • whyrusleeping
        awesome... every so often, if ipfs runs out of memory in a stress test, my chromebook crashes
      • *most likely* because its trying to swap and my kernel says "whats swap?"
      • pfraze has quit
      • inconshreveable joined the channel
      • btc__: hows the testnet going?
      • btc__
        it's there. i started writing a Golang wrapper to allow remote control of daemons. so we can run tests on them.
      • implemented Add and Cat
      • need to implement Config
      • whyrusleeping
        schweet. when you get that working link me to the code, ill work on making dhthell use it
      • btc__
        we'll want this to run in test suites. So it may be worthwhile to think about how dht can be adapted to that use case
      • eh
      • perhaps not
      • bengl has quit
      • whyrusleeping
        yeah, thats on my plate too.
      • it currently *can* be run in test suites
      • but doing those from within ipfs is difficult
      • i currently have a test box that runs a bunch of dhthell tests and records their outputs and performance metrics
      • btc__: also, im rebasing bithack, and i cant figure out what ContextWithMetadata is supposed to be
      • btc__
        ContextWithLoggable i believe
      • it's in eventlog/context.go
      • util/eventlog/context.go
      • whyrusleeping
        cool cool, thanks!
      • bengl joined the channel
      • ipfsbot
        [go-ipfs] whyrusleeping force-pushed bithack from 618a4c7 to b10543b: http://git.io/YzK52g
      • go-ipfs/bithack 7eea2f5 Jeromy: beginnings of a bitswap refactor
      • go-ipfs/bithack 3759134 Jeromy: dont panic on empty wantlist
      • go-ipfs/bithack 6a55a21 Brian Tiger Chow: test(bitswap)...
      • chriscool has quit
      • chriscool joined the channel
      • whyrusleeping
        jbenet: can mars move to 2 cpus and 2GB ram?
      • ipfsbot
        [go-ipfs] maybebtc created bitswap-blockservice-diplomacy (+5 new commits): http://git.io/EDTjDA
      • go-ipfs/bitswap-blockservice-diplomacy 4289606 Brian Tiger Chow: style(blockservice) s/Remote/Exchange...
      • go-ipfs/bitswap-blockservice-diplomacy a40f51e Brian Tiger Chow: feat(bitswap) make offline exchange query datastore...
      • go-ipfs/bitswap-blockservice-diplomacy dde80f6 Brian Tiger Chow: feat(blockstore) write cache...
      • chriscool has quit
      • [go-ipfs] maybebtc closed pull request #388: refactor(bitswap, blockservice) Give bitswap responsibility for local datastore Gets/Puts (bithack...power+responsibility,peter) http://git.io/_JPcaQ
      • whyrusleeping
        btc__: whats this?
      • btc__
        whats what?
      • "Vim is love, Vim is life." i concur.
      • whyrusleeping
        lol
      • this new branch, whats on it?
      • your branch names always make me smile
      • btc__
        lol. a write cache for blockstore. to cache Put operations
      • whyrusleeping
        ah! gotcha
      • i didnt know commas were allowed in branch names, lol
      • btc__
      • i didnt know until i tried. an example of art driving innovation
      • whyrusleeping
        looking good!
      • inconshreveable has quit
      • inconshreveable joined the channel
      • btc__
        question about some of the failing tests
      • we've got a bunch of non-deterministic tests that sometimes fail due to timing issues
      • in one, i was thinking about calling `t.SkipNow()` if a context deadline is exceed due to non-determinism
      • does that sound reasonable?
      • either that or delete the test =/
      • in both cases the test isn't entirely useful =/
      • whyrusleeping
        well, i am okay with the skipnow idea
      • the tests are still 'useful'
      • but its hard to differentiate between a true timeout and a 'computer is being slow' timeout
      • btc__
        yeah
      • the failures mostly happen when the computer is slow
      • whyrusleeping
        yeah... thats unfun
      • can you make a list of the tests failing due to timeouts on slow machines?
      • i want to look over the,
      • m
      • btc__
      • 106 failures drawn from a sample of 1000 runs
      • whyrusleeping
        cool, only 4 tests
      • TestConnectCollision is testing for a hang
      • not sure how to avoid timers there, should probably just increase the timeout on that one
      • 1 second seems a bit small anyways, especially for one computer to do two crypto handshakes
      • TestNotFound should probably be looked at... that seems a bit odd
      • TestProvidesAsync is ugly, i want to fix it now
      • btc__
        yeah. if we combine longer timeouts with t.Short/t.Skip, all will probably be well
      • whyrusleeping
        agreed
      • btc__
        i mourn for the experience of the dev running tests locally
      • but we'll have to get used to running subsets of tests when building stuff i guess
      • whyrusleeping
        yeah
      • running a full regression will become painful
      • right now it takes around 30 seconds on my machine
      • mildred joined the channel
      • ipfsbot
        [go-ipfs] maybebtc created feat/expose-cmds (+2 new commits): http://git.io/T0l26A
      • go-ipfs/feat/expose-cmds 843d761 Brian Tiger Chow: style: sort command list...
      • go-ipfs/feat/expose-cmds 5bea05d Brian Tiger Chow: feat(core/commands): expose commands to allow for the development of high-level interface...
      • patcon joined the channel
      • [go-ipfs] whyrusleeping created idcmd-fix (+1 new commit): http://git.io/_6WDNw
      • go-ipfs/idcmd-fix 28a3291 Jeromy: fix for #381
      • chriscool joined the channel
      • tilgovi has quit
      • chriscool has quit
      • inconshreveable has quit
      • fiatjaf joined the channel
      • domanic_ joined the channel
      • domanic_ has quit
      • domanic_ joined the channel
      • jbenet
        whyrusleeping: sure we can boost it up but we should find the memory hogs. One other possibility is all the buffered channels on the network stuff. But I doubt it... There not much there. Does the profiler give us mem usage vis? (I believe it does)
      • In any case, modern servers can handle 10s-100s K tcp req/s
      • btc__
        2MB per connection really adds up quickly
      • hmm
      • how important is this one to you guys?
      • domanic_
        req/s is not the same as concurrent connections
      • btc__
        regarding memory usage, it's not surprising that the numbers are higher than we'd like. we just haven't been mindful of our footprint until now. while it's important, reliability and usability are probably more important. tuning memory allocations won't really move the needle on key metrics
      • (talking myself down after considering spinning up a network to collect a memprof)
      • compleatang joined the channel
      • compleatang has left the channel
      • patcon has quit
      • domanic_ joined the channel
      • pfraze joined the channel
      • ipfsbot
        [go-ipfs] maybebtc force-pushed misc/2014-11-11th-hour from 135195d to f4a8aaa: http://git.io/FU-OGw
      • go-ipfs/misc/2014-11-11th-hour ffc92b7 Brian Tiger Chow: fix(net/multiconn) data race in test...
      • go-ipfs/misc/2014-11-11th-hour 962326b Brian Tiger Chow: feat(core/commands): expose commands to allow for the development of high-level interface...
      • go-ipfs/misc/2014-11-11th-hour 2f0d3ff Jeromy: fix for #381
      • domanic_ is now known as domanic
      • mildred has quit
      • [go-ipfs] whyrusleeping pushed 1 new commit to idcmd-fix: http://git.io/katISg
      • go-ipfs/idcmd-fix b0e915b Jeromy: cleanup useless debug statement
      • whyrusleeping
        jbenet: its the number of concurrent connections im worried about, if mars is our base bootstrap node, id love to have it not crash when a flash mob occurs
      • ipfsbot
        [go-ipfs] whyrusleeping opened pull request #396: Fix ID command lookups of unconnected nodes (master...idcmd-fix) http://git.io/sO3ttw
      • jbenet
        whyrusleeping: agreed, and we should def beef it up. But we need to be able to sustain 1,000s/10,000s not just 100s
      • whyrusleeping
        yeah, definetly agreed
      • jbenet: we need to fix the memory usage of msgio
      • jbenet
        whyrusleeping: is it really just that? I suspect it's other stuff. Also since magic
      • whyrusleeping
        well, its half of the problem
      • jbenet
        Since msgio sends the total length in a buffer we theoretically never have t read more than 1 msgio length (I think it's uint32 atm?)
      • whyrusleeping
        jbenet: but every single msgio instance always has at least one buffer held
      • and the buffers are set at 1MB
      • jbenet
        (Never have to commit memory to read more than*)
      • Yeah we shouldn't need to. I can go fix it later