#dat

[dat] maxogden pushed 1 new commit to master: http://git.io/deayfQ

dat/master 22091b8 Max Ogden: slightly better write stream feedback

mafintosh: i think our stats should switch from happening on stream write to happening at a fixed interval (i think you have said this before)

yea

mafintosh: but in the meantime i switched dat import to show rows/sec based on the parser stream in write-stream, not the write stream results data which comes in batches thanks to byte stream

mafintosh: now it updates a little better, but still has some pauses as the batches are writing and the stream slows down

maybe we should change the progress stream a bit as well

i mean the stream being piped to dat.progressStream

mafintosh: i exposed the parser stream https://github.com/maxogden/dat/blob/master/lib...

mafintosh: and pipe that to the progress stream now

also im gonna use 'processed' instead of 'parsed'

how about emitting a progress event instead having a progress stream?

it could emit something like {blobs:4, documents:42, bytes:10000} etc

well the progress pauses sometimes

e.g. when stuff is waiting

so the setTimeout/setInterval logic has to happen somewhere

we could change the progress stream to start a timer on init

and write at a fixed interval

then expose the .blobs , .documents .bytes as well on the return value and emit 'progress' when these change

and then clean up on finish

mafintosh: yea that sounds good

then i could do the same in the replicator

ogd: i found the replication bug! :D

pushing out a fix

it only replicated the first blob for each doc

restarting the clone on the other machine...

ogd: i'm gonna do this interface for progress monitoring, https://gist.github.com/mafintosh/e64e1199645a4...

then dat.progress can just poll the props every 0.5s etc

750 secs into clone and sofar 25gbs transferred

mafintosh: nice!

much better.

mafintosh: +1 to that progress api

mafintosh: you wanna rip out progressLogStream or should I?

i can do it tomorrow

kewl

mafintosh: hows the clone going?

87gb cloned sofar

2800s in

ah nice, tailing output.log now

dang didnt beat an hour

ogd: re conf notes + broader project notes, use github.com/datproject/meta ?

jbenet: +1

yay it finished!

there it is

a little over an hour for 114gb :)

i think theres a bug where it doesnt call .close when clone finishes....

no it closed

oh ok cool

your tailing the output :)

oh right haha

mafintosh: ok lets start 10 clones at once

or 3

one on the server and one on each of our laptops

and watch the original to see how it does w/ serving

kk

cloning now

mafintosh: did you push your attachment fix?

yeah - you need to npm install again

k

i deleted the npm clone on the second DO server

pulling on the good side of 2mb per sec here

to my laptop

my local npm is being dumb... one sec

time is approaching 4 in the morning here so I'm probably going to bed soonish

haha

i can help! i dont have 100 GB free, but I can run it for a while to give it load.

plus am in a coffeeshop, packet loss guaranteed :)

latest dat, dat clone npm.dathub.org

git dat, not npm i dat

yeah, git pulled :)

ok pulling on DO#2 and my laptop

cloning*

ogd: thoughts on `dat [--]version` ?

oh yea that should be there

should be easy, i attach a .version property to dat instances (i think its .version)

i'll send a pr shortly. do you prefer subcommand or flag?

both -v and --version should work

yea sgtm

dat version might as well

mafintosh: looks like the server is pretty busy, throughput on DO#2 is lower

mafintosh: go to bed though!

yea DO#2 is eating all the cpu

that was expected though

since the network is super good between DO#1 and DO#2

we could consider adding a throttle stream to the replication api impl

that would probably distribute the load better

alright guys - gonna call it a night. see you tomorrow :)

night!

ogd: finished writing. do you want me to push to PR or push to master? (dont think i have collab access)

https://github.com/jbenet/dat/compare/maxogden:... <-- chnages

(tests pass, except 1: live rep, but think that's ok

jbenet: just added you

jbenet: live rep tests should be passing now.... hmm...

[dat] jbenet pushed 1 new commit to master: http://git.io/vsPINA

dat/master bc76da1 Juan Batiz-Benet: added version cmd

w00t

ogd: they failed for me on prev head too

(22091b8)

:D

jbenet: can you try deleting the dat/node_modules folder, then doing npm install && npm test again?

ahh, that might be it, sec.

[dat] maxogden pushed 1 new commit to master: http://git.io/JCIT7Q

dat/master 159d29c Max Ogden: update collaborators

there should be a way to add post receive hooks to git that ship with the repo. so like, we could make it npm install every time a dep changed.

agreed

npm install takes forever on coffeeshop-fi.

hah

soon you can just run npm locally w/ dat :)

ogd :D yay

also soon, ipfs mounted node_modules cache. "you already have it" before you download it.

hah yes