#dat

if it's gonna be json without keys could it be csv?

Keys might be nice for lookups if there's a lot of commits. It's hard to know

At least csv or other kind of delimited is less content to download total, with lots of commits

If we have lots of commits it actually looks like a feed but maybe we don't expect that many?

ya i think it could be a csv. im not sure how many we'd expect. but i think the idea behind json is that it'd be easier to put in feed later if we want to (not that csv will be that much more difficult)

yoshuawuyts: I once tried to make a diffing tool quickly http://github.com/Karissa/knead

karissa: that looks neat!

yoshuawuyts: something like this but desktop app and w front end modules would be doppee

ayee, yeah for sure

yoshuawuyts: https://github.com/mafintosh/pick-random-stream (my fav example of a streaming algorithm that normally would be sync)

mafintosh: oh I like that

hmmmm, streaming algorithms for to manipulate / analyze data

yoshuawuyts: statistical analysis

yoshuawuyts: there was a great talk about this at jsconf.eu some years ago. i'll see if i can find it

yoshuawuyts: http://2013.jsconf.eu/speakers/julien-genestoux...

mafintosh: thanks!

yoshuawuyts: i had an idea to use pick-random-stream to build a realtime lottery

mafintosh: lol, what would be the prize?

yoshuawuyts: i dunno lol

yoshuawuyts: but i like the idea of it being realtime

yoshuawuyts: like a simple website that shows the "current winner"

and then you can sign up to join the lottery

and after two weeks, the "current winner" wins

hahaha

I like that

yoshuawuyts: oh. what if we made a "the-best-?" website

yoshuawuyts: where you could start a lottery for picking some sort of picture

and that was the entire concept lol

mafintosh: haha, so it randomly selects images and eventually one wins

mafintosh: like submitted by people?

yoshuawuyts: yea

mafintosh: haha neat

yoshuawuyts: so i could start one to pick "cat picture of the week"

eljee: i'm workin gon the gateway thing now

eljee: unless you already made progress?

mafintosh: https://github.com/mafintosh/hypercore-protocol...

mafintosh: https://gist.github.com/juliangruber/b9468fe46a...

juliangruber: added a PR for testing the mtime bug in hyperdrive-import-files (https://github.com/juliangruber/hyperdrive-impo...). I tried with the raf update & your hyperdrive mtime PR but it is still failing. I can't quite figure out why the hyperdrive test works but this fails...

jhand: awesome! yeah there is still a pr that needs to be merged after some refactoring, one sec

^ jhand cc mafintosh

juliangruber: ya i was trying with that one and the mtime still was changing

jhand: thanks for writing the tests!

oh ok interesting

juliangruber: i *think* the test shoudl work with that PR, but maybe I was reading it wrong

(the test I just wrote in hyperdrive-import)

jhand: do you have latest raf too?

juliangruber: yep

it needs to have the .end method

ok

juliangruber: but I couldn't get a hyperdrive mtime test to fail, so maybe I'm missing something

jhand: could be another bug in hyperdrive-import-files maybe

juliangruber: ya that was my thought but all the mtime stuff there looks okay, the mtime is still changing on my fs when I do import with raf

jhand: maybe add console.log(fs.statSync(file).mtime) at relevant parts

juliangruber: ya I'll investigate with that a bit more. I think I need to go find more coffee first =)

that's always a good idea :)

@mafintosh good way to check if stream is object or string mode?

cool dataset, NW oregon (mostly forest) LIDAR, 75gb https://coast.noaa.gov/htdata/lidar1_z/geoid12a...