#dat

mafintosh: yeah, great job! Super readable

karissa: thanks :)

how's things going with the linux containers?

i keep dreaming about breaking down the code/executable/data divisions

ogd: see stackvm - mikolalysenko has an interesting idea about generic hash set replication :)

karissa: o/

I'm wondering about dat hooks. I want to know when someone has pushed new changes to a dat repository so I can do things in response to those changes. Example: index the updates into ElasticSearch. Has anyone done any work on this? The git equivalent are git hooks https://git-scm.com/book/en/v2/Customizing-Git-...

oh rad

flyingzumwalt: you can do a createChangesStream({live: true}) in javascript to get a change and then do something with it

flyingzumwalt: here's the doc on it (small tho) https://github.com/maxogden/dat-core#stream--db...

flyingzumwalt: we don't have the git-hooks interface in the config file (package.json) but it could be done

thanks karissa so with the createXXStream approach I would need to run some sort of node daemon that watches for changes? (as opposed to hooks, which would be triggered by dat when events occur)

flyingzumwalt: yeah, that's right.

karissa++

I'd like to look into writing a hooks implementation. I'll poke through the code and see what I can figure out.

flyingzumwalt: wow thanks!

flyingzumwalt: it'd be good to implement it in 'dat' not 'dat-core' because we are changing 'dat-core'

are there instructions anywhere on how to set up the dev environment and run the test suite?

flyingzumwalt: do these work for you? https://github.com/maxogden/dat#developers

it's a start. I'm spoiled by projecthydra. They require something like this in the README https://git-scm.com/book/en/v2/Customizing-Git-...

oops. this: https://github.com/projecthydra-labs/curation_c...

but I can figure it out ;-)

flyingzumwalt: ok let me know if something is confusing and i'd be happy to help improve the docs!

will do. thanks.

hey cool http://www.se.rit.edu/~opensd/

senior academic project

(found via tomorrows mozilla science call agenda)

ogd: nice!

“This is an extention of the work done by @arfon, @hubgit, @kaythaney and others on Code as a Research Object / fidgit.”

ps: did everyone hear about Mozilla archiving etherpad.mozilla.org?

see for instance: https://old.etherpad-mozilla.org/oscodefest-sof... and I don’t think these docs will persist much longer

best to archive your own copies

here’s the notice: https://discourse.mozilla-community.org/t/ether...

mafintosh: any thoughts on a better cp -r module? https://github.com/maxogden/electron-packager/i...

todrobbins: hmm good to know thanks

ogd: i usually do tarfs.pack().pipe(tarfs.extract()) hehe

lol

ogd: its a surprisingly fast way to do it

alright ill probably just do that then

(graph diagram inspiration)

also apparently this paper defines merkle dags http://truthsayer.cs.ucdavis.edu/model.12.6.pdf

though it doesnt call them that

" In this case, Cassandra and Riak implement a Dynamo-inspired process called anti-entropy. In anti-entropy, replicas exchange Merkle trees to identify parts of their replicated key ranges which are out of sync. A Merkle tree is a hierarchical hash verification: if the hash over the entire keyspace is not the same between two replicas, they will exchange

hashes of smaller and smaller portions of the replicated keyspace until the out-of-sync keys are identified."

mafintosh: we should see how o/ is implemented

ogd: unrelated i just read how that works two hours ago

lol

mafintosh: you gotta post more links to irc :)

then we can use it as a metric in our next biannual grant report: 'Posted 3288 links to IRC'

'Posted 3288 links to IRC' ogd++

ogd: will do. i was on an airplane though

mafintosh: you have to click 'download full text' here http://www.researchgate.net/publication/2202241...

mafintosh: but apparently that proposes something called a 'search DAG'

mafintosh: (i think its a newer version of a paper i linked earlier)

ogd: reading it

ogd: the dynamo merkle tree replications just sorts the entire dataset then builds a merkle tree

ogd: it then sends the root hash to the other peer (who also sorts his dataset and generates a merkle tree)

these are all about security and dont really talk about append-only, distributed incremental use cases. but maybe we can get some ideas

ogd: if the hashes match they have the same data - if not peer one will send the next two hashes

ogd: and it then applies the same algorithm to those two hashes

ogd: so in log(n) roundtrips they'll know the diff

mafintosh: isnt that a lot of traversals

mafintosh: cause they have to sort it all on every round

ogd: they have to sort the dataset once at the beginning of the replication plus generate the merkle tree

ogd: but that might also take a while if the dataset is big

mikolalysenko came up with a very clever way of maintaining that merkle tree by using a merkle trie

because its "easy" to add/remove things from a trie

in practice its still a bit tricky though to calculate how big the diff is gonna be though using these approaches without buffering the diff to disk before sending it back (i.e. progress bar is difficult unless you buffer the diff to disk before sending it)

how is sql COUNT implemented? must be some indexing so they can quickly estimate without doing a full scan of the data

and i think if you'd only wanna generate a single branch you'd have to generate a new trie for that

ogd: they probably just index the number of nodes in the index

ogd: like we do in the new dat-graph impl

what if we just store the node count in each node of the graph?

its like a merkle tree but instead of a hash its just the sum

thats what i do now in dat-graph

ah ok. so cant you get quick counts that way?

yep

but i cannot combine that with the merkle trie/tree approach

(easily)

ogd: but it works with our path/log approach

ogd: because that sends "old" nodes first

mafintosh: started writing up a crappy outline for the merkleization/merkle trie thing https://gist.github.com/mikolalysenko/fa1403784...

mikolalysenko: looking good!

if you want to add anything let me know

mikolalysenko: you should add dat under merkle dags

roger

that search DAG link is basically the same idea as a merkle dag I think

mikolalysenko: also i think a really nice feature about your trie idea is that it sounds pretty simple to implement

i started a discussion issue https://github.com/datproject/discussions/issue...

yeah!

still not sure how well this will work, but I am excited to try it out

currently wrapping up a few things at plotly but will work on this tonight!

merkle fingers http://www.verisign.com/en_US/resources/img/spe...

the general idea though is: pointer machine -> functional data structures -> merkle dags

or put another way, merkle dags are just verified functional data structures

oh yea i didnt even think of looking at functional programming theory since its all append only

multiple people have told me this is all really similar to Datomic which is closed source