#dat

mafintosh: you around?

notwes: a bit yea

jhand: yes we can

it can wait, or maybe I can DM you on twitter. I just did a bit of benchmarking of your protobuf package vs that one I mentioned a month ago or so, and was hoping you could take a look to make sure my results seem resonable

notwes: sure thing

if you want to take a look

(im in this)

mafintosh: really I just didn't want to post them as an issue and have it open to people's typical "your benchmark is wrong for this reason" before getting a second set of knowledgeable eyes

notwes: let me try and run it

ogd: "Mr. Ogden"

:P

(wesleytodd) Mafintosh: thanks!! Again, no rush

nice

@wesleytodd you have a mistake in the bench

@wesleytodd you need to call .finish() on all your protobufjs encodes. otherwise you arent testing serialization

(wesleytodd) Hahah. I knew I must have missed something. I accidentally removed that during a refactor

(wesleytodd) My original test had much closer results and then I changed stuff and they were suddenly much further apart

@wesleytodd fyi for cases where you *really* care about perf i use the buf argument to protocol-buffers .encode(msg, buf) and encode into a preallocated buffer

eases gc + gives you a boost as it then does zero allocs per encode

mafintosh: awesome suggestions! I will update with that, thanks!

mafintosh: does dat-next-next still rabin? should we turn that off for now?

mafintosh: or put it behind a flag

ogd: i turned it off for now in next-next

having a flag to re-enable it sgtm though

mafintosh: cool

why is it disabled?

have you had good experience with rabin, dedup-wise?

pfrazee: lgierth we have been benchmarking stuff and rabin is a big bottleneck right now, it's causing imports to be slower than network transfers without it would be. so we decided to turn it off by default in the cli but leave it in the code, and add it in based on smarter heuristics in the future

ogd: does turning it off just result in fixed-size chunks?

pfrazee: yep

ogd: ok cool

yeah it's been slower than fixed-size chunks in ipfs too

did you see any gains in deduplication though? in ipfs it has been slighty better, and slightly worse, depending on the settings

better=more dedup

but always slower

is rabin only run on initial write, or is it run during verification of a file too during replication?

only when files are read

not metadata

yeah. Well I wonder if the dedup gains are more than the fingerprinting cost

lgierth: we cant answer that scientifically yet cause we dont have enough dats out there in the wild, which was another factor as to turning it off for now

fast dat = more users

word :>

we've only run small-scale benchmarks so don't have a definitive answer either

haha

:>

fasterrrr

found it https://github.com/ipfs/archives/issues/137 -- lots of prose but also numbers

lgierth: nice thx

[dat-core] cshum closed pull request #38: patching sublevel (master...master) https://git.io/vswFo

whoa what is dat-core

o/ Good read on the long tail of research data from director of COAR

(for the RDA interest group)

dat-core isn't based on hypercore/-drive?

dat-core looks like it's old

oh ya just never saw it before

ah, ok

Anyone want to apply to https://conferences.oreilly.com/jupyter?

jhand dat-core was dat-node 0.1 ;)

heh, just got a mail from beaker in my spam folder

it's the other beaker project though

thought this project fell asleep

the beaker notebook project?

yeah

I ran into this before I even knew about dat

yeah

jhand: man thats a great article

jhand: good arguments for distributed data tools

ogd: ya! distributed data tools and low barriers to use