#dat

network pears

lol

(matrixbot) `icarito` if I accidentally added files to a dat, can I remove them, even from history? I added some build artifacts that are bloating my dat needlessly

@icarito mafintosh if you delete a file it will get 'garbage collected' right? o/

(matrixbot) `icarito` I thought dat preserved file history?

(matrixbot) `icarito` i.e. if I delete something from git it will linger in history, I assumed the same was true for dat?

(matrixbot) `icarito` normally that's desired except when adding files in error either realing secrets or bloating the repo

(matrixbot) `icarito` *revealing

the metadata lingers but not the content

so it's not slow like git

(matrixbot) `icarito` I added `node_modules/` dir before putting it in `.datignore` and that increased my dat size significantly

(matrixbot) `icarito` but no worries it's not slow or humongous

(matrixbot) `icarito` :-)

(matrixbot) `icarito` thanks for DAT I love it !

woohoo.

@icarito in case you (or anybody else) is using beaker, beaker is sloppy right now and doesnt delete history. We decided to change that for 0.8

(Redni) Thanks @joehand :) And yes, my mistake I wrote pears instead peers xD

Ogd ya it'll be a dangling pointer

I hear scope crepes go well with network pears.

mafintosh: ooo nasty crash using hyperdb... unclear if related to hyperdb though https://gist.github.com/maxogden/42304aafd95ad8...

mafintosh: doesnt take that long to crash either... 10s or so

mafintosh: yea doesnt crash if i take the db.puts out

ogd: ouch

ogd: how far does it get before that happenns?

mafintosh: around 300 cchanges

mafintosh: do you think its related to the size of the value? some of these have lots of versions

ogd: try smoething: https://gist.github.com/maxogden/42304aafd95ad8...

instead of passing cb directly

pass a function that *just* forwards the error and not the value to cb

db.put('/latest-seq', data.seq, function (err, val) { cb(err) })

ya

mafintosh: still crashed, at like ~550

mafintosh: it sits for a while doing nothing then crashes....

mafintosh: like no puts succeed (no new console entries in log), for like 30 seconds

then the core dump

ogd: okay, let me try and run it the

owrks locally

mafintosh: what seq do you get up to?

mafintosh: see if you can get past 2636090

ogd: seq=285961

mafintosh: hmm maybe it didnt start crashing till it had lots of keys then

ogd: i thinks its a backpressure bug

ogd: try this on the server. simply comment out the hyperdb stuff but *dont* call the cb

mafintosh: ahh

ogd: seems to be ndjson that is buffering forever

ogd: its split2 that is the issue

always the italians

mafintosh: can't you live code sometime?

cblgh: haha sure

i think i'd learn so much lol

we need dat.tv

im home for a bit now so no excuse

dat.tv++

mafintosh: wow split2 doesnt do backpressure

ogd: yea im sending a PR to fix it

mafintosh: how have we made it this far without finding this out lol

ogd: i was just thinking that

my network was too slow at home for it to be the bottlenenck

haha

mafintosh: nice https://github.com/mcollina/split2/pull/11/files

i have commit rights apparently

nice haha

push it

mafintosh the repo czar

not on npm though

i guess mcollina doesn't trust me that much

haha

the facade of trust

mafintosh: so if your database was faster i would have never found this bug :)

ogd: thats another way to put it haha

mafintosh: ok i imported all the metadata into a hyperdb

mafintosh: im ready to fetch tarballs on demand now, i wanna do it through a REST API that you can hit that tells my server to download the tarball, put it into the dat, and return the GET call when the dat has the tarball you want, then you can fetch it by filename using hyperdrive

ogd: ya sounds good

use the hyperdrive js api

mafintosh: i can make a hyperdrive and a hyperdb on the same hypercore instance ok?

ogd: use a new hypercore but they can easily replicate together

just like how hyperdrive uses two cores

ogd: are you storing the package.json in hyperdb also? didn't see it in the gist

mafintosh: nope just versions

ogd: anyreason why?

mafintosh: just to minimize metadata overhead

ok

mafintosh: its 266M currently (just array of versions)

ogd: did i talk to you about protobuffing package.jsons or did i dream that?

mafintosh: sounds like something id say

hehe

i was thinking that if we protobuf it + remove readme's etc it'd be pretty cool

cause then you can sync just the hyperdb and do any dependency resolution offlinne

mafintosh: do you need the whole package.json to do resolution?

ogd: just the dependencies/devDependencies/optionalDependencies etc

yea maybe ill add those

/module/version -> protobuf with those

oh yea

i'm hacking on iterators now

/module -> all-versions

or you could simply iterate /module/*

mafintosh: could you see hyperdb as an abstract-leveldown compliant module?

(iteration will be slower)

ralphtheninja[m]: our iterators are a bit different

nods

ralphtheninja[m]: moved

ogd: the version list can be easily compressed as an array of varints

one big array, each pair of 3 varints is the version

mafintosh: danke!