#dat

or maybe you want to move a materialized view from hypercore storage to a different db and want to delete it

yosh answered part of this irl just now

are u guys in berlin?

yes

yea you can delete

recently found out that you can actually unlink data inside files

using fcntl

fcntl is the coolest

or just discovering these low level things are cool

even though you're at the mercy of the kernel gods

emilbayes: found another cool one the other day. It's possible to have a zero copy file-to-network interface using the sendfile(2) call ✨

yoshuawuyts: Yeah i've seen that in the nginx source :D

yoshuawuyts: Is that exposed in node? would be a cool module for static file serving

emilbayes: hah, not afaik

yoshuawuyts: I remember it always being on of the reasons people wrapped their node.js apps in nginx servers when I was green in node

emilbayes: also the pagecache interface is really cool; write files to disk, but actually not really it's magically kept in memory for a while - kafka makes heavy use of both sendfile and pagecache; never stores anything in memory explicitly

so pagecache is like the inverse of mmap?

mmap being writing virtual memory to disk, when out of space

emilbayes: oh yeah, sounds close enough

yoshuawuyts: I want to explore stuff like this when I get more time

yoshuawuyts: Maybe we should do a low level hack camp duing fall sometime

yoshuawuyts: mafintosh keeps talking about doing a hacking trip

would be cool to dive into more OS stuff ^^

emilbayes: I'd be game

yoshuawuyts: We were looking a kenya

pfrazee: here?

All test passing on blake2b :O

mafintosh: yo

pfrazee: almost have your del bug fixed

mafintosh: awesome

yoshuawuyts: https://github.com/thlorenz/libuv-dox/tree/mast...

emilbayes: looks the closest they come is to emulate the API, but not actually using the sendfile() call https://github.com/libuv/libuv/blob/d251dee33b0...

emilbayes: wonder if that's the case b/c connecting streams is in JS land, and data needs to cross boundries

emilbayes: b/c that'd mean https://github.com/indutny/uv_link_t could perhaps mean this can change

ah

Blake2b for your browser (convenience) https://github.com/emilbayes/blake2b

mafintosh: Couldn't you potentially reuse the ctx object in blake2b because js is single threaded?

mafintosh: That would save some allocation time for many small hashes. I'm already reusing the parameter_block

pfrazee: okay, pushed a fix to append-tree master

mafintosh: I'll give it a shot

not on npm yet

pfrazee: you need to re-add the missing files in your dat

mafintosh: they're dead for good?

pfrazee: yea sorry about that

corruption bug

mafintosh: no worries

need to write a fuzz'er

was a pretty serious bug actually

can you push to github?

oh yeah?

pfrazee: it is on github

in append-tree master (not npm yet)

oh I was looking at the package.json

wanted to see if i could write a fuzz'er to verify my fix

ok

can't seem to break to fix with various tests

which is good!

so gonna risk a npm release haha

the fix actually makes the code a lot simpler

which is good

my main concern is why i did not do it this way in the beginning

maybe i'm just smarter now

emilbayes: soooo i found a way where the kv is auto compacting

aka, all alive values are auto clustered over time

mafintosh: What do you mean?

emilbayes: so you have a kv store with 3 key/values, a, b, c

emilbayes: so if i delete b the store looks like this now

[a, (nil), c, del(b)]

so instead of doing that!

we can just do this

[(nil), (nil), a, c]

sorry [(nil), (nil), c, a]

aka, instead of deleting b we just rewrite a (closest to b) but! do not link the old b value

So this is the pointers?

a, b, c etc corresponds to the nodes

so they have key,value,pointers

the point is. do not write deletes

just rewrite an old value and do not link the node you want to delete

mafintosh: smarter every day

pfrazee: it's like an incrementing compaction

pfrazee: okay published the append-tree fix

mafintosh: cool

gonna force bump it in hyperdrive

pfrazee: please let me know if anything remotely weird happens after deletes hehe

mafintosh: ok will do

hyperdrive 9.2.9 is out fix a force bump

jhand: o/ reinstall deps

*with not fix

mafintosh: I think I need a ELI5 version :p

emilbayes: i'll show you in person

yeah haah

need a higher bandwidth medium

mafintosh: o/

mafintosh: have you been thinking about multi-user {hypercore,hyperdrive,dat}s?

multi-write, I mean

like hyperlog

noffle: think mafintosh is out rn, but believe he's been working on multi-writer last week

noffle: ya

what repo ought I be looking at?

not public yet

cause its not working yet

mafintosh: is there a branch I can poke at? what's the approach you're taking?

noffle: i'll add you to the private one

its only private cause we are force pushing a bunch

hehe

mafintosh: I know youve probably told me this twice, but remind me: does hyperdrive currently do any deduplication within an archive? Eg if I add foo.txt, cp foo.txt foo2.txt, then add foo2.txt, will it double add?

or will foo.txt and foo2.txt both point to the same content-feed blocks?

pfrazee: simple authorization doesn't seem to work with beaker, it doesn't present the user/pw popup

pfrazee: currently double adds

pfrazee: and gesture navigation works fine in file listings but not on webpages

TheLink: you mean like Basic auth?

pfrazee: yes

ok I'll file a bug

mafintosh: ok cool

mafintosh: should i force bump hyperdrive also?

in cli

jhand: ya good idea

mafintosh, is there a hyperdrive fuse binding yet?

or code to automatically import a tree recursively into hyperdrive (not that it's hard)

basically, I'm trying to make a dropbox-like interface. Add a folder as "shared" and access it remotely

I guess desktop dat does some of this

That would be cool

also, I'm thinking of ways I can use a web of known-trusted peers to speed up discovery

(for example, I have a machine at my house and another at my parents'. I want to share a file with them without having to use global dns or dht)

our machines know and trust each-other through something like dht, but optimized for locality

if I create a dat using the desktop client and then try to view it in datproject.org, it normally never shows up.

is this because I'm behind a NAT and DHT is the only way for the public site to find me?

creationix: you can use mirror-folder for importing, that's what we do

jhand, what do you mean?

You can also whitelist IPs but still use the global Discovery

creationix: it's a module

Does recursive & live importing

so, to use this over a network, I just use hyperdrive for one of the `fs` instances?

or I guess this is unrelated to hyperdrive actually since it handles sync logic

recursive file watchers are a problem though. I need to support sharing entire hard-drives

(or at least entire home folders)

creationix: ah ya. I think some fs support recursive watching well but not others

nice work taravancil

pfrazee: we need a github to dat thing

windows and osx support it better at the os, linux require manually setting lots of watchers

I need to find a way to use dat for daplie, then I can help write apps

jhand: like a way to export repos to dat?