#dat

bret: re: how many dats can hypercore handle, good question! I'm not sure

5000 is probably on the high end

creationix: interesting. your shell piplines are wild!

do you intend to rely on dat signatures on the security model?

I haven't really read much about it, but The Update Framework (TUF) describes a whole bunch of corner cases

eg, if secret key is stolen/lost

if something like caify was ever used to fan out to thousands of devices it would probably be worth thinking through things like that

also makes me think of debian's (abandoned?) jigdo system: http://atterer.org/jigdo/

mwarning: I think the connection handshake stuff is well covered in the whitepaper

mwarning: ... or maybe they aren't. should be in wire protocol DEP in a couple weeks

there's a brief note in my (now out of date) notes:

basically, the two peers send unencrypted Feed messages to each other on connection

each peer creates a random nonce which they include in the Feed message

as well as the discovery key

if the discovery keys match, then the peer is presumed to have the actual public key, and thus access to the dat archive

and the connection proceeds

all further messages go through a stream encryption pipe

using the XSalsa20 cipher of libsodium

you pass libsodium the appropriate nonce (your/local nonce for sending, remote for receiving) and the current offset (how many bytes had been sent/received up until this message, not including the unencrypted Feed message)

and the dat archive public key as the key

both sides know the key and both nonces; the same stream is generated, and XOR-ing on the receive side returns the plaintext

there is a bit of fiddling necessary because the libsodum function (stream_xor_ic()) works on 64 byte chunks, and the stream is byte-level

(aka, implementing clients need to handle fiddling)

we've discussed switching to a nwew libsoduium "secretcret stream" pattern

[1;3D[1;3C

(oops)

"secret stream" pattern. but is a new upstream feature, and a lower priority than getting current behavior documented

(IMHO)

oh mwarning isn't even here

I emailed them

bnewbold_: I haven't yet thought through the security model. For now I'm just using ssh and trusting my server isn't compromised. I should add a signature to the index file at a minimum.

By this summer there will be thousands of consumer devices using this, so I should think more on it. I just also need to get something working now for the next batch of hardware prototypes. The crazy shell piping allowed me to not have to implement any network code.

I love me some crazy shell piping

(make sure to set -o pipefail though)

might be orthogonal to what you're doing, but from when I used to do embedded stuff I remember coming across this "lights out" remote upgrade scheme:

I don't think buildroot/openwrt have anything similar out of the box

one of the openwrt-based community firmwares has this: https://gluon.readthedocs.io/en/latest/features...

on some router-ish devices you have enough diskspace to have two rootfs partitions, as a safeguard in case a newly flashed image doesn't boot

ah yes, "lights out" is a whole different idea of upgrades

most openwrt-supported devices simply don't have any removable storage so lights out often isn't an option. most bootloaders (u-boot) support tftp though

oh I wish I could use BSD, but I barely have hardware support in linux as it is

lgierth: yeah, I'm going with A/B partition. The caify tool will download the index, sync down the objects and then write to the block device of the other partition all while online. Then set some flags for uboot and reboot. It will rollback if the new partition doesn't pass self inspection. We even have a MCU watchdog that will reset the main CPU if it doesn't ping a keepalive hardware line.

I was going to just use tar, but I hate re-uploading and re-downloading all those repeated files for every release. I wanted something smart enough to de-duplicate between releases, but also support uid, gid, full mode, rdev, mtime, etc...

Since caify just syncs the ext4 partition as-is is preserves anything ext4 supports :) Similar to @mafintosh's ext4 in dat experiments.

hmm, no pipefail because I'm not using bash. I wonder if ash has it.

juliangruber: you around? :)

made a new impl of abstract-random-access

Does ipfs come with searchable metadata capabilities? I cannot find any info on this..

What would be extremely cool is if ipfs had the ability to hash a binary files with an embedded metadata header, so that a change in metadata would not necessarily affect the file hash..

..or perhaps my question is mute since the file is chunked up and delta copied anyway.. never mind then

But I'm still interested in searchable metadata support, or any other way to effectively query files based on built in or arbitrary metadata.

(scriptjs) @mafintosh hello, where did you locate this? I don’t see this in a branch

@scriptjs hi

(scriptjs) @mafintosh hey there :)

contemplating making a new repo called random-access on the dat org for iy

it

and then deprecating the old one

(scriptjs) kk, ther seems to be a relatively good ecosystem of modules currently

(scriptjs) s/ there

(scriptjs) but this is a great place to grow some new things

(scriptjs) Are they all relative to dat though. I seem them as generic

true

(scriptjs) why not a random access org on github

(scriptjs) to group them

ya cool

@scriptjs you wanna make one? then i'll make the new repo there

(scriptjs) yeah sure

(scriptjs) I’ll ping when ready.

cool, i'll write some tests for my thing while you do that

(scriptjs) kk

(scriptjs) @mafintosh invited you. I could not get random-access so made it random-access-org. Is that ok?

@scriptjs ya fine, also good would be random-access-community :)

(scriptjs) ok let me see if that is availabe

(scriptjs) @mafintosh ok, got random-access-community

(scriptjs) you should have your invite

accepted

(scriptjs) kk, I also invited julian

@scriptjs which naem do you like best for the 'base class' module? random-access or random-access-storage?

(scriptjs) I think the abstract is best with random-access since it possible it may not be persisted

(scriptjs) how are you leaning?

i've called it random-access so far

but now i think random-access-storage better conways it's meaning

random-access is bit fluffy

(scriptjs) sure, lets do storage then

(scriptjs) then you don’t ask what does random-access mean

(scriptjs) bad way to start something with a q of what is it

(scriptjs) I can try to reach out to the random access module owners to invite them to consolidate their modules under the community

cool!

(scriptjs) Then perhaps we have a better chance to have something nice growing

(scriptjs) with all sorts of interesting back ends