#dat

jimpick: in case it's useful, I started working through this a little while back, mainly to make it easier to spy on the protocol stream https://github.com/andrewosh/hypercore-protocol... -- it's an attempt to separate out encoding/decoding from feed management

nice

needs a bunch of work, but the decoder stream does make for easy spying :)

better than a bunch of console.logs :-)

i just got hypercore-protocol-proxy to retrieve a single value ... so i think i'm making progress

jimpick: uhhhh

(fsteff) A little question about the discovery:

(fsteff) is there a difference between the feed.discoveryKey and the value actually used for discovery?

(fsteff) Because the discovery key I get from a hypercore is longer than the one I previously sniffed from the DNS traffic beaker is generating.

(fsteff) (also I am getting errors in my dns module when I use this key, because it is too long)

@fsteff the one used in discovery is just the disc key truncated to 20 bytes so it fits in a subdomain

(fsteff) mafintosh: ok thanks, so I'll do that too

(fsteff) mafintosh: is the DNS server ok with requests that do not contain any additional TXT data, or does it throw errors internally?

(fsteff) currently I am doing it that way (so it does not support announcing to the dns server), and it seems to work

@fsteff no that should be fine

:)

(fsteff) I am building a java version of the server part of @RangerMauve 's discovery-swarm-stream

(fsteff) Once it works one can build android apps with dat in a webview, without needing a node runtime

wowo

cblgh: ah, someone else had the same problem! always good to see =)

FWIW I think it's pretty important we get away from the 20 byte truncation sooner than later

(RangerMauve) It's at 20 because of the DHT, right? Subdomains can be 63 chars long for the DNS discovery.

It is yes

pfrazee: curious why?

mafintosh: just seems like the false positives are a problem and it gets harder to change this as more people adopt the toolset

pfrazee: you mean the hashes clashing?

mafintosh: yeah

is it 20 bytes of 64 (string encoded) or 20 bytes of 32 (binary encoded) ?

last one

It seems unlikely to me that you have hit a clash

But we can easily increase it to 31 bytes

Or use alpha numeric lowercase encoding and get below the limit

jhand: yeah but seems like they solved it by doing what i thought we were doing with cabal (calculating the maximum amount of lines to display)

hmmmm

(RangerMauve) mafintosh, pfrazee: What about using the max amount of bytes that a given protocol supports? So for the mainline DHT, use 20 bytes, for DNS use 63, etc.

(RangerMauve) I don't think there's any point in using the same subset of the discovery key on different protocols.

mafintosh: gets harder to increase it when there are more codebases

I guess we'd just need the lookup logic to support truncated inputs

(RangerMauve) Yeah, so the discoveryKey is the canonical key, and then you use as much as possible based on the limitations of the discovery network

why truncate at all? 63 is the limit for a single name... but you can nest those pretty deeply

true, we should just do 32-chars.32-chars.dat.local

pfrazee: o/

mafintosh: now all you need to decide is the equivalent of endianness ordering =0)

mafintosh: yeah I'm +1 for that

yay, i got hypercore-protocol-proxy working as a web gateway! https://hypercore-protocol-proxy-test.glitch.me/

here's the glitch app for it https://glitch.com/edit/#!/hypercore-protocol-p...

it's different from the old web gateway i was using in that it doesn't instantiate a full hyperdrive on the node.js side of things ... it's just proxying the protocol. should be a lot less memory usage

hypercore-protocol-proxy is a bit naive at the moment ... eg. if there are 3 peers, it will transfer the same data 3 times because each peer gets the same 'request' message

except it's not working :-)

it's working now ... might have been a network glitch

sometimes it works, sometimes it doesn't

working for me! thanks for sharing

i do get an occasional "Error: premature close"

that's sort of normal

ahh

i tried removing the 'live' flags ... it might be an issue with multiple people using it at once

it was keeping a lot of connections open

iirc, there's been some work/research done regarding the use of an anycast equivalent on p2p networks

(though you can get some of that for free via ipv6)

jimpick: hey has glitch been able to connect to the dat network?

yeah, i've been using it for my demos

jimpick: okay awesome. I'm looking for a good way for people to deploy dat-based node services once @beaker/dat-daemon is done

https://github.com/beakerbrowser/dat-daemon <-- nice!

i'm going to play around with getting https://github.com/maxogden/linux packaged up into an electron app with a little admin interface ... primarily for running little bots. i realized i needed it for my pipette backend

it worked really nicely with dat-container

i tweaked the gateway a bit ... i wasn't closing the swarm connections, so it was leaking swarms :-)

ooh, i blew up node

trying node 10.x ... glitch is really slow at installing things. i think their servers are overloaded

(RangerMauve) jimpick: What's HyperOS? 😲

RangerMauve: it's just a mini linux distribution with just enough stuff to run dat-container

(RangerMauve) jimpick: Cool. :O

the demo needs a lot of work, it seems ... it only works some of the time

although i'm amazed it works at all ... proxying multiple peers into a single replication stream

(austinfrey) jimpick: what do you gain by using hypercore-protocol vs using hypercore directly for your proxy?

austinfrey: the main thing is avoiding duplication of data in the gateway... for a public gateway, it could add up to a lot

austin: also, i want to do “sparse” replication, so it’s tricky to tell the gateway what blocks to sync - that would probably need some extra magic

(RangerMauve) The intermediate processing of the data is also added overhead

pfrazee: that dat-daemon is different from https://github.com/soyuka/dat-daemon ?

(RangerMauve) jimpick: Hopefully I have that discovery-swarm-stream thing done this week! I feel so bad that I haven't been working on it more. 😭

Vito`: lmao of *course* that exists. Im having such a hard time naming this module. Yeah it's different. @beaker/dat-daemon is a toolset for making dat services

another approach might be https://github.com/maxogden/multiplex

(RangerMauve) jimpick: I considered using amultiplexing implementation, but in the end I think that using protobufs with a length-prefixed-stream is easier to implement on different systems.

i rather suspect that there isn’t one “winning” to doing this gateway thing, and different approaches will work better depending on the application

(RangerMauve) I really want this to be a standard to let people implement their own discovery-swarm servers in different langauges and environments

the mafintosh hypercore-protocol-proxy library might be a neat way to send the same data to multiple peers without having to send the same data multiple times ... good for mobile apps

i think mafintosh has some more ideas for it

RangerMauve: it would be so great if we had some solutions that “just work”

(RangerMauve) Yeah, with @fsteff 's java implementation it would enable cordova apps (or similar) apps, and potentially Java based apps with a webview

(RangerMauve) Plus it'd be a good reference implementation for an MVP discovery-swarm that others could base of for, say, iOS and desktop apps.

maybe I should name it `@beaker/dat-daemon-tools`