#dat

noffle: that sounds pretty easy using some of the libraries. I might make that.

mafintosh: is there a way to make a dat peer that doesn't store stuff locally?

it's just a network proxy and fetches from peers on demand

well, not full dat, it just needs hypercore and dat compatable discovery

creationix, not a full answer, but you can replace https://github.com/mafintosh/random-access-file with https://github.com/mafintosh/random-access-memory

creationix: ideally I'd type 'drop-dat file1 file2 ...' and I'd get some http url on stdout

re interface

noffle: the http part will require someone to host a service (freemium business model or self host I'd guess)

but I can open source the software at least :)

if self hosting was a nice little one-liner cli that'd be great

if you don't need to push updates to the fileset later, it makes bookeeping easy. Just throw away the private key after the files are added

yeah, a node CLI host tool seems pretty trivial

you'll need some way to auth with your self-hosted instance

and configure it's public URL

there was this issue https://github.com/datproject/dat/issues/527 Share a single file with the cli (close now)

so my plan is to create a hyperdrive using memory or some temporary storage, adding the file(s) and then pushing to a server over websocket before letting it go

noffle: state 1 done. Create a temporary dat and share with network `npm i -g drop-dat`

so `drop-dat some-file.txt another-file.log` and copy-paste the url into beaker or dat clone somewhere

step 2 is to make a hosting service with an API for adding DATs and integrate it with the CLI tool

creationix: oh karissa just told me you can do https://datproject.org/KEY -- nice

though I can't seem to get it working

wow nice progress :)

noffle: yeah, there is that existing gateway, but it doesn't always work

npm install'ing..

woah, async/await stuff? what version of node do I need? I'm on 6.9.5

that's probably fine

nah, it crashes

let me know if there are issues and I can tame the JS a bit

async function main (argv) {

^^^^^^^^

SyntaxError: Unexpected token function

bleh, I'll test on 6.x since many people are still on that

:(

noffle: I published a new version using babel to compile down to node 6. syntax. Installing 6.x now to test

creationix: excited about your hacking!

so I think I'll make a second CLI tool where you pipe it a dat url and it converts it to an http url

noffle: still around?

nevermind, it doesn't work over actual internet

timeouts break stuff

creationix: cool stuff!

creationix: why do you read the first 32 bytes off the socket?

if you use hypercore-protocol it'll emit feed with the discoveryKey

does dat's version history save the date of each version?

currently thinking about dat wrt archiving websites, and if dat makes sense for that usecase

i mean ipfs makes arguably more sense, but it's also a bit cumbersome and i just like dat more atm:~

cblgh: only the unix metadata for each file. we can add a field for "user data" though. i have other use cases for it

cblgh, do you mean like dat://{hash}{[+version]}/{path} ? (version is 1, 2, 3, 4...)

it's supported by beaker at the very least https://github.com/beakerbrowser/beaker/issues/152

dat cli should support that

(i don't think it does atm, but i'm saying it should)

mafintosh: yeah i was just thinking that for that case it would be nice to know exactly when the website was scraped

and thought of git commits and how they all have a datetime

cblgh: we've been going back on fourth on how to do this actually

cblgh: and the latest thinking was that it'd be nice if explicit snapshots were just files

like: "Scraped yahoo.com" 2017/10/05 12:31 version=42

in a file called .versions or something

mafintosh: yeah, i like that. seems similar to tagging release commits on github

yea

also easy to interact with a simple file

we were talking about having this file be a third hypercore feed in the dat as well

but you can do it today by just using a simple flat file :)

i'll prolly do that

a json file with a map of {link: <link>, last_scraped: <datetime>, version: <latest dat archive version for link>}

anyone know of prior art for dat scrapers, yt to dat and the like?

brb lunch

cblgh: i have one for soundcloud

but i just used ytdl hehe

ogd is my go to scraper question person though

ohh yeah mafintosh i loved that one

(e-e-e) hey mafintosh: have you had a chance to glance at those pr’s I opened on hyperdrive

@e-e-e oh you are too fast for me haha, didn't realised you fixed all the issues!

will take a look + merge after i give this talk

(e-e-e) cool. Thanks mafintosh:

(e-e-e) are you giving a talk?

ya

in dublin

about containers

(e-e-e) ah cool - hope you kill it!

@e-e-e your prs look great

@e-e-e https://github.com/mafintosh/hyperdrive/pull/19...

(e-e-e) ahh - easy

@e-e-e in the future it'd be really nice if we could pass a prefix to .download to only download a single folder / file

just food for thought

looking at your code that might be easy to add

(e-e-e) super easy i think

cblgh: if you wanna crawl and share https://wiki.archlinux.org i'd use it :)

(e-e-e) mafintosh: removed the .only - would be simple to add path option, but perhaps best as another pr, with its own tests.

@e-e-e https://github.com/mafintosh/hyperdrive/pull/19...

@e-e-e if the / was an arg it would just work right?

an arg that defaulted to /

(e-e-e) mafintosh: yeah - thats what I was thinking too.

(e-e-e) recursion as default would not be an issue?

@e-e-e nice, i'm just gonna merge the PR as is now and we can make another pr

for that feature

(e-e-e) ah dont worry I am doing it right now

@e-e-e added a couple of minor comments here https://github.com/mafintosh/hyperdrive/pull/19...

we don't emit errors else where, so just to stick to be pattern

(non "critical" errors)

critical meaning that drive is messed up beyond repair

(e-e-e) ah...

(e-e-e) thanks so much

(e-e-e) worderful comments

ha

thanks

mafintosh: haha hell ya

(e-e-e) mafintosh: also realised I had not added docs for the function - in the process I realised that callback was not optional too - so i made it optional.

(e-e-e) now path can be passed to download specific files.

(e-e-e) :)

Sweeeet

mafintosh: good point about the protocol emitting the key. I wonder how I would do that though. I can't create an archive before I k ow the key and I can't speak the protocol without the archive instance to call replicate on right?