#dat

yoooo, has anyone considered building a chrome extension that talks UDP / TCP

people here at IPFS brought up the idea of shimming a possible WebUDP spec using a chrome extension

reckon it might be like very cool haha

(and apparently chrome extensions get raw UDP access? :O)

hey yoshuawuyts - https://github.com/datproject/dat.land-roadmap/...

eljee: cool, posted :D

mafintosh: hey can you point me at your server that streams movies from dat archives?

(the code)

hey all, I just received a 4TB NAS in the mail today and would like to use dat to share the .WAV files and metadata database from https://radiowitness.io

I'm going to mirror the Amazon S3 bucket to some folder structure, seems that dat handles this simply enough

but I'm not sure the best way to share the metadata databases

SQL dump, or weekly .csv, are there any existing best practices for this?

pfraze: might be https://github.com/substack/spellcast

rhodey_orbits: 4tb of csv isn't that fun

rhodey_orbits: or how big is the metadata?

karissa: Im thinking of the adaptive streaming example he did

pfraze: ah

karissa: do you think dat could handle 4tb right now?

that's a heavy heavy load

the .wav files total about 600GB right now, the metadata is just a couple GB

@karissa, my NAS just happens to be 4TB right now :) don't have that much data yet

rhodey_orbits: dat should be able to handle 4tb of data it will just take a bit to scan it

rhodey_orbits: a few gigs of csv isn't too bad

rhodey_orbits: but if you're comfortable with SQLite it's nice

Nicer

karissa: cool :) yeah I'm happy with SQLite I'm just not sure how it stores it's DB on disk and if diffs to the tables would play nicely with dat

pfraze: since we stopped storing block data in the leveldb directly it handles larger repo sizes much better

pfraze: now it's just using the fs rw

rhodey_orbits: it just stores binary data so think of it as closer to bit torrent sync right now

rhodey_orbits: it's on the roadmap to have better hooks into the history w human readable commits and diffing

karissa: well if dat handles a 4TB load, I'll be impressed! That's such a large workload, if you dont manage your RAM usage carefully, it'd be easy to exhaust the system

rhodey_orbits: but none of that is really available in the cli, desktop app, or browser right now

karissa: ok, great, I'll probably go with sqlite then. any magic number of files per folder to boost performance/

performance?*

rhodey_orbits: that's a mafintosh question :)

pfraze: I haven't tried it personally but mafintosh designed hyperdrive with large datasets in mind.

mafintosh: I wanna store 600+GB .WAV files in dat, there is no inherit folder hierarchy for these files, can you recommend any magic number of files per folder to boost dat performance?

files are very small on average, I got ~8.6M of them right now

pfraze: afk right now. will link you later

mafintosh: thanks

rhodey_orbits: 8.6M files??

rhodey_orbits: do they ever change?

rhodey_orbits: I wonder if you can concatenate them somehow and denote the start/stop time

karissa: lol yeah 8.6M, no they don't ever change, append only for the database too

karissa: ohhhh, 0.o yeah maybe

rhodey_orbits: dat is cool because you can stream blocks from the middle of a file

rhodey_orbits: so you could in theory have a few larger wav files and skip to the middle if you want to listen to a particular start time

karissa: so I get the sense I'll get a large performance boost if I trade file size for file count?

rhodey_orbits: there's an example of this with a visualization using a movie here https://github.com/mafintosh/hypercore-stats-ui

rhodey_orbits: yes I think so. lots of little files is a strain on any file system

karissa: ok :) I'll think on this then, maybe I'll do daily .WAV files

rhodey_orbits: sweet ya

rhodey_orbits: I used to collect Twitter data and we did daily 34gb dumps, much easier to handle that kind of thing I think

rhodey_orbits: this is a cool use case, being wav files.

karissa: sweet, I'll probably start by seeding the metadata DB first, then figure out the WAV file concatenation

rhodey_orbits: nice

damn I'd love to repackage radiowitness.io web app as an online/offline hybrid using dat to sync the .WAV & DB

^.^ the dream ^.^

maybe this time node.js and I will finally become bffs

are wave files seekable? I would think so

substack: yes

wave files are just raw pcm data

is it planned to add daemonization capabilities to dat cli?

like karissa did with dat-menubar

or was it dat-manager?

I forgot

TheLink: both used the same API

TheLink: yeah we might need that for people who want to host a ton of datasets on one machine

the server part of a dropbox like dat app would also need sth like that I guess

using systemd or such

TheLink: we might just need a swarm module that can handle multiple swarms