#dat

ogd: mafintosh I've setup Docker with ext4+OverlayFS driver in our new fat node I while ago (however, I haven't got time to play with it yet)

[dat-core] mafintosh pushed 1 new commit to master: http://git.io/ve8gJ

dat-core/master 03f8116 Mathias Buus: rewrite

[dat-core] mafintosh pushed 1 new commit to master: http://git.io/ve8gU

dat-core/master daa54a9 Mathias Buus: 0.0.2

ogd: when's that talk?

karissa: the berkeley one on friday

ogd: aye

ogd: i think its going to be pretty interdisciplinary.. hmm. i wonder if it'll be at the right level to talk about the internals

karissa: also im doing a lunchtime talk at 18F in SF on thurs if you are interested in seeing their situation

ogd: yeah, sounds good

karissa: we're hacking on dat beta now. im using this for the CLI https://github.com/maxogden/subcommand

nice

[dat-core] mafintosh pushed 2 new commits to master: http://git.io/ve8Pe

dat-core/master b07ad65 Mathias Buus: add missing return and use npm hyperlog

dat-core/master 68f7684 Mathias Buus: 0.0.3

[dat] maxogden pushed 1 new commit to beta: http://git.io/ve8Pl

dat/beta 792f529 Max Ogden: implement init

woohoo

karissa: imagine you type 'dat init'. what should happen - it created a new folder called: '.dat', 'data.dat', or 'dat' (or something else)

karissa: also what do you think about the crazy idea of using package.json with our config under a 'dat' key (instead of dat.json)

ogd: will it adhere to the npm package.json?

spec

?

karissa: yea but we prob wont touch any keys except name, description, author and repository

karissa: all dat specific stuff will be under 'dat'. that way if there is already a package.json in a dir it will use it

karissa: but if we are first then npm can read ours later

hmm

karissa: i am ok either way. pro of dat.json is... simplicity? con is its yet another json file

i think it should be dat.json because it might diverge

ogd: how will the user interact with the .dat/dat folder?

karissa: thats what i was thinking. mafintosh suggested making it not hidden. i dont really think they'll go in there

ogd: if they are editing stuff inside it (like readme, dat.json, etc) then it makes sense to not have it be 'dotted'

yeah

karissa: the argument for making it not hidden is that it might have a ton of data, and its confusing UX to have that be hidden

ogd: yeah

karissa: cause users might be like 'what is taking up all this space'

yeah

im going to think about it a minute

ogd: for the berkeley talk are you planning on any walk through demos/

karissa: depends on how far we get with the CLI in the next few days

karissa: but i can demo some of our pilot project use cases

ogd: ok. i was thinking it'd be useful to at least walk through a typical collab scenario in the beginning so they can grok it. even if the cli isn't done

karissa: oh yea definitely

karissa: we just had an idea for new cli

karissa: `dat move foo.jpg .`

karissa: would remove foo.jpg from the blob store and put it into the users current dir as 'foo.jpg'

interesting

karissa: and `dat copy foo.jpg .` would keep a copy of the data in .dat but make a copy in cwd

karissa: that way theres a easy story for 'i just cloned a dat with a big file in it and i wanna now use the file'

karissa: and move/copy are alternatives depending on whether or not you wanna have 2 copies of the file on your machine

karissa: and then for filesystems that support copy-on-write, or FUSE, we can do really cool stuff

karissa: because you can do a copy that actually takes up 0 space until you edit it

[dat-core] mafintosh pushed 1 new commit to master: http://git.io/ve8My

dat-core/master 8ea4652 Mathias Buus: better cli

ogd cool

ogd: i think there's a more fundamental ux problem here

ogd: what is the likelihood that someone will have a file in the blob store and they won't want it copied/moved?

karissa: depends on if they want to redistribute the file to other peers

karissa: e.g. i think dat right now is pretty server-centric

[dat-core] mafintosh pushed 1 new commit to master: http://git.io/ve8DI

dat-core/master 25101b0 Mathias Buus: always expose meta

ogd: so when a user clones a dat, i wonder if they're more likely than not going to be confused if the file isn't 'visible' in some way

blobs attached to rows are part of the data

but blobs attached to an entire dataset might more likely be breakdowns, models, visualizations

ogd: or might be data themselves as an entity (thinking non-tabular data stores)

karissa: the problem is.. if you dont keep the data in a blob store then you lose version control, and also the ability to seed that file to other peers

karissa: i agree though with your ux assessment

right i'm not really talking about technicalities but its a ux thing

making it 'appear' as though the file is there, or something.

might be useful

but idk, we still haven't really done user trials so its hard to say how people will use dat. but this is just my hunch that 'copy' and 'move' might be confusing for most domain specific experts

maybe if we can liken doing 'dat clone' to downloading a zip file

now you have a .zip but it isnt unzipped yet

mm

yeah

for example, if you do dat clone and a thing called data.dat shows up

when you cd in, what if you see each dataset as a directory, or something?

and when you clone you get output like 'Downloaded 4388 rows and 324 files into data.dat'

mm

yeah

ogd: i think it might be more likely to see like "3 files" in most cases

karissa: there could be different clone commands too

karissa: like one that downloads blobs into your cwd

karissa: and one that 'syncs' them into your .dat

ogd: mmm

ogd: yeah

ogd: like, development vs. fetch

so if you just wanna get the files and you dont care about having a backup copy locally

dat clone vs. dat download/fetch

I think differnt clone commands are a good idea

karissa: what if you could do e.g. 'dat download <hash>' and that would: fetch metadata for <hash>, then download all blobs in the db at the time of <hash>, at the correct versions they were at at <hash>, and puts the files into cwd

ogd: yep

ogd: sounds about right. you might also want the table as a csv.

ogd: im assuming ppl who use this are going to do one-off analysis or want to export their dat somewhere else

karissa: yea basically the people who are using other peoples dat repos to do the 'last mile' stuff

surprised we overlooked this use case so long haha

ogd: indeed

ogd: well, it takes time. ux is hard

erikg: https://github.com/maxogden/dat-core now has snapshotting support (i.e. checkouts) using your algorithm

yyayayayay!

[dat-core] mafintosh pushed 3 new commits to master: http://git.io/ve8bd

dat-core/master 4f7f5c1 Mathias Buus: remove dead code