#dat

mafintosh: check out http://asdf-standard.readthedocs.org/en/latest/...

karissa: checkout by above gist also

mafintosh: i think the good parts are: the block stuff, compression, exploded form

mafintosh: bad parts are: yaml (because complexity), scientific specific stuff like ndarray, coordinate system (because not everyone needs it)

mafintosh: how about metadata as an object instead of 'author'?

mafintosh: also good is optional block index

karissa: yea for sure - the author was just an example

the gist wasn't clear on that though :)

ogd: reading it now

mafintosh: theres also a talk https://www.youtube.com/watch?v=YRBgL8_tt9U

mafintosh: i kind of want to open an issue on the asdf repo and be like 'we like these parts, but dont care about these other parts, would you consider make the spec even more modular'

ogd, karissa: re: my gist, we might wanna call it some else than commit as well

ogd: i'd do it, would surely at least generate a good discussion, and it does seem like the "science-specific" parts could be siphoned off

karissa: search is cool! just queried figshare for "brains", tons of hits... what other targets could we have?

freeman-lab: thinking CKAN (datahub.io), socrata, github, harvard data verse

freeman-lab: opendatacache.com

freeman-lab: glad you took a spin! :D :D

we also probably need 'transformers' for each of these, that turns a result from the search into a script that can be executed to get the data

karissa: sweet! was also thinking about the nasa satellite imagery apis

freeman-lab:ooo sounds fun

yup, was just thinking about that as i tried to go from a figshare result to the actual data

maybe this api would be good to think about first https://github.com/karissa/dps#dps-add---execsc...

ah cool so "script" could be some kind of filter or extractor

looking at figshare it seems the pattern would be something like, hit the url if its a fileset or dataset, then hit the download_url for one or more files of a certain type... can see how standardized that is and try to spec out as an example

we can run indexer nodes that use the search but then download files, hash them, put the hashes into a dat with a link to the original http resource

then you can clone the dat metadata and verify the hashes when you download the real data, or ask peers for the data references by the hashes

freeman-lab: that'd be awesome

ogd: oo great idea.

ogd: yeah that sounds sweet

mafintosh: http://bitsup.blogspot.com/2015/09/thanks-googl...

ogd: neat!