#dat

sfkeller: no ckan stuff yet, though i feel like finnp looked into it

sfkeller: and can you elaborate on the schema mapping question? we store everything in protocol buffers so theoretically there could be something that uses protobuf schemas to do schema mapping to other things, but i dunno what a general purpose api would look lie

sfkeller: i experimented on ckan a bit: https://github.com/finnp/ckanDOWN

ah right

sfkeller: however i think using ckan datastore as a dat level backend is not too clever

sfkeller: because ckan is for making data available. and people are not interested in viewing the internal representation of dat online ;)

[dat] maxogden closed pull request #207: change log ids in data keys and current table (master...master) http://git.io/Lv3l3g

finnp: Ok. But using ckan as frontend / catalog? like here http://opendata.admin.ch/en or search engine http://docs.ckan.org/en/ckan-2.2/api.html ?

[dat] maxogden pushed 17 new commits to master: http://git.io/0fzwqw

dat/master 9d5510c Finn: custom working version

dat/master 452f943 Finn: content negotiation with format-data module

dat/master 709b518 Finn: update format-data, more pluggable format-opts interface

sfkeller: I think you could link dats on ckan, like i did here: http://demo.ckan.org/dataset/earthquake-data/re...

[dat] maxogden pushed 2 new commits to master: http://git.io/8x7Y-g

dat/master 1a519bc Anton Podviaznikov: fix example. Should be an array

dat/master b1e7628 Max Ogden: Merge pull request #224 from podviaznikov/patch-2...

ogd: Regarding my schema mapping question: In OGD we often have heterogeneous schemas around a similar topic, like building addresses (e.g. coming from different states/cantons). I'm looking for an aggregation service which "harmonizes" these sources and offers a "unified" (e.g. nation wide) data set. This needs "adapters" which do schema transformation like this http://webhelp.esri.com/arcgisdesk

top/9.3/datainterop/mergedProjects/FME_Workbench/gifs/schema_example2.gif

oh yea i forgot in switzerland its OGD, hahaha

:-)

sfkeller: in that screenshot its a 1:1 mapping?

ogd: OGD = Open Government Data (of course :-) )

yea, i was at OKCon last year and first saw that

pretty awesom eheheh

sfkeller: so i have been conceptualizing a thing called the 'dat external table sync API'

sfkeller: i need to write it up, but it would basically be a section in the dat.json config that you could add to configure an external table that your data should get written to automatically

so if data gets written to dat, and you have an external table setup, dat would automatically push data there

the actual push mechanism would be some sort of module that can take a row of data from dat and insert it into a remote table

ogd: Yes, but this is more complex (from BizTalk Map): http://stackoverflow.com/questions/867686/how-t...

another aspect of the table sync api would be a schema mapping

so you could choose which how to map the data from dat into the external table

so e.g. you could write a postgres-external-table module and add it to dat.json, along with the pg table you wanna write to and how to map the schema when you write it

sfkeller: ahh interesting (the biztalk map)

Actually: I'd wish some schema mapping language like this https://github.com/sourcepole/ogrtools (or a better one :-) ) of which I was and still am involved.

I mean, the OGR/GDAL open source converter should anyway become part of dat.

ogd: table sync api sounds super useful for ckan

there are node bindings now :) http://npmjs.org/gdal

i recently updated https://www.npmjs.org/package/shp2json to use them

awesome...

Still: https://github.com/sourcepole/ogrtools is something on top of OGR.

[dat] maxogden pushed 1 new commit to master: http://git.io/N8xgQg

dat/master 3565bc5 joyrexus: detect .tsv file extension

[dat] maxogden closed pull request #229: detect .tsv file extension (master...master) http://git.io/9HmG7A

[dat] maxogden closed pull request #231: Usage information for each command with --help / -h (master...usage) http://git.io/ijOTqw

running npm test locally, i'm getting a hang on "CLI dat clone remote that isnt running"

fresh pull from git

it turns out i hadn't killed a previous listening process

i think this caused the hang

I've been hit by that before. We need to make the 'dat discovery' stuff more robust

erikg: I did some work on that while back here, https://github.com/mafintosh/http-party

Just needs to be integrated

mafintosh: so it's like a stack of HTTP servers?

on the same port?

erikg: its like a pool of local http servers that do automatic failover (on one machine)

erikg: similar to what dat does internally just a bit more fault tolerant and as a separate module

i see

mafintosh: what about the idea of multiple dats on the same port

mafintosh: do you see that as orthogonal or resolved somehow by the http-party?

erikg: from a high-level view http-party runs multiple http servers on the same port so yeah :)

initially that was why i wrote. i just never got around to integrate it :(

mafintosh: yeah, it's just the simpler case of some kind of dat multiplexing

erikg: yea its basically just failover

ive been getting npm fetch failed errors in travis the last couple days :(

yeah same

Attempting to stream csv files...

What exactly is a mulit-part encoded file? http://docs.python-requests.org/en/latest/user/...

pkafei: its an encoding for http where you can send multiple files in one PUT/POST

pkafei: usually its used in either email or html form file uploads

pkafei: its kind of hard to parse though. how are you using it?

ogd: https://github.com/maxogden/dat/pull/232 is rebased

erikg: i was rerunning the tests cause travis crapped out the first time

erikg: oh cool they passed

ogd: csv files are encoded in a similar fashion like email?

[dat] maxogden pushed 5 new commits to master: http://git.io/X0erqQ

dat/master 5630973 joyrexus: detect .tsv file extension

dat/master 53ada6b Erik Garrison: add --tsv command-line option...

dat/master 61b74ff Erik Garrison: test for csv and tsv import from stdin

pkafei: nope