#openspending

georgianab: hey there!

:)

pudo: hello!

how are you doing?

quite well, i'm at my day job

cool. what is that?

backend developer in ruby

coool

indeed

so adriana has now formed a little secret huddle? that's really brilliant.

anyway, victor showed me aleph

yes she did

and the funny thing is that apparently both projects bring me to you :))

on aleph: I'm currently working on a related day job and hope to convince them to let me develop it a bit further

This could use an upgrade: https://investigativedashboard.org/search/

it has much potential

this one is shiny!

Yes, designers rock :)

I've already got a new search backend up for them for entities/networks, once that is stable we'll do more on documents

(btw really need to import TED into that network thingie :)

would be nice

so, getting back to adriana's project

which now has the codename datavis

yeah, tell me more about that if you don't mind!

i have the structure and some data

I just saw her mid-August but we didn't chat about it enough

i.e. the TED DB?

yes

sort of

we are only interested in the fields that are included in opencontracting standard

i also plan to use elasticsearch for that one

man, this might actually be really relevant to you - did we talk about it yet? https://github.com/occrp/datamapper#example-con...

no, but seems like we could be good friends

so it takes tables (db or spreadsheet) and maps them to JSON Schema

which I think OCDS is

it should work

right now it indexes to ES

but I also want to make it run against RDF

which is scary as shit

but then I get graph queries

where the graph is defined by json schema :)

pudo: who has sparql endpoints for procurement data?

nightsh: I don't actually know. I don't yet :)

there's a few EU projects that might

e.g. http://publicspending.net/

somebody needs me here

will get back as soon as i can

ok ciao

sorry to rant on you :)

and pudo, what is barn>=0.4 in aleph's requirements.txt and where does it come from? :)

aargh

it's an old library I used

you can just remove it apparently

I was about to try, thought I'll search for unfortunate module imports before doing that

grep gives me nothing, I suppose it's safe-ish to get rid of it, thx

pudo: docpipe is also MIA :)

can you try with the openoil branch applied?

I think I sorted out the errors in that

FWIW, barn -> archivekit, docpipe -> loadkit

I'm slow ATM skype-ing, but sure :)

sorry :) I'll be around all day in any case

pudo: i'm back

hey hey georgianab

you were speaking about rdf and having graph queries

I'm very sorry about that

but yes

why are you sorry

because I brought up RDF

it's akin to a war crime

it's great to know something that cool will come up

haha

so I'm a bit cynical about it

but it's useful for making data integrate

so there's this thing I've been working on: https://github.com/pudo/jsongraph

i worked on a project before

oh cool, what did it do with RDF?

when we investigated tenders in romania

and we had a loot of xml's

but we has to build a relational db that was similar with the one where we got them from

because it was hard to make sense of them otherwise

yeah I get that sense a lot

i think the rdf functionality you were talking about would have been priceless for that

sharing db dumps moght be easier in many cases

it wasn't a dump

we got them through soap

messy business

outch

that's hard core

but it happens a lot

because most state instituions work on windows

at least in Eastern Europe

and that's how they roll

:))

old-skool :)

which is always bad in software

getting back to it, i will take a look at your tool

but i wanted to ask

I need to write this all up...

for the aleph project, did you use es as the main datastore?

so aleph is based on this thing called archivekit

which basically manages a sort of virtual filesystem

in production, that would be S3 or something like that

so all index can be rebuilt from an S3 bucket

which has all the actual source documents

if you click on [cached] next to anything on aleph, it actually forwards you there

oh, i thought that was a db that stored everything

damn, with an HTTPS warning

no it's just flat files on S3

interesting

for adriana's project however

there's also some metadata like this: http://aleph.grano.cc/api/1/manifest/saflii/b4f...

pudo: got it past pip, diving into the thing now

haha, awesome nightsh!

so for adriana's project i think i need to store them all in a db because there are some calculations and things i need to do on that data

what kind of DB were you thinking of?

SQL would be really interesting, but there's so much stuff you can also do with elastic now

i was thinking about mongo because there are several levels of nesting

since, i assume, in TED they were kept as flat files

oh we're talking about TED? I thought you meant documents?

well, they are documents

that come from ted

as far as i know

is it the actual TED web pages?

no, there are some csv's

that contain only some of the fields in documents

let me give you a sample

please

oh, i got it all wrong

sorry

so, the thing is that i have some documents whom fields need to be stored into a db that we are going to use + es and also be able to map their fields to a csv structure and export that

so all of that minus the database is in aleph

you can index documents and extract any sort of field youlike

and then export these as a table

indeed

hooking in a database should not be very hard, but it's a bit of work

that's thing i wanted to ask, because i'm slightly undecided about how to make that db arhitecture

where does the structured data come from, are you pulling it out of the docs or is it in a separate dataset?

out of teh docs