#theodi

Nearly standup time! Here's the hangout URL: https://plus.google.com/hangouts/_/calendar/Z2F...

is it just me or is this the clearest the office hangout has ever been?

Yo floppy

yo

Bit worried about OpenAddresses

Count is stuc at just under a million

But the script is still running

And no errors

WUT

joy

Can't see anything in the logs either

hmm

that's bad

not going up at all any more?

I don't think so

Just jumping on the console

Yup. Count is at 994983

And no errors in the log

I wonder if we can just kick it off from a later page maybe

maybe try locally

see if anything explodes

This was the last one that was ingested

so at least we know where to start then

At 5:30 last night

Right, so this was the last page http://ernest.openaddressesuk.org/addresses?pag...

So, I think we should start from a few pages back

Maybe 111650

ok - it won't duplicate, so yes

going to try from local?

Yup

I'll just try a few pages first

There's a surprsing amount of duplicates, but it seems fine

I'll kick it off again

pkqk: do you mind if I optimise the JSON response to the datasets search on ODCs?

I hacked it in a while ago, but it's WAY too slow and doesn't need to be

so if it won't conflict with you, I'll improve it

sure, I fixed some of it but it’s probably still quite slow

you mean the json response generation bit?

just looked through the jbuilder stuff, yeah it’ll be slow, also doesn’t seem to do pagination which you might want, what’s it used for as it appears to be giving a lot more information than the .atom feed

floppy: Right. I've stopped the workers and tweaked the procfile, and added some logging

Once it's deployed, I'll switch them on again

pezholio: ok

pkqk: yeah, I did a quick hack and reused the full partial, so it's getting loads of info at once

I'll take a little look

mainly because I need screenshots for my blog post :)

cool, just keep it on a branch/PR as I might be deploying things today too

sure

I'll run it by you first

👍

morning all

Mornin' Davetaz

floppy: It's creating addresses happily now

good news

Should get to a million soon!

996764

strange that it stalled

has it sped up?

Yeah, back to (sort of) normal

But

Don't forget, as it goes through, the likelihood of there being duplicates increases

That's why it probably seems like it's slowing down

oh

No idea why it stalled though

because companies house isn't unique of course

many companies registered at same address

H'eaxctly

you mean

ahaa

so we're not expecting 3 million final addresses

Yup

gotcha

New Wu Tang BTW 996764

Wu Tang Clan - Das neue Album "A Better Tomorrow" (Album Stream) - YouTube

"A better Tomorrow" von WU-TANG CLAN erscheint am 28.11. Hier jetzt das Album kaufen: amazon: http://wmg.cc/wutang-abettertomorrow-amzn iTunes: http://wmg.cc...

Even

We have ONE MILLION ADDRESSES

and a LAAASSSSEERRR

"LASER"

I need that laser

to DESTROY EVERYTHING

What up?

unnecessary bullshit

don't worry

oh balls

'sup?

i'm in Malaysia the week before xmas

Bad planning that

grrrrr

I'm beginning to think github pages doesn't like me

Page builds are JUST NOT HAPPENING

When I clone and build locally, it's fine

And I get no error email

blame it on queues

Yeah, but it's normally a few minutes at the most

This has been over 10 mins

I’ve learnt from talking a friend that works at Heroku that things working is an illusion and anything that big is basically failing all the time

Ha!

That's encouraging anyway

But then we do know that the whole Internet is help together with blu tak and bits of string

Right. Lunch

pkqk: https://github.com/theodi/open-data-certificate...

Optimise JSON search by Floppy · Pull Request #978 · theodi/open-data-certificate · GitHub

By removing most response set data. This cuts the response time by roughly 2/3 for /datasets.json

deleting things, the best way to speed things up

yeah, it was always slow because it was doing too many queries to get info that wasn't necessary

btw, I'm the only person using this I think. It's not documented

too many queries is most of the app