#amara

nicholasreville: no

ehazlett: do we know what caused it?

nicholasreville: i think we have some long running queries that API clients hit

nicholasreville: we had K, T, and some others all hitting the API at once

ehazlett: are we able to throttle that in some way so that we're not vulnerable?

nicholasreville: once those queries finish things usually come back

nicholasreville: well we've talked about it before and didn't want to throttle -- T didn't like the idea

nicholasreville: what i did today is add capacity

is there anything else we can do to be more robust?

even 10 minutes like that in the middle of the day can be very bad

nicholasreville: no because our customers don't run on a schedule

nicholasreville: sometimes it's 10am sometimes it's 5p

ehazlett: we can't add capacity on the fly? or could we throttle some of our customers but not all?

nicholasreville: that's what i did (add capacity)

ehazlett: i mean, can we add capacity without having that period of downtime before it gets added

like can we add it instantly / automatically as the load increases before we get timeouts?

nicholasreville: yeah i mean right now it was about 5 min to bring up new servers

nicholasreville: if we new when they were going to hit it we could pre-emptively add servers

it's not feasible to have something that detect when queries are high and adds capacity automatically?

nicholasreville: well they are detected which is why i scale

nicholasreville: the problem is they hit at once

ehazlett: can we just have a total request limit and reject requests if we are exceeding that level?

i would rather reject API requests than have 5-10 minutes of downtime once in a while

nicholasreville: i'm going to dive into it to see if i can find a pattern to trigger

k, that would be great

all: back

all: call?

sounds good to me

jumping on skype

sjl: bendk janeted ehazlett nicholasreville Skype sounds good

works for me

deanjansen: works for me

sjl https://unisubs.sifterapp.com/issues/2334 - ticket that just needs merge

all: brb

all: back

sjl: just got testing feedback from cogi – will you have a moment to review it before you take off today? I'd like to get any followup questions to them before end of day

sjl: see you just responded

deanjansen: so look at that before other tickets?

I think they've given us all they can

sjl: this should be considered a blocker

sjl: and since it'll take coordination w/ them, it's prob good to take care of asap

ok, make a ticket and set it as critical I guess

sjl: ok

janeted: I have no idea on the billing thing, it seems like the migrations have run. I'll look deeper into it after I take care of this cogi stuff

ok thanks sjl

janeted: actually the URL maggie mentioned as erroring out seems to work now

janeted: is it possible the migrations got run at some point since the last time someone checked it was broken?

I suppose anything is possible

i don't do that stuff

heh

anyway - it should have worked so you can mark as fixed again

well, it looks like the last time someone saw it error was 3 days ago, maybe it's been fixed now

and can just retest it - it was all fine on the demos and locally before

yeah I'll mark it as fixed, we've been deploying staging off and on so there's a good chance it was just the migrations

deanjansen: ah ok, I think I see the problem (or at least A problem)

sjl: cool

deanjansen: are we trying to maintain backwards compatibility at the API level, or are we having partners update their code?

deanjansen: basically the "language" parameter is now called "language_code"

deanjansen: so we can either have them send "language_code" or we can make our API accept both

sjl: I think we should maintain backwards compatibility, if it's not weird/crazy – we haven't really done anything to have them prep to update their API calls

yeah

oh nevermind, it has nothing to do with that, it's just a bug.

all: deploying staging

Project unisubs-staging build #1981: STILL FAILING in 9 min 59 sec: http://dev.universalsubtitles.org:8888/job/unis...

Steve Losh: Update integration repo.

deanjansen: fixed the bug, I also emailed vietor and cced you and the others

sjl: awesome – thanks for getting that out so quick

yeah, luckily it was pretty easy since they gave us all the info to duplicate the request

sjl: cool

all: brb

all

all: back

all: heading out -- bass lesson tonight. migrations are still going strong, I'll peek in on them when I get back just to be sure

sjl: later!

all: heading out ; later

ehazlett: later!

all: out for a bit, brb in about half an hour