#amara

all: deploying production

all: afk for a bit, will be back later

widespread issues saving subtitles

widespread saving issues again

bendk: opened a pull request for gh-758

ehazlett: franontanaya is reporting save errors and I can see some database locking errors again in sentry: https://sentry.amara.org/amara/amaraproduction/...

got multiple translators with intermittent problems through the euro morning (it always seems to be worse in euro mornings). I had one failed save and one successful one

janeted: looking

thx ehazlett

ehazlett: I think I heard you say once that we re-index videos once a day. How does that happen? Is it a celery task, do we just run the management command from cron?

bendk: i think it's a periodic_task in celery

ehazlett: if you want to review bendk 's fixes for gh-758: https://github.com/pculture/unisubs/pull/832 - that will need the index updated once we merge and deploy staging. It will be good to get that on staging sooner as it's a high-pri TED ticket and they'll probably want to check it out.

ehazlett: yeah, if you can review that one soon that would be good, they were hoping to test today

janeted: I don't think we need to worry too much about the index.. Just tell them it will be fixed tomorrow or so

k

can we just updated it manually though - to avoid a flurry of tickets, please bendk ehazlett

bendk: janeted what branch did that come from? it doesn't look like it was sync'd before the PR

janeted: it takes a few horus

*hours

ehazlett: I can manually run it for TED videos

ehazlett: I didn't sync before the pr.

I don't go there

ugh ok

well bendk i'll apologize in advance if i stomp on any commits

maybe bendk can fix it?

ehazlett: yeah, I'll fix it

1 sec

ok

ehazlett: merge should be good now

bendk: ok

bendk: this was from the general_log for the save issues this morning http://dpaste.com/1333102/

bendk: that thread was in "sleep" for over 300 seconds

300 - Display a message on empty revision (version) pages - https://unisubs.sifterapp.com/issue/300

bendk: can you take a quick peak at this - https://github.com/pculture/unisubs/issues/833 I need to do selenium test for all the speaker name stuff, and I just want to be sure I've got all the behaviors down correctly.

janeted: one thing is that for the site search, it should return results based on the speaker name for all languages + the speaker name set on the video

for the tasks search, it's only the speaker name on the video

ok thanks bendk

and for the activity pages, we will show the title at the time the video was deleted

right

so for existing activity, before the changes it's just title

after the change, it's speaker-name: title (for ted videos)

sounds good

we also should test that if speaker name isn't set, then for TED pages we just return the title

not ": title" or something like that

that's all my comments, the list looks good

ok

thanks

bendk: janeted merged gh-758 -> staging ; want me to deploy staging?

ehazlett: yes please

janeted: ok

all: anyone using staging?

I want to test something out there

bendk: no -- it should be deployed

ehazlett: can you give univsubs_staging process access?

I'm going to test what happens when we run the rebuild index command at midnight

I wonder if that is related to stability

looking at the newrelic graphs there may be a somewhat of a correlation

bendk: which graph?

just the main one

the big spikes seem like they happen around 00:00

bendk: right but look at solr -- not much there

i would think if it was from indexing solr would be high too

bendk: for example you can see when you indexed the specific team at 10:15

bendk: ah wait -- that link you sent doesn't show midnight

one sec

yeah same -- not much solr

does look like something automated though

perhaps one of the partners does a sync or something

I'm wondering about the database usage

i don't think we have that many periodic tasks around that time

in particular, I think we have an issue with one of our tasks that queries the database, then reindexes the results

right

while it's reindexing it's keeping that transaction open, which may keep database locks open

hmm ok -- i would just think we would see the same spike in solr; maybe not

I'm wondering if the site reindex would do that as well

although I'm sort-of doubting it at this point

bendk: actually that graph is the majority in python

db is really low

yeah, I know

but I'm still doubting it, especially since it seems like the outages were correlating to when we had transactions that were keeping open row locks

maybe the time spent waiting for the lock is counted as python

honestly, I don't really have a good explanation, but I don't think that python usage should take down the site. At worst it should just make things slow

exactly

well there's definitely something going on

bendk: http://d.pr/i/w6wZ

yeah, so that's the reindexing definitely

bendk: here's the celery search index: https://rpm.newrelic.com/accounts/207361/applic...

the task is search.tasks.update_search_index

not sure why it's not on that list though, it must have a really long execution time

well if so it should definitely be there

I just needed to change the time span

hmm

hmm indeed

that seems like it definitely could be an issue

yep

however

it's always been like this

and we never had issues before

so it must be like a domino effect

I think it is

it's not the tasks themselves it's how they tie up the DB (if my theory is correct)

i mean it really should eat a bunch of time

and so changes to our DB could make things worse

it indexes every video

right

did we switch to InnoDb as part of the DMR change or have we always used it?

it's been innodb since i got here

well, the switch from the table of subtitles to all subtitles stored in 1 row may have done it

ah

the other related task is update_search_index_for_qs, which does like 200 videos at once

both of those seem like they could be causing the issues to me

i agree

anyways, I'm going to try to make a new system for indexing... It should be more fine grained, like index 1 video per task

ok -- well we'll need more workers for sure

otherwise we'll add ~300k tasks at night

maybe not, the other part is some sort of lock so that only N workers work on it at a time

right

I think it's like 600k

ok

even better

we don't have queue priorities in place

so we might want to look there also

we can maybe make a second queue

so we can dedicate workers to only doing the indexing

ugh

yeah

I think that's the normal celery way

right

but I'm not sure if we need it yet though

if i can get some solid time on the containers we would be able to bring up any number of workers on any queue

that's the last piece (the workers)

okay, don't worry about this yet