#amara

Hi sylvainc -

before you get going today - could you check these 2 pulls:

morning janeted, sure

thanks -

have a good weekend?

yes, it was great weather, how about youre?

yours

ok - more rainy, windy and cold

not very springlike

or I guess taht can be spring

but it looks like we're past that now

oh, too bad, better rain during the week

yes -

it's much nicer today - figures

I saw Ben's improvements last Friday, and I was planning to apply the same thing to other slow functions, such as moving videos and sending bulk messages, there are also some code structure improvements suggested by Ben. Do you see anything else to be done at first? I think there is also 1456 (moving videos within a team)?

I think both would be good

I tested 1458 this morning and it seems fine now

great, thanks

sylvainc, - so that I can test perf improvements more accurately - I made a selenium test that add a bunch of users then sends the team a message, and measures the time from when the send button is clicked to the messages sent notificaiton on the screen

so here's the numbers I got:

1000 ysers 6 seconds

3000 users 18 seconds

5000 users 31 seconds

10000 users 60.94

it never actually 504'd for the test - maybe there are different settings, but it's appears to be a pretty linear relationship for the time to send message

thanks janeted, that's a great baseline, since you have the test set up, how about trying a simple experiment to check whether the problem comes from notifications or actual message sending?

sure sylvainc -

how would I do that?

great, thanks, that's easy: in apps/messages/views.py:

comment out lines 134 and 135"

for pk in new_messages_ids:

send_new_message_notification.delay(pk)

got it

thanks again, that'd be a very helpful data point

1000 users = 1.11 seconds

5000 users 4.88 seconds

sylvainc, ^^ - do you need a number for 10000 users?

no need, thanks again

k

can do the same with the task approvals as well,when you want

ok, thanks a lot,

I did not expect the notifications to be the part that was taking time as they are done in a worker, but it seems like event just starting the tasks is taking time

yes - a lot of time

sylvainc, on 1461 and I'm not seeing the difference in the times

janeted: you mean it still takes long?

or no differences with test with no notifications, janeted?

sylvainc, it still takes long with the notifications on

exact same amount of time as before

sylvainc, I have to go, will be back online in about 1.5 hrs

argh, sorry, so I did not get the problem

tty later!

i can push my test case

maybe that would help?

would be great, thanks, if you have time before leaving

sylvainc, pushed it to the branch

./test.py webdriver_testing.check_teams.test_messages:TestCaseMessageUsers.test_large_team_message

is the test case - it's set to 1000 users now.

ok, wonderful, thanks a lot!

and it just logs the time for sending

you probably have to comment out del Logging in docker_dev_settings_test (2nd to last line)

so that the logs will display

back soon

ok, got it, tty later then

all: staging deployed

although looks like there are timeouts on staging now

ehazlett: do you know if staging uses dev_settings.py environment variables?

sylvainc: no it has different settings to connect with the staging services, etc.

thanks ehazlett, so we should have CELERY_ALWAYS_EAGER = False (tasks being asynchronous)?

ehazlett, I see the timeouts on staging as well

but it works fine for me locally

any other sources of errors messages?

could it be any type of indexing thing?

sylvainc, we are getting 100% timeouts on staging - just opening the main page

can you think of anything that could cause that?

staging branch works for me locally

ehazlett, is also looking into it

janeted: no idea, i don't see anything recent that should affect all pages

do you think the previous staging was on 8b17dd265632417af3fd0d8cc6938c4a132da8b6?

so any issue should have been after that?

janeted: it is a gateway timeout, so I don't think it is in amara code

yes - it was

on that rev, I mean

all my local tests pass

hmm ok nothing is showing in sentry

janeted: ok so i reverted to 2014-05-02 deply for staging and it's fine

so it looks like something with the latest commits

theh commit from the current working version is 8b17dd26

thanks ehazlett, is there a way to get django's output?

sylvainc: sentry but it's not showing anything

ehazlett, can we pull them in 1x1 or something?

this is the diff between what is running now and current staging

hey, bendk - we are having issues with deploying staging with the latest batch of commits

something https://github.com/pculture/unisubs/compare/8b1...

is causing the whole thing to time out

it's curretnly on production

?

it's reverted for now - and there aren't any sentry errosr

no o

just getting it onto staging is the problem

and what's timing out?

starting the server up?

bendk, seems like it's the whole thing

every page was a 504

bendk, should ehazlett deploy it again so you can see?

sure

yeah I think that's the best

ehazlett, can you deploy it again, please?

yeah

bendk: do you think this could come from this commit? It works on dev environment but I am not 100% it is fine. I call directly a function that has a @task decorator:

sylvainc: it should be fine

all: staging deployed and that fixed it

janeted: ^^

ehazlett, sylvainc bendk - so I am still getting a timeout when sending a whole-team message on staging

janeted: can you give me the page URL?

I"m signed in a darren.bridenbeck - trying to message the whole Volunteer team

what's the name of the team?

volunteer

nvm