#amara

sjl: crap, sorry

sjl: i'll jump on it when i get off this call

I think after the DMR we should look into improving performance a bit at some point, because these video pages are excruciatingly slow when you're trying to test crap

resetting these branches is a pita

honza: you mean switching between?

ehazlett: yeah, vereset essentially :)

ehazlett: can't wait for this stuff to get merged

honza: yeah that's what i do: git checkout <branch> ; vereset ; dbreset

honza: no kidding :)

ehazlett: the command itself is great but man it takes forever

ehazlett: honza i totally have two vms. because i just can't deal with two branches

honza: we have too many packages ;)

fernandotakai: honza yeah

ehazlett: you could probably do some symlink magic to keep both and just switch but who wants to mess with that

fernandotakai: honza it does take forever to build our ve

honza: yeah

ehazlett: friggin pip

:)

all: have any idea why we're getting a shitload of messages on a celery vhost (rabbitmq)? http://d.pr/i/Scei

all: has any one did anything with celery recently?

ehazlett can be my import script

arthurdebert: ah ok

arthurdebert: what are you using for the vhost?

ehazlett where can I see the stats for the rabbit queue?

ehazlett production>rabbitmq>>overview>>queus_total__messages_ready reads weridly for us

arthurdebert: sudo rabbitmqctl list_queues

ehazlett I mean historical data, I want to see when it grew

arthurdebert: hmm those are odd

ehazlett is this what we want https://graphite.amara.org/render/?width=1105&a... ?

arthurdebert: yeah -- i need to check those stats ; they look odd -- may be a whisper storage mis-config

ehazlett ah

arthurdebert: i got the first alert about 45 min ago

ehazlett hm, I wonder if I should stop the import or if we can increase the # of celery workers?

arthurdebert: looks like the unisubs queuea are fine

arthurdebert: well that's the issue

arthurdebert: it's putting the messages on the wrong queue

arthurdebert: so there are no workers consuming

ehazlett which queues are not fine?

ehazlett what?

ehazlett it's running with --settings=unisubs_settings...

arthurdebert: there are 70000+ messages on the "celery" queue on the "/" vhost -- it needs to be on the "unisubs" vhost

arthurdebert: what directory?

ehazlett arthur@data-00-production:/opt/apps/production/unisubs$ python unisubs-integration/ted/tedximporter.py --datapath=/home/arthur/postprocessed_sans_dupes.csv --settings=unisubs_settings --import >> /home/arthur/tedx-import.log

arthurdebert: can you stop it? i want to see if the messages stop

ehazlett hm, I can but I'll probably loose some time as it restarts and goes through the list again… if you fell it's important, I'll do it

arthurdebert: well right now those jobs are just sitting

ehazlett hm, ok, let me stop it

ehazlett stopped

all: leaving for the weekend, see you on monday!

fernandotakai later

arthurdebert: later

arthurdebert: yeah they stopped

ehazlett ok, and how would that be causing us to use the wrong queue? I can't fanthom it

arthurdebert: i'm still digging -- have no idea

ehazlett k

arthurdebert: ping

cz looking

arthurdebert: echo "from django.conf import settings ; getattr(settings, 'BROKER_VHOST')" |/opt/ve/production/unisubs/bin/python manage.py shell --settings=unisubs_settings

arthurdebert: >>> 'unisubs'

ehazlett how is that possible? I didn't change anything on that env

arthurdebert: i have no idea

arthurdebert: digging

arthurdebert: here's a hunch

arthurdebert: did you run this on staging?

ehazlett nope

arthurdebert: ok -- here's my current wild guess ;)

arthurdebert: actually nevermind -- i forgot you were running with --settings=unisubs_settings

all: brb 15m

arthurdebert: perhaps this: https://github.com/pculture/unisubs-integration...

ehazlett that's normal

arthurdebert: well it's overriding the django settings module, right?

arthurdebert: which would reset all values to defaults

ehazlett but wouldn't that screw up the other settings, such as the DB connection?

arthurdebert: yeah i would think

ehazlett we're picking up the right settings, or else nothing would work (mainly the db connection)

arthurdebert: that's true

arthurdebert: so something is overriding the BROKER_VHOST

arthurdebert: ok so i temporarily edited the script to output BROKER_VHOST and it's right

ehazlett ah

ehazlett are you running it?

arthurdebert: no i just made it output BROKER_VHOST and exit

arthurdebert: and it was right

ehazlett k

ehazlett must be something else

sjl: pushed

ehazlett but the messages stopped, right?

arthurdebert: yep

arthurdebert: http://d.pr/i/qVLF

arthurdebert: it's been there

arthurdebert: shit i have no idea

arthurdebert: try running it again -- i'm going to watch the queue

ehazlett will do

arthurdebert: yep climbing again

ehazlett but it's barely run anything

ehazlett like it processes about 1 video per second

arthurdebert: did you stop it?

arthurdebert: http://d.pr/i/85x9

ehazlett nope, it's running

arthurdebert: yep celery is climbing

ehazlett here's my dilemma a) I'm really going to need to leave on the next 30 minutes b) I can't seem anything wrong with the video's I'm pulling

arthurdebert: stop it -- can you add it to print the output of settings.BROKER_VHOST right before it processes the video?

arthurdebert: i only checked before it called process_all

arthurdebert: and i'm not familiar with it enough to start hacking ;)

ehazlett will do

arthurdebert: what is it putting on the queue?

honza: thanks

arthurdebert: this is one of the decoded jobs, this look right? https://gist.github.com/a5bf2047f73e833a14f3

all: back, finally

ehazlett I think this is related to stuff that fernandotakai and honza are working on

arthurdebert: ugh

arthurdebert: i wonder why it stops when you stop your importer

ehazlett is there a human readable way to get those parameters?

arthurdebert: actually wait, i think what i did was just pull a random job

fernandotakai hey, sorry, if you're around, let me ask something

arthurdebert: nope -- it's the jobs on the celery queue on /

fernandotakai is it possible that just by importing videos from yt and saving them locally I can be triggering the adding amara credits things?

ehazlett if that is being created by me, than we NEED to drop those

arthurdebert: i can destroy the queue

ehazlett can we get a human readable output of that? I want to check the parameters

arthurdebert: let me check

arthurdebert: trying

ehazlett I don't seem to miss those tasks in what I'm doing, but it makes me nervous to not know what's going on

arthurdebert: exactly -- same here

ehazlett yeah, I can see why my script would run that, but I could totally drop them

ehazlett nor I understand why it run that on the other queue

arthurdebert: i know -- very odd

arthurdebert: {'retries': 0, 'task': 'videos.tasks.add_amara_description_credit_to_youtube_video', 'eta': None, 'args': (u'aPThfwSc2tA1',), 'expires': None, 'callbacks': None, 'errbacks': None, 'kwargs': {}, 'id': '96f549bc-04bc-4f63-b77c-01ffdac4f97b', 'utc': True}

arthurdebert: had to perform some magic ;)

ehazlett tks, let me check

ehazlett ok, so, this is caused by my script, for sure

ehazlett I also know where it's calling but

arthurdebert: k

ehazlett a) I can totally drop those b) I don't know why they're being queued on the wrong vhost

ehazlett I 'm thinking about letting this running

arthurdebert: i know -- the only thing i can think of is either 1) somethings changing the vhost or 2) celery is being initialized with something like celery = Celery() which would use the default settings

ehazlett I'm positive this will have no ill effects on what I'm doing

arthurdebert: ok -- it will just send alerts (no big deal)

arthurdebert: there are no workers pointing to that vhost

ehazlett ok, so I'll restart it and we can figure it out

arthurdebert: ok

ehazlett it's still pretty weird

arthurdebert: thx for checking

arthurdebert: 100% agree ;)

ehazlett tks for the heads up

arthurdebert: yw

ehazlett I want to know what's going on, I just can't dive in right now

arthurdebert: sure -- np

arthurdebert: i'll leave the messages there so we can check later (as long as they don't expire)

ehazlett tks, that's good thinking