#celery

or part of the same site?

same site

do you have celery beat running?

or rather, where do you have celery beat running?

right now each server has celery beat running

ah ha

yeah

that is your problem

yes i know

what is your solution

im thinking a separate server

yea thats what we did

that hosts redis, and a celery beat

[] = server

and have every webapp server use that redis server

so that they can help each other out?

{} = autoscale group

so if one server is being lazy it can use more resources on its celery workers?

kk

{[web],[web]} {[worker], [worker]} {[beat]}

basically set your celery beat to its own autoscale group

it can be tiny

since its just kicking off jobs

set the desired capacity to 1

why even bother put it in an auto scale group

im thinking redis server with the beat attached to it

in case you need to change the server configuration

or in case the beat server dies

its AWS, stuff can happen

we also wrote our own wraper process that we run celery beat under, in order to catch the edge cases of 2 celery beat servers being online at the same time when we update the server configuration

and do you host your redis/rabbit on the beat server?

noooooo

Use Elasticache

crohling88: Thx for pinging me ;) Will be testing sooooooooon ;)

you want a central broker shared by your beat and all your workers

holy fkk i can host a redis cache in AWS?!?!

that your web talks to as well

damn straight

While your at it, run Postgres on RDS

im running mysql on RDS

yea ok

Also, AWS protip: use CloudFormation

and check out github.com/cloudtools/troposphere

Djarnis: try setting BROKER_POOL_LIMIT = None

Its a performance overhead, but may solve some (dis)connection problems

well ty for elasticache, that is going to be awesome to use

no pronlem

*problem

dashx: its important to use only one broker for your site

otherwise its like running mysql on each web server

micah_chatt, should i use the elasticache as the broker, and the celery workers can use the local redis cache for the result backend?

does that make any sense?

do your tasks return anything meaningful?

right now no

set it to None

and if they did?

return something meaninful, that is?

then set the backend to somewhere you would want to collect/observe it

If you use RabbitMQ, that is a bad backend

Redis isn’t a bad idea

what about the same broker?

yea that is fine

k

but the DB backend also works and may be easier to query

It’ll make your DB useage go up a bit (all depending on the task artifact/number of tasks)

micah_chatt: I think BROKER_POOL_LIMIT = None was the catch … Had it uncommented in the code … Can’t wait til the test results comes in ;)

ok

Thanks for the help btw micah.

no problem

dashx: do you know about celery flower?

that was the best thing we had to do monitoring with Redis as the broker

micah_Chatt: I saw it today. IM thinking it will go into the same server thats running my beat

micah_chatt: For rabbitmq it's any argument supported by amqp.Connection, or librabbitmq.Connection

brilliant, thank you!

asksol: it's time for my daily ping ;-)

re https://github.com/celery/celery/issues/2492#is...

bmbouter: what did I miss? :)

I randomly drop links to that comment every few days

there's just a lot going on so I could see missing it

I'm looking for feedback on that plan

I don't normally read the backlog, if there is a backlog that I need to read it's my mail :(

The plan is good, the downside is adding yet another setting we need to manage

brb, interview

and 4.0 is so complete we just need to sprint a bit to complete the documentation

asksol: I can't imagine you could read all of these things. Can you +1 the plan and re-open the issue

perhaps someone on my team can work on it

if os, we would do it for both 3.1 and 4.0

bmbouter: a huge issue is that worker_direct() won't actually have access to the configuration

but I guess we could make it lazy and use a WorkerDirect type so that send_task fills it out later

The setting name is too specialized I also think, it would be better if there was a CELERY_WORKER_DIRECT_QUEUE, that enables you to set the queue/exchange settings

yes I agree

CELERY_WORKER_DIRECT_QUEUE = {'auto_delete': True}

that will work for 4.0

for backwards compatability on 3.1 I'll suggest the separate setting for auto delete

We need to continue on 3.1 for a while longer and we want that in 3.1 so it's our carrot so to speak

but that means we have to maintain both of them

that's the downside of backwards compatability

we could make a new setting that does it the 4.0 way in 3.1 and just name it differently in both

that would be additive

yeah, but it means 4.0 needs to support WORKER_DIRECT_QUEUE and WORKER_DIRECT_QUEUE_AUTO_DELETE

no 4.0 could do it totally different than 3.0

3.1 rather

according to semantic versioning that would be appropriate

it could, but that would be breaking the deprecation policy :)

if we add this setting to 3.1 we need to support it until at least 5.0

oh interesting I'm not familiar with that

where can I read more about that in the docs?

we use the same policy as Django: https://docs.djangoproject.com/en/1.9/internals...

note that there is nothing stopping you from doing what you want in 3.1 already

asksol: I'll think more about this and try to suggest a more cohesive plan

I have to go now unforutnately

can you describe this more though "a huge issue is that worker_direct() won't actually have access to the configuration"

and this "we could make it lazy and use a WorkerDirect type so that send_task fills it out later"

I'll see what you write even if I can't respond right now