#celery

hm

hello friends, I keep getting the following error: NotImplementedError: No result backend configured. Please see the documentation for more information.

though my result backend is indeed set

I will upload my config in a second

lknd: celery --version?

3.1.21 (Cipater) malinoff

I noticed that when I'm running my workers this is showing on the config

results: disabled://

lknd: lowercase settings are celery4-only

oh

on celery3 you have to use RESULT_BACKEND

thank you, I suppose I'm running off the wrong version all along then

I wanna thank you for all the help the other day by the way malinoff

I learned a whole lot by going through the documentation later

lknd: welcome

mostly regarding the task calls, and how the broker came into play

this is a lot of fun, I wish I had an elective class for this at university

malinoff: the result of a task should be whatever is being returned right?

lknd: yes

hm, for some reason I'm returning a string from my task.. the task executes properly but when checking the celery log I get the task id and the result as None

and common issues that would lead to my result not being the return value?

lknd: show your task code

will do

lknd: are you sure you've updated the task on worker side?

oh sorry malinoff I thought that happened on touch

you're right, I have not restarted the worker

thought it was done automatically if the task was touched

does it make sense to say that the result backend of celery is simply adding rows into a table with the state and result as fields?

if I wanted to generalize the process malinoff

and perhaps emulating without relying on celery magic

in case we need to switch later on to say Hadoop due to the increased amount of data throughput

lknd: if you need to publish results into a custom DB, you should really implement a proper backend class

using one of these https://github.com/celery/celery/tree/master/ce... as example

rather than a custom DB, I was thinking more along the lines of decoupling the worker process

and not relying on that celery magic

in case we change from celery to some other queue task down the line

That's not how celery is supposed to be used :) Celery is a framework, and changing it to an other task queue implementation won't be easy anyway

Just imagine how much work you'll need to do in order to replace django with, say, flask

(not a hello world app, but a somehow mature project, of course)

true, what I thought was definitely that it would be a lot of work

I've been tracing through the source code

and definitely not something to get done in a couple of days, to understand at a micro level at least

our lead has this urge for us to look into decoupling and making everything as modular as possible (this included!) but I believe that for time's sake I will talk him out of it

I believe it would take longer to literally re-write celery magic than to adjust the code down the line if needed to for a new task queue

meh

celery is already modular, you only need to use its modular system - that's all

I can see that, but what I meant was more along the lines of hm

our lead wanted to emulate what celery does at a low level

so that if we switch to say hadoop, it can sort of "work" without much modification

lknd: are you going to use hadoop as message broker? :)

have never used hadoop tbh, but I thought it can be used as a job scheduler?

malinoff:

Oozie is an hadoop job scheduler

But yarn is a more low level job scheduling with queues system and a FIFO / fair share rules

well, if you're going to use hadoop you're likely to use hadoop-specific tools

honestly I need to better wrap my head regarding the necessity to decouple the worker part in case of future scaling

lknd: imo if you're choosing hadoop to solve one problem, you'll have two problems

i think that was more of a generic example rather than what would really happen!

in general the idea is that decoupling the worker part will allow us to easily switch between Celery/Hadoop/whatever third party API if needed for scaling

at least that's what I can make out of it

lknd: you can't be prepared for everything. There's no generic solution to switch implementations on demand - there always be implementation-specific warts and gotchas

and you can't do anything with it

you _will_ need to spend developers effort in order to replace parts of your system

but you know the funniest thing - you might not need to replace anything at all!

oh?

I agree with your thought regarding it

at least, from my ignorance

that's what I made it out to be

what about using apache kafka?

does anyone know what the result in celery_taskmeta is encoded in?