#celery

Hello everybody,

I'm having a hard time using a Celery task to work on in-memory data structure of my Django program

Hello,

I'm having a hard time using a Celery task to work on a in-memory data structure of my Django application

(Python app would be the same)

I think it's because they don't share the same memory space, being on different processes..

Is there any way Celery could run in the same process as the Django app

maybe in a different thread?

cyberjoac, can an application run in the same process as another application? (tip: think of cryptographical services)

Indeed that sounds impossible...

So what would be the way to go to work on an in-memory data structure with Celery

cyberjoac, drop this structure on the disk

2GB

so?

file?

don't you have 2gb on the disk?:)

I'm not sure what confuses you

You simply cannot manipulate the address space of an other application

All right so basically

I should serialize the data structure to the disk

to a file? or Redis?

the thing is that in order to work on the data structure, I have to load it in memory

so it confuses me a bit

when should I drop it to disk

before sending it to the Celery task?

It sounds to me like a very expensive operation

to pickle and unpickle each time...

cyberjoac, you can put the data in Redis and do not transfer it each time, but teach your task to get the data from redis

Yes but it means that each time I need to make a query on my data structure

I need to fetch it from Redis

and therefore deserialize it

isn't it a very expensive operation?

cyberjoac, it is, but you're working with a huge data

cyberjoac, you can write a django view and query the data via api

or controller

or whatever it named

In order to query the data, it has to be in memory

I cannot query the data from Redis

it's not a standard data structure

for the sake of precision, it's a KD-Tree

If you write an api to this structure using django, you will have access to the data via api

but not from Redis

from the memory, correct?

uh

yes

from the memory

You will not have a direct access to the memory, only via api

But it doesn't solve the problem of celery accessing a different memory space

cyberjoac, it solves - just use api you wrote via http request

you're saying that the celery task

you send query parameters, django queries the data in memory and returns the result

doesn't use the data structure

ok this solves querying the data structure

but it doesn't solve constructing it with Celery

My task in celery is supposed to reconstruct this data structure every once in a while

cyberjoac, why?

Can't you just use the api?

From celery?

yes

That's interesting

It's kind of a "Proxy" design pattern?

it's kind of... "real world" design pattern

that's how things work

you have a service with api which works with in-memory data, and you can access the data via api

Then how do I call the API from inside a celery task?

I'm so sorry for all those noob questions...

You're helping me tons here!!

:) You said you have a django app - django can expose it's api via http requests

well, that what django was built for

But can the Celery task send an Http request?

of course - celery task is just an arbitrary python code

you can do anything you want

So in resume

I put my task inside my Django code

false

:)

I mean inside a tasks.py code

You should write an api inside your django code

that is inside one of the django apps

tasks.py can be anywhere

even on a different server

and then in the @app.task I just call an HttpResponseRedirect

no actually inside the @app.task I would need to send an Http Request

to the URL that reconstructs the data structure

yes, something like that

but then it's not asynchronous anymore is it?

When was that asynchronous?

You call the api - you wait for the result

That was the whole purpose of doing it inside the task in the first place

to have a long CPU-bound request done asynchronously

constructing this data-structure could take 30 seconds

cyberjoac, you can construct the structure once on the django side, and each http request will work with already constructed data

yes but, I need to "reconstruct" the structure every once in a while

this is why I looked at Celery on the first place

I'm not sure i understand the purpose of reconstructing 2GB data on each call

not each call

but each minute

Sounds like you have a design problem

or each hour

So

Django can do the job

right?

You expose the functionality via api

celery should not know anything about what's going on with the data

But once again, I lose asynchronousness..

am I wrong?

cyberjoac, you never got it

you never had it*

i guess that is correct one

that's right but I'm trying to have it :-)

Anyway that still sounds like a much better solution

Does it sound reasonable to call via API once every minute through a Celery task a reconstruct of the Data structure?

won't the Django server "blocK" until it finishes constructing the data structure?