#celery

asksol: ping

or anyone?

stumbling over https://github.com/celery/django-celery/issues/183

installing the 3.0 branch doesn't fix the problem

jezdez: are you using django < 1.4?

no

honestly I don't know how this can happen

hmm, maybe_make_aware sets it only if tzinfo is not set, right?

could be a race condition, though didn't think that could happen here

btw, why are you not using flower? :)

the timer thing is running in a thread, so could be a race condition I guess

or, no the value is not shared there

unless task.eta already is a datetime and something else is changing it

no idea, the date is always the same

how could that be the same date all the time?

how can I delete it?

so this only happens for a specific date?

yes, continuously spamming our logs

you'd have to go into the database

and do what?

delete the row

right, but what table?

djcelery_taskstate ?

may be another _

I have to think about it, maybe two threads are changing the object at the same time

you should look at flower though, it will replace the admin monitor thingy

dude, we have millions of tasks in that table

sigh

flower never came up since django-celery used to work

yeah, celerycam can quickly put strain to your database

it should clean it up once a day or so, but deleting things from a database is a challenge

flower doesn't yet persist the data though. so everything is lost if you restart it

I wish it would at least put a fucking task id in the log

the error message?

yeah

not too much to ask, right?

not sure how that would work, guess it could use some logging magic

btw, your traceback is different from the issue?

it would be nice if the django warning included the name of the field ;)

btw, could it be that these are old values created before an upgrade?

it says 2013-02-15, which seems a bit off

maybe an outer space timezone ;)

I have no fucking idea

this is ridiculous

which field is that even?

eta or tstamp?

tstamp is the field that states when the task was received, right?

I don't know, it could be eta, expires, tstamp

eta when it's supposed to be sent to the worker?

ah, eta, that is maybe it, or expires

considering it's in the future

timestamp is the time of the last event

related to this task

could just have update_task make sure all of the datetimes have timezones

asksol: couldn't parse your last sentence?

update_task saves the model

it could fix all datetime fields

how can I make sure all datetimes have timezones?

I'm only using celery stuff

where with stuff I mean the crontab thingie

that should work, I'm not spending a lot of time on issues for this since flower is a much better solution

btw, adding new information to exceptions is hard in py 2 :(

that didn't do the trick

then I guess we have to find out what field it is

either there's an unexpected date field, or maybe_make_aware did not return an aware datetime

you can wrap the exception in a new exception but then you lose the stack, or you can re-raise the exception with a new value, but then the exception class may not support the first argument being a string. So adding the task_id can only safely be added to the exception in Python 3 (which keeps a list of stack traces instead of just one)

try adding if defaults['eta']: assert defaults['eta'].tzinfo

same with expires and tstamp

or just add a assert value.tzinfo around maybe_make_aware itself

yeah, there are additional datetime fields for this model

no additional

I don't get it, when I run celerycam outside of supervisor it works

but within it doesn't

argh

so it only happens as a daemon?

I just run celerycam

supervisor handles the rest

at least it should

seriously wtf.

daemonization can cause a lot of side effects though

please find out if maybe_make_aware actually sets the tzinfo

django does set up some locale and default timezone and things still, right?

that could be affected by daemonization maybe

hey, the make_aware implementation, it only does something if USE_TZ is False?

no, is true

ok, here's some new information

we started to schedule tasks with apply_sync's countdown parameter

with exactly 7 days

it uses that 7 days in seconds to calculate the eta

I think when getting the self.app.now() from django-celery it returns a naive datetime

yeah, celery uses naive datetimes internally and expects them to be utc (if CELERY_ENABLE_UTC is on, otherwise it assumes local tz)

could it be that the same eta is later used to be written to the taskstate?

the eta here is received by the worker and converted to local timezone

the worker also sends the event received by celerycam

but that shouldn't really matter since maybe_make_aware should set the tzinfo

technically I think the datatime is aware (with UTC timezone) when sent in the event, but it's converted to iso8601 format first, and I don't think the timezone is reconstructed on the other side. But still, maybe_make_aware should set the tzinfo, it could be the wrong timezone but the value returned from maybe_make_aware shouldn't be naive

(that is unless USE_TZ=False)

make_aware does nothing if USE_TZ=False

I think you wrote that function

but if USE_TZ=False I guess the model wouldn't complain about naive datetimes either

timezones must be like the hardest computer problem ever :)

so to debug this we need to make sure that maybe_make_aware sets the .tzinfo

settings.USE_TZ is True

ok

mher: do you have any plans adding persistence support to Flower?

it could just dump the state to a shelve or something at exit

and possibly sync every n minutes

asksol: I moved the definition of eta and expires out of the dictionary declaration and it works

maybe I'm just going crazy

in that case it's probably a problem with maybe_make_aware