#twisted

Tickets pending review: https://tm.tl/#8966 (the0id), #9052, #9086, #9100, #9101, #4964 (jameshilliard), #9131, #5739, #9138, #9143

Tickets pending review: https://tm.tl/#8966 (the0id), #9052, #9086, #9100, #9101, #4964 (jameshilliard), #9131, #5739, #9138, #9143

Hello, I'm (still) struggling with adbapi + sqlite3. In short: I'm able to create tables and successfully write entries, but I'm unable to retrieve said entries. If it's a concurrency issue, I don't see where it is... Could someone please have a look at my question on SO? Thanks! https://stackoverflow.com/q/44160375/1156707

blz: what is the output from the trial test run?

blz: and what's TestCase? is that twisted.trial.unittest.TestCase?

blz: i can't run that, dunno what mk_dummy_inode should be

blz: okay, i hacked together an implementation for it, and fixed the other nameerrors and stuff, and it passes for me

blz: what version of twisted, what python version?

blz: i did it on python 2.7.9 and the latest twisted

blz: https://gist.github.com/cdunklau/b8999bdbbbe15b...

blz: also works on python 3.4.2

cdunklau: sorry, I was afk (lunch)

blz: no worries

cdunklau: I'm on python 5.3.3 running twisted 17.1.0

cdunklau: and lastly here's an implementeation for inode: https://hastebin.com/aqayimuqag.py I can't believe I forgot to include it ;/

https://paste.pound-python.org/show/y0m1QlJWR27... (repasted for blz)

blz: is that a pypy version? :D

or did you mean 3.5.3

cdunklau: oh whoops. Yes, 3.5.3!

blz: what's dumps

json?

cdunklau: I'm going to try to reproduce with your gist because I've in my attempts to fix it, I've realized that commenting out certain tests causes others to function ... which is a bit odd...

cdunklau: pickle

aaaaah noooo

cdunklau: man I did a bang-up job with my examples. Sorry about that!

heh

cdunklau: pickle vs json shouldn't change much, though, should it?

blz: not for this test, but storing pickles is a terrifying thought

cdunklau: gawd yeah, I know

cdunklau: It shouldn't matter too much insofar as this is just being locally stored. It's not travelling down the wire or anything, but it's still ugly as sin

blz: security isn't the only problem with pickle

cdunklau: what did you have in mind? I'm in a position where I can definitely convince the bossman to change it, and it sounds like there's good reason to do so

store data, not code

blz: use a reasonable serialization format. json, xml, protobuf, capnproto, etc

Right, but what's the immediate risk/problem for an application that's exclusively reading from the db in questino?

blz: you will load old code

Yeah, if it had been me, I would have definitely gone with json (or something similar)

cdunklau: and? I'm not trying to be contrarian -- it's just that I need to express the problem in a clear an articulable manner if I'm going to convinve anyone in the office.

blz: pickle gets really hairy if you try to unpickle an object of which the implementation has changed since you pickled it

Their initial reaction will be "it works right now"\

Ah okay, like changing python versions and discovering that the dict implementation changed?

and python can change the pickle protocol

blz: more with custom types

Hmm okay. Good to know, thanks :)

blz: http://www.benfrederickson.com/dont-pickle-your...

blz: also, it's python-only. all of the other methods i mentioned are supported in $LANGUAGE

right, of course

storing pickles means you _must_ use python to read them

blz: pickle *may* be okay for data you don't care about throwing away. like a cache or something

but if you *need* to successfully read the data at some future point, pickle is the wrong choice

okay

cdunklau: https://hastebin.com/lofehusaje.txt Any idea what's up?

https://paste.pound-python.org/show/8BHUZSFfvtK... (repasted for blz)

blz: um...

oh. trial test_sqlite

i noticed that too when i tried on python 3

aah bitten by py3 again ...

didn't get that error though. i got this https://bpaste.net/show/0ba5858cd835

blz: looks like you could do trial ./test_sqlite.py

God this is enraging. My test is line-for-line the same as the one in your gist, and I'm still getting https://hastebin.com/loguposiga.txt

https://paste.pound-python.org/show/473e4l9fZZK... (repasted for blz)

cdunklau: I'm sorry to do this, but I could really use a fresh set of eyes. Do you see anything different/wrong in the actual (i.e.: non-example) test code? https://hastebin.com/ohunidutah.py

https://paste.pound-python.org/show/qMslZ9v9A0i... (repasted for blz)

Other than the import paths, it should be strictly identical...

blz: can you pastebin the test?

the whole file

sure -- would it be easier if I pointed you to a github repo? It's an open-source project, and I can show you the relevant files

cdunklau: https://github.com/pydio/pydio-sync-tx/blob/dev...

You should just be able to clone & checkout the dev branch, then `python setup.py develop`, then `trial pydio`.

blz: hmmm.... take out the `del self.d`

cdunklau: done, no change.

blz: and when you comment out the rmtree(self.meta) and actually check the db manually after the test fails, the row is there, right?

Yes. The way I checked previously was to set a breakpoint right before the runQuery statement and check if the row exists, but I'll check it your way as well.

gimme a sec

cdunklau: yeah, it's there...

Strange, eh?

blz: i wonder... here https://github.com/pydio/pydio-sync-tx/blob/dev...

the empty line?

blz: put in `from twisted.internet import task, reactor` and then `yield task.deferLater(reactor, 2, lambda: None)`

so you wait 2 seconds before doing the query

hmm yeah worth a shot

cdunklau: well f--k me running...

blz: okay now cut the time down until it fails

cdunklau: that shouldn't happen, right? i.e.: that's a bug, no?

blz: i *think* so

cdunklau: the threshold is in the neighborhood of .000009 (seconds, right?)

yes, the second arg to deferLater is seconds

Hmm okay, well the short term fix is to put a sleep, then :)

blz: well... use the deferLater thing so you don't block the event loop[

That's the good news. The bad news -- I guess -- is that there seems to be a bug in adbapi.ConnectionPool

Yeah, "sleep", not _sleep_ ;)

In any case, thanks a lot. I was seriously losing hair over this one ...

Should I open an issue or something?

blz: well, first cut it down to as minimum of a reproduction as possible

blz: i have a feeling this will be difficult to reproduce

Hmm true... as evidenced by the fact that your attempt at running my stackoverflow code worked.

that's what i mean

If you're interested in teh SO bounty by the way, feel free to put an answer up.

blz: adbapi is really quite simple

blz: nah

Yeah I read the spec and it seems straightforward

blz: it's basically just a wrapper around the dbapi driver that dumps stuff in a thread

blz: is your project polyglot?

blz: i wonder if this happens on python 2 as well

cdunklau: I expect it should run on python2. I can try.

blz: also a detailed log of what sqlite sees happening might help you pinpoint it

i suspect that the transaction commit isn't actually complete by the time you do the select

Yeah, but that's still very odd, since I'm waiting for the deferred to fire

btw I just tried running under py2 and our earlier discussion about pickle was ... prescient...

sure but maybe sqlite does stuff after it says "transaction ok'

or whatever

Yeah maybe there's some dead space between when sqlite says ok and the FS flushes its buffers?

right

i.e. sqlite will stil write in-order, but you can get a dirty read

but still... that would be a pretty major bug in sqlite, no?

Oh but wait...

adbapi manages a connection pool

blz: yes