#appengine

/

      • Lenolium
        plantedseed: If your Player models are at the top level, each player can have one update per second.
      • But the other side of that is that any update made without an ancestor takes time to propagate throughout the system.
      • kingarmadillo joined the channel
      • buffal0 joined the channel
      • buffal0 has quit
      • buffal0 joined the channel
      • buffffal00 joined the channel
      • buffal0 has quit
      • buffffal00 has quit
      • buffal0 joined the channel
      • plantedseed
        Lenolium: I get the error BadRequestError: queries inside transactions must have ancestors
      • are transactions pointless without ancestors? i dont get it
      • Strom
        don't use queries in transactions
      • do only key based lookups
      • build your own indexes to keep track of whatever you want
      • if necessary run cron jobs that will do a query and then save that query's result to some entities that you can fetch by key in a transaction
      • but in most cases you don't need the query job and can design the data model in a way that you can just chain key based lookups
      • plantedseed
        i dont get it. I signup, that means checking if the username and email is taken, if not fine => insert else return error to the user
      • Strom
        yeah, for checking that I suggest lock entities for username & email
      • so an entity UsernameLock, that would use the username as key, and also have a nother property which contains the user's key
      • then you can fetch that userlock entity by key (which is the username) to check in a consistent way whether the username has been takend or not
      • plantedseed
        so I need to query and insert in a transaction s no other player comes between and also checks, sees that it is fine at the same time and then i get two inserts of the same username
      • so i create a parent?
      • Strom
        no, why do you need to query? don't query
      • plantedseed
        how can i not query?
      • Strom
        what is that you want to do? check for if username is in use?
      • plantedseed
        yes and email
      • Strom
        I suggest lock entities for username & email
      • so an entity UsernameLock, that would use the username as key, and also have a nother property which contains the user's key
      • then you can fetch that userlock entity by key (which is the username) to check in a consistent way whether the username has been takend or not
      • plantedseed
        Problem 1: username and email must be unique Problem 2: Save progress, email, etc. Saving means checking what level the user is at in the daatabase and only store if higher. Then I have to query right?
      • Strom: but is that a real lock? cant someone other user do the same at the same time and both see that the username is free and insert?
      • buffal0 has quit
      • Strom
        that's what transactions are efor
      • you do it all inside a single transaction
      • 1) check for UsernameLock by key (which is the username) -- if it doesn't exist, then the username hasn't been taken and you can proceed
      • 2) creat a new user with auto-generated numeric id
      • 3) save a UsernameLock entity, with using the username as the key, and also putting the auto-generated numeric id as a property
      • if two users start trying to do it at the same time, then the transaction will succeed only for one of them
      • the other user will get an error -- the transaction will first fail, and then the automatic retry will see that the UsernameLock is in place and the username has been taken
      • -- and for your saving / user level check, you can do that in a transaction without queries as well
      • plantedseed
        ok thanks. so i need 2 parents? UserNameLock and EmailLock?
      • Strom
        you can fetch the entity by key that contains the level information
      • parents? no don't use the ancestor system at all really
      • plantedseed
        will this means that "If your Player models are at the top level, each player can have one update per second." will not be true anymore?
      • Strom
        it will
      • it will be ture, to be clear
      • *true
      • plantedseed
        OK, so if I keep updates two one particular entity with a minimum of 10 seconds then i should never run into problems with transactions failing?
      • two->to
      • Strom: OK i am not sure still. UsernameLock would be a class UsernameLock(ndb.model) and there would be one for each username? why not make the username the key for the Player model then? WOuld the Player and the UsernameLock reference each other somehow?
      • I need a model with 2 keys? is that possible? Or I need the Player model/entity type with username as Key and then i need another model with email as a Key?
      • plantedseed has quit
      • kloff joined the channel
      • kingarmadillo joined the channel
      • kingarmadillo has quit
      • kloff has quit
      • plantedseed joined the channel
      • telex has quit
      • telex joined the channel
      • [1]slugmandrew joined the channel
      • slugmandrew has quit
      • [1]slugmandrew is now known as slugmandrew
      • kingarmadillo joined the channel
      • kingarmadillo has quit
      • plantedseed has quit
      • kingarmadillo joined the channel
      • kingarmadillo has quit
      • kingarmadillo joined the channel
      • aatish910 joined the channel
      • aatish910
        When will memcache be available on app engine flexible?
      • aatish910 has quit
      • kingarmadillo joined the channel
      • Method__ has quit
      • kingarmadillo joined the channel
      • kingarmadillo joined the channel
      • kingarmadillo has quit
      • kingarmadillo joined the channel
      • plantedseed joined the channel
      • plantedseed
        Strom
      • Can someone please helpt me with those questions+
      • plantedseed has quit
      • plantedseed joined the channel
      • troberti
        plantedseed: you can use cross-entity group transactions ("xg" in ndb )
      • this allows you to use a transaction across more entities groups that do not share a common ancestor
      • xg-transactions are very useful for rare situations that would otherwise cause you to change your datamodel in weird ways
      • so you just get start a xg-transactions, get the User entity, get the Email entity and then do your thing
      • plantedseed
        ty. so user and email will be different entities, not properties?
      • right now i have them in the same entity/model
      • troberti
        Yes think you want to have them different entities
      • and of course they need to refer to each other through a property
      • plantedseed
        and how does this affect the possibility to query often? i mean with a million users there will be quiet a lot of signups and logins
      • troberti
        what do you need to query for?
      • plantedseed
        the user needs to be able to update his progress every 5-10 seconds (that wont happen very often but in the worst case)
      • not query but use the database
      • as in "check value , write"
      • troberti
        datastore performance will not be your issue :)
      • as long as you write to different entity gorups
      • buffal0 joined the channel
      • datastore has excellent 'horizontal' scaling.. that is you should model your data so that each user's data is in its own entity group
      • buffffal00 joined the channel
      • plantedseed
        ok, so how many entities/models do i need to describe a Player which has the properties: username and email which are unique and then level, country and some other misc fields
      • wont a cross-group transaction affect the write contention for all entities?
      • troberti
        only those in the transaction
      • the entity groups used in the transaction
      • I can't make your app for you.. i would just start making it to get experience
      • plantedseed
        but if i get 1 million users that 1 million entity groups right? the limit is at 25...
      • troberti
        yes
      • you can refer to max 25 entity groups in an xg transaction.. but you only need to access 2
      • your User/Player entity and the "Email" entity
      • Player entity has the username as its keyname, Email entity has the email address as keyname..
      • and Email entity of course needs a property that stores the associated username and viceversa
      • I don't think I can explain it any better, sorry
      • bbl going for coffee
      • plantedseed
        but each entity is in its own group isnt it? if it does not hve an ancestor?
      • if they have ancestors then all the sudden you can only make 1 update per second to the dataset
      • greg_f joined the channel
      • you are saying the XG transaction applies only to the WRITE part? looking up by KEY for all entities is not included in the 25 entity groups?
      • buffffal00 has quit
      • Strom
        it is included in the 25, if it's inside the transaction
      • smw has quit
      • buffal0 joined the channel
      • plantedseed
        OK, im still confused by all this.
      • kingarmadillo joined the channel
      • User 1 signs up as "user1" and "[email protected]";, is it not possible to prevent anyone else from signing up with the same username at the same time without a huge penalty that will not work?
      • smw joined the channel
      • Strom
        it is possible
      • I gave you the transaction actins step-by-step yesterday
      • buffffal00 joined the channel
      • plantedseed
        yes thanks but i dont really get it.
      • Strom
      • what don't you get?
      • plantedseed
        why would i have a separate UsernameLock, why not just have the username as key in the real User/player model? And then I need a EmailLock for the email?
      • buffal0 has quit
      • For one user, what do I need and how would the transactions look? 2 or 3 models? Main model/entity with data, 2 lock entities?
      • Strom
        1) you could use the username as a key for the actual User entity, but then it would be difficult to let users change their username
      • plantedseed
        and those entities, there will be more than 25 users then there are more than 25 XG and I cannot do a transaction over ALL users, then it falls...
      • Strom
        2) you would still need a separate entity for other things you want to ensure are unique as the e-mail
      • plantedseed
        Usernames will not change
      • email can be changed
      • Strom
        why do you need to do a transaction over all the users?
      • (it's possible, but not performant)
      • plantedseed
        how else would i ensure noone tries the same username?
      • Strom
        did you look at the link I posted?
      • just to be clear, let me paste it out one more time
      • 1) check for UsernameLock by key (which is the username) -- if it doesn't exist, then the username hasn't been taken and you can proceed
      • plantedseed
        isnt that what the transaction is for: 1. Lock database. 2. Check if username exists. 3. Insert if not existing.
      • so noone can come in between point 2 and 3 and think hey username does not exist lets insert and then you get 2 identical usernames