Google I/O 2008 - App Engine Datastore Under the Covers
Why Stack Overflow recommends it
Stack Overflow users link it when they need to understand Datastore internals, indexes and the design limits that come from scaling on a Bigtable-style store.
Recommended by 8 people across 10 different months, 2010 to 2018.
What it is
A Google I/O 2008 talk by Google engineer Ryan Barrett explaining how the App Engine Datastore works internally. It covers why joins are missing, why queries hit deadlines, how indexes are chosen, and why both parent/child relationships and reference properties exist.
What people said
Strong rec Jason Hall · 2010-04 · Stack Overflow · on “Schema-less design guidelines for Google App Engine Datastore and other NoSQL DBs”
Don't duplicate the properties if they'll always be the same between the SearchResult and a Search . If a SearchResult should have a reference to a Search , keep a ReferenceProperty [LINK] pointing to the Search. This basically stores the related Search 's Key in the model. class SearchResult(db.Model): search = db.ReferenceProperty(Search, required=True) # other stuff... I also highly recommend you watch some of the App Engine videos from last year's Google I/O [LINK] (and from 2008 [LINK] ), in particular this one [LINK] by Brett Slatkin, and this one [this video] by Ryan Barrett. They're all pretty helpful videos if you have the time, but I found those two in particular to be really great.
Recommends Xion · 2011-09 · Stack Overflow · on “What is a good resource for understanding GAE datastore architecture?”
You could start with presentation from Google I/O: App Engine Datastore Under The Covers [this video] The presentation on Appstats [LINK] also gives some insight into how datastore (and RPC calls in general) work, as well as giving some optimization tips.
Recommends johngoche9999 · 2012-01 · Stack Overflow · on “JDO and Google App Engine datastore: key representation in code”
I have just finished watching the following videos in an attempt to understand JDO and Google App Engine datastore: [LINK] [LINK] [LINK] [this video] Now I wonder, take the example where we have an entity of kind Grandparent having an entity of kind Parent having an entity of kind Child as in one of the videos. The key for one of the the Child entities could be: Grandparent:Jane/Parent:Jack/Child:Joe How do I code the class for this instance in JDO (presumably there will be three classes) but I would like to see an implementation where we can see the key values set as part of the key explicitly. Any ideas? I also wonder, what is the difference between using JDOQL to access my data and iterating t
Recommends Shay Erlichmen · 2012-07 · Stack Overflow · on “In Google App Engine/J, what is the maximum row count in a table?”
If your referring to the datastore, for all practical reasons the only limit is your budget, the more data you store you will pay more for storage. In regard for query speed, the major overhead in query is the amount of items that query fetches, it takes almost the same time to retrive X items from 1,000 entities set and from 100,000,000. There is a very interesting video from Google IO 2008 [this video] about the datastore (the indexes part is still relevant)
Recommends stun · 2013-04 · Stack Overflow · on “How to define Key Name in Datastore GAE Python?”
Quick Advice Working on project, which need to have input form to extend its content(name,email,phone..) with defined Key Name in Datastore, preferably to increase in define way . Took me like a minute or two trying to understand what you are saying. Don't rush typing out what you want to ask without thinking. If you want a well thought out answer from other people, try to ask the question clearly. My Questions to You Why do you think you need to define the Key Name manually ? What benefit do you think you are getting? From what I can infer, you seem like you don't understand the basics of the GAE HRD Datastore. If you don't understand the HRD datastore, you can't code effectively on GAE. My
Recommends stun · 2013-04 · Stack Overflow · on “Improving database record retrieval throughput with appengine”
I will second what @illia-frenkel said above. Maybe consider designing your application to include using `memcache` and being cacheable? Since you're not providing any more info, I'd advise watching **App Engine Datastore Under the Covers** [this video] and **Building Scalable Web Apps with App Engine** [LINK] from Google IO 2008. Also read Jeff Dean's [LINK]
Recommends Nick · 2014-12 · Stack Overflow · on “Do the unused properties consume size in gae datastore?”
If I understand your question correctly, you are envisioning a system where you have: A Kind in your Datastore where the Entities for that Kind can have differing subsets of a common property-key space W . Entity 1's property set might be {W[0], W[1]} , and Entity 2's property set might be {W[1], W[2], W[5]} . You want to know whether this polymorphism (or "schemalessness") will cost you space, and whether each Entity, as in some naive MySQL implementations The short answer is no - due to the schemaless nature of Datastore, having polymorphic entities in a kind (the entities have all different names and combinations of properties) will not consume extra space. The only way to have these "unu
Recommends Nick · 2014-12 · Stack Overflow · on “How to load seed data into app engine's datastore using golang?”
So, to begin, I'll gently remind you that there are no "tables" in Datastore. Thinking using the terms of the RDBMS world will only confuse you. Please take the time to really understand the underlying storage mechanisms and data structures. I'd recommend this video for an in-depth look at what Datastore actually is. [this video] So to get to your actual use-case, "application restart" is also something of a tricky term to use in relation to app engine. I'd recommend getting familiar with the actual infrastructure that runs your app - when instances of what scaling type are turned on and off, how they can communicate, etc... [LINK] The best way, apart from these nitpicks, to have the list of word
Recommends Nick · 2015-01 · Stack Overflow · on “How do I retrieve representation metadata for properties in the datastore?”
I just created a sample project to reproduce your issue. JDO is saving the entities in the Datastore, but not naming the primary key property according to the member variables of the PersistenceCapable class. Check your datastore viewer and grab a random Container entity to see what's happening. If you need to do metadata queries on the id property in this manner, you should be aware that the ID/Name property is part of the key. Check this interesting talk [this video] to learn more about how the Datastore internals work (it's based on BigTable).
Recommends Patrice · 2015-02 · Stack Overflow · on “How to build datastore indexes (PHP GAE)”
As Alex Martelli mentioned in a comment, most of the time, your indexes are built when you run your app on the devserver and have your datastore get queried there (this adds the required indexes for any question into your index.yaml file. So you have two ways you can go at it. 1- Run your app on local devserver, go to your dev " Developer Console [LINK] " to add one or two entities to your datastore. Run your queries, that'll populate your index.yaml with all required indexes. You can then run appcfg.py update_indexes to just deploy your index.yaml (bottom of this page [LINK] ) 2- Your other solution would be to read this [LINK] , a page on how datastore indexes work. Then read this [LINK] a