Bugzilla – Full Text Bug Listing
| Summary: | [Mongo] Create a MongoDBStore | ||
|---|---|---|---|
| Product: | [Modeling] EMF | Reporter: | Eike Stepper <stepper> |
| Component: | cdo.core | Assignee: | Eike Stepper <stepper> |
| Status: | CLOSED FIXED | QA Contact: | Eike Stepper <stepper> |
| Severity: | enhancement | ||
| Priority: | P3 | CC: | esp1, robert.blust, venkata.krishnamoorthy, vroldanbet |
| Version: | 4.0 | Keywords: | noteworthy |
| Target Milestone: | --- | Flags: | stepper:
indigo+
|
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | Appealing to a Broader Community | ||
|
Description
Eike Stepper
Keeping in cdo.core for now... After setting up a MongoDB server, creating driver/source bundles and a cdo.server.mongodb plugin skeleton I started again to study the MongoDB docs. My current main concern is that MongoDB does not know the concept of a transaction. The CDO repository framework used to assume that the backend technology already provides a mechanism to: 1) apply multiple changes either altogether or none of them, 2) ensure that readers do not see the results of partial change sets CDO already ensures on framework level that multiple concurrent writers do not write to the same objects in the same branch. But the framework does not implicitely acquire read locks on read operations. That'll be quite hard to tackle... > 1) apply multiple changes either altogether or none of them, No trivial solution if server crashes have to be considered. Since crash recovery is marked "nice to have", I'll first go for an in-memory commit/rollback protocol. > 2) ensure that readers do not see the results of partial change sets This first iteration will introduce the concept of pluggable isolation level and implement a simple strategy with a single R/W lock per repo. According to http://www.mongodb.org/display/DOCS/Atomic+Operations MongoDB supports atomic operations on single documents. MongoDB does not support traditional locking and complex transactions. In the DBStore the CDOCommitInfos table only contains some "meta" information like userID or commit comment. The real change set data is worked into the various tables mapped from the model. This is possible because most RDBMS support proper transactions. In MongoDB that's not easily possible. What about the following design: If atomicity is only supported for single documents we should probably see that a CDO commit creates just a single document, a CDOCommitInfo document and adds it to a system collection. This is possible because, when the commit info arrives at the store layer, CDO has already checked for object-level write conflicts, including containment cycles and optional xref checks. Writes would immediately have ACID properties. Working the change set data into the mapped collections could be done asynchronously. Crash recovery would get way easier and it seems that this approach brings much higher concurrency because it wouldn't need a single R/W lock per repo. Committed revision 7094: - trunk/plugins/org.eclipse.emf.cdo.tests - trunk/plugins/org.eclipse.emf.cdo.tests.db - trunk/plugins/org.eclipse.emf.cdo.tests.db4o Committed revision 7095: Committed revision 7099: MongoDB Thread-Safety Questions ============================ What is a "connection" in MongoDB Java, com.mongodb.Mongo or com.mongodb.DB? Can multiple threads safely share a single com.mongodb.Mongo instance, com.mongodb.DB instance respectively? I've read somewhere that certain drivers support connection pooling (whatever that connection really is). Does the Java driver support this pooling and what does that mean for the responsibilities of MongoDBStore and MongoDBStoreAccessor? Example: Responsibilities in DBStore ========================== A DBStore instance does not maintain a JDBC connection, but only an IDBConnectionProvider (e.g. DataSource). Each DBStoreAccessor maintains its own JDBC connection and associates it with the CDO context (CDOSession or CDOTransaction). Because of the JDBC connection DBStoreAccessor instances are expensive to create they are pooled within their DBStore. Ok, many questions are answered in http://www.mongodb.org/display/DOCS/Java+Driver+Concurrency : The Java MongoDB driver is thread safe. If you are using in a web serving environment, for example, you should create a single Mongo instance, and you can use it in every request. The Mongo object maintains an internal pool of connections to the database (default pool size of 10). [...] DB and DBCollection are completely thread safe. That would mean MongoDBStore can maintain a single MongoDB/DB instance pair and share it with all MongoDBStoreAccessors. What about the lifecycle of the MongoDB/DB instance pair? Should it be controlled completely outside of the MongoDBStore (i.e. we'd just pass in a DB instance before store activation)? Currently I prefer to pass in a Mongo-URI/DB-name pair and let the store open/close the connection in doActivate/doDeactivate... The above mentioned page also says something about "complete consistency" and "write heavy environments". I must admit that it's an entire mystery to me what that's supposed to mean. Can somebody understand that and explain it to me? Committed revision 7100: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb Committed revision 7101: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7111 Committed revision 7112: Committed revision 7113: Committed revision 7116: org.eclipse.emf.cdo.server org.eclipse.emf.cdo.server.db org.eclipse.emf.cdo.server.db4o org.eclipse.emf.cdo.server.hibernate org.eclipse.emf.cdo.server.mongodb org.eclipse.emf.cdo.server.objectivity org.eclipse.emf.cdo.tests.mongodb Committed revision 7117: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.db - trunk/plugins/org.eclipse.emf.cdo.server.db4o - trunk/plugins/org.eclipse.emf.cdo.server.hibernate - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7118 This commit includes a basic MongoDBBrowserPage which is registered early in the tests: http://localhost:7778 Committed revision 7119: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.net4j.util Committed revision 7120: - trunk/plugins/org.eclipse.emf.cdo.server.db - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7121: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7122: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7127: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb Committed revision 7128: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7129 Committed revision 7130 Committed revision 7131: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb The more I learn about MongoDB the clearer our strategy becomes: MongoDB's atomicity constraints directly implies that a logical CDO commit must be persisted with a single new document. My original idea to store the change set deltas in this commit document and update per-class collections later (possibly in a background tread) turned out to be quite complex and functionally sub-optimal. It would still require costly and complex crash detection/recovery. Now I learned that MongoDB can easily index and query fields of embedded objects. Hence we will no longer store change set deltas in the commit document, but hte entire resulting state, i.e. CDORevisions for the new, changed and detached objects. With this design the database will only have a single collection for the revisions of all classes. No need for additional mapping to per-class collections, no concurrency issues, no distribution issues, no threading issues ;-) Committed revision 7132: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.hibernate - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.server.objectivity - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb I've added a BidiMap<EClassifier, Integer> to reduce persistent footprint and costly URI lookups. Committed revision 7133 Committed revision 7134: Committed revision 7135: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7136 Committed revision 7137 Committed revision 7138 Committed revision 7139 Committed revision 7140: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests Committed revision 7141: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb Committed revision 7142: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7143: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7144: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests Committed revision 7145: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7146: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7147: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7148 Committed revision 7149: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests Committed revision 7150: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests Committed revision 7151: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests Committed revision 7152: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.db - trunk/plugins/org.eclipse.emf.cdo.server.db4o - trunk/plugins/org.eclipse.emf.cdo.server.hibernate - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.server.objectivity - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb Committed revision 7153: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7154 Committed revision 7155: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7156: - trunk/plugins/org.eclipse.emf.cdo.tests - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb - trunk/plugins/org.eclipse.emf.cdo.workspace Committed revision 7157: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7158: - trunk/plugins/org.eclipse.emf.cdo.server - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7159: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7160: - trunk/org.eclipse.emf.cdo.server.mongodb - trunk/org.eclipse.emf.cdo.tests - trunk/org.eclipse.emf.cdo.tests.mongodb Committed revision 7162: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7163: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb Committed revision 7164: - trunk/plugins/org.eclipse.emf.cdo - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb *** All non-auditing tests are passing now (1102 cases) *** Committed revision 7165 Committed revision 7167: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb Committed revision 7168: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb Committed revision 7169: - trunk/features/org.eclipse.emf.cdo.server.mongodb-feature - trunk/features/org.eclipse.emf.cdo.site-feature - trunk/features/org.eclipse.emf.cdo.tests-feature - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests - trunk/plugins/org.eclipse.net4j.tests - trunk/releng/org.eclipse.emf.cdo.releng Committed revision 7172: - trunk/features/org.eclipse.emf.cdo.server.product - trunk/plugins/org.eclipse.emf.cdo.server.db - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.server Updated the wiki: http://wiki.eclipse.org/CDO/Server_Configuration_Reference#Element_store http://wiki.eclipse.org/CDO/MongoDB_Store Resolving... The MongoDB driver bundle is available at, e.g., http://net4j.sourceforge.net/update/ Committed revision 7591: Committed revision 7595: - trunk/features/org.eclipse.emf.cdo.sdk-feature - trunk/features/org.eclipse.emf.cdo.server.mongodb-feature - trunk/releng/org.eclipse.emf.cdo.releng The MongoDB driver is now part of the shipped SDK. Committed revision 7600: - trunk/plugins/org.eclipse.emf.cdo.server.mongodb - trunk/plugins/org.eclipse.emf.cdo.tests.mongodb Available in R20110608-1407 |