Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 337152 - [Mongo] Create a MongoDBStore
Summary: [Mongo] Create a MongoDBStore
Status: CLOSED FIXED
Alias: None
Product: EMF
Classification: Modeling
Component: cdo.core (show other bugs)
Version: 4.0   Edit
Hardware: All All
: P3 enhancement (vote)
Target Milestone: ---   Edit
Assignee: Eike Stepper CLA
QA Contact: Eike Stepper CLA
URL:
Whiteboard: Appealing to a Broader Community
Keywords: noteworthy
Depends on:
Blocks:
 
Reported: 2011-02-14 13:51 EST by Eike Stepper CLA
Modified: 2011-06-23 03:41 EDT (History)
4 users (show)

See Also:
stepper: indigo+

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Eike Stepper CLA 2011-02-14 13:51:24 EST 
Must have
=========

* Base functionality
* Flexibility is not a priority
* Long IDs are fine
* No need for connection pooling.
* Auditing
* Commit Infos


Nice to have
============

* Automatic crash repair
* Query Handler
* Branching (Lower priority than auditing, can be deferred)
Comment 1 Eike Stepper CLA 2011-02-14 13:52:03 EST 
Keeping in cdo.core for now...
Comment 2 Eike Stepper CLA 2011-02-15 05:17:59 EST 
After setting up a MongoDB server, creating driver/source bundles and a cdo.server.mongodb plugin skeleton I started again to study the MongoDB docs.

My current main concern is that MongoDB does not know the concept of a transaction. The CDO repository framework used to assume that the backend technology already provides a mechanism to:

1) apply multiple changes either altogether or none of them,
2) ensure that readers do not see the results of partial change sets

CDO already ensures on framework level that multiple concurrent writers do not write to the same objects in the same branch. But the framework does not implicitely acquire read locks on read operations.

That'll be quite hard to tackle...
Comment 3 Eike Stepper CLA 2011-02-15 23:17:56 EST 
> 1) apply multiple changes either altogether or none of them,

No trivial solution if server crashes have to be considered. Since crash recovery is marked "nice to have", I'll first go for an in-memory commit/rollback protocol.

> 2) ensure that readers do not see the results of partial change sets

This first iteration will introduce the concept of pluggable isolation level and implement a simple strategy with a single R/W lock per repo.
Comment 4 Eike Stepper CLA 2011-02-15 23:36:27 EST 
According to http://www.mongodb.org/display/DOCS/Atomic+Operations MongoDB supports atomic operations on single documents. MongoDB does not support traditional locking and complex transactions.

In the DBStore the CDOCommitInfos table only contains some "meta" information like userID or commit comment. The real change set data is worked into the various tables mapped from the model. This is possible because most RDBMS support proper transactions. In MongoDB that's not easily possible.

What about the following design:

If atomicity is only supported for single documents we should probably see that a CDO commit creates just a single document, a CDOCommitInfo document and adds it to a system collection. This is possible because, when the commit info arrives at the store layer, CDO has already checked for object-level write conflicts, including containment cycles and optional xref checks. Writes would immediately have ACID properties.

Working the change set data into the mapped collections could be done asynchronously. Crash recovery would get way easier and it seems that this approach brings much higher concurrency because it wouldn't need a single R/W lock per repo.
Comment 5 Eike Stepper CLA 2011-02-16 00:47:29 EST 
Committed revision 7094:
- trunk/plugins/org.eclipse.emf.cdo.tests
- trunk/plugins/org.eclipse.emf.cdo.tests.db
- trunk/plugins/org.eclipse.emf.cdo.tests.db4o
Comment 6 Eike Stepper CLA 2011-02-16 00:52:34 EST 
Committed revision 7095:
Comment 7 Eike Stepper CLA 2011-02-16 00:59:56 EST 
Committed revision 7099:
Comment 8 Eike Stepper CLA 2011-02-16 01:11:01 EST 
MongoDB Thread-Safety Questions
============================

What is a "connection" in MongoDB Java, com.mongodb.Mongo or com.mongodb.DB?

Can multiple threads safely share a single com.mongodb.Mongo instance, com.mongodb.DB instance respectively?

I've read somewhere that certain drivers support connection pooling (whatever that connection really is). Does the Java driver support this pooling and what does that mean for the responsibilities of MongoDBStore and MongoDBStoreAccessor?
Comment 9 Eike Stepper CLA 2011-02-16 01:15:59 EST 
Example: Responsibilities in DBStore
==========================

A DBStore instance does not maintain a JDBC connection, but only an IDBConnectionProvider (e.g. DataSource).
Each DBStoreAccessor maintains its own JDBC connection and associates it with the CDO context (CDOSession or CDOTransaction).
Because of the JDBC connection DBStoreAccessor instances are expensive to create they are pooled within their DBStore.
Comment 10 Eike Stepper CLA 2011-02-16 01:32:41 EST 
Ok, many questions are answered in http://www.mongodb.org/display/DOCS/Java+Driver+Concurrency :

The Java MongoDB driver is thread safe. If you are using in a web serving environment, for example, you should create a single Mongo instance, and you can use it in every request.  The Mongo object maintains an internal pool of connections to the database (default pool size of 10). [...] DB and DBCollection are completely thread safe.

That would mean MongoDBStore can maintain a single MongoDB/DB instance pair and share it with all MongoDBStoreAccessors.

What about the lifecycle of the MongoDB/DB instance pair? Should it be controlled completely outside of the MongoDBStore (i.e. we'd just pass in a DB instance before store activation)?

Currently I prefer to pass in a Mongo-URI/DB-name pair and let the store open/close the connection in doActivate/doDeactivate...

The above mentioned page also says something about "complete consistency" and "write heavy environments". I must admit that it's an entire mystery to me what that's supposed to mean. Can somebody understand that and explain it to me?
Comment 11 Eike Stepper CLA 2011-02-16 01:43:43 EST 
Committed revision 7100:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 12 Eike Stepper CLA 2011-02-16 01:58:07 EST 
Committed revision 7101:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 13 Eike Stepper CLA 2011-02-16 03:00:04 EST 
Committed revision 7111
Comment 14 Eike Stepper CLA 2011-02-16 03:03:10 EST 
Committed revision 7112:
Comment 15 Eike Stepper CLA 2011-02-16 03:04:02 EST 
Committed revision 7113:
Comment 16 Eike Stepper CLA 2011-02-16 04:41:00 EST 
Committed revision 7116:

org.eclipse.emf.cdo.server
org.eclipse.emf.cdo.server.db
org.eclipse.emf.cdo.server.db4o
org.eclipse.emf.cdo.server.hibernate
org.eclipse.emf.cdo.server.mongodb
org.eclipse.emf.cdo.server.objectivity
org.eclipse.emf.cdo.tests.mongodb
Comment 17 Eike Stepper CLA 2011-02-16 05:08:08 EST 
Committed revision 7117:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.db
- trunk/plugins/org.eclipse.emf.cdo.server.db4o
- trunk/plugins/org.eclipse.emf.cdo.server.hibernate
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 18 Eike Stepper CLA 2011-02-16 06:25:42 EST 
Committed revision 7118

This commit includes a basic MongoDBBrowserPage which is registered early in the tests: http://localhost:7778
Comment 19 Eike Stepper CLA 2011-02-16 11:00:33 EST 
Committed revision 7119:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.net4j.util
Comment 20 Eike Stepper CLA 2011-02-16 11:45:18 EST 
Committed revision 7120:
- trunk/plugins/org.eclipse.emf.cdo.server.db
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 21 Eike Stepper CLA 2011-02-16 11:47:06 EST 
Committed revision 7121:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 22 Eike Stepper CLA 2011-02-16 13:48:06 EST 
Committed revision 7122:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 23 Eike Stepper CLA 2011-02-17 00:48:32 EST 
Committed revision 7127:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 24 Eike Stepper CLA 2011-02-17 00:58:25 EST 
Committed revision 7128:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 25 Eike Stepper CLA 2011-02-17 02:17:33 EST 
Committed revision 7129
Comment 26 Eike Stepper CLA 2011-02-17 02:45:40 EST 
Committed revision 7130
Comment 27 Eike Stepper CLA 2011-02-17 04:27:27 EST 
Committed revision 7131:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 28 Eike Stepper CLA 2011-02-17 05:22:47 EST 
The more I learn about MongoDB the clearer our strategy becomes:

MongoDB's atomicity constraints  directly implies that a logical CDO commit must be persisted with a single new document. My original idea to store the change set deltas in this commit document and update per-class collections later (possibly in a background tread) turned out to be quite complex and functionally sub-optimal. It would still require costly and complex crash detection/recovery.

Now I learned that MongoDB can easily index and query fields of embedded objects. Hence we will no longer store change set deltas in the commit document, but hte entire resulting state, i.e. CDORevisions for the new, changed and detached objects. With this design the database will only have a single collection for the revisions of all classes. No need for additional mapping to per-class collections, no concurrency issues, no distribution issues, no threading issues ;-)
Comment 29 Eike Stepper CLA 2011-02-17 08:29:56 EST 
Committed revision 7132:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.hibernate
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.server.objectivity
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 30 Eike Stepper CLA 2011-02-17 08:33:31 EST 
I've added a BidiMap<EClassifier, Integer> to reduce persistent footprint and costly URI lookups.
Comment 31 Eike Stepper CLA 2011-02-18 13:07:18 EST 
Committed revision 7133
Comment 32 Eike Stepper CLA 2011-02-19 00:52:51 EST 
Committed revision 7134:
Comment 33 Eike Stepper CLA 2011-02-19 01:42:07 EST 
Committed revision 7135:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 34 Eike Stepper CLA 2011-02-19 03:51:35 EST 
Committed revision 7136
Comment 35 Eike Stepper CLA 2011-02-19 04:08:15 EST 
Committed revision 7137
Comment 36 Eike Stepper CLA 2011-02-19 06:04:38 EST 
Committed revision 7138
Comment 37 Eike Stepper CLA 2011-02-19 06:26:15 EST 
Committed revision 7139
Comment 38 Eike Stepper CLA 2011-02-19 07:53:44 EST 
Committed revision 7140:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
Comment 39 Eike Stepper CLA 2011-02-19 09:37:12 EST 
Committed revision 7141:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 40 Eike Stepper CLA 2011-02-19 13:12:19 EST 
Committed revision 7142:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 41 Eike Stepper CLA 2011-02-20 04:21:59 EST 
Committed revision 7143:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 42 Eike Stepper CLA 2011-02-20 05:25:30 EST 
Committed revision 7144:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
Comment 43 Eike Stepper CLA 2011-02-20 05:41:10 EST 
Committed revision 7145:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 44 Eike Stepper CLA 2011-02-20 05:56:39 EST 
Committed revision 7146:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 45 Eike Stepper CLA 2011-02-20 06:16:30 EST 
Committed revision 7147:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 46 Eike Stepper CLA 2011-02-20 06:35:32 EST 
Committed revision 7148
Comment 47 Eike Stepper CLA 2011-02-20 07:11:16 EST 
Committed revision 7149:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
Comment 48 Eike Stepper CLA 2011-02-20 07:14:42 EST 
Committed revision 7150:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
Comment 49 Eike Stepper CLA 2011-02-20 07:46:18 EST 
Committed revision 7151:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
Comment 50 Eike Stepper CLA 2011-02-20 08:53:03 EST 
Committed revision 7152:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.db
- trunk/plugins/org.eclipse.emf.cdo.server.db4o
- trunk/plugins/org.eclipse.emf.cdo.server.hibernate
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.server.objectivity
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 51 Eike Stepper CLA 2011-02-20 09:00:54 EST 
Committed revision 7153:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 52 Eike Stepper CLA 2011-02-20 09:24:03 EST 
Committed revision 7154
Comment 53 Eike Stepper CLA 2011-02-20 09:58:09 EST 
Committed revision 7155:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 54 Eike Stepper CLA 2011-02-20 10:11:40 EST 
Committed revision 7156:
- trunk/plugins/org.eclipse.emf.cdo.tests
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
- trunk/plugins/org.eclipse.emf.cdo.workspace
Comment 55 Eike Stepper CLA 2011-02-20 12:47:21 EST 
Committed revision 7157:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 56 Eike Stepper CLA 2011-02-20 13:05:37 EST 
Committed revision 7158:
- trunk/plugins/org.eclipse.emf.cdo.server
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 57 Eike Stepper CLA 2011-02-20 13:48:47 EST 
Committed revision 7159:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 58 Eike Stepper CLA 2011-02-20 14:26:04 EST 
Committed revision 7160:
- trunk/org.eclipse.emf.cdo.server.mongodb
- trunk/org.eclipse.emf.cdo.tests
- trunk/org.eclipse.emf.cdo.tests.mongodb
Comment 59 Eike Stepper CLA 2011-02-21 01:13:23 EST 
Committed revision 7162:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 60 Eike Stepper CLA 2011-02-21 01:48:59 EST 
Committed revision 7163:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 61 Eike Stepper CLA 2011-02-21 03:08:48 EST 
Committed revision 7164:
- trunk/plugins/org.eclipse.emf.cdo
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 62 Eike Stepper CLA 2011-02-21 03:10:51 EST 
*** All non-auditing tests are passing now (1102 cases) ***
Comment 63 Eike Stepper CLA 2011-02-21 03:27:51 EST 
Committed revision 7165
Comment 64 Eike Stepper CLA 2011-02-21 04:46:59 EST 
Committed revision 7167:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
Comment 65 Eike Stepper CLA 2011-02-21 05:02:36 EST 
Committed revision 7168:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 66 Eike Stepper CLA 2011-02-21 06:42:00 EST 
Committed revision 7169:
- trunk/features/org.eclipse.emf.cdo.server.mongodb-feature
- trunk/features/org.eclipse.emf.cdo.site-feature
- trunk/features/org.eclipse.emf.cdo.tests-feature
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests
- trunk/plugins/org.eclipse.net4j.tests
- trunk/releng/org.eclipse.emf.cdo.releng
Comment 67 Eike Stepper CLA 2011-02-21 14:53:21 EST 
Committed revision 7172:
- trunk/features/org.eclipse.emf.cdo.server.product
- trunk/plugins/org.eclipse.emf.cdo.server.db
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.server
Comment 69 Eike Stepper CLA 2011-02-22 10:47:34 EST 
The MongoDB driver bundle is available at, e.g., http://net4j.sourceforge.net/update/
Comment 70 Eike Stepper CLA 2011-04-06 01:11:59 EDT 
Committed revision 7591:
Comment 71 Eike Stepper CLA 2011-04-06 01:51:09 EDT 
Committed revision 7595:
- trunk/features/org.eclipse.emf.cdo.sdk-feature
- trunk/features/org.eclipse.emf.cdo.server.mongodb-feature
- trunk/releng/org.eclipse.emf.cdo.releng
Comment 72 Eike Stepper CLA 2011-04-06 01:51:57 EDT 
The MongoDB driver is now part of the shipped SDK.
Comment 73 Eike Stepper CLA 2011-04-07 03:02:27 EDT 
Committed revision 7600:
- trunk/plugins/org.eclipse.emf.cdo.server.mongodb
- trunk/plugins/org.eclipse.emf.cdo.tests.mongodb
Comment 74 Eike Stepper CLA 2011-06-23 03:41:17 EDT 
Available in R20110608-1407