Community
Participate
Working Groups
This is a request to generalize Mylyn Connector Discovery Framework such that it is accessible to other projects for other types of add-ins. This will necessitate moving/refactoring the code to either platform/p2 or perhaps a separate project. The core requirements can be summarized as follows: 1. The system should maintain a catalog of add-ins, where each catalog entry is composed of at least name, description, provider, license, update site URL, feature id to install and a collection of tags. 2. It should be possible to access the "install add-ins" wizard globally from the main Eclipse menu system. This would show all add-ins. 3. It should be possible for adopters to invoke this wizard programatically while specifying tags to filter on. The tags would be things like "mylyn.connector", "wtp.server.adapter", "dtp.connector", etc. This feature can later expand in following ways: 1. Submitter-maintained catalog entries via interface on eclipse.org. 2. Paid add-ins with the transaction interaction happening directly in the add-in wizard. 3. User feedback, rating, all your typical app store junk. I don't have a direct experience with Mylyn Connector Discovery feature, but I have extensive experience being on the user and adopter end of WTP server adapter discovery. Some aspects of that feature work extremely well, others not so much. The following few paragraphs try to summarize some of the problems that we've hit in hopes that these would be avoided in the common framework. WTP server adapter catalog was by design extremely basic and minimal in amount of information it contained. Each entry was just the update site URL. All entries were embedded directly in WTP plugins. This made things simple, but caused quite a few problems. 1. Updates to the catalog required a patch to WTP. If your product ships cycle doesn't align with WTP, this leads to problems. We hit this a couple of times. Most recently by forgetting to file the request to update our update site URL in WTP's Galileo stream on time. Our product update sites have different URL's for each platform release. Now we are stuck with our users not being able to install WebLogic Server adapter through this wizard in Galileo until Galileo SR1 comes out. The version hosted on our Ganymede site is not compatible with Galileo. Had the catalog been maintained on eclipse.org, we could have updated our catalog entry while Galileo is in the lock-down and before any users noticed the problem. 2. The WTP framework uses update site feature categories as the tagging mechanism for identifying features that repesent WTP adapters (because this info is not in the catalog). This created a lot of problems for us as categories are visible to users. We ended up having to publish two update sites. One for the full product with feature categories the way our PM's wanted our users to see them and another hidden one to be given to WTP server adapter discover catalog that is shaped how that mechanism expects it. This obviously creates more releng and QA overhead on adopters. It would be much better to have a single update site with the catalog entries carrying more information. 3. The WTP framework uses feature names as the string displayed to the user in the adapter installation wizard. This became a problem for us because PM's wanted one string for the feature name when visible from the standard update manager UI along side our other features and another string to be visible in the server adapter discovery wizard. Since that was not possible, finding a single string that worked ok in both contexts was a major pain in the behind. The moral is that the catalog entry should contain add-in name. 4. Since WTP framework didn't know what was contained on those update sites that were in the catalog, the server adapters discovery wizard had to contact all of these update sites to build up the list to display to the user. Much effort was put into making this more streamlined, multi-threaded, etc., but the user experience was still less than ideal, especially on slow connections or when some updates sites are slow to respond or go dead for a while. Part of this problem can be solved simply by putting enough basic information into the catalog entry. Unfortunately, there is still the problem of how to determine if the selected add-in (or more specifically, the feature it specifies) is compatible with the current install or already installed. Perhaps the thing to do here is to delay this check until user selects an add-in. Another more advanced solution is to have a daemon running on Eclipse.org that queries the various update sites and stashes the p2 metadata in the catalog. If p2 metadata is in the catalog entry, then presumably validity check can happen fairly quickly when the wizard comes up. 5. Testing the installation of your add-in ahead of a product release is unnecessary difficult. The update site that you'd like to test through the add-in wizard is private at this stage and the public catalog might already contain an entry for your product that points to your public update site with an older version. Our current practice is to patch WTP to update the catalog to point to our internal update site and give that build to QA to test. Too difficult. Ideally, we would have a facility to easily specify a local catalog for testing purposes and perhaps selectively suppress entries coming from other catalogs. This can be implement in a variety of ways. Something as simple as command line flags passed to eclipse process would work. That's enough from me for now. If this moves forward, I'd like to contribute to this effort. I am copying Tim deBoer from WTP Server Tools and Brian Fitzpatrick from DTP. I would recommend to start with DTP as the first non-Mylyn adopter of this framework. Brian has expressed interest in bringing such technology to DTP and there is no backwards compatibility with an existing framework to worry about as there would be with WTP. For our part, we would love to be able to make it easy for our users to install Oracle Database connector in the same way as it's easy for them now to install WebLogic Server adapter.
DTP would definitely be a good candidate for this sort of discovery mechanism, as has been discussed in the past. Unfortunately, I most likely won't be able to help. I'm leaving the DTP leadership and will just be on the fringes of DTP as I shift jobs. So I apologize, but wish you guys the best of luck.
> DTP would definitely be a good candidate for this sort of discovery mechanism, > as has been discussed in the past. Unfortunately, I most likely won't be able > to help. I'm leaving the DTP leadership and will just be on the fringes of DTP > as I shift jobs. So I apologize, but wish you guys the best of luck. Thanks, Brian. Oracle is interested in bringing this functionality to DTP, so we will help make that happen. Perhaps you or one of the other DTP committer will at least help integrate our patches.
Great to see interest in generalizing Mylyn Connector Discovery. Connector Discovery was designed with this in mind, however there are several connector-specific concepts that are present in the design, mostly related to naming of connector metadata. Mylyn Connector Discovery was however purpose-built for Mylyn and does not address all of the requirements listed above. For example, the UI is very connector-centric (eg: filtering mechanism). You'll be pleased to hear that Mylyn Connector Discovery addresses many of the design defficiencies/concerns presented in the description of this bug, including 1 (no patch needed to change a listing), 2 (tagging mechanism is not related to update sites), 3 (names are distinct from feature names), 4 (listings are not discovered via an update site), 5 (discovery directory location can be altered with a system property for testing purposes) At a high level Mylyn uses the concept of a directory and connector descriptors. The directory is a declarative (XML) resource that exists at a well-known URL, and points to connector descriptors. Connector descriptors describe an installable component, and include branding, description, images, etc. So Connector Discovery first accesses the directory to discover where the connector descriptors can be found, then downloads the descriptors to discover what the installable units are. This design provides a central point of control for updating what is discoverable, and avoids the need to install any plug-ins to have contributions to the discovery UI. It also gives vendors the freedom to update their contributions without involving changes or involvement by committers, providing that their contribution is already listed in the directory. Contributions may also be provided by installed plug-ins if desired. Interested parties may wish to take a look at the following Mylyn projects: * org.eclipse.mylyn.discovery.core * org.eclipse.mylyn.discovery.ui * org.eclipse.mylyn.discovery.tests I hope that helps with getting started! Please let me know if you have any questions.
So how should we get started on this generalization and where? I think this framework's eventual scope (when you consider possible expansion into a full-on app store) makes it easily big enough to justify a separate project. Thoughts? We can fairly quickly put together a project proposal, seed the project with code from Mylyn and start refactoring. Given limited initial scope, advanced state of code as it exists in Mylyn and interest from a variety of parties, we should be able to easily graduate and join Helios train where Mylyn and DTP can pick it up as a dependency. I just went through setting up build and all the related infrastructure for the technology.fproj project and it is much easier to do now between Athena and Hudson. Took only a couple of days for me to set all of this up without knowing anything about Athena before hand. Should be trivial to copy my working config at this point. I will volunteer for that task. So, who were the primary devs on the connector discovery framework for Mylyn and are these individuals available to work on refactoring it into a common format?
Glad to hear that you're eager to move forward. I was the lead developer on this feature with help from the team at Tasktop. As far as availability and prioritization of work I suggest that you contact Mik at Tasktop.
Konstantin: Sorry for slowness on this. I'll be able to review once the Galileo release madness is over.
I have created a bug 284501 in order to track the generalization of discovery. This bug is a good summary of the needs of DTP and WTP, so I made it specific to tracking those needs. I'll comment on the generic bug with sugguestions on how to proceed. Regarding the specific comment to create a new project, in all of my experiences with cross-project reuse, I think it's better to take a more bottom-up approach in order to ensure that there are enough developer resources available to warrant a new project before creating that new project. We have some good strategies for how to ensure minimal friction when growing the general project/component in order to ensure that it is as easy as building against a newly-created project. See bug 284501 for more details. Konstantin: Since you have offered up time to help make this happen, I've reassigned this bug to you for the time being. Let's discuss the general case on the parent bug, then once we've converged we can figure out the details here.
Moving bug to P2 which now hosts discovery. Basic information how P2 discovery can be integrated is on the wiki at http://wiki.eclipse.org/Equinox/p2/Discovery and a first binary release is available on the Helios staging site. Konstantin, have you had a chance to look at P2 discovery? It would be great if you could provide feedback what the outstanding requirements for adoption by the webtools project are. The current state of P2 extension install discovery will also be discussed in this EclipseCon talk: http://www.eclipsecon.org/2010/sessions/?page=sessions&id=1205 .
I am moving this to MPC since the Marketplace Client offers the feature set requested in the description and could be a better fit for DTP and WTP than p2 Discovery. Konstantin, it would be great if you could take a look at the latest MPC and list specific enhancements that you would need to adopt MPC for DTP and WTP.
I have spent some time playing with MPC and have opened some issues for limitations relevant to use of MPC in embedded extension discovery context. Bug 313449 - Marketplace client needs to filter out incompatible software https://bugs.eclipse.org/bugs/show_bug.cgi?id=313449 Bug 313450 - Marketplace should support composite listings https://bugs.eclipse.org/bugs/show_bug.cgi?id=313450 The first one is a show-stopper for WTP adoption as this scenario currently works due to every version of WTP having its own version of extensions catalog. The second one should not block adoption for the embedded scenario, but not resolving it will cause a large spike in the number of Marketplace listings, all pointing to the same update sites.
@Kontantin I am not sure if the MPC is going to meet your requirements. It certainly wasn't the design or intention to be as general as you are envisioning. I think a suitable replacement for the WTP adapter discovery would be the p2 discovery. It allows for a focus list of add-ins to be presented in a Mylyn Discover-like UI and controlled from a server.
> @Kontantin I am not sure if the MPC is going to meet your requirements. It > certainly wasn't the design or intention to be as general as you are > envisioning. I think a suitable replacement for the WTP adapter discovery > would be the p2 discovery. It allows for a focus list of add-ins to be > presented in a Mylyn Discover-like UI and controlled from a server. I will admit being rather confused about the difference between MPC and p2 discovery. I thought that they were one and the same. It all started with Mylyn Connector discovery UI, right? I think it makes a lot of sense to allow MPC (driven from filtered data drawn from Marketplace) to be used in embedded contexts. The requirements are 99% identical. The only difference is that when you access MPC from the Help menu you expect to see all types of solutions. When you access MPC from a specific functional context (creating WTP server runtime or defining database connection), you want a more focused list of solutions.
(In reply to comment #12) > I will admit being rather confused about the difference between MPC and p2 > discovery. I thought that they were one and the same. It all started with Mylyn > Connector discovery UI, right? MPC is built on top of the p2 discovery, so they have the same base but MPC is a specialization. > > I think it makes a lot of sense to allow MPC (driven from filtered data drawn > from Marketplace) to be used in embedded contexts. The requirements are 99% > identical. The only difference is that when you access MPC from the Help menu > you expect to see all types of solutions. When you access MPC from a specific > functional context (creating WTP server runtime or defining database > connection), you want a more focused list of solutions. The challenge is that we have limited moderation on the Marketplace, ie. anyone can post a listing. I believe the p2 discovery is ideal for projects that want to have control over the listing.
> MPC is built on top of the p2 discovery, so they have the same base but MPC is > a specialization. Ok. I think I understand. The wizard and server API is the same. MPC client is then p2 discovery UI hooked up to marketplace.eclipse.org. Correct? > The challenge is that we have limited moderation on the Marketplace, ie. anyone > can post a listing. I believe the p2 discovery is ideal for projects that > want to have control over the listing. I don't think that embedded use necessarily requires tighter control. On the other hand, one of the problems we need to solve is to give ability for people to self-administer their listing. Marketplace has solved that one already.
(In reply to comment #14) > > MPC is built on top of the p2 discovery, so they have the same base but MPC is > > a specialization. > > Ok. I think I understand. The wizard and server API is the same. MPC client is > then p2 discovery UI hooked up to marketplace.eclipse.org. Correct? No, that's not correct. MPC uses p2 discovery for the purposes of code reuse. MPC is also similar to p2 discovery in that it drives the p2 APIs in order to install IUs into Eclipse. That's where the similarities end. The server APIs are different. Intended audience and use cases are different. > > The challenge is that we have limited moderation on the Marketplace, ie. > anyone > > can post a listing. I believe the p2 discovery is ideal for projects that > > want to have control over the listing. > > I don't think that embedded use necessarily requires tighter control. On the > other hand, one of the problems we need to solve is to give ability for people > to self-administer their listing. Marketplace has solved that one already. p2 discovery allows people to self-administer their listing as you require. With p2 discovery the administrator of the p2 discovery catalog has control over which listings appear, but beyond that everything is up to the solution provider. The interface to do so is not web-based as with Marketplace, but based on implementing extension points with PDE tooling. Everything that I've seen so far on this bug makes me think that p2 discovery is the right technology for you to use. p2 discovery is designed specifically for your use case: it's intended to facilitate a targeted installation experience where it's desirable to add extensions and components to an existing product that enable integration with optional external components (ie: server adapters, connectors, etc.) Trying to change Marketplace to suit your needs is like trying to fit a square peg into a round hole. Feel free to reopen this bug if you think that my assessment is incorrect or if you wish to continue the discussion. Marking as won't fix, since this is not an MPC issue.
*** Bug 313450 has been marked as a duplicate of this bug. ***
David, I am re-opening as I disagree with your assessment that Marketplace requirements and embedded installation requirements are disjoint. It makes little sense for projects that want to support embedded installation to re-invent the server side architecture necessary to publish and administer the catalog. It makes good sense to re-use Marketplace infrastructure on the server and MPC on the desktop to fill these requirements. As a user, I want to see product information, reviews, rankings, etc. whether I am installing from the Help menu or from an embedded context. I can see p2 discovery being useful to someone building a commercial IDE where it is necessary to tightly control available extensions. However, in the context of Eclipse projects wishing to support targeted installation, it is not an ideal solution.
Great, thanks for the clarification. Would it be sufficient to have a category (as we do for Mylyn Connectors for example) and have MPC API that could launch the MPC client with the category preselected?
> Great, thanks for the clarification. Would it be sufficient to have a category > (as we do for Mylyn Connectors for example) and have MPC API that could launch > the MPC client with the category preselected? I think tags would work better than categories as you can multi-tag for different contexts. Then as you say we would need an API to launch MPC with a particular tag pre-entered as a filter. For user simplicity, I would argue that this should be a permanent invisible filter rather than a pre-selection behavior that user can change after opening the wizard.
I'm closing this ancient issue. I believe after years of having p2 discovery ui, there is nothing left to do here.
