Bugzilla – Full Text Bug Listing
| Summary: | discuss mapping of Red Hat Koji architecture to Builds api | ||
|---|---|---|---|
| Product: | z_Archived | Reporter: | Ken Leung <plcnkl> |
| Component: | Mylyn | Assignee: | Project Inbox <mylyn-triaged> |
| Status: | CLOSED WONTFIX | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | P3 | CC: | akurtakov, plcnkl, swagiaal |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| URL: | http://koji.fedoraproject.org/koji/index | ||
| Whiteboard: | |||
| Bug Depends on: | 354624, 354625 | ||
| Bug Blocks: | |||
|
Description
Ken Leung
Thanks for the description of the architecture. I have pasted a few thoughts below. > First, Koji is a simple build system that does not run tests, therefore, there > are no build numbers or culprit. That should not cause any problems. You should be able to leave any values as null (except for the ID maybe). If this is not working as expected for a particular field, please file a specific bug and we will investigate fixing this. > Second, each build request will be treated as a task where each task may > trigger multiple child tasks (which corresponds to an actual build) for > different build targets (like fedora15i386, fedora15x86_64, > redhatEnterprise6i386, etc.) or dependencies (like one must first build the > source rpm before the actual rpm can be built). What would a user typically want to monitor in the builds view? A list of recent tasks as displayed here: http://koji.fedoraproject.org/koji/index ? From a users perspective, would it make sense to subscribe to packages? > If the user would like to > restart the building process, he/she will push it as another task again. > Therefore, we are interested in monitoring the tasks instead of a build. It's maybe just a matter of terminology. You can think of IBuildPlan as a lightweight instance of a build (or task) that is kept in memory. In the case of Koji this might always be corresponding to one particular task and not represent a build job like in Hudson. I am wondering though if package might be a better match for the plan. It looks as if each package is associated with builds just like a Hudson job is associated with builds. > Finally, the artifacts are tied to a build instead of a task in Koji, and due to > the fact that we are interested in monitoring the task, we will not be able to > fill in the artifacts field of the Build class. In Koji, is the hierarchy like this? Package Build Task Task We don't have an explicit representation for Task in the framework but I could imagine adding something like that. Bamboo has stages that contain jobs for instance. Each stage/job is a step executed as part of a build and seems fairly similar to a Koji Task. Hi Mr. Pingel:
Thank you very much for the reply! I think we are interested in monitoring the root tasks (parent tasks) because the successfulness of the root task depends on all its subtasks, where each of the subtask can be a reflection of an actual build. So in Koji's case, according to my understanding at least, will have a hierarchy similar to:
owner
|
interested ---> task -------- package (i386)
| | | L package (x86-64)
| | | L ......
| | L Channel
| L Host
|_________________________.....................
| |
subtask1 - build(i386) subtask2 - build(x86-64).....
|L Channel | L Channel
L Host L Host
In the meantime, given a task ID, I can query the koji server for the package ID, owner ID, build ID and all its subtasks. So I think it makes more sense to monitor a task. And I would say it is difficult to monitor a package because the koji status codes are tied tasks and builds instead of package and there is no way to travel around the entity tree I showed above with a package ID.
I also recall that there is an interface called an IArtefact, probably that's where a package should fit in (but I have doubt that I can reach that part of the connector with my remaining summer work term or I should hand over to somebody else..) So, I wonder, if it is possible for me to leave IArtefact as null for the now?
When we designed the Builds view we had the following use-case in mind: Build servers usually host dozens or hundreds of project builds. Developer are typically only interested in a small subset that relates to the projects they are currently working on. Therefore, the builds view supports a subscription model to select that subset. In the case of Koji I was wondering to which item developer would typically want to subscribe to? Maybe there is no simple answer since there are so many different items such as packages, tags, builds etc. and this may vary depending on users goals. You can certainly ignore IArtifact for now but from a quick look at the page of a build the produced RPMS would seem like a very good fit. If you have a URL and name for each RPM it should be straight forward to populate the data model. Just to clarify, where does a Koji build fit into the picture? Looking at http://koji.fedoraproject.org/koji/buildinfo?buildID=217521 IBuild would seem like a good fit to model builds. (In reply to comment #3) > When we designed the Builds view we had the following use-case in mind: Build > servers usually host dozens or hundreds of project builds. Developer are > typically only interested in a small subset that relates to the projects they > are currently working on. Therefore, the builds view supports a subscription > model to select that subset. > > In the case of Koji I was wondering to which item developer would typically > want to subscribe to? Maybe there is no simple answer since there are so many > different items such as packages, tags, builds etc. and this may vary depending > on users goals. > > You can certainly ignore IArtifact for now but from a quick look at the page of > a build the produced RPMS would seem like a very good fit. If you have a URL > and name for each RPM it should be straight forward to populate the data model. > > Just to clarify, where does a Koji build fit into the picture? Looking at > http://koji.fedoraproject.org/koji/buildinfo?buildID=217521 IBuild would seem > like a good fit to model builds. Hi Mr. Pingel: I must admit that the model of Koji is quite complicated and its really hard to decide which entity to monitor (honestly, I took a about a day of mental fighting against myself when deciding which entity should I be monitoring). We had a team meeting yesterday and I agree with the team's vision: when a Koji user pushes a build (reflected by a task), it will be broken into architecture-specific builds (sub tasks). So regardless the successfulness of an individual architecture-specific builds, the task is still considered a fail if any of the subtasks fail and it will require a fix. Therefore, a Koji user is interested in monitoring the task, as a group of builds. I also got another question that I am not sure am I allowed to ask here. It turns out that my seniors are particularly interested in one use case and I wonder if it is possible to happen - there is another Red Hat in-house eclipse project called fedora packager, which allows its users to push a Koji build from within eclipse. So we want to make fedora packager dependent on Mylyn builds as well as the Koji connector so that whenever fedora packager pushes a build, the build is automatically added to Mylyn builds and a build editor with the build will show up in eclipse. If this is not the right place to ask this question, please let me know and I will post it to the mailing list, thank you very much! Ken Leung (In reply to comment #4) > We had a team meeting yesterday and I agree with the team's vision: when a Koji > user pushes a build (reflected by a task), it will be broken into > architecture-specific builds (sub tasks). So regardless the successfulness of > an individual architecture-specific builds, the task is still considered a fail > if any of the subtasks fail and it will require a fix. Therefore, a Koji user > is interested in monitoring the task, as a group of builds. Sounds good. I would still think about how the build fits in since it has artifacts (RPMs) and a changelog that map nicely to the Mylyn Builds data model. So, it sounds like all that is needed on the framework end is a representation of (sub-)tasks for a build? > ... So we want to make fedora packager dependent on Mylyn builds as > well as the Koji connector so that whenever fedora packager pushes a build, the > build is automatically added to Mylyn builds and a build editor with the build > will show up in eclipse. That should be fairly straight forward to do. Depending on how the subscription is implemented for Koji in the end you should be able to add the corresponding items. Feel free to file an enhancement require once you start working on that feature and we can add the necessary API to the framework if anything is missing. (In reply to comment #5) > So, it sounds like all that is needed on the framework end is a representation > of (sub-)tasks for a build? Hi Mr. Pingel: I believe so, and is it possible to also add Builds view support for subtasks? Ken Leung To add one more POV as a frequent koji user. It makes most sense to me to subscribe to packages and this would map closer to mylyn-builds better. While having a per user home page showing latest builds/tags and etc. I don't think this can be abstracted enough to be part of mylyn and it would fit better elsewhere assuming there is a good api allowing querying and showing things in an easy way using mylyn api (like one can query for a task, see it's common attributes and query for bugzilla specific one if he knows exactly what is he doing). One more comment about Koji and tests. Koji do run tests if this is specified in the RPM spec file. Much in the same way ant builds are running tests only if there is a target running them in the build.xml. The output can be parsed from the build.log output file but I'm not sure how the type of tests run is determined - there can be junit output (from ant, maven-surefire), testng or even cppunit, ctest or whatever custom unit testing system might be used. (In reply to comment #6) > I believe so, and is it possible to also add Builds view support for subtasks? We generally try to avoid hierarchies http://wiki.eclipse.org/Mylyn/UI_Design#Avoid_Hierarchies where ever we can. My initial thought was that we would add support for subtasks to the data model and UI in the build editor to navigate from the parent task to its children. I am not sure if we would support this in builds view as well but we could consider it later. The current UI is mostly designed around reviewing results of completed builds. It does not yet support monitoring jobs that are in progress very well. I am wondering if it makes sense to extend the Builds View to show running tasks (similar to the Hudson Builds Executor Status table). Does Koji have API to retrieve that type of status? (In reply to comment #7) > To add one more POV as a frequent koji user. It makes most sense to me to > subscribe to packages and this would map closer to mylyn-builds better. While > having a per user home page showing latest builds/tags and etc. I don't think > this can be abstracted enough to be part of mylyn and it would fit better > elsewhere assuming there is a good api allowing querying and showing things in > an easy way using mylyn api (like one can query for a task, see it's common > attributes and query for bugzilla specific one if he knows exactly what is he > doing). That sounds good to me. We already have a limited toolkit for constructing the current editor UI. It would certainly make sense to make that reusable allowing connectors to provide custom pages that are specific to the respective server structure. (In reply to comment #8) > One more comment about Koji and tests. > Koji do run tests if this is specified in the RPM spec file. Much in the same > way ant builds are running tests only if there is a target running them in the > build.xml. The output can be parsed from the build.log output file but I'm not > sure how the type of tests run is determined - there can be junit output (from > ant, maven-surefire), testng or even cppunit, ctest or whatever custom unit > testing system might be used. The Mylyn Builds data model for tests is independent of the particular test framework (although modeled after JUnit). The only existing Eclipse integration is for JUnit but we could consider supporting other frameworks as well or make it extensible. The only reason that we haven't done is that it hasn't come up so far. I have filed bug 354624 and bug 354625 to discuss extensions to the builds model. (In reply to comment #9) > We generally try to avoid hierarchies > http://wiki.eclipse.org/Mylyn/UI_Design#Avoid_Hierarchies where ever we can. My > initial thought was that we would add support for subtasks to the data model > and UI in the build editor to navigate from the parent task to its children. I > am not sure if we would support this in builds view as well but we could > consider it later. Hi Mr. Pingel: In the meantime, I am still working on the server client's functionality for querying each of the entity, so it may take a bit more time to complete the data model and post it here. > The current UI is mostly designed around reviewing results of completed builds. > It does not yet support monitoring jobs that are in progress very well. I am > wondering if it makes sense to extend the Builds View to show running tasks > (similar to the Hudson Builds Executor Status table). Does Koji have API to > retrieve that type of status? I believe the Open task status is the same as building. Ken Leung As I mentioned in my last comment, I will now post the revised data model for the Koji Connector:
interested --------> Package (IBuildPlan)
(recent builds) | L------------ Build 1 - task (IBuild)
| (IBuild) |||||L Owner
Build 2 - task ||||L Channel
. |||L Host
. |L-----------Sub task 1
. | ||L Owner
| |L Channel
Sub task 2 L Host
||L Owner
|L Channel
L Host
So, for each package (which will appear as an IBuildPlan), there will be one or more recent builds (sub-builds, not supported by GUI yet, so only the package will be shown as an IBuildPlan in the builds view, and when double clicked, the build editor wills how the most recent task/build).
Due to the fact that scratch builds (builds that are not entering the next Fedora/Red Hat Enterprise Linux release) are not associated with a task, each Koji build may or may not have an instance of a task. In this case, we will try to map a task as an IBuild if it is available or a build as an IBuild for the build editor. If a task is available, we will also show its sub tasks in the build editor as its descendants.
If anybody has any idea about how to improve this model, please feel free to comment against it, thank you very much in advance!
Ken Leung
Dear all, Considering the performance of the final product and what can be displayed by Mylyn Builds, I think I should remove the channel and host out of the data model in my previous comment. In the meantime, the code related to those fields are all implemented but commented out. If anybody has any idea about this decision, please post on this bug. Thank you very much! Ken Leung (In reply to comment #14) > Dear all, > > Considering the performance of the final product and what can be displayed by > Mylyn Builds, I think I should remove the channel and host out of the data > model in my previous comment. > > In the meantime, the code related to those fields are all implemented but > commented out. If anybody has any idea about this decision, please post on > this bug. Thank you very much! > > Ken Leung Ken, would you please tell the URL where to find the implementation? (In reply to comment #15) > Ken, would you please tell the URL where to find the implementation? Hi Alex, Its actually on my github, sorry that I thought everybody from the team knows my git hub url from my status report few weeks ago, my bad. So, here you go! https://github.com/kleung/org.eclipse.mylyn.koji Its still under development, so it might look messy, I must apologize for that and thanks for the comment! Ken Leung (In reply to comment #16) > > Hi Alex, > > Its actually on my github, sorry that I thought everybody from the team knows > my git hub url from my status report few weeks ago, my bad. > > So, here you go! > > https://github.com/kleung/org.eclipse.mylyn.koji > > Its still under development, so it might look messy, I must apologize for that > and thanks for the comment! > > Ken Leung I know it but Steffen doesn't :) All effort on this one has died long ago. Better to close it. If someone suddenly gains interest again please reopen. |