Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 422498

Summary: Please create HIPP instance for simrel
Product: Community Reporter: David Williams <david_williams>
Component: CI-JenkinsAssignee: CI Admin Inbox <ci.admin-inbox>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: denis.roy, mknauer, thanh.ha, webmaster
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Whiteboard:

Description David Williams CLA 2013-11-25 12:49:24 EST
I'm making this request to begin "experimenting" with running simrel aggregation jobs in its own "HIPP Instance". 

Not sure if it matters, but current "simrel" aggregation is partially done on Hudson, but partially done directly on /shared/simrel directory (via scripts) but the idea of doing this in a "HIPP instance" would be it'd be completely "in Hudson". I mention this since there is a pretty large disk requirement ... each "build" taking 2 to 4 Gig. 

Not sure if/how jobs are normally initialized, but if its done from "current jobs", the configs to copy-over are from 
https://hudson.eclipse.org/hudson/view/Repository%20Aggregation/
(but, we would not want them "active" until changes made so they would not interfere with "production builds" on shared instance). 

My intuition is that this would not replace the current shared instance until at least after Kepler SR2 (February, 2014).
Comment 1 David Williams CLA 2013-11-25 12:52:55 EST
Markus, may I suggest this same instance be used as initial/experimental area for "EPP builds" done completely on Hudson?  

Seems they are closely tied together, and not sure if there's any advantage to have separate "HIPP Instances" -- though having EPP here would significantly increase "disk space" required.
Comment 2 Thanh Ha CLA 2013-11-25 12:56:29 EST
(In reply to David Williams from comment #1)
> Markus, may I suggest this same instance be used as initial/experimental
> area for "EPP builds" done completely on Hudson?  
> 

We setup a HIPP for EPP last week per bug 421800. If your going to share an instance perhaps we can just add simrel to the packaging HIPP?
Comment 3 Markus Knauer CLA 2013-11-25 13:07:10 EST
(In reply to David Williams from comment #1)
> Markus, may I suggest this same instance be used as initial/experimental
> area for "EPP builds" done completely on Hudson?  

I requested an EPP HIPP instance - see bug 421800, and it is already available:

  https://hudson.eclipse.org/packaging/

So far I'm using it only for Tycho-based Luna experiments, and I don't have any plans to migrate any old Kepler jobs to the new instance.

I don't know the architecture behind HIPP, but I assume they are using their own local disk space. In that case it could make sense to consume the simrel artifacts directly and not via network/http.

It's up the the webmasters to decide... I'm open for all suggestions

Some criteria

- access rights to download area (simrel committers vs. EPP committers)
- disk space requirements
- HIPP restart by committers
- I/O / network
- ...
Comment 4 David Williams CLA 2013-11-25 13:08:14 EST
Well, let's leave them separate then ... sounds like Markus is further along and may be doing things differently than I have in mind. 

I assume a job in one HIPP instance can trigger a job in another HIPP instance? (If that's not the case ... then, there is the first issue to solve :)
Comment 5 Thanh Ha CLA 2013-11-25 13:12:05 EST
(In reply to Markus Knauer from comment #3)
> - access rights to download area (simrel committers vs. EPP committers)

It'll be difficult to differentiate since there is only 1 HIPP user if your sharing and the user will be a part of both committer groups.

> - disk space requirements

We mount the same partitions that is on build.eclipse.org so if you copy things to downloads or other places then it's shared disk. The job output is on local storage though in the case if you don't move your build output.

> - HIPP restart by committers
> - I/O / network
> - ...
Comment 6 Thanh Ha CLA 2013-11-25 13:15:13 EST
(In reply to David Williams from comment #4)
> Well, let's leave them separate then ... sounds like Markus is further along
> and may be doing things differently than I have in mind. 
> 
> I assume a job in one HIPP instance can trigger a job in another HIPP
> instance? (If that's not the case ... then, there is the first issue to
> solve :)

No, a HIPP instance can not trigger a job in another instance. Unless the other instance lets them on a per job basis by enabling the job config "Trigger builds remotely (e.g., from scripts)". In this case you'd have to share a secret token with the job from the other HIPP instance.
Comment 7 David Williams CLA 2013-11-25 13:27:44 EST
(In reply to Thanh Ha from comment #6)
> (In reply to David Williams from comment #4)
> > Well, let's leave them separate then ... sounds like Markus is further along
> > and may be doing things differently than I have in mind. 
> > 
> > I assume a job in one HIPP instance can trigger a job in another HIPP
> > instance? (If that's not the case ... then, there is the first issue to
> > solve :)
> 
> No, a HIPP instance can not trigger a job in another instance. Unless the
> other instance lets them on a per job basis by enabling the job config
> "Trigger builds remotely (e.g., from scripts)". In this case you'd have to
> share a secret token with the job from the other HIPP instance.

First issue solved. (We can do that). Thanks Thanh.
Comment 8 Thanh Ha CLA 2013-11-27 14:15:31 EST
I was going to create this today but I'm a little confused. When I "ls -l /gitroot/simrel" I noticed the group for the repos were "callisto-dev". I think this will make our usual pattern of using the group name as the project HIPP user confusing.

Usually we create a HIPP user genie.<project> and then assign them a HIPP URL of http://hudson.eclipse.org/<project> where "project" is the last part of a group name.

In this case I'm guessing we want the URL to be http://hudson.eclipse.org/simrel but everything else would be callisto-dev.

Denis, do you have any opinions on what should be done here?
Comment 9 David Williams CLA 2013-11-27 14:26:43 EST
(In reply to Thanh Ha from comment #8)
> I was going to create this today but I'm a little confused. When I "ls -l
> /gitroot/simrel" I noticed the group for the repos were "callisto-dev". 

Yeah, just in case you are interested ... "callisto" was the first one we did ... 7 years ago, or so ... and it just stuck. It's only been more recently we've began to think ahead and noticed, "hey, we are doing about the same thing each release ... we should have some more generic names."

Not sure what/how many complications it would cause, but if there was an (easy) way to start using "simrel" group, that'd be fine with me. (I'm sure its more work for you all, than me ... so up to you). 

But yes, would want the URL to be .../simrel
Comment 10 Denis Roy CLA 2013-11-27 14:30:32 EST
The only complication is that the HIPP Control buttons for start/stop/restart/upgrade won't show up on anyone's web UI since it's not attached to a project.
Comment 11 Thanh Ha CLA 2013-11-27 16:40:16 EST
The simrel HIPP instance is now available at https://hudson.eclipse.org/simrel/

Committers in the "callisto-dev" group should be able to login and create / modify jobs. Keep in mind that you will need to use your _email_ address to login unlike in the shared instance.

I added the simrel HIPP user to the callisto-dev group since I'm guessing you need it to be able to access the simrel directories.

If there's any additional plugins required for Hudson feel free to reopen this bug and list any plugins needed.
Comment 12 Markus Knauer CLA 2013-11-27 17:27:32 EST
(In reply to Thanh Ha from comment #11)
> Committers in the "callisto-dev" group should be able to login and create /
> modify jobs. Keep in mind that you will need to use your _email_ address to
> login unlike in the shared instance.
> 
> I added the simrel HIPP user to the callisto-dev group since I'm guessing
> you need it to be able to access the simrel directories.

Just thinking a bit about it... it may be better to use "callistoadmin" instead of "callisto-dev". Opinions?
Comment 13 David Williams CLA 2013-11-27 22:23:25 EST
(In reply to Markus Knauer from comment #12)
> (In reply to Thanh Ha from comment #11)
> > Committers in the "callisto-dev" group should be able to login and create /
> > modify jobs. Keep in mind that you will need to use your _email_ address to
> > login unlike in the shared instance.
> > 
> > I added the simrel HIPP user to the callisto-dev group since I'm guessing
> > you need it to be able to access the simrel directories.
> 
> Just thinking a bit about it... it may be better to use "callistoadmin"
> instead of "callisto-dev". Opinions?

That would be better ... for those that can create/modify jobs to have a more smaller group of people. 

I'm assuming from there we can assign "project level security" where ROLE_CALLISTO-DEV (or similar) could granted permission to start/stop jobs?
Comment 14 David Williams CLA 2013-11-27 22:25:47 EST
And, can you clarify ... I thought part of the purpose of this HIPP setup was that we could "manage" our own instance ... decide what plugins to install, etc. ... but I do not see a "manage" button (like I normally do).
Comment 15 Thanh Ha CLA 2013-11-28 09:34:40 EST
(In reply to David Williams from comment #14)
> And, can you clarify ... I thought part of the purpose of this HIPP setup
> was that we could "manage" our own instance ... decide what plugins to
> install, etc. ... but I do not see a "manage" button (like I normally do).

We assign admin privileges only when requested and also only assign it to specific people rather than the whole committer group. I've added you to the admin list so you should be able to manage the instance now.
Comment 16 Markus Knauer CLA 2013-11-28 14:13:30 EST
(In reply to David Williams from comment #13)

Yep, that's exactly what I thought. For job creation a smaller group would be better, and nobody prevents us from granting additional permissions to persons and/or to groups.
Comment 17 Thanh Ha CLA 2013-11-29 09:42:45 EST
I had a look and callisto-dev does have quite a few people on that list. I will defer the question of creating a new group to Denis.

Perhaps this topic is worth it's own separate bug?
Comment 18 David Williams CLA 2013-11-29 11:36:35 EST
(In reply to Thanh Ha from comment #17)
> I had a look and callisto-dev does have quite a few people on that list. I
> will defer the question of creating a new group to Denis.
> 
> Perhaps this topic is worth it's own separate bug?

There is a linux group, at least on "build.eclipse.org" which Markus suggested, 

$ getent group callistoadmin
callistoadmin:*:8403:david_williams,kmoir,mward,mknauer

Or ... is the "group" you mean something defined in LDAP or something? If this latter case, then I'll open a new bug ... as we'd probably want to name it something more generic :) such as "simrel-releng" or something, and slightly different set of members (e.g. kmoir wouldn't need to be a member, and perhaps we'd add Doug Schaefer .... if he didn't object).
Comment 19 Thanh Ha CLA 2013-11-29 14:24:51 EST
(In reply to David Williams from comment #18)
> (In reply to Thanh Ha from comment #17)
> > I had a look and callisto-dev does have quite a few people on that list. I
> > will defer the question of creating a new group to Denis.
> > 
> > Perhaps this topic is worth it's own separate bug?
> 
> There is a linux group, at least on "build.eclipse.org" which Markus
> suggested, 
> 
> $ getent group callistoadmin
> callistoadmin:*:8403:david_williams,kmoir,mward,mknauer
> 
> Or ... is the "group" you mean something defined in LDAP or something? If
> this latter case, then I'll open a new bug ... as we'd probably want to name
> it something more generic :) such as "simrel-releng" or something, and
> slightly different set of members (e.g. kmoir wouldn't need to be a member,
> and perhaps we'd add Doug Schaefer .... if he didn't object).

I had not realized this was an existing group. In that case I swapped out callisto-dev for callistoadmin on the simrel instance. So now only callistoadmin group members should have access to create new jobs in the instance.