Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 371496

Summary: increase quota for "tools" and provide more fine grained reporting
Product: Community Reporter: David Williams <david_williams>
Component: ServersAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: P3 CC: daniel_megert, jarthana, markus.kell.r, srikanth_sankaran, stephan.herrmann
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Whiteboard:

Description David Williams CLA 2012-02-14 09:19:03 EST
Once a month, I get a note that says something like 
> 27 GB of space are used, and the quota is set at 2 GB.

Not very meaningful or helpful. Our Tools quota should be increased quite a bit. Its my understanding the quotas as they are used now are not an absolute cut-off, but as a reminder to projects if they start to grow beyond what's reasonable, for them, and need to do some cleanup. 

Plus, it would help, for Tools project, if each subproject had their own quota and report. 

I've taken snapshots during December, and recently in February, with following usage results. The biggest "disk hog" is PTP, which not only uses a lot, but also is growing at an unreasonable rate (well, at least, seems unreasonable, to me, not knowing more details). Other projects might have grown some, but appear relatively stable around the size they are, so seems like a quota slightly above what they are using seems a reasonable next step. 

That do you think? Can the detail level be implemented? In quotas and reports? 
Is this a reasonable approach to setting quotas? 

I've send not to tools-leads@eclipse.org so the project leads may comment in this bug with more details. 


proj     	Dec   Feb       Recommend Quota

./ptp		8.3G	 12.0G       10G
./cdt		7.1G	  8.1G       10G
./orbit		4.9G	  3.7G        5G
./pdt		4.2G	  4.3G        5G

./ajdt		877M	  1.1G        1G
./aspectj 	987M      0.6G        1G 
./atf 		462M	  467M        1G
./buckminster   258M      262M        1G
./gef		452M	  954M        1G

Tools Total:     		      35G 



Thanks,
Comment 1 Stephan Herrmann CLA 2012-02-14 10:39:07 EST
Object Teams is currently using more than usual because setting up dual build (Indigo + Juno) happened in a hurry after migrating to git.

Once Indigo SR2 is out of the door, I can wipe out about 1 Gig or so.
Comment 2 Eclipse Webmaster CLA 2012-04-20 10:47:46 EDT
I've made some changes to the way the quotas are calculated/reported.  

Previously 'parent' projects like tools ended up being used as catch alls for their 'children'.  Now we try to work out which children tools has and subtract their usage from the parent project(and notify the children individually).  

Next time the update script runs, let me know if you still think it needs work.

-M.
Comment 3 David Williams CLA 2012-04-30 20:32:36 EDT
It's probably ok the way it is. I got a report saying we used 4 G with a 2 G quota, and the only thing I see that is not part of a child project is about 500 M under /tools/downloads (and, yes, that could likely move to archives ... very old stuff). 

But, not sure where the other 1.5 G is coming from. Maybe you could just set our limit at 5 G so would only get the report if TL Project exceeded that? 

Or, you can figure out where the other 1.5 G is coming from and perhaps tweak your reports? 

But, it is much better ... and I'm glad the "children" are getting the reports directly. 

[20:21:42] david_williams@build:/home/data/httpd/download.eclipse.org/tools
 
$ du ./* -sh
1.4G    ./ajdt
800M    ./aspectj
470M    ./atf
309M    ./buckminster
7.3G    ./cdt
531M    ./downloads
8.0K    ./emf
1.4G    ./gef
4.0K    ./index.html
7.3G    ./orbit
4.3G    ./pdt
6.6G    ./ptp
24M     ./tcf
4.0K    ./TIME
8.0K    ./uml2
8.0K    ./ztime
Comment 4 Eclipse Webmaster CLA 2012-05-01 14:33:07 EDT
I suspect that there are 2 issues at work.

1) I forgot to turn the 'check current usage' on, so the data was a bit stale (fixed)
2) Rounding issue(s) in the calculations.  Basically if you aren't over the exact value of a Gig, you count 0.

I've upped the tools quota to 4G which should silence it.

-M.
Comment 5 Markus Keller CLA 2012-06-18 08:01:26 EDT
As JDT UI lead, I still get mails like this:

"The downloads area disk quota has been reached for project eclipse.jdt. You are receiving this e-mail because you are one of the committers who has write access to the project's downloads area.

150 GB of space are used, and the quota is set at 2 GB. [..]"


An email from Matt explained the situation:
> David is spot on.  Both the Foundations internal DB and the project meta 
> data list the JDT downloads area as '/eclipse' so when the quota tool 
> runs it's 'blaming' JDT for the entire Eclipse project space usage(minus 
> any child projects it can find).

Can we do anything to avoid these mails for sub-projects of the Eclipse top level project?
Comment 6 Eclipse Webmaster CLA 2012-06-18 16:23:44 EDT
(In reply to comment #5)

> 
> Can we do anything to avoid these mails for sub-projects of the Eclipse top
> level project?

Well the easiest/fastest solution is to provide a download url other than download.eclipse.org/eclipse (like say download.eclipse.org/eclipse/jdt) in the project metadata (I can update our internal records if they fail to sync the change).  

-M.
Comment 7 Dani Megert CLA 2012-06-19 02:55:52 EDT
(In reply to comment #6)
> Well the easiest/fastest solution is to provide a download url other than
> download.eclipse.org/eclipse (like say download.eclipse.org/eclipse/jdt) in 
> the
> project metadata (I can update our internal records if they fail to sync the
> change).  
> 
> -M.

Why would we change the correct download URL to something that gives 404? This workaround is no good ;-).
Comment 8 Markus Keller CLA 2012-06-19 05:53:56 EDT
(In reply to comment #6)
> Well the easiest/fastest solution is to provide a download url other than
> download.eclipse.org/eclipse (like say download.eclipse.org/eclipse/jdt)

Sounds good, can you create a directory and add a simple redirect to http://download.eclipse.org/eclipse/downloads/ , so that all sub-projects can just use that url?
E.g. http://download.eclipse.org/eclipse/downloads/subprojects/
Comment 9 Eclipse Webmaster CLA 2012-06-21 10:20:14 EDT
I talked over the best way to handle the requested re-directs with Denis, and in the end his suggestion was simply to raise your quota, so I did.

-M.