Community
Participate
Working Groups
I think our "getMapFiles" have some issues, and thought I'd document a few here. I think they have "evolved" many times over the years, so they don't necessarily work as expected (and, I probably broke the "working even though not expected to" behavior with a few innocent changes), but as looked into deeply, I saw a few concrete issues that could be improved/if not fixed. First, we often "check if local" (and not re-get them) based purely on the existence of a directory ${buildDirectory}/maps. Well, that's not too accurate, first, for them to "really" exist, there needs to be a directory more like ${buildDirectory}/maps/org.eclipse.releng/maps Next, even if that exist, there's no assurance they are "fresh", and/or from the right branch or tag of the git repo we get them from. Next oddity, which may involve my minimal understanding of git, the the pattern we use looks odd ... and doesn't seem to be working as anyone would think it would, from appearances. Our typical pattern is similar to below: First,, we define some properties, and clone the (whole) repo: <property name="mapsGitRepo" value="${buildDirectory}/commonrepo" /> <mkdir dir="${mapsGitRepo}" /> <exec executable="git" dir="${mapsGitRepo}"> <arg line="clone ${commonrepo}" /> </exec> Next we check out the version we want: <exec executable="git" dir="${mapsGitRepo}/eclipse.platform.releng.maps"> <arg line="checkout ${mapVersionTag}" /> </exec> All that seems to work, and appears "on disk" as expected. Now here's here it gets fuzzy (at least to me) We use git to get a new?! copy of the maps files, all tarred up and then untar them into our maps location. <exec executable="git" dir="${buildDirectory}/maps" output="${buildDirectory}/maps/maps.tar"> <arg line="archive --format=tar --remote=${mapsGitRepo}/eclipse.platform.releng.maps ${mapVersionTag} org.eclipse.releng" /> </exec> <untar src="${buildDirectory}/maps/maps.tar" dest="${buildDirectory}/maps" /> This doesn't seem to work as expected, if I look at the tar directly, it looks right, but if I look at the what ant untar created, some files are in the wrong place (at "top" of file hierarchy, instead of in the folder they belong in). So, I added a bunch of debug and "failonerror" condidtions, but still see the incorrect behavior. So ... I'm just wondering we do all the tar stuff to begin with? We've already cloned the repo, checked out the right version, why not copy over that we need? Unless there's some reason that's a bad idea, that's what I'm I'm going to try next. For what its worth, we we tag the maps, we seem to use the repo we cloned and checked out, so ... while very small window of time ... seems like there is opportunity to tag a different set of maps than what was actually used, from the tarred version.
Just to keep track of observations, if/when we do "tag maps", we use a pattern as below. As we discussed in bug 366279 comment 7 ... we may need to specify the version we checked out, for the case where we might have multiple versions checked out (though, admittedly, currently we have "just one" per build, but believe a long term goal is to "share" a more common location? <target name="tagMapFiles" if="tagMaps"> <exec executable="git" dir="${mapsGitRepo}/eclipse.platform.releng.maps"> <arg line="tag ${buildId}" /> </exec> <exec executable="git" dir="${mapsGitRepo}/eclipse.platform.releng.maps"> <arg line="push origin ${buildId}" /> </exec> </target> But, also, if I am beginning to understand autotagging ... we'd never tag the maps file from this part of the customTargets.xml code as long as we use autotagging?
(In reply to comment #0) > All that seems to work, and appears "on disk" as expected. Now here's here it > gets fuzzy (at least to me) > > We use git to get a new?! copy of the maps files, all tarred up and then untar > them into our maps location. > > <exec executable="git" dir="${buildDirectory}/maps" > output="${buildDirectory}/maps/maps.tar"> > <arg line="archive --format=tar > --remote=${mapsGitRepo}/eclipse.platform.releng.maps ${mapVersionTag} > org.eclipse.releng" /> > </exec> > <untar src="${buildDirectory}/maps/maps.tar" dest="${buildDirectory}/maps" /> > > > This doesn't seem to work as expected, if I look at the tar directly, it looks > right, but if I look at the what ant untar created, some files are in the wrong > place (at "top" of file hierarchy, instead of in the folder they belong in). > FWIW, I took out this "get new tar and untar it" section and put in a "copy what we've already cloned" section and it worked as expected: <copy overwrite="true" failonerror="true" verbose="true" todir="${buildDirectory}/maps"> <fileset dir="${mapsGitRepo}/eclipse.platform.releng.maps"> <include name="**" /> </fileset> </copy> code review welcome. I don't plan any mass changes at the moment, since essentially we want to get these map directories once, early in the process, so I've fixed that one "early" method. The rest can wait, though, I think they are essentially incorrect (we don't actually use them). I'll admit I'm not positive how this interacts with "autotagging". We could do a better job of coordinating that, perhaps using/getting from same location where we've already fetched and tagged maps files, and in those case where we have already "auto-tagged" the maps files with, say ${buildId} when we should use that tag ${buildId}, instead of ${mapVersionTag} to "get" (checkout) our working copy of the map files. <exec failonerror="true" executable="git" dir="${mapsGitRepo}/eclipse.platform.releng.maps"> <arg line="checkout ${mapVersionTag}" /> </exec> would become <exec failonerror="true" executable="git" dir="<gitCache location of>/eclipse.platform.releng.maps"> <arg line="checkout ${buildIdOrWhateverItIsWeUse}" /> </exec>
I'll transform this bug to track/discuss how "getMaps" can better interface with autotagging. As I understand it, the autotagging methods updates the maps files, based on changes it sees, then tags those map files with ${buildId}. It makes sense for the eclipsebuilder to have/get its "own copy" of the map files, just for "safety" but, it should just get a copy based on the $buildId produced by the autotagger, probably from local clone, for efficiency, during a "live build". But, have the ability to get from remote repo, in the case of reproducing a build at a later time? Other ideas/suggestions welcome. As far as I know, this isn't urgent, and could wait until after the release. Let me know if I am misunderstanding.
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. If the bug is still relevant, please remove the "stalebug" whiteboard tag.
