Community
Participate
Working Groups
In the Paho project we now have some C components, and more may be coming in the future. I would like to know if there is a plan for cross-platform building of C components (I've not been able to find one). If I should be asking this question somewhere else (a mailing list/forum?) please tell me. I work for IBM, and we currently build the components for Windows, Mac and other Unixes on our servers. We are happy to continue with this approach - but I am wondering if the Eclipse build infrastructure might support any of this now or in the future? For Linux, we can look to getting our components into Linux distributions. Basic build and testing can be done on the current Hudson slaves. There are Hudson slaves for Windows and Mac testing. Is using them for basic building and testing of C components appropriate? (For Windows, using Microsoft C/C++ compilers is the norm). Again, apologies if I'm asking in wrong place. This does have the advantage of easily referring other people to the answer. Thanks. Ian
I just happened to be looking into cross compiling for a hobby of mine and thought I'd mention this cross compiler project for Windows binaries on Linux called mxe [1]. Might be worth looking into. [1] http://mxe.cc/
> Again, apologies if I'm asking in wrong place. This does have the advantage > of easily referring other people to the answer. This is exactly the right place, and mostly for the reason you've mentioned. Cross-compiling would definitely be the best solution. As we are a vendor-neutral OSS shop, we use only OSS on our infrastructure. We do employ Windows and Mac on two Hudson slaves for the sole purpose of performing UI testing, which Linux cannot replicate.
Thanks, that answers the main question :-) I've used cross-compilers for Linux in the past, using crosstool-ng http://crosstool-ng.org/. Thanks for the pointer to MXE. I'm guessing that as projects at Eclipse contain more non-Java code, this question will become more common.
Just to add another potential tool to the list. There is also "xchain" for building Mac binaries on Linux. Apparently it's based on what the OpenTTD project uses. https://github.com/tatsh/xchain
As well as the existing C client in Paho this will also apply to: - the proposed Mosquitto project (C building on Windows/Mac) So this question will now also apply to a .Net MQTT
Wrong key hit while typing my last comment. As well as the existing C client in Paho the question of proprietary build tools will also apply to: - the proposed Mosquitto project (C building on Windows/Mac) - a yet to be contributed .Net MQTT client in Paho - Objective C MQTT client in Paho (if it gets legal approval) A discussion between Mike Milinkovich and Wayne Beaton indicates that we should be able to get proprietary build tools in the Eclipse infrastructure, if those tools are ones which an open source developer would use on a proprietary OS. These tools would be: - Visual Studio for Windows (C and C# components at a minimum) - Xcode for Mac If there's anything I can do to help...
Ian, So here is what we would like you to do to help us out. First off, let me say that we do not want to install IDEs on our build servers. So we don't want to install Visual Studio or Xcode. We do, however, want to install the right headless compiler + SDK on the build servers so you can compile your code. So could you help us out by telling us exactly what we need to get on Windows and the Mac to get you going? We have great contacts at Microsoft and Apple, so I'm not worried about getting these things once we know what to ask for. (If I'm wrong, I'll let you know!) Make sense?
Sure Mike. I'll start off by explaining that the only way I have installed headless compilers on both Windows and MacOS in recent years is to install the IDEs. The headless compilers come with them. In about 2008 I could install gcc for MacOS, but in recent years I've had to install Xcode. I'll find out whether there is a way to install the headless compilers on their own, but it would not surprise me if we can't. Would it be a problem installing the IDEs even if they are never used? Disk space?
*** Bug 372091 has been marked as a duplicate of this bug. ***
(In reply to Ian Craggs from comment #8) > Would it be a problem installing the IDEs even if they are never used? Disk > space? That's Denis' call. I hope that as long as GUI's are never used it wouldn't be a problem.
> Would it be a problem installing the IDEs even if they are never used? Disk > space? Our experience with apps designed for GUIs is that the CLI tools are afterthoughts, and at some point, they will call on the GUI. We regularly see builds hang on Windows and Mac because there's a dialog asking for user input, and this is problematic. The other issue is the multi-user nature of Hudson and our build farm. Even Eclipse (as an IDE) is not designed to be run by multiple users at the same time. Dedicating one server for only one concurrent build becomes costly. Having true command-line tools that can be called by Hudson or from the comman line would be the most reliable way to make this work.
FWIW, I'm trying real hard not to stonewall this, but my fear is that you'll eventually be asking me to open RDP and VNC to the world...
I think I may have missed out on some discussion so maybe someone could get me up to speed. Has cross compiling been removed as a potential option? or is it still worth considering / investigating as an alternative option?
Denis. Ok, I understand the problem of not wanting builds to hang. I'm totally with you on that one. For C/C++ on Windows, we use the cl.exe command line compiler, for our automated builds in IBM too. I've been using this since the mid-90s. For C/C++ on the Mac, we use Apple's build of gcc. For both of these compilers, you used to be able to get them standalone. But I had to change to getting them by installing the IDEs in the last 5 to 10 years, as Microsoft and Apple put more emphasis on their graphical IDEs. But these compilers are true command line programs. I can't see either Microsoft or Apple wanting to remove these programs either, as people like you and me will always want automated builds. I'll check with the build guys here to see if how they install these tools. I don't understand the connection to VNC/RDP :-)
Thanh. It was I who suggested that cross-compiling was not a profitable use of time for people working on projects that target proprietary operating systems, and for which proprietary build tools are the norm (C/C++, .Net, Apple Objective C). There are some important proprietary platforms which we want to target in the m2m/mobile world: - Windows desktop (including .Net) - Windows Phone - Windows CE/Embedded - MacOS - iOS Cross-compiling has the disadvantages of: 1) it can be (very) time-consuming to get right 2) the cross-compiled libraries may not be compatible with applications built with the proprietary tools 3) we wouldn't be using the same tools as just about every other developer on the planet, for these platforms. When they build our programs, they will want to use the standard tools 4) it will complicate build tests as the tests will have to be run on a different slave to where they are built I wasn't suggesting that the Eclipse organization *should* use Microsoft and Apple development tools. But I am suggesting that in Paho and Mosquitto at least we will want to use them, and that if we can't use the Eclipse build servers, then I would look at others, say at IBM. Does that help?
(In reply to Ian Craggs from comment #15) > Does that help? Yes, thanks for the explanation.
> I don't understand the connection to VNC/RDP :-) If your build stalls because some application is showing some blocking dialog (such as, "Unable to find mouse. Please click Ok") you'll want to see the screen.
Thanks Denis for the explanation. I've never had that problem with builds I've put together for Windows because I use the command line build and compile. But I can see it's a risk.
Just to cross-reference, see also bug 405753 which is similar question about some of the Eclipse Platform "natives".
Just moving to Hudson since that will likely be the vehicle that provides this service.
Ian, can you point me to the repository and any documentation on how to build your c projects? I'd like to look into testing building one of your projects as a POC for setting up a Windows build slave. Thanks in advance.
Thanh, the repository for the C client is: git.eclipse.org/c/paho/org.eclipse.paho.mqtt.c.git/ build instructions for Windows are on this page: http://www.eclipse.org/paho/clients/c/ I am interested to know if you find the build instructions adequate.
(In reply to Ian Craggs from comment #22) > build instructions for Windows are on this page: > > http://www.eclipse.org/paho/clients/c/ > Thanks, I was able to build on Linux just fine, however for Windows I'm missing some dependencies. I guess on Linux I must have already had them installed since it's my regular use system, however on Windows I seem to be missing "openssl/ssl.h". I'll try to get through this and note down any dependencies that should be documented along the way. Do you have any recommendations on how I should get OpenSSL for Windows?
(In reply to Thanh Ha from comment #23) > (In reply to Ian Craggs from comment #22) > > build instructions for Windows are on this page: > > > > http://www.eclipse.org/paho/clients/c/ > > > > > Thanks, I was able to build on Linux just fine, however for Windows I'm > missing some dependencies. I guess on Linux I must have already had them > installed since it's my regular use system, however on Windows I seem to be > missing "openssl/ssl.h". > I got this installed but the build isn't able to pick it up. How do I tell Windows builds where the headers are?
(In reply to Thanh Ha from comment #24) > (In reply to Thanh Ha from comment #23) > > (In reply to Ian Craggs from comment #22) > > > build instructions for Windows are on this page: > > > > > > http://www.eclipse.org/paho/clients/c/ > > > > > > > > > Thanks, I was able to build on Linux just fine, however for Windows I'm > > missing some dependencies. I guess on Linux I must have already had them > > installed since it's my regular use system, however on Windows I seem to be > > missing "openssl/ssl.h". > > > > I got this installed but the build isn't able to pick it up. How do I tell > Windows builds where the headers are? The paths are currently hardcoded in the solution to: c:\work\openssl\windows_ia32\include with the library files libeay32.lib and ssleay32.lib in: c:\work\openssl\windows_ia32 just because that was quickest at the time. If that's impossible for you I'll parameterize the solution (I'm assuming that must be possible).
(In reply to Ian Craggs from comment #25) > The paths are currently hardcoded in the solution to: > > c:\work\openssl\windows_ia32\include > > with the library files libeay32.lib and ssleay32.lib in: > > c:\work\openssl\windows_ia32 > > just because that was quickest at the time. If that's impossible for you > I'll parameterize the solution (I'm assuming that must be possible). Thanks, this helps. I also had the 64 bit version. I'll try installing to this path.
(In reply to Thanh Ha from comment #26) > (In reply to Ian Craggs from comment #25) > > The paths are currently hardcoded in the solution to: > > > > c:\work\openssl\windows_ia32\include > > > > with the library files libeay32.lib and ssleay32.lib in: > > > > c:\work\openssl\windows_ia32 > > > > just because that was quickest at the time. If that's impossible for you > > I'll parameterize the solution (I'm assuming that must be possible). > I was able to get further (and turns out Windows 7 supports symlinks so I symlinked my install path to your hardcoded directories to get it working) Unfortunately I'm getting numerous unresolved external symbol errors now when building. I'm guessing perhaps it has to do with the version of OpenSSL I installed. Which version are you using to build on Windows? I installed the latest 1.0.1g.
(In reply to Thanh Ha from comment #27) > (In reply to Thanh Ha from comment #26) > > (In reply to Ian Craggs from comment #25) > > > The paths are currently hardcoded in the solution to: > > > > > > c:\work\openssl\windows_ia32\include > > > > > > with the library files libeay32.lib and ssleay32.lib in: > > > > > > c:\work\openssl\windows_ia32 > > > > > > just because that was quickest at the time. If that's impossible for you > > > I'll parameterize the solution (I'm assuming that must be possible). > > > > I was able to get further (and turns out Windows 7 supports symlinks so I > symlinked my install path to your hardcoded directories to get it working) > > Unfortunately I'm getting numerous unresolved external symbol errors now > when building. I'm guessing perhaps it has to do with the version of OpenSSL > I installed. Which version are you using to build on Windows? > > I installed the latest 1.0.1g. Same here, 1.0.1g. What sort of entries are missing - compile or link time? Maybe you could send me the log.
Created attachment 244015 [details] log Attached build logs. The errors seem to be with linking. I copied the 2 files libeay32.lib and ssleay32.lib to C:\work\openssl\windows_ia32 but it doesn't seem to work.
(In reply to Thanh Ha from comment #29) > Created attachment 244015 [details] > log > > Attached build logs. The errors seem to be with linking. > > I copied the 2 files libeay32.lib and ssleay32.lib to > C:\work\openssl\windows_ia32 but it doesn't seem to work. Those look pretty fundamental. How about trying with the OpenSSL libraries that we have built? They are 13M and 2M in size, so I don't know about attaching them to this bug. I did try to send them to my home directory on build.eclipse.org temporarily so you could get them, but that stopped before finishing, and now I can't get log back in... I suspect I might have exceeded a quota (oops), unless it's a coincidence. Where's the best place to put them for you?
(In reply to Ian Craggs from comment #30) > They are 13M and 2M in size Wow really? my copies seem to be 763KB and 65KB > Where's the best place to put them for you? Anywhere on build.eclipse.org would be the best place for me actually but I believe Webmaster is working on that system right now and it seems to be down. You could try emailing them to me, not sure how well that will work...
(In reply to Thanh Ha from comment #31) > > Where's the best place to put them for you? > > Anywhere on build.eclipse.org would be the best place for me actually but I > believe Webmaster is working on that system right now and it seems to be > down. > Looks like build.eclipse.org is back online. Can you try there again?
Actually I figured it out! My copy of libeay32.lib and ssleay32.lib were from the 64-bit package. When I swapped it for the copy from the 32-bit package it started working. I'm able to successfully build the project now. I guess this project cannot be built with the 64-bit version of the libraries?
(In reply to Thanh Ha from comment #33) > Actually I figured it out! > > My copy of libeay32.lib and ssleay32.lib were from the 64-bit package. When > I swapped it for the copy from the 32-bit package it started working. > > I'm able to successfully build the project now. > > > I guess this project cannot be built with the 64-bit version of the > libraries? This particular project is set for 32-bit builds. I'd have to create another project for 64-bit builds.
(In reply to Ian Craggs from comment #34) > This particular project is set for 32-bit builds. I'd have to create > another project for 64-bit builds. Okay that's good to know. I'm not sure how Visual Studio handles additional paths and variables (is it all static or can you pass parameters?) but I think instead of these paths we should use: c:\work\openssl\windows_ia32\include -> c:\%OpenSSL_Path%\include c:\work\openssl\windows_ia32 -> c:\%OpenSSL_Path%\lib and maybe default %OpenSSL_Path% to C:\OpenSSL-Win32\ for 32-bit and C:\OpenSSL-Win64\ for 64-bit but allow it to be override-able via parameter. This way we can put in the instructions to install OpenSSL and if the user leaves defaults it works out of the box.
I think I found a decent solution. I will post a Gerrit patch soon. I can provide a patch for the build instructions too but the paho www.eclipse.org repo is not available in Gerrit. I guess I can attach that patch to this bug?
Patch for build code: https://git.eclipse.org/r/28135
Created attachment 244049 [details] patch
Just to document, this is what my HIPP Windows slave job calls as a shell script: SET OpenSSLDir=C:\common\OpenSSL-1.0.1g-Win32 CALL "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\vcvars32.bat" msbuild "Windows Build\Paho C MQTT APIs.sln" /p:Configuration=Release
Thanks. Out of interest, did you change the paths in the solution using the IDE or by directly editing the files?
(In reply to Ian Craggs from comment #40) > Thanks. Out of interest, did you change the paths in the solution using the > IDE or by directly editing the files? I modified the files by hand. I found that using the IDE always caused it to modify every line instead of just the lines I was interested in so it would have made change review confusing.
We cannot do Windows and Mac slaves on our infra. Please see bug 375991. It may be possible to connect HIPP to external slaves run on Cloud provider resources. Please see: https://wiki.eclipse.org/Hudson/Connecting_HIPP_to_an_external_slave
