add gcc and binutils packages #60

cosmicexplorer · 2018-03-21T22:28:59Z

gcc (like clang) is being provided for both OSX and Linux. binutils does not support OSX (and I've tried to change that, believe me), so the linker will stay an unsolved problem for now. The scripts for all 3 of these tools will mkdir -p the appropriate output directories and copy the packaged archive over to build-support/bin/${tool}/${arch}/${version}/${tool}.tar.gz (and also, all are being provided as tgz archives -- BinaryTool is great!).

The packages produced by these scripts have been used to successfully compile and link code in an extremely bare Linux environment. The same level of testing has not been done with OSX, but I have done my best to make as much of the script as cross-platform as possible so the only feature gaps are the ones in the software we're depending on.

Be forewarned that gcc can take literal days to build unless you dial up the parallelism (using MAKE_JOBS), but that you can't do that on OSX due to race conditions (unclear why, but it has always failed when I've tried).

Let me know if there's a ton more work to be done here or if these look alright. Obviously these packages are big, and they are also hard to slim down -- that's not saying it's impossible.

Also, here is my pants PR which consumes these subsystems. Please let me know if there are any challenges to getting either of these PRs merged that I haven't considered. Thanks!

I know the history is all screwy, github stopped letting me upload files > 100MB to this repo (even though I'd successfully uploaded them before e.g. clang?) and I'm not sure why. I can spend some effort making the version history cleaner to merge if necessary, especially for a repo with huge binaries where everything can be slow sometimes.

…ild#56)

…oaded are too big for their service

illicitonion

Looks great! Thanks for putting this together :) A few pretty minor comments

I think @stuhood was saying we're going to stop checking in binaries at all, which hopefully should make your pushing large files problem go away...

illicitonion · 2018-03-22T10:48:59Z

build-binutils-2.30.sh

+mkdir -p "$BINUTILS_BUILD_TMP_DIR"
+pushd "$BINUTILS_BUILD_TMP_DIR" # root -> $BINUTILS_BUILD_TMP_DIR
+
+curl -L -v -O "$BINUTILS_RELEASE_URL"


Throw in a --fail, and throughout the PR (which I really, really wish was the default)

illicitonion · 2018-03-22T10:55:01Z

build-clang-6.0.0.sh

@@ -0,0 +1,160 @@
+#!/bin/bash


It would be nice to re-use the same script for 5.0.1 and 6.0.0, given the differences seem to be:

Actual version numbers

A 'final' being present or absent in a string

Extracting two extra directories

I agree, just wasn't sure if that was something people would find useful. This works for clang 5.0.1 and 6.0.0 now using e.g. ./build-clang.sh osx 5.0.1 (or it worked when I last tried running it, will focus on correctness later today).

illicitonion · 2018-03-22T10:55:38Z

build-gcc-7.3.0.sh

+      --build="x86_64-apple-darwin${_osx_uname_release}"
+      --host="x86_64-apple-darwin${_osx_uname_release}"
+      --target="x86_64-apple-darwin${_osx_uname_release}"
+      --with-system-zlib


Is statically linking zlib an option? Fewer system deps are better imo...

Agreed, that was ripped from the homebrew recipe without further thought. I have since tried investigating how difficult it would be to build all of these from the CentOS docker image, which changed these again (this file is now build-gcc.sh).

illicitonion · 2018-03-22T10:57:05Z

build-gcc-7.3.0.sh

+## Some functions, despite bash making it unnecessarily hard to use them.
+
+# Check if a directory exists, and error out or return its absolute
+# path. Passing -f allows testing a file as well.


-e just tests "exists" - is -d vs -f actually a distinction you care about?

Nope! And random meaningless arguments in shell scripts get confusing pretty fast, so a great point to note. See get_existing_absolute_path in utils.bash for what this function ended up turning into.

cosmicexplorer · 2018-03-23T11:08:42Z

Changed everything a lot, now we have build-{clang,binutils,cmake,gcc}.sh which can be invoked with a target platform (osx or linux) and a release version (e.g. ./build-gcc.sh osx 7.3.0). This is partially inspired by @illicitonion's comment above about duplicating whole scripts for different versions, and partially because it would be extremely nice if we could compile everything from our trusty docker container (if we can't, then we have lost nothing whatsoever).

Each new script (and functions within those scripts) echoes some single distinct "product" to stdout upon completion, and tries to work within its own subdirectory whenever possible. This idiom may not be necessary but seemed to be make this type of script easier to write and follow.

The idea with this is that ./populate-native-toolchain.sh can be edited and then invoked to fill out the entire directory structure for these scripts. Like anything with enough shell scripts, one begins to wonder whether another language is necessary -- I don't think we're there yet, but I could be wrong. If you run ./populate-native-toolchain.sh you'll find that it fails (that or I'm being rate-limited by the gcc mirror). I haven't verified that all the combinations work, and will do that ASAP -- just want to know whether this seems like a potentially-useful approach for this PR.

illicitonion

This is genuinely some of the best crafted bash I've ever seen. Kudos!

I haven't run this, but I'm happy with it assuming it runs. Handing over to @stuhood for how things should look in a no-binaries-committed world :)

illicitonion · 2018-03-23T14:28:59Z

build-clang.sh

+
+## Interpret arguments and execute build.
+
+# $1 =


illicitonion · 2018-03-23T14:41:01Z

utils.bash

+  local -r path_arg="$1"
+  # -f is an "illegal option" for OSX readlink, so this may have e.g. `/../`
+  # within it.
+  local -r abs_path="$(pwd)/${path_arg}"


Nit: Check that path_arg is relative - if it's absolute, we don't want to double-absolutify it :)

illicitonion · 2018-03-23T14:43:42Z

utils.bash

+
+function var_is_set {
+  local -r var_name="$1"
+  [[ "${!var_name+x}" != '' ]]


What's the +x here?

(Also, this is perfectly fine and readable, but -z (empty string) and -n (non-empty) are handy tests to know about :))

cosmicexplorer and others added 23 commits February 26, 2018 16:43

add compiler.tar.gz for osx and linux using cmake 3.9.5

aee1351

refactor packaging to use common function for macos and linux

215c1de

...osx find doesn't support '+' in -exec

bcb3062

change package name to clang

ad67ff4

run make with one job due to race conditions

7911496

add comment describing the include/ subdir

7753893

refactor to add c++ stdlib

423efa4

add linux-only linker packaging script

e168d63

add binutils

e63816a

add gcc package script

4dbe79f

add binaries (REBASED: NO MORE BIG BINARIES HERE)

31b436a

make gcc script executable

d45d02a

update gcc

1852bee

make a small update to the gcc packaging script

aaa7eab

finish recovering a ton of work i lost

9b016be

minor comment tweaks

a5b67e8

Add buildozer 0.6.0-80c7f0d45d7e40fa1f7362852697d4a03df557b3 (pantsbu…

9bf404d

…ild#56)

Synchronize docker instructions with pantsbuild/pants. (pantsbuild#58)

1598e82

Add clang (pantsbuild#57)

f0ab05c

tone down the verbosity because i think it's making the build slower

3a2d27d

made a few docs updates

bbc3ae3

gotta figure out why github is telling me the files i had already upl…

a7aad2b

…oaded are too big for their service

osx gcc doesn't have a libexec dir........

662f917

cosmicexplorer mentioned this pull request Mar 21, 2018

combine Clang, GCC, Binutils, and XCodeCLITools to form the NativeToolchain subsystem and consume it in BuildLocalPythonDistributions for native code pantsbuild/pants#5490

Merged

illicitonion requested changes Mar 22, 2018

View reviewed changes

try revamping the toolchain generation for maintainability

10e4d82

illicitonion approved these changes Mar 23, 2018

View reviewed changes

cosmicexplorer added 2 commits March 23, 2018 15:47

change how we generate gzipped packages

5bffa44

make populate-native-toolchain.sh work

c3b9f0a

stuhood closed this Mar 29, 2018

cosmicexplorer mentioned this pull request Mar 30, 2018

there's no usable open-source OSX (Mach-O) linker right now pantsbuild/pants#5640

Closed

stuhood changed the base branch from gh-pages to master March 30, 2018 17:47

cosmicexplorer mentioned this pull request Mar 30, 2018

add gcc 7.3.0 and clang 6.0.0 cross platform, binutils 2.30 for linux, edit clang 5.0.1 to conform to #59 #62

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add gcc and binutils packages #60

add gcc and binutils packages #60

cosmicexplorer commented Mar 21, 2018 •

edited

Loading

illicitonion left a comment

illicitonion Mar 22, 2018

cosmicexplorer Mar 23, 2018

illicitonion Mar 22, 2018

cosmicexplorer Mar 23, 2018

illicitonion Mar 22, 2018

cosmicexplorer Mar 23, 2018

illicitonion Mar 22, 2018

cosmicexplorer Mar 23, 2018

cosmicexplorer commented Mar 23, 2018

illicitonion left a comment

illicitonion Mar 23, 2018

illicitonion Mar 23, 2018

illicitonion Mar 23, 2018

add gcc and binutils packages #60

add gcc and binutils packages #60

Conversation

cosmicexplorer commented Mar 21, 2018 • edited Loading

illicitonion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Mar 23, 2018

illicitonion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Mar 21, 2018 •

edited

Loading