support Linux and macOS curl binaries #53
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vszakats
changed the title
vszakats
force-pushed
the
port-to-unix
branch
7 times, most recently
from
d651c9a
to
259872f
Compare
Also add future targets: linux and mac
…KE_RC_FLAGS [ci skip]
Also document a zlib bug showing up as a CMake warning after setting `CMAKE_INSTALL_LIBDIR`.
This is a bad solution. Tried `--force` and `--overwrite` without success. ``` ==> Installing llvm dependency: python@3.11 ==> Pouring python@3.11--3.11.5.monterey.bottle.tar.gz Error: The `brew link` step did not complete successfully The formula built, but is not symlinked into /usr/local Could not symlink bin/2to3 Target /usr/local/bin/2to3 already exists. You may want to remove it: rm '/usr/local/bin/2to3' [...] Possible conflicting files are: /usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3 ``` Also avoid updating dependencies of preinstalled packages.
Fixes this error with the ancient bash (3.x) shipping with macos: ./_dl.sh: line 222: opts[@]: unbound variable When `opts` is an empty array.
`linux-headers` package required to fix openssl builds to avoid:
```
../crypto/mem_sec.c:60:13: fatal error: linux/mman.h: No such file or directory
60 | # include <linux/mman.h>
| ^~~~~~~~~~~~~~
```
Alternative workarounds:
- disable 'secure-memory' feature with `no-secure-memory` `./Configure` option
- CC="${CC} -static -idirafter /usr/include/ -idirafter /usr/include/$(uname -m)-linux-gnu/"
Ref: https://github dot com/openssl/openssl/issues/7207#issuecomment-880121450
Without this, Homebrew or components are picked up from there instead
of the path we explicitly specify. And only at compile time, not link
time, resulting in confused builds using headers from one build of a
dependency and libs from another. Seen with OpenSSL (quictls).
That said the mac builds produced by autotools and cmake are very
different (even with an identical `-V` output).
More info:
I'm running into the OpenSSL-confusion issue in local tests with
autotools. That is on macOS with Apple's CommandLineTools.
Despite passing the correct `./configure` option, what actually gets
picked up is the Homebrew OpenSSL under `/usr/local`. But not at link
time, where it want to link to the correct, custom OpenSSL. The latter
is with `no-engine`, the Homebrew one is with, so it fails to link.
```
$ ./configure [...] --with-openssl=/curl-for-win/quictls/x64-sys/usr
[...]
CPPFLAGS: -DNDEBUG -DHAS_ALPN -DCURL_STATICLIB -isystem /curl-for-win/quictls/x64-sys/usr/include
LDFLAGS: -framework CoreFoundation -framework SystemConfiguration -framework CoreFoundation -framework Security -L/curl-for-win/quictls/x64-sys/usr/lib
[...]
/bin/sh ../libtool --tag=CC --mode=compile clang -arch x86_64 -DHAVE_CONFIG_H \
-I../../include -I../lib -I../../lib -DBUILDING_LIBCURL -DCURL_HIDDEN_SYMBOLS \
-DNDEBUG -DHAS_ALPN -DCURL_STATICLIB \
-isystem /curl-for-win/quictls/x64-sys/usr/include -fvisibility=hidden \
-mmacosx-version-min=10.13 -O3 -Qunused-arguments -pedantic -Wall -Wextra [-W...]
-c -o vtls/libcurl_la-openssl.lo `test -f 'vtls/openssl.c' || echo '../../lib/'`vtls/openssl.c
[...]
. vtls/openssl.h
.. /usr/local/include/openssl/ssl.h
... /usr/local/include/openssl/macros.h
.... /usr/local/include/openssl/opensslconf.h
..... /usr/local/include/openssl/configuration.h
.... /usr/local/include/openssl/opensslv.h
[...]
ld: -undefined dynamic_lookup cannot be used to find '_ENGINE_by_id' in dylib in dyld shared cache for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
Changing `-isystem` to `-I` fixes it, though I don't yet know how to
teach autotools to do that. Another fix is to pass this flag to clang:
`-isysroot "$(xcrun --show-sdk-path)"` (this drop /usr/local from the
list of built-in header search paths)
TODO: might make sense to make a small script to handle these combinations automatically, like `_clean-bin.sh` does for executables.
Before this patch these option caused the llvm builtins rt library to be included as a whole inside libcurl.a when using autotools. Resulting in these issues: ``` *** Warning: Linking the shared library libcurl.la against the *** static library /usr/lib/llvm-13/lib/clang/13.0.1/lib/linux/libclang_rt.builtins-aarch64.a is not portable! ``` ``` + llvm-strip-13 --enable-deterministic-archives --strip-debug a64-musl/usr/lib/libcurl.a llvm-strip-13: error: unsupported object file format ``` This patch changes the way we pass that library: Instead of passing it as a full name with an absolute path (affectively passing it as a object to include as-is), we pass it using -L and -l like any other lib. autotools still complains (as of bullseye, can't see this on trixie), but the build succeeds: ``` configure: using LDFLAGS: -Wl,-z,relro,-z,now -fuse-ld=lld-13 -Wl,-s -Wl,--build-id=none -static -nostdlib -nodefaultlibs -nostartfiles -L/usr/lib/aarch64-linux-musl /usr/lib/aarch64-linux-musl/crt1.o /usr/lib/aarch64-linux-musl/crti.o -lc -L/usr/lib/llvm-13/lib/clang/13.0.1/lib/linux -lclang_rt.builtins-aarch64 /usr/lib/aarch64-linux-musl/crtn.o -Wl,-Map,curl.map configure: LDFLAGS note: LDFLAGS should only be used to specify linker flags, not libraries. Use LIBS for: -lc configure: LDFLAGS note: LDFLAGS should only be used to specify linker flags, not libraries. Use LIBS for: -lclang_rt.builtins-aarch64 configure: WARNING: Continuing even with errors mentioned immediately above this line. ```
"all executables" include the easily forgotten ones built by autotools
configure phase, even when not building any actual useful executable
for an autotools project (only static libs, like we do here in most
cases). To have `./configure` test results consistent with build-time
behaviour, we must always populate `LDFLAGS` and `LIBS` with custom
options. We use global custom options to override glibc and libgcc for
debian llvm musl builds.
Option mismatch at detection and use manifested in this build issue for
example, when building libressl (arc4random_buf detected in configure
with glibc, but indeed missing because of using musl headers. configure
doesn't need this declared in the headers (only libs!) to detect it
successfully):
```
../../crypto/bn/bn_rand.c:153:2: error: call to undeclared function 'arc4random_buf'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
arc4random_buf(buf, bytes);
^
```
This also fixes these warnings with autotools debian llvm musl builds:
```
configure: LDFLAGS note: LDFLAGS should only be used to specify linker flags, not libraries. Use LIBS for: -lc
configure: LDFLAGS note: LDFLAGS should only be used to specify linker flags, not libraries. Use LIBS for: -lclang_rt.builtins-aarch64
```
Also in _ci-linux-debian.sh: - Allow CW_CCSUFFIX override. - Also restore libgcc install for non-musl builds.
Fixing (seen on bullseye): ``` + checksec --format=xml --fortify-file=x64-musl/usr/bin/curl <?xml version="1.0" encoding="UTF-8"?> <fortify-test name='x64-musl/usr/bin/curl' libc_fortify_source='yes' binary_compiled_with_fortify='no'> Command exited with non-zero status 1 ```
It builds correctly now. It also doesn't use nasm.
From a build perspective it behaves almost exactly like BoringSSL, and
we're assuming the same codepaths by default.
AWC-LC has a built-in way to skip building tests and command-line tools.
We do use that. It also has versioned releases and a way to retrieve
this version number at build-time, so no need for the commit hash hack
we use with BoringSSL.
It suffers from the same pthread dependency issue on Windows, by the
looks of its source code.
Actual testing on Windows reveals that the project no longer compiles
with llvm-mingw:
```
awslc/crypto/refcount_win.c:23:15: error: expected parameter declarator
static_assert(alignof(CRYPTO_refcount_t) == alignof(LONG),
^
```
It means we lost BoringSSL for Windows and also AWS-LC. Comments tell
that their team only tests Visual Studio, though they were open for
other compilers. That was in 2021.
This patch is experimental and AWS-LC support might be removed in the
future.
_umul128 should be supported by llvm, but apparently not on Windows? https://reviews.llvm.org/D25353 A similar issue has been fixed in BoringSSL on 2014-12-02: google/boringssl@af9d941
This option strips the UUID. On one hand this may cause problems down the line according to the man page, on the other, this UUID is calculated from the output, and doesn't contain randomness or depends on timestamps, meaning: it's reproducible.
When trying to run autoreconf for libssh2 and curl git repos: ``` + cd libssh2 + rm -r -f x64-musl bld + '[' -f configure ']' + autoreconf --force --install Can't exec "aclocal": No such file or directory at /usr/local/Cellar/autoconf/2.71/share/autoconf/Autom4te/FileUtils.pm line 274. autoreconf: error: aclocal failed with exit status: 2 ``` Ref: https://github.com/vszakats/curl-for-win/actions/runs/6392970919/job/17351293395#step:3:4308
We disabled it partly because in macOS CMake builds, LDAPS did not enable automatically. We wrongly assumed this also applies to autotools builds, but it turned out this was an issue with CMake builds only and fixed here: curl/curl#12006 (in curl 8.4.0) With this fix landed, we can re-enable it in CMake and re-enable with autotools for all curl versions. With 8.3.0 this will only enable LDAP, without LDAPS. The other reason for disabling it was that Apple deprecated the LDAP API in 10.10. To avoid extra deprecation warnings coming with LDAP on macOS, we target OS X 10.9 Mavericks instead of High Sierra. The build also runs fine with these warning present, but we only enable LDAP/LDAPS on macOS when targeting 10.9. Also fix `_OSVER` value for `10.[0-9]` inputs.
Unless building for cross-make compatibility tests. CMake is the only build method doing unity builds and these binaries will inherently be much different than non-unity ones. Do not enable when this is goal. EXPERIMENTAL. Unity mode has been under works for a while now, but it's not as thoroughly tested as regular builds. These builds require all static symbols to be unique throughout the whole curl project, which may fail in build configurations that are not CI tested and that have not been manually tested either. Regressions are also possible after curl updates. I expect this to improve over time.
|
Merged with a07dc75 (and the 146 commits before it). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add the ability to create Linux and macOS binaries. We are
keeping the name
curl-for-winthough. Non-Windowsbinaries are experimental at this point and they are not
official curl binaries for these platforms.
Highlights:
either Alpine, Debian bookworm / bullseye / trixie.
relro,_FORTIFY_SOURCE=2and-fstack-protector-allfor Linux.
(experimental)
both llvm-clang and Apple clang. These also feature LDAP/LDAPS
and SecureTransport support.
both daily and per-commit. This is subject to change.
(it requires MSVC, so Windows not supported for this backend.)
bldtstbuilds that use no 3rd-partydependencies except zlib. To make build tests/porting easier.
nozliboption to fully opt-out from3rd-party dependencies when used together with
bldtst.dos2unixandunix2dostools.realpathcommand.gpgdiron abort.Caveats:
some of these listed in the TODO section of
_build.sh.not functionally tested yet.
reproducible builds are not yet possible for macOS. (unless
using llvm/clang.)
Linux:
macOS (quictls + SecureTransport):
Closes #53
Merge without squashing!