Detecting OpenSSL's version on Windows

[HertzDevil]

Crystal detects libssl and libcrypto's versions during compilation time to bind to their APIs correctly. This is normally achieved using pkg-config; Windows doesn't have it, but we currently control all third-party libraries distributed together with Crystal, at least for portable packages (insofar as places like neatorobito/scoop-crystal and crystal-lang/install-crystal already rely on the Windows CI build artifacts). That means we can hardcode OpenSSL or LibreSSL's version number somewhere.

The current situation is to set the CRYSTAL_OPENSSL_VERSION environment variable. One way to improve it is writing the version number in a file called openssl_VERSION (or libressl_VERSION) placed somewhere under Crystal::LIBRARY_PATH, so that it can be read with macros:

{% ssl_version = nil %}
{% for dir in Crystal::LIBRARY_PATH.split(';') %}
  {% unless ssl_version %}
    {% config_path = "#{dir.id}\\openssl_VERSION" %}
    {% if config_version = read_file?(config_path) %}
      {% ssl_version = config_version.chomp %}
    {% end %}
  {% end %}
{% end %}
{% ssl_version ||= "0.0.0" %}

This is the original approach pursued in #11477.

[oprypin]

Thanks for resuming the discussion.
#11477 (comment)

Yes, this needs to be solved. If we don't come up with an even better solution than the one you initially suggested with storing a version file, I think we should actually revert directly to that solution.

One user of https://github.com/crystal-lang/install-crystal already expressed confusion. Quote:

on windows I have the following error using crystal-lang/install-crystal:

Error: Cannot determine OpenSSL version, make sure the environment variable `CRYSTAL_OPENSSL_VERSION` is set

And indeed that's all you see whenever trying to require OpenSSL currently. And despite the fact that this never worked in the first place, now it doesn't work in a confusing way.

And I don't think that hardcoding this variable in downstream solutions like install-crystal would be a good solution at all.

[straight-shoota]

We have a unique situation regarding libraries: On Linux, BSDs and Macos, it's common that libraries are globally available and discoverable on the system. And the OS usually provides an easy tool for installing dependency packages. So we don't have to care about providing libraries with Crystal, just naming them as dependencies is usually enough.

That's different on Windows. AFAIK, except for system libs, there are typically no libraries shared between different applications. Programs usually bring all their dependencies themselves, including shared libraries. Even package managers like chocolatey and scoop don't provide libraries, they seem to be used only for applications.

So I would assume that we need distribution packages for Windows to ship with all libraries used by stdlib. And yes, that means we're in full control of which versions are available. I suppose you can still provide other versions of the libraries, but you may need to change some configuration for that.

Placing the information about library versions in the lib directory sounds like a good idea. I'm wondering though, if maybe we should use a single file for that, or have one for each library. We need version information for libllvm as well.
But I guess individual files is more flexible. You can store them in the same location as the library itself, if you use other than the default. And it should be picked up correctly.

[HertzDevil]

llvm-config.exe is more or less guaranteed to exist if the compiler is built on Windows, unlike pkg-config, so I just set the LLVM_CONFIG environment variable. If we want to avoid that too, then llvm-config.exe needs to be in PATH and find-llvm-config would need a CMD / Powershell equivalent.

But I guess the libllvm version number is relevant if a project uses the LLVM::* types directly? Does it make sense not to have the full LLVM suite in that scenario?

[straight-shoota]

Yeah, I'm talking about distribution, not the compiler build environment. LLVM is part of stdlib, so I would expect that we should ship libllvm as well for all batteries included.

I'm not sure we actually need llvm-config for that. When we're packaging the lib ourselves, we should have all the configuration info.

[HertzDevil]

We would essentially need to replicate this part without llvm-config:

crystal/src/llvm/lib_llvm.cr

Lines 11 to 17 in 07b2065

	@[Link(ldflags: {{"`#{LibLLVM::LLVM_CONFIG} --libs --system-libs --ldflags#{" --link-static".id if flag?(:static)}#{" 2> /dev/null".id unless flag?(:win32)}`"}})]
	lib LibLLVM
	VERSION = {{`#{LibLLVM::LLVM_CONFIG} --version`.chomp.stringify}}
	BUILT_TARGETS = {{ (
	env("LLVM_TARGETS") || `#{LibLLVM::LLVM_CONFIG} --targets-built`
	).strip.downcase.split(' ').map(&.id.symbolize) }}
	end

On my machine this gives:

• %LLVM_CONFIG% --libs: 73 static libraries totalling 284 MB (more for every extra target architecture built)
• %LLVM_CONFIG% --system-libs: psapi.lib shell32.lib ole32.lib uuid.lib advapi32.lib
• %LLVM_CONFIG% --ldflags: -LIBPATH:...\llvm\lib
• %LLVM_CONFIG% --targets-built: X86
• %LLVM_CONFIG% --version: 13.0.0

If we move to dynamic linking on Windows then most likely there will be a single libLLVM.dll instead. In theory, each option here can be encoded in a separate text file.

[oprypin]

For now, regarding OpenSSL, shall we commit the file-based solution?

https://github.com/crystal-lang/crystal/compare/0e01d43033b85ab577ac9721e138ea200b100665..0e01d43033b85ab577ac9721e138ea200b100665%7E

[oprypin]

@HertzDevil , please indeed open a pull request with such content.

https://github.com/crystal-lang/crystal/compare/0e01d43033b85ab577ac9721e138ea200b100665..0e01d43033b85ab577ac9721e138ea200b100665%7E

[HertzDevil]

I think the LLVM static libraries would be prohibitively large compared to the rest of the files we distribute. Most people won't use the LLVM types in the standard library. It is a different story if we manage to support DLLs though.