WIP: Pywin32 updates

[joankaradimov]

Note: this package does not build correctly (and never did, I believe).

I've updated the code to the latest version. I've fixed a conflict with the Visual Studio SDK headers being used instead of MinGW's.

I also got a couple more modules to build. Most changes are not here, though. I'll make a PR directly in the pywin32 with my changes.

[ZachBacon]

according to the build log, it's erroring out at

error: no matching function for call to 'max(long long unsigned int, unsigned int)'
  790 |  cb = std::max((long long unsigned int)cb, SECURITY_DESCRIPTOR_MIN_LENGTH);

Could be a simple thing to fix potentially.

[joankaradimov]

Could be a simple thing to fix potentially.

There are a bunch of those and they are easy to fix, yes. But I don't want to patch the code. This should be fixed by adding CFLAGS="-include minmax.h" to the environment. That doesn't work, though. I'll try to figure out what wrong with it within the next few days.

P.S.
There are quite a few more errors after this one is fixed.

[ZachBacon]

My suggestion is patching, But, in a way that we don't break things for the standard way to compile it, but working towards getting it upstream. That way once it's done, there isn't a burden to maintain a patch

[joankaradimov]

I've further updated the version from 221 to 224.

but working towards getting it upstream. That way once it's done, there isn't a burden to maintain a patch

Yes, I'm definitely looking to do just that.

[joankaradimov]

Update:
I've submitted a PR to pywin32: mhammond/pywin32#1371

[lazka]

Update:
I've submitted a PR to pywin32: mhammond/pywin32#1371

👍

[joankaradimov]

I've made some more PRs in pywin32. Some of them - already merged:

Fix char/wchar/tchar inconsistencies in PythonService.cpp
Fix gcc/mingw errors for PyWinObject_From(LARGE|ULARGE)_INTEGER
Update some mingw-specific macros
Make some compiler options work with mingw too

And with #5659 fixed, I should be able to have something working within a few days.

[Alexpux]

@joankaradimov is any progress on this?

[joankaradimov]

I have the package compiling, installing, and running locally. I was able to use the emrun command in emscripten that relies pywin32.

A couple of modules rely on MFC and cannot be fixed. At least not any time soon. I will replace them with code that raises NotImplementedError.

Additionally I intend to wait for a new version of pywin32 to be released. It should happen very soon. It has a bunch of my MinGW compatibility fixes in it.

Another blocker is #5616. I use a patched mingw-w64-crt-git locally.

Right now I'm working on some SEH code. Luckily it's C++ code and I should be able to use _set_se_translator.

[ZachBacon]

@joankaradimov how goes progress on this front? I see your PR has been accepted upstream, but are you still working on the SEH code?

[joankaradimov]

@ZachBacon Yes, I'm still working on it. It's a bit of a mess. I'm unable to link to _set_se_translator. The symbol is decorated with MSVC's name mangling scheme. There are a few things I want to try before giving up and trying another approach.

Other than that - I've opened another PR in pywin32.

[joankaradimov]

I've updated to version 225, squashed all commits up to that point, and I've added myself as a maintainer (I don't know if Ray Donnelly has interest in maintaining this; maybe I should drop a mail).

I'm still struggling with __try/__except blocks. Turns out _set_se_translator is useless in MinGW, since the equivalent of MSVC's /EHa does not exist.

[mingwandroid]

Hi @joankaradimov, I don't have a lot of time to maintain things on MSYS2 unfortunately. Thank you for helping out though, it is very much appreciated.

[joankaradimov]

This turned out to be way more complex and time consuming then I thought.

I experimented with vectored exception handling and came up with nothing.

I read the LibSEH code and documentation, but the documentation says it does not support 64 bit SEH. It also looks too complex. I don't like the idea of downloading all of that in a PKGBUILD script.

Then I decided to go with mingw's excpt.h and its __try1 and __except1 macros. But they didn't work. The linker kept insisting that its l_startw and l_endw labels are duplicate symbols (even though they come from different functions). On top of that they differ syntactically from what's in MSVC++.

Eventually I decided to take a shot at my own implementation of __try/__except. I managed to get something working on x64. I'm still working on the x86 code. But at least the compilation passes that phase now. In any case I'll keep improving the SEH handling.

[Alexpux]

@joankaradimov any news?

[joankaradimov]

@Alexpux I have a handling of 64-bit SEH that I am happy with. I'm still working on an implementation for 32-bit, but I don't think that's a blocker -- it can be patched in with a later release.

Additionally I had another PR merged: mhammond/pywin32#1444

Right now I am waiting for pywin32 228 to be released. It should be fairly soon. It will include another MINGW compatibility fix.

It will also drop compatibility with Python 2. I notice a lot of MSYS2 packages have already dropped Python 2. I believe that should also be done here.

[joankaradimov]

I've updated to version 228. Some new errors are popping up now :(

I'm working on fixes (and potentially on backporting as much as possible to pywin32)

[joankaradimov]

This is on hold again, unfortunately. After building and installing I'm experiencing the problems described here. It appears that pywin32 has problems with Python 3.8.

On top of that some of the modules cannot be compiled. But that is expected for the moment. Here are some of the failures:

 win32help: No library 'htmlhelp'
 adsi: No library 'ADSIID'
 axdebug: No library 'axscript'
 exchange: No library 'Ex2KSdk'
 exchdapi: No library 'Ex2KSdk'
 bits: No library 'Bits'
 ifilter: No library 'ntquery'
 directsound: The header 'dsound.h' can not be located.
 win32ui: The header 'afxres.h' can not be located.
 win32uiole: The header 'afxres.h' can not be located.
 dde: The header 'afxres.h' can not be located.
 Pythonwin: The header 'afxres.h' can not be located.

Additionally - I've disabled two more modules - mapi and propsys. I'm getting some linker errors for missing symbols. These would need to be resolved separately.

P.S.
I built and tested some basic calls with Python 2. They worked. So it's not very likely that any of the patches cause the issue. But for this PR I've removed the Python 2 build. I can bring it back if someone needs it.

[lazka]

So, should we just merge this as is, so it doesn't bitrot even more, or wait?

[joankaradimov]

There's a problem in the installation I haven't fixed yet.

I'd say we wait for the next version. It should drop Python 2 support and rewrite most of setup.py. I hope it fixes the aforementioned installation problem. It should help drop/simplify some of the patches too.

It will also include another MINGW compatibility fix I made in pywin32.

[lazka]

ok, thanks for the update

[podsvirov]

Just noticed that the b300 is released. Perhaps we can get some progress here?

[mkhon]

pywin32.patch.txt

This patch fixes the build with the latest windmc.exe (missing trailing LF causes windmc to omit #endif in the generated .h file)
Also fixes link errors due to missing symbols

[mkhon]

Another thing is that os.add_dll_directory() in mingw python is somehow broken. So I had to add this patch:

--- 000/win32/Lib/pywin32_bootstrap.py	2020-06-13 12:39:03.000000000 +0700
+++ 001/win32/Lib/pywin32_bootstrap.py	2021-06-20 15:15:20.687719200 +0700
@@ -28,7 +28,7 @@
             os.add_dll_directory(pywin32_system32)
         # This is to ensure the pywin32 path is in the beginning to find the
         # pywin32 DLLs first and prevent other PATH entries to shadow them
-        elif not os.environ["PATH"].startswith(pywin32_system32):
+        if not os.environ["PATH"].startswith(pywin32_system32):
             os.environ["PATH"] = os.environ["PATH"].replace(os.pathsep + pywin32_system32, "")
             os.environ["PATH"] = pywin32_system32 + os.pathsep + os.environ["PATH"]
         break

[lazka]

Another thing is that os.add_dll_directory() in mingw python is somehow broken. So I had to add this patch:

yeah, we reverted the DLL lookup changes for 3.8 because they broke too much, and that affects add_dll_directory(). Something that needs to be fixed at some point.

[lazka]

Rebased on master. I'm going to merge this so it doesn't bit rod more.

Feel free to create a new PR improving things.