build wheels for linux and windows?

[skshetry]

Hi, is it possible to provide wheels for Linux and Windows platforms? We are using it indirectly through webhdfs from fsspec, and were wondering if it's possible to build wheels.

[jborean93]

Unfortunately for Linux this isn't really feasible due to the issues mentioned in pythongssapi/python-gssapi#200. Essentially Python wheels on Linux need to embed the libkrb5 library in the wheel which means a process could have 2 copies of this library loaded. This would be problematic when trying to share data that is stored in memory or set thread specific settings as it would only apply to the one embedded with the wheel. MacOS is a different story as we can link to the system wide framework and thus would share the same krb5 implementation as other libraries.

Technically we could make wheels for Windows linked against the MIT krb5 software for Windows just like python-gssapi does but this isn't something I've got much time to work on at the moment. If you willing you could have a look at what it would take to implement but at a minimum I would need the following:

• Ability to compile on Windows with MIT kerberos
• The above but on CI
• At least a minimal set of tests to run on Windows in CI

Once the above is done we can then add the wheels to the publish step in CI.

[deepika094]

hi, I suddenly started getting below error while doing pip install krb5
subprocess.CalledProcessError: Command '('krb5-config --cflags krb5',)' returned non-zero exit status 1.

Even if i just clone your git project and run requirements.txt it throws me this error. Could you help please?

[deepika094]

git clone https://github.com/jborean93/pykrb5.git
pip install Cython
python setup.py bdist_wheel -> Even this building wheel command throws same error..

[jborean93]

What happens when you run krb5-config --cflags krb5 manually. Does it fail? It uses this to determine the compile and linker flags required to build the extensions.

[deepika094]

No it gives me -I anaconda installation path/include for linux

[jborean93]

And what's the error code, here is what I typically expect on the output

If it's returning a non-0 return code then the setup script is determining that it failed for some reason. I find it interesting that it would have anaconda paths in there as it's a system package and not really related to Python.

[deepika094]

for --flags it gives me /anaconda/include
for --libs krb5
-L/anaconda3/lib -Wl,--enable-new-dtags -Wl,-rpath -Wl,/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,-rpath,/anaconda3/lib -L**/anaconda3/lib -L**/anaconda3/lib -Wl,--disable-new-dtags -lkrb5 -lk5crypto -lcom_err

I tied two methods-

1. Where we try to download this package from Jfrog which picked the same latest cached version . It found tar.gz and tried to build wheel -
   logs- Installing build dependencies: started
   [06:57:15] [Step 4/5] Installing build dependencies: finished with status 'done'
   [06:57:15] [Step 4/5] Getting requirements to build wheel: started
   [06:57:15] [Step 4/5] Getting requirements to build wheel: finished with status 'error'
   [06:57:15] [Step 4/5] error: subprocess-exited-with-error
   [06:57:15] [Step 4/5]
   [06:57:15] [Step 4/5] × Getting requirements to build wheel did not run successfully.
   exit code: 1
   [06:57:15] [Step 4/5] ╰─> [25 lines of output]
   [06:57:15] [Step 4/5] /bin/sh: krb5-config: command not found
   [06:57:15] [Step 4/5] In distributed package, building from C files...
   [06:57:15] [Step 4/5] Using krb5-config at 'krb5-config'
   ......
   ...

subprocess.CalledProcessError: Command '('krb5-config --cflags krb5',)' returned non-zero exit status 127.
[06:57:15] [Step 4/5] [end of output]
[06:57:15] [Step 4/5]
[06:57:15] [Step 4/5] note: This error originates from a subprocess, and is likely not a problem with pip.
[06:57:15] [Step 4/5] error: subprocess-exited-with-error

I logged in to this build host and ran krb5-config --cflags krb5 it gave me anaconda path and if i do echo $? its 0.. (so no error)

Method 2- I tried to git clone in pycharm
git clone https://github.com/jborean93/pykrb5.git
pip install Cython
python setup.py bdist_wheel -> Even this building wheel command throws same error..
subprocess.CalledProcessError: Command '('krb5-config --cflags krb5',)' returned non-
zero exit status 1

[jborean93]

[06:57:15] [Step 4/5] /bin/sh: krb5-config: command not found

This seems suspect, it seems like in whatever environment your build step is running in has stripped out the PATH env vars it needs to locate krb5-config. Try doing KRB5_KRB5CONFIG="$( which krb5-config )" python setup.py bdist_wheel. You may need to run which krb5-config manually beforehand to make sure it picks something up.

[deepika094]

now failing for subprocess.CalledProcessError: Command 'krb5-config --libs gssapi' returned non-zero exit status 127.
[19:00:30] [Step 5/6] [end of output

[jborean93]

That will be a separate library https://github.com/pythongssapi/python-gssapi which does the same thing but for gssapi rather than krb5. It uses a similar mechanism to detect the compiler and linker flags and if any of them can't find krb5-config in the PATH it's going to fail. You need to fix that problem in your build process.

[csjab]

Does https://cibuildwheel.readthedocs.io make anything easier for automatically publishing wheels for as many supported systems as possible?

[jborean93]

Unfortunately for Linux it will not be possible to provide wheels for this library. PyPI does not allow you to have a wheel that contains a dynamically linked library outside of a specific set like libc. It does this because there are no guarantees that the host that is installing the wheel would contain the library that was used to compile the wheel in the first place. Or even if it did, that it was compatible ABI wise with what the wheel was compiled with. We could theoretically do a static link or bundle libkrb5 with the wheel but now any Python process that loads this library will be unable to work with other Python packages that are trying to load libkrb5 from the system. It also causes problems with system provided configurations for libkrb5 not being applied to the one in this package.

For macOS it is possible to do, and we currently do so.

For Windows it is also possible to do but:

• We would need to write the code to handle this scenario
• It would need MIT krb5 (not SSPI that comes natively with Windows)

It's in my view that if you want to use Kerberos on Windows then you are better off using SSPI itself. There are a few Python modules out there already for that.

[iamthebot]

FYI for linux, the conda package published here for this repo pulls in conda-forge's krb5 for you (bypassing the need to have a dynamic link to the system krb5 or even install krb5 on the system at all).