Build platform-specific wheels containing libmagic #294

ddelange · 2023-09-06T12:59:42Z

This PR builds self-contained wheels as discussed in #233. For Windows users, this renders python-magic-bin from @julian-r obsolete.

pip install these wheels

pip can install from GitHub Release assets from my fork:

# space separated list of URLs for --find-links
export PIP_FIND_LINKS='https://github.com/ddelange/python-magic/releases/expanded_assets/0.4.28.post8'
pip install --force-reinstall python-magic

- python-magic-0.4.27.tar.gz
+ python-magic-0.4.28.tar.gz
- python_magic-0.4.27-py2.py3-none-any.whl
+ python_magic-0.4.28-py2.py3-none-macosx_10_9_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-macosx_11_0_arm64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_s390x.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-win32.whl
+ python_magic-0.4.28-py2.py3-none-win_amd64.whl

The wheels:

Come bundled with libmagic (.dylib on mac, .so on nix, .ddl on win) - no additional user action needed
- latest libmagic (5.46), except on windows (5.44, used the latest GitHub Release at https://github.com/julian-r/file-windows)
- building from source on linux and macos with maximum backwards compatibility
- skipping i686 linux wheels as they don't pass pytest ref ca4def3
Are forward compatible, they will install on cpython 2.7 and all current and future cpython 3.5+ distributions
Are built for the major cpu architectures (and also for alpine linux)
- ~~Linux architectures limited by availability: https://pkgs.org/search/?q=file-libs~~ now building from source on linux
- For windows I couldn't find compiled libmagic for ARM64, skipped it
Are uploaded as GitHub Release assets, as well as upload to PyPI, whenever a GitHub Release is (pre)released.
- 🚩 @ahupp for this, you still need to add wheels.yml as trusted publisher here

CI/CD

dists build with official cibuildwheel on GitHub Actions, and they build in parallel:

The PyPI publish step is failing here ^ because my fork is not a trusted publisher
For any system not covered by these wheels, pip will fall back to the source distribution, which will check for proper libmagic on the systen at install time.

fix #137, fix #288, fix #225, fix #276, fix #248, fix #87, fix #139, fix #233, fix #73, fix #60, fix #34, fix #293, fix #233, fix #278, fix #262, fix #248, fix #238, fix #145, fix #61, fix #12, fix #295, fix #311, fix #312, fix #313, fix #321, fix #332, fix #249, fix #333

apirogov · 2023-09-12T13:49:39Z

This is nice! Hope this will be merged soon!

Just ran into issues with my library being not usable by Mac and Windows users because I rely on python-magic. If there are wheels, I don't need to find a workaround or replace the library :)

python-magic-bin did not work for some of them, by the way.

stumpylog · 2023-09-12T14:40:45Z

.github/workflows/main.yml

+      with:
+        files: dist/*
+
+    - name: Upload to PyPI


A small improvement here might be to use the PyPa Action instead: https://github.com/pypa/gh-action-pypi-publish

The big advantage is trusted publishing, instead of storing a password or token as a secret

that's cool, thanks for sharing!

@ahupp shall I make that change and you set it up on PyPI side?

Sounds like trusted publishing is the way to go. I recently got this email:

Hi ddelange!
Earlier this year, we announced that PyPI would require all users to enable a form of two-factor authentication on their accounts by the end of 2023.

Keeping your PyPI account secure is important to all of us. We encourage you to enable two-factor authentication on your PyPI account as soon as possible.

What forms of 2FA can I use?
We currently offer two main forms of 2FA for your account:

Security device including modern browsers (preferred) (e.g. Yubikey, Google Titan)
Authentication app (e.g. Google Authenticator)
Once one of these secure forms is enabled on your account, you will also need to use either Trusted Publishers (preferred) or API tokens to upload to PyPI.

What do I do if I lose my 2FA device?
As part of 2FA enrollment, you will receive one-time use recovery codes. One of them must be used to confirm receipt before 2FA is fully active. Keep these recovery codes safe - they are equivalent to your 2FA device. Should you lose access > to your 2FA device, use a recovery code to log in and swap your 2FA to a new device.

Read more aboutrecovery codes.

Why is PyPI requiring 2FA?
Keeping all users of PyPI is a shared responsibility we take seriously. Strong passwords combined with 2FA is a recognized secure practice for over a decade.

We are requiring 2FA to protect your account and the packages you upload, and to protect PyPI itself from malicious actors. The most damaging attacks are account takeover and malicious package upload.

To see this and other security events for your account, visit your account security history.

Read more on this blog post.

If you run into problems, read the FAQ page. If the solutions there are unable to resolve the issue, contact us via support@pypi.org.

Thanks,
The PyPI Admins

@ahupp so the last thing for you to do is adding this repo as trusted publisher to https://pypi.org/manage/project/python-magic/settings/publishing/

.github/workflows/main.yml

ahupp · 2023-09-28T17:18:10Z

This is huge, thank you! Apology for the delay I thought I'd commented earlier but guess not. I'll look this over soon; I didn't quite understand how bad the binary dep situation was expecially on windows.

.github/workflows/main.yml

jean-humann · 2023-12-03T17:02:05Z

@ahupp @stumpylog could we have this one merged (and released) by the end of the year please ?

estarfoo · 2024-06-26T15:37:48Z

Hi! Great PR, is there any real contention about it beyond the scope of OS/distro support in add_libmagic.sh and README.md?

Most users will only ever want the wheels from this repo. In particular, this looks like it will solidly cover usage in Docker images. Anyone who wants to use the source version and provide libmagic themselves, probably knows best how to do the latter in their environment. given info on where this package will look for the library. (Those who package python-magic for their Linux distro of choice will already have a preferred way of ensuring libmagic presence. This will probably not exactly match anything suggested in python-magic docs.)

Even for those particularly invested in having a range of setup instructions, the PR in its current state should look like a clear improvement on master, and further improvements in that are will come more easily as separate PRs (because they won’t be tied to CI scripts).

So: how about merging this without completing libmagic setup instructions for every possible platform? Seems like it already does what the PR title says.

Privat33r-dev · 2024-06-26T15:42:39Z

So: how about merging this without completing libmagic setup instructions for every possible platform? Seems like it already does what the PR title says.

Totally for it. My suggestions just show the way to improve it, but I would merge it "as is" since it already provides a huge value. "Perfect is the enemy of good".

@ahupp hopefully you can find some time to review this most discussed PR in the python-magic's history :)

ferdnyc · 2024-07-30T08:06:41Z

add_libmagic.sh

+        tmpfile="$(mktemp)" &&
+        curl -sSLo "${tmpfile}" "https://astron.com/pub/file/${version}.tar.gz" &&
+        tar xvf "${tmpfile}" &&


@estarfoo

Not a big deal, but if going with the piped curl output, personally I'd pair that with an explicit tar xvf -. Again, for the paranoid.

add_libmagic.sh

+import platform, sysconfig, io, zipfile, urllib.request
+assert platform.system() == "Windows"
+machine = "x86" if sysconfig.get_platform() == "win32" else "x64"
+url = f"https://github.com/julian-r/file-windows/releases/download/v5.44/file_5.44-build104-vs2022-{machine}.zip"


ddelange · 2024-06-29T07:20:12Z

@ahupp this also fixes failing CI on master (looks like the github actions linux runner image no longer ships libmagic by default)

ashtonpaul · 2024-07-31T19:32:08Z

@ahupp @ddelange Any update on this PR and the release? 👀 Is there anything that needs to be tested or are we waiting on anything other than a review?

ddelange · 2024-09-02T07:51:58Z

@ahupp please review, merge, add trusted publisher (see PR description) and release 🙏

wilhelmklopp · 2024-10-18T08:17:58Z

Also would love to see this merged, it's causing some issues for windows users of our package (which depends on mocket which depends on python-magic). Thank you! 😄

cclauss · 2024-10-18T09:25:34Z

@ashtonpaul @ddelange @wilhelmklopp @ferdnyc #294 (comment)

ferdnyc · 2024-10-21T03:08:06Z

@ahupp Speaking personally, this is a big, complex PR that I don't really feel qualified to approve given the scope and nature of all the changes here. I guess I'm in agreement with your #294 (comment)

Perhaps it would be useful to break it into three separate PRs that are easier to review and land.

One PR that deals only with macOS and another that deals only with Windows might be easier to land. Once that is done then this PR could be rebased to deal only with Linux and friends. I know it is extra work but I sense that new momentum is needed.

OTOH, the PR has received a number of approvals as-is, so there's ample indication that others are satisfied with it in its current form. I left some comments, which were addressed, so my input certainly shouldn't be viewed as disapproval by any means.

Haiz14 · 2024-12-10T06:48:09Z

When will 0.4.28 be released? Like, this year?

edit: sorry if it came out as rude. It takes quite some complexity to merge complex PR's. And asking a developer about when there is a new release, is never really the right thing.

C0rn3j · 2024-12-28T17:40:52Z

@ahupp This PR has 6 approvals, and from the looks of it, 0 controversy, would it be possible to review & pull it sometime soon?
I just ran into one of the issues this fixes myself.

Thanks in advance!

nightlark · 2025-02-04T20:50:18Z

I'm trying to understand what the current situation is with using libmagic for file type identification in Python, and I've found at least 13 separate packages on PyPI that are in various states of not being maintained -- of them, this PR seems to be the version that includes the most complete set of wheels with pre-compiled copies of libmagic (aside from not being available on PyPI).

Would it make sense to create a fork of python-magic based on this PR by @ddelange, perhaps under an org with at least 2 maintainers that can review PRs and publish updated wheels to help minimize the burden for a single person and hopefully avoid adding to the unmaintained package count?

@ahupp has done an great job with python-magic and this is in no way a criticism of their work or minimizing the time they've spent working on and maintaining it -- of the libmagic packages I found on PyPI, python-magic seems to be in by far the best overall shape. Life happens and situations change so time isn't as plentifully as it once was, and I know how it is for things to get really busy and maintaining an open source project can be draining, leading to burnout. This happens literally all the time (as evidenced by all of the libmagic PyPI packages that haven't been updated since before 2019).

ddelange · 2025-02-04T22:20:15Z

@ahupp fwiw I'm happy to become a maintainer of this project 👍

There is an ongoing debate on how to embed libmagic library in the pypi package. For windows we have to rely on python-magic-bin for the time being. See ahupp/python-magic#294

stumpylog · 2025-02-14T16:11:35Z

I would also be happy to help maintain. We use this in paperless-ngx, so I'm happy to work on keeping it up to date, resolving issues, etc

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: update magic/compat.py simplify tests into something more delarative

ddelange · 2025-02-14T18:08:13Z

Hi @ahupp 👋

I fixed the broken tests I just pulled from master. Did you forget to track the test.py file by any chance? I commented out that test for now.

I also upgraded this PR to build wheels and run CI using the newly released file-5.46.

Any response from you on this PR would be greatly appreciated 🙏

.github/workflows/ci.yml

cclauss · 2025-02-14T18:33:33Z

.github/workflows/wheels.yml

+      with:
+        python-version: '3.x'
+
+    - run: sudo apt-get install -y libmagic1


This will tell you that it is already installed on both Intel and ARM.

Right. But do we want to rely on github bundling libmagic in their ubuntu-latest runner image?

.github/workflows/wheels.yml

Co-authored-by: Christian Clauss <cclauss@me.com>

ddelange changed the title ~~Build ABI3 wheels containing libmagic~~ Build platform-specific wheels containing libmagic Sep 7, 2023

ddelange force-pushed the abi3-wheels branch 2 times, most recently from a98f13b to dc9c393 Compare September 7, 2023 07:26

ddelange mentioned this pull request Sep 7, 2023

Build platform-specific wheels containing libmagic ddelange/python-magic#1

Closed

ddelange force-pushed the abi3-wheels branch 12 times, most recently from d672b91 to 14f7dbb Compare September 7, 2023 10:03

Build platform-specific wheels containing libmagic

ec952d7

ddelange force-pushed the abi3-wheels branch from 14f7dbb to ec952d7 Compare September 7, 2023 10:09

jspraul mentioned this pull request Sep 9, 2023

Update file to 5.44 hey-red/Mime#49

Closed

apirogov mentioned this pull request Sep 12, 2023

python-magic has no wheels for Windows/Mac Materials-Data-Science-and-Informatics/metador-core#85

Open

stumpylog reviewed Sep 12, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

Move persmissions into job scope, remove ABI3 reference

a437409

ahupp mentioned this pull request Oct 10, 2023

Binary distribution for libmagic on Windows #293

Open

Switch to PyPI trusted publishing

4a715e2

ddelange mentioned this pull request Nov 23, 2023

bugfix: add path to load libmagic bin on Windows #249

Open

ddelange commented Nov 23, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

estarfoo approved these changes Jun 26, 2024

View reviewed changes

nitely mentioned this pull request Sep 16, 2024

django-spirit[files] python-magic-bin cannot be installed on mac silicon (m3) nitely/Spirit#331

Closed

wilhelmklopp mentioned this pull request Oct 18, 2024

Dependency python-magic-bin seems to be unmaintained kolofordjango/kolo#95

Closed

ashtonpaul approved these changes Nov 2, 2024

View reviewed changes

ddelange mentioned this pull request Dec 10, 2024

magic.from_file(file) generate "File Not Found" exception when file is a linux (symbolic) link #336

Open

C0rn3j mentioned this pull request Dec 28, 2024

Tauon blindly trusts file extension, causing files to be read wrong Taiko2k/Tauon#1362

Open

ddelange added 4 commits February 14, 2025 19:28

Merge branch 'master' of https://github.com/ahupp/python-magic into a…

43c0c99

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: update magic/compat.py simplify tests into something more delarative

Fix tests

3e51048

Use older typing syntax

d7b1171

Use older typing syntax

620d78f

cclauss reviewed Feb 14, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

cclauss reviewed Feb 14, 2025

View reviewed changes

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

ddelange commented Feb 14, 2025

View reviewed changes

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

Apply suggestions from code review

d3e886c

Co-authored-by: Christian Clauss <cclauss@me.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build platform-specific wheels containing libmagic #294

Build platform-specific wheels containing libmagic #294

ddelange commented Sep 6, 2023 •

edited

Loading

apirogov commented Sep 12, 2023

stumpylog Sep 12, 2023

ddelange Sep 12, 2023

ddelange Oct 26, 2023 •

edited

Loading

ddelange Nov 13, 2023

ddelange Nov 22, 2023

ahupp commented Sep 28, 2023

jean-humann commented Dec 3, 2023

estarfoo commented Jun 26, 2024

Privat33r-dev commented Jun 26, 2024

This comment was marked as resolved.

ferdnyc Jul 30, 2024

This comment was marked as resolved.

ddelange commented Jun 29, 2024

ashtonpaul commented Jul 31, 2024

ddelange commented Sep 2, 2024

wilhelmklopp commented Oct 18, 2024

cclauss commented Oct 18, 2024 •

edited

Loading

ferdnyc commented Oct 21, 2024

Haiz14 commented Dec 10, 2024 •

edited

Loading

C0rn3j commented Dec 28, 2024 •

edited

Loading

nightlark commented Feb 4, 2025 •

edited

Loading

ddelange commented Feb 4, 2025

stumpylog commented Feb 14, 2025

ddelange commented Feb 14, 2025

cclauss Feb 14, 2025

ddelange Feb 14, 2025

Build platform-specific wheels containing libmagic #294

Are you sure you want to change the base?

Build platform-specific wheels containing libmagic #294

Conversation

ddelange commented Sep 6, 2023 • edited Loading

pip install these wheels

CI/CD

apirogov commented Sep 12, 2023

stumpylog Sep 12, 2023

Choose a reason for hiding this comment

ddelange Sep 12, 2023

Choose a reason for hiding this comment

ddelange Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

ddelange Nov 13, 2023

Choose a reason for hiding this comment

ddelange Nov 22, 2023

Choose a reason for hiding this comment

ahupp commented Sep 28, 2023

jean-humann commented Dec 3, 2023

estarfoo commented Jun 26, 2024

Privat33r-dev commented Jun 26, 2024

This comment was marked as resolved.

ferdnyc Jul 30, 2024

Choose a reason for hiding this comment

This comment was marked as resolved.

ddelange commented Jun 29, 2024

ashtonpaul commented Jul 31, 2024

ddelange commented Sep 2, 2024

wilhelmklopp commented Oct 18, 2024

cclauss commented Oct 18, 2024 • edited Loading

ferdnyc commented Oct 21, 2024

Haiz14 commented Dec 10, 2024 • edited Loading

C0rn3j commented Dec 28, 2024 • edited Loading

nightlark commented Feb 4, 2025 • edited Loading

ddelange commented Feb 4, 2025

stumpylog commented Feb 14, 2025

ddelange commented Feb 14, 2025

cclauss Feb 14, 2025

Choose a reason for hiding this comment

ddelange Feb 14, 2025

Choose a reason for hiding this comment

ddelange commented Sep 6, 2023 •

edited

Loading

ddelange Oct 26, 2023 •

edited

Loading

cclauss commented Oct 18, 2024 •

edited

Loading

Haiz14 commented Dec 10, 2024 •

edited

Loading

C0rn3j commented Dec 28, 2024 •

edited

Loading

nightlark commented Feb 4, 2025 •

edited

Loading