macOS 11 is darwin 20.0.0. This confuses the configure script, resulting in defining _POSIX_C_SOURCE and friends.

New changeset 8ea6353f60625c96ce96588c70ff24a77f8c71f9 by Ronald Oussoren in branch 'master':
BPO-41100: Support macOS 11 when building (GH-21113)
https://github.com/python/cpython/commit/8ea6353f60625c96ce96588c70ff24a77f8c71f9

New changeset ad7a66731f2681c626f32f097e1e91cdc02330ab by Miss Islington (bot) in branch '3.9':
BPO-41100: Support macOS 11 when building (GH-21113)
https://github.com/python/cpython/commit/ad7a66731f2681c626f32f097e1e91cdc02330ab

New changeset c4a53e48a98b3d8fdcfc11d1b5af3ea0dd987a88 by Miss Islington (bot) in branch '3.8':
BPO-41100: Support macOS 11 when building (GH-21113)
https://github.com/python/cpython/commit/c4a53e48a98b3d8fdcfc11d1b5af3ea0dd987a88

New changeset cfbc759f918d646a59acb99251fc10b3900248a6 by Miss Islington (bot) in branch '3.7':
BPO-41100: Support macOS 11 when building (GH-21113) (GH-21155)
https://github.com/python/cpython/commit/cfbc759f918d646a59acb99251fc10b3900248a6

New changeset 2f168c6356f92c38ae7751d2faf2b266a9356229 by Ned Deily in branch 'master':
bpo-41100: clarify NEWS item about macOS 11 support (GH-21174)
https://github.com/python/cpython/commit/2f168c6356f92c38ae7751d2faf2b266a9356229

New changeset 1a0c581a94892fbd8fa70905130776ef7a059d97 by Miss Islington (bot) in branch '3.9':
bpo-41100: clarify NEWS item about macOS 11 support (GH-21174)
https://github.com/python/cpython/commit/1a0c581a94892fbd8fa70905130776ef7a059d97

New changeset 027bba2eada175358a7f34cad155b25e1b805082 by Miss Islington (bot) in branch '3.8':
bpo-41100: clarify NEWS item about macOS 11 support (GH-21174)
https://github.com/python/cpython/commit/027bba2eada175358a7f34cad155b25e1b805082

[Moving the discussion from Issue41164 to here.]

The change in PR 21224 may be needed to successfully build but I'm not sure whether that LIBTOOL_CRUFT stuff is still needed. With a squick search, I don't see any references to it anymore. But it won't hurt to add it.

New changeset 604d95e235d86465b8c17f02095edcaf18464d4c by Lawrence D'Anna in branch 'master':
bpo-41100: fix _decimal for arm64 Mac OS (GH-21228)
https://github.com/python/cpython/commit/604d95e235d86465b8c17f02095edcaf18464d4c

A first testrun on the arm-mac resulted in test failures in test_distutils (in particular test_deployment_target_default in the distutils test).  That's something I have to investigate further.

I've created 3 patches for ctypes

* use system libffi instead of vendored one on Mac OS 10.15 and up
* use correct ABI for variadic functions on arm64
* check for libs in the dyld shared cache

With those, test_ctypes should all pass on arm64

PR21564 is my attempt to merge all changes needed to support macOS 11 and arm64.  To be honest I had lost track of what's needed.

With this the patch the tests pass on my DTK system. 

TODO:
- Verify builds on older macOS versions
- Maybe: support building using non-system libffi on macOS


PR 21583 is a separate attempt at doing the code changes that are needed to build with the latest SDK and deploy on macOS 10.9.  This patch is still in a fairly rough state, and needs more testing (I haven't done any testing on older platforms yet)

TODO:
- Test on macOS 10.9
- Tweak the build-installer script and possibly _osx_support.py: the minimal deployment target for arm64 should be macOS 11.0.
- Tweak the build-installer script to build an optimised version

TDB:
- Keep this as two separate PRs or merge them? 
  The latter makes testing easier, but review harder

Hi,
What you can do is to verify builds on older macOS versions and you can also support building using non-system libffi on macOS.

Hope this hhelps...

Regards,
Mika Hawkins

New changeset a0ad82959652ff64c99231f457fd740b17330514 by Ned Deily in branch '3.7':
bpo-41100: additional fixes for testing on macOS 11 Big Sur Intel
https://github.com/python/cpython/commit/a0ad82959652ff64c99231f457fd740b17330514

https://github.com/python/cpython/pull/22855 is currently the most complete PR, and should be almost ready for merging (although there hasn't been any code review at this point).

New changeset 41761933c1c30bb6003b65eef1ba23a83db4eae4 by Ronald Oussoren in branch 'master':
bpo-41100: Support macOS 11 and Apple Silicon (GH-22855)
https://github.com/python/cpython/commit/41761933c1c30bb6003b65eef1ba23a83db4eae4

Over at NumPy we are getting reports that some python built for macOSx 11 is not accepting wheels with the `macosx_10_9_x86_64` platform tag. Could it be related to this issue, another open issue, or some other provider's python? xref https://github.com/numpy/numpy/issues/17784

@mattip, it could also be due to https://github.com/pypa/packaging/pull/319. I’m away from the keyboard fir the moment and can’t investigate further.

New changeset 442746af649cc2c20d690acfabf44ab0e06c36b4 by Ronald Oussoren in branch 'master':
bpo-41100: Stripping '-arch arm64' didn't work after all (GH-23280)
https://github.com/python/cpython/commit/442746af649cc2c20d690acfabf44ab0e06c36b4

New changeset e8b1c038b14b5fc8120aab62c9bf5fb840274cb6 by Ronald Oussoren in branch '3.9':
[3.9] bpo-41100: Support macOS 11 and Apple Silicon (GH-22855) (GH-23295)
https://github.com/python/cpython/commit/e8b1c038b14b5fc8120aab62c9bf5fb840274cb6

New changeset 5aa6c99da1087ded0ccb99bb12591e5ab31d75c3 by Ned Deily in branch '3.9':
bpo-41100: Update Whatsnew and installer ReadME for 3.9.1 (GH-23472)
https://github.com/python/cpython/commit/5aa6c99da1087ded0ccb99bb12591e5ab31d75c3

New changeset 936533ca0415c40dc64ccb5f8857720f32b3fcb4 by Ned Deily in branch 'master':
bpo-41100: minor build installer fixes (GH-23480)
https://github.com/python/cpython/commit/936533ca0415c40dc64ccb5f8857720f32b3fcb4

New changeset 3266991e721286e942125350ca41e1403d8384d6 by Miss Islington (bot) in branch '3.9':
bpo-41100: minor build installer fixes (GH-23480)
https://github.com/python/cpython/commit/3266991e721286e942125350ca41e1403d8384d6

New changeset c0c23ea72b76b06b7db0d09415fa90bab8ded63a by Ned Deily in branch 'master':
bpo-41100: in test_platform, ignore 10.16 (GH-23485)
https://github.com/python/cpython/commit/c0c23ea72b76b06b7db0d09415fa90bab8ded63a

New changeset 96d906b144e6e6aa96c5ffebecbcc5d38034bbda by Miss Islington (bot) in branch '3.9':
bpo-41100: in test_platform, ignore 10.16 (GH-23485) (GH-23486)
https://github.com/python/cpython/commit/96d906b144e6e6aa96c5ffebecbcc5d38034bbda

Are there plans to backport PR 22855 to any branches older than 3.9?

> Are there plans to backport PR 22855 to any branches older than 3.9?

The plan is to also support 3.8 on Big Sur and Apple Silicon as 3.8 is still in bugfix mode.  There are no plans to backport support to 3.7 and 3.6 which are in the security-fix-only phase of their release cycles.

Is there an ETA for having Big Sur support backported to the 3.8 branch, or is it already there and I'm just not seeing it here?

There is no ETA. I'm spending more time away from the computer and on other projects when I'm not at work. I also have no personal need for 3.8 support.

If anyone else is up to the job: the back port to 3.9 should be a good starting point, but I don't trust the output of the cherry_picker tool to 3.8.

I ran into a minor issue with the ctypes extension and the _dyld_shared_cache_contains_path dependency that was added as part of this work (commit e8b1c038b14b5fc8120aab62c9bf5fb840274cb6).

When building on Intel macOS 10.15 using an 11.x SDK, the configure check for this symbol passes (#define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH 1) and the extension compiles (albeit with a lot of deprecation warnings spewing from the Apple SDK headers). However, at run-time, we blow up due to a missing symbol error:

```
2021-02-21T21:57:21.4116370Z cpython-3.9> clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/_decimal.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/basearith.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/constants.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/context.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/convolute.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/crt.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/difradix2.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/fnt.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/fourstep.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/io.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/mpalloc.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/mpdecimal.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/numbertheory.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/sixstep.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/transpose.o -L/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/tools/pyhost/lib -L/usr/local/lib -lm -o build/lib.macosx-11.0-x86_64-3.9/_decimal.cpython-39-darwin.so
2021-02-21T21:57:27.1635100Z cpython-3.9> *** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so, 2): Symbol not found: __dyld_shared_cache_contains_path
2021-02-21T21:57:27.1637220Z cpython-3.9>   Referenced from: build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so (which was built for Mac OS X 11.0)
2021-02-21T21:57:27.1638560Z cpython-3.9>   Expected in: /usr/lib/libSystem.B.dylib
2021-02-21T21:57:27.1639770Z cpython-3.9>  in build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so
```

(This build output is from https://github.com/indygreg/python-build-standalone/runs/1947611547?check_suite_focus=true)

The missing symbol is provided by libdyld.dylib, which doesn't appear to be linked directly by CPython's build system.

I'm only able to reproduce this on a GitHub Actions 10.5 machine using the 11.1 SDK: it seems to work fine on both an 11.1 Intel and M1 device.

I reckon the correct fix is to have _ctypes link libdyld is this build configuration. I was also able to work around it in python-build-standalone by unsetting HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH (hacky solution but this feature isn't needed by the build that was failing). And of course linking a non-Apple libffi should also work, which is potentially why pyenv/Homebrew/etc users aren't noticing.

python-build-standalone uses its own Clang built from source, so it is entirely possible there's something differing between the Apple Clang and my Clang causing this to fail. However, at least 1 other person ran into this in the wild (https://stackoverflow.com/questions/65853539/symbol-not-found-dyld-shared-cache-contains-path-on-osx-after-installl-from), so it might be generic.

While I'm here, I've taught python-build-standalone to cross-compile CPython 3.9 from both Intel and M1 Apple devices (without using universal binaries). I also have some (hacky) support for targeting iPhone/iWatch/iTV. If there's interest, I could start sending patches upstream. Let me know!

I had some time today, so I took a crack at merging this into Python 3.8. I've filed GH-25274 with my proposal.

To summarize, I'm suggesting back porting all changes from bpo-41100 (as well as the subsequent bpo-42688) but without the weak-linking changes to `posixmodule` and co, the installer changes and the `universal2` variant. This means we don't support `arm64` officially, but someone can build for it from source if they'd like.

This should unblock Homebrew (which currently manually patches based on some obsolete PRs) as well as folks who build from source and have a particular need (as we would for example at Dropbox).

Is there any hope of a backport to 3.7? Or has that ship sailed?

> Is there any hope of a backport to 3.7? Or has that ship sailed?

Python 3.7 doesn't accept bugfixes anymore, nor new features, nor new platofrm support (macOS 11).

The 3.7 branch now only accept security fixes:
https://devguide.python.org/#status-of-python-branches

Trying to build from source to Apple M1 Big Sur (macOS 11), and it's failing to compile the standard library extension modules. It seems to be that the .c files are not being located properly, but looking in Modules/ I can see all the required source files are there. attached is a typical build log, have tried a variety of options to no avail. Similar build configurations have worked for previous MacOS releases, which leads me to believe this is something else. I've searched fairly exhaustively for any information related to this and come up with nothing.

@harlantc, in general, please open new issues for new problems. That said, I took a quick look at the build.log you provided and it was not obvious exactly why things went wrong. The most obvious difference is that when I do a 3.9.4 build, the extension module source paths (the -c arguments to the failing compiles) have been normalized by setup.py to absolute paths, whereas in the failing build.log they are unprefixed relative paths. I suggest you look at the top level setup.py file to figure out why that is the case in your build. I also note that the build appears to be on a very early release of macOS 11 Big Sur; it would be best to upgrade to the latest version of it and ensure you have the latest version of the Command Line Tools installed. Also, your configure arguments appear to be unnecessary complex: the -m64 and the --build and --target arguments shouldn't be necessary and we rarely test with --enable-shared on macOS. It's possible they are contributing factors.  If you do find something that appears to be a Python build issue, please open a new issue with the details.

@ned.deily sorry about that, will do that in the future. Thanks for replying.

in regard to the extra args, those were all added in an attempt to make things work, I should probably have shown an earlier log that excluded them. I wasn't sure setup.py would be the right place to look for more, I'll take a look there.

Thanks

No problem. Also you should not need to specify sysroot if you are; the Apple compiler front-end takes care of that for you (see man xcrun for more details).

Thanks, Max, for doing the backport to 3.8 hard work!  While PR 25274 covered most of the changes needed, there were some additional fixes and changes needed and I felt it was easier to cherry-pick all of the changes into a new PR. I have done extensive building and testing on Big Sur on both Apple Silicon and Intel Macs and regression building and testing on some older macOS versions including macOS 10.9 and python.org macOS installers. So far, it looks good to me. Assuming the 3.8 release manager has no objections, I will merge this in time for 3.8.10 and we can finally close this issue.

New changeset b29d0a5a7811418c0a1082ca188fd4850185e290 by Ned Deily in branch '3.8':
[3.8] bpo-41100: Support macOS 11 Big Sur and Apple Silicon Macs (#25806)
https://github.com/python/cpython/commit/b29d0a5a7811418c0a1082ca188fd4850185e290

New changeset 518f8b5dd50b73499282a1ef02b197af106ad118 by Ned Deily in branch 'master':
bpo-41100: Update Misc/ACKS (GH-25808)
https://github.com/python/cpython/commit/518f8b5dd50b73499282a1ef02b197af106ad118

New changeset bb3ec9ceae7032457f6e7fb72b39aee95976051c by Miss Islington (bot) in branch '3.9':
bpo-41100: Update Misc/ACKS (GH-25808) (GH-25809)
https://github.com/python/cpython/commit/bb3ec9ceae7032457f6e7fb72b39aee95976051c

Thank you to everyone who contributed to this major undertaking!  A particular thank you to Lawrence for doing much of the initial work and paving the way. Now that 3.8 also supports Big Sur and Apple Silicon Macs as of the imminent 3.8.10 release, it's time to close this issue. If new concerns arise, pleasa open or use other issues.