Skip to content

Add the Android build factory#26

Closed
xdegaye wants to merge 1 commit intopython:masterfrom
xdegaye:android-build
Closed

Add the Android build factory#26
xdegaye wants to merge 1 commit intopython:masterfrom
xdegaye:android-build

Conversation

@xdegaye
Copy link

@xdegaye xdegaye commented Dec 8, 2017

The Python test suite succeeds when run with buildbot strenuous settings with all the Android architectures on API 24: x86, x86_64, armv7 and arm64.

This PR proposes to update master.cfg to run buildbots for Android. The Android build system is implemented at issue bpo-30386 and PR 1629, it adds the Android/ directory and the configure-android script to the root of the source directory without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.

The Android emulator is actually qemu, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop when the test suite is run with -u all -j1. Since none of the changes to Python that have been made for Android is specific to armv7 or arm64, maybe their test suite could be run just weekly.

The buildbot.sh script simulates the operations executed by a buildbot worker (except for the git step) and explains the changes to master.cfg proposed here. Its content is listed here:

#! /bin/bash
# Run the commands triggered by a Python buildbot master.

# makesetup step
PYTHON_DEVPT_VERSION=3.7 \
    WITH_LIBRARIES=libffi,ncurses,openssl,readline,sqlite \
    EMULATOR_CMD_LINE_OPTIONS="-use-system-libs -no-window" \
    ANDROID_API=24 \
    ANDROID_ARCH=x86_64 \
    Android/makesetup --with-pydebug > makesetup.log 2>&1
echo "makesetup step done with exit status: $?"

# compile step
make all > compile.log 2>&1
echo "compile step done with exit status: $?"

# pythoninfo step
make pythoninfo > pythoninfo.log 2>&1
echo "pythoninfo step done with exit status: $?"

# test step
make buildbottest TESTOPTS=-j2 TESTPYTHONOPTS= TESTTIMEOUT=900 > test.log 2>&1
echo "test step done with exit status: $?"

# clean step
make distclean > clean.log 2>&1
echo "clean step done with exit status: $?"

Running this script on x86_64 gives the following results (the corresponding five log files are attached in the next comment) when it is run the second time, on the first run the native python and the external libraries are built and compile.log contains a lot more lines:

$ /path/to/buildbot.sh
makesetup step done with exit status: 0
compile step done with exit status: 0
pythoninfo step done with exit status: 0
test step done with exit status: 0
clean step done with exit status: 0

@xdegaye
Copy link
Author

xdegaye commented Dec 8, 2017

Here are the log files:
One must sometimes use double-click instead of drag-drop to attach files :-(

makesetup.log
compile.log
pythoninfo.log
test.log
clean.log

@vstinner
Copy link
Member

vstinner commented Dec 8, 2017

I'm worried that there are still open issues related to Android, especially issues on the Python test suite.

For example https://bugs.python.org/issue32246

I would prefer to make sure that the Python test suite is very stable before adding this worker.

Anyway, amazing work @xdegaye! Thank you for your continuous effect on supporting Android!

@xdegaye
Copy link
Author

xdegaye commented Dec 8, 2017

It seems that the only open issues (not enhancement) specific to Android are the two ctypes issues and it is not clear when those two will be fixed:

  • The issue 32246 that you mention seems to be related to the fact that the test suite is run from inside the temporary directory used by test_regrtest. I will try to reproduce this on linux. Now that the tests are run outside this directory, test_regrtest is Ok as can be seen from the results of buildbot.sh above.
  • ctypes is disabled in Modules/Setup for x86_64 (issue 32202) and arm64 (issue 32203). Those problems may be related to libffi.
  • issue 26855 and issue 32210 are about the future implementation of platform.android_ver() which is an enhancement, so this should not prevent the test suite to be run.
  • Issue 32033 is about pwd and is not specific to Android, it is a mismatch between the implementation of pwd, its tests and its documentation. It was detected because test_pwd runs getpwall() which does not exist on Android. I have temporarily removed _getpwall() in issue 26856 and left it open since issue 32033 is languishing.

On the other hand:

  • Issue 30386 (the issue that implements this build system) is still open and its branch is not merged.
  • I will shortly post a detailed note about the status of the support on python-dev and the ensuing discussion may bring some changes.
  • It still remains to find the hardware to run these buildbots.

So we can postpone the review of this PR until all the above three conditions are met.

@zware
Copy link
Member

zware commented Dec 11, 2017

I've had a look at this, and it looks fine to me. I think we can merge it anytime to be ready as soon as a worker host is available and the necessary CPython changes are merged.

@xdegaye
Copy link
Author

xdegaye commented Dec 11, 2017

@zware Thanks for the review Zachary. I do not have the access rights to this repo so I will let you guys do the merge.

@vstinner
Copy link
Member

vstinner commented Jan 26, 2021

No update since 2017, I close the PR.

@vstinner vstinner closed this Jan 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zware zware zware approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xdegaye @vstinner @zware