Skip to content

BUILD: use system python3 in the chroot#12299

Merged
charris merged 2 commits intonumpy:masterfrom
mattip:chroot-python3
Nov 1, 2018
Merged

BUILD: use system python3 in the chroot#12299
charris merged 2 commits intonumpy:masterfrom
mattip:chroot-python3

Conversation

@mattip
Copy link
Member

@mattip mattip commented Oct 31, 2018

The chroot travis test used python2. Update it to use python3.

libatlas-base-dev gfortran python-dev python-nose python-pip cython \
python-pytest"
libatlas-base-dev gfortran python-dev python3-dev python-pip python3-pip \
cython python3-pytest python-pytest"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need all the non-python3 versions if we are running Python3?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possibly, but then we cannot run chroot python2 tests anymore. If that is the direction I will change this

# run again in chroot with this time testing with python3
sudo linux32 chroot $DIR bash -c \
"cd numpy && PYTHON=python PIP=pip IN_CHROOT=1 $0 test"
"cd numpy && PYTHON=$PYTHON PIP=$PIP IN_CHROOT=1 $0 test"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does PIP=$PIP make any sense here, it is already in the environment.

@charris
Copy link
Member

charris commented Oct 31, 2018

This seems over complicated for its stated purpose.

@mattip
Copy link
Member Author

mattip commented Oct 31, 2018

There are four changes:

  1. change the test matrix to define the python and pip explicitly
  2. percolate those choices into the chroot
  3. install into the chroot python3 packages that were missing
  4. replace pip with $PIP

If we wish to only support python3 it can be simplified by removing 1, hardcoding python3 and pip3 into 2, and dropping the python2 packages from 3.

I think 4 works outside the chroot because it is using a virtualenv, not clear to me if inside the chroot it is also in a virtualenv.

@charris
Copy link
Member

charris commented Oct 31, 2018

I wonder if we can run 32 bit linux tests on azure-pipeline? We still have some slots before we hit the 10 parallel build limit. That doesn't help for releases yet, as they are still built on appveyor and travis, but it tmight be an easy way to get 32 testing on linux.

As it is running in bionic, the default python3 is 3.6, which should be good for a while. Did you try just changing things to *3 in the chroot? And then in the calling lines

  sudo linux32 chroot $DIR bash -c \
    "cd numpy && PYTHON=python PIP=pip IN_CHROOT=1 $0 test"

use python3 and pip3?

@tylerjereddy
Copy link
Contributor

32 bit linux tests on azure-pipeline

Pretty sure numba does this--I'll try to take a look today. There's also a recent Azure Pipelines blog post that seems to announce availability of Raspberry Pi arch testing and various 32 bit archs.

@charris
Copy link
Member

charris commented Oct 31, 2018

Also, might be worth simplifying the i386 tests as suggested in the note

  # this can all be replaced with:
  # apt-get install libpython2.7-dev:i386
  # CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" \
  #   linux32 python setup.py build
  # when travis updates to ubuntu 14.04

EDIT: travis was to have switched to 14.04 as default in Aug, 2017.

@mattip
Copy link
Member Author

mattip commented Nov 1, 2018

Did you try just changing things to *3 in the chroot?

@charris Then we can only run chroot python3 tests. Is that OK, or do we wish to preserve the possibility of running python2 by changing .travis.yml only?

@mattip
Copy link
Member Author

mattip commented Nov 1, 2018

this can all be replaced with ...

There are two parts to the travis_test.sh file: building and testing. That section may build a 32-bit numpy, if it links correctly with OpenBlas, but then we will need further changes to run the 32 bit python for testing.

@mattip
Copy link
Member Author

mattip commented Nov 1, 2018

This seems over complicated for its stated purpose.

Do you mean my changes or the whole chroot 32 bit testing option in travis-test.sh?

Travis itself suggests using docker, here is one example of a 32 bit docker command, it does not look less complicated than what we now have.

I could try pursuing the linux32 option if so desired

@charris
Copy link
Member

charris commented Nov 1, 2018

or do we wish to preserve the possibility of running python2.

We will be dropping Python2 support in a couple of weeks.

@mattip
Copy link
Member Author

mattip commented Nov 1, 2018

changed to only use python3 in chroot

@charris charris merged commit b4b6277 into numpy:master Nov 1, 2018
@charris
Copy link
Member

charris commented Nov 1, 2018

Thanks Matti.

@tylerjereddy
Copy link
Contributor

The 32-bit 3.x Azure linux build seems prone to stalling during unit tests, so that's where I'm stuck at the moment on that.

@tylerjereddy
Copy link
Contributor

I wonder if i.e., 32-bit build / test on a 64-bit OS without linalg acceleration libraries is prone to issues -- I think that's my current config there. My first pass at getting the :i386 linear algebra stuff working was a bit of a mess, but the library otherwise builds and runs the first 30 % of the unit tests in full mode before stalling.

@charris
Copy link
Member

charris commented Nov 1, 2018

Interesting. Does it always stall at the same test?

@tylerjereddy
Copy link
Contributor

I think so, and same stall for 2.7 & 3.5. libpython3.6-dev:i386 doesn't exist on the Azure Ubuntu version, but I think I can perhaps try libc6-dev-i386 + 3.6-7 now that I'm mimicking the numba approach with 32-bit conda env + gcc_linux-32 gxx_linux-32

@mattip mattip deleted the chroot-python3 branch March 4, 2019 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments