With just ssize_t somehow I get errors on windows (older python?). See https://github.com/numpy/numpy/actions/runs/7944728697
Try changing to Py_ssize_t.

Or maybe it should just be fortran_int???

Looks like it is called with fortran_int, but the size_t goes back before c++, but was still called with a fortran_int. That seems a dangerous combination, as C casts signed to unsigned without warnings. The affected routines are only called by cholesky, so changing the argument type should work. Special casing zero might be a good idea, a zero check should be very cheap.

The original code goes back to 2012. I was wondering, as we seldom use size_t.

@charris - do you think it makes sense to just use fortran_int in the loop? It is not like this is going to be anything other than int32 or int64 anyway.

EDIT: though tests pass now, at least.

Lets go with fortran_int. That is what is passed in, and we are dealing with a matrix returned by an LAPACK call.
If inconsistencies turn up, they can be dealt with at a higher level.

The functions triu_matrix and tril_matrix could use a rename, as what they do is zero the unused part of the matrix returned by LAPACK, maybe something like clean_fortran_triu, etc., with a documentation comment. But that might be out of scope here.

If there was no chance of other uses in the future, I would just pass in params.

@charris - Sounds good, I went with all your suggestions - the zeroing is done for Cholesky only, so it makes sense to keep it close (especially as it is meant to be inlined).

Good name choice! Thanks Marten.

The failing test is unrelated, not sure what is going on there.