Increase coverage and other niceties for the operator/elementwise tests by honno · Pull Request #89 · data-apis/array-api-tests

honno · 2022-01-26T18:21:32Z

This PR:

Utilities ndindex.iter_indices() to greatly improve values testing
- This is used for every test via the newly introduced *_assert_against_refimpl() utils
Implement values testing for all remaining op/elwise tests
Test scalar scenarios for binary parametrized tests (i.e. the ones that cover both operators and elementwise functions)
Greatly improves error output (no more million random variables popping up when there's an error)

Additionally:

Resolves Pretty print indices #71 with sh.fmt_idx()
Resolves Test shape broadcasting for _all_ elementwise/operator tests #84

* Use `xps` dtype strategies where possible for better repr and dtype filtering * Helper for asserting broadcasted shapes * Helper `sh.iter_indices()` to wrap `ndindex` equivalent * Update `test_equal` with `sh.iter_indices()`

Also updates `test_logical_and`

Tuples give the impression of `in_dtype` being hetereogenous

Slices are not hashable!

Also moves `ah.int_to_dtype()` and renames it `mock_int_dtype()`

- Fix old usage of `mock_int_dtype` - Infer `in_stype` - Allow scalar `right` for `binary_assert_against_refimpl()` - Use util in `test_add`

Also `ignorer` -> `filter_`

honno · 2022-02-01T20:04:22Z

@asmeurer Not quite ready yet, but you might wanna skim over to see if you can identify any issues with the "reference implementation" approach I'm using here... it's a big change 😅

Notably this does still pass-but-filter special cases (and without using boolean masks). I wrote a high-level note here:

array-api-tests/array_api_tests/test_operators_and_elementwise_functions.py

Lines 45 to 65 in 9d1f4da

    
           # This module tests elementwise functions/operators against a reference 
        
           # implementation. We iterate through the input array(s) and resulting array, 
        
           # casting the indexed arrays to Python scalars and calculating the expected 
        
           # output with `refimpl` function. 
        
           # 
        
           # This is finicky to refactor, but possible and ultimately worthwhile - hence 
        
           # why these *_assert_again_refimpl() utilities exist. 
        
           # 
        
           # Values which are special-cased are generated and passed, but are filtered by 
        
           # the `filter_` callable before they can be asserted against `refimpl`. We 
        
           # automatically generate tests for special cases in the special_cases/ dir. We 
        
           # still pass them here so as to ensure their presence doesn't affect the outputs 
        
           # respective to non-special-cased elements. 
        
           # 
        
           # By default, results are casted to scalars the same way that the inputs are. 
        
           # You can specify a cast via `res_stype, i.e. when a function accepts numerical 
        
           # inputs but returns boolean arrays. 
        
           # 
        
           # By default, floating-point functions/methods are loosely asserted against. Use 
        
           # `strict_check=True` when they should be strictly asserted against, i.e. 
        
           # when a function should return intergrals.

asmeurer · 2022-02-01T20:32:49Z

Let's make sure we document that we're doing this and that any errors should usually mean the function is wrong, but it could also mean our tolerances need to be loosened.

asmeurer · 2022-02-01T20:34:59Z

My biggest worry here is float32, since the reference implementation will operate on float64. And if we ever add a 16-bit float, that will be even worse. You might want to test this on something like cupy just to see if it works on an accelerator.

asmeurer · 2022-02-01T20:39:40Z

array_api_tests/test_operators_and_elementwise_functions.py

+# when a function should return intergrals.
+
+
+def isclose(a: float, b: float, rel_tol: float = 0.25, abs_tol: float = 1) -> bool:


I guess since we are testing one float at a time, just using math.isclose here is fine. In general though, we will want an isclose, and corresponding assert_approximately_equal method that only use the array API methods so that they can be called on arrays (I will need this for the linalg tests).

I don't think it will be an issue, but if we do run into issues with float32 and the fact that Python float is 64-bit only, we may need to make this use array operations too. That would also give us more control over the exact formula used for "closeness", though I doubt that will be an issue either given our very loose tolerances.

In general though, we will want an isclose, and corresponding assert_approximately_equal method that only use the array API methods so that they can be called on arrays (I will need this for the linalg tests).

The *_assert_against_refimpl utils do this along an array/arrays, only using the Array API endpoints of indexing and scalar casting. I've kept this in just the elwise/op file for now as it's quite particular to those tests, but you might want to draw inspiration from it.

I don't think it will be an issue, but if we do run into issues with float32 and the fact that Python float is 64-bit only, we may need to make this use array operations too.

Ah I see, good to keep in mind.

Keeps all refimpl logic near eachother

honno · 2022-02-02T12:08:21Z

Happy with this PR now.

honno added 14 commits January 28, 2022 11:09

Updates to op/elwise tests

f661a23

* Use `xps` dtype strategies where possible for better repr and dtype filtering * Helper for asserting broadcasted shapes * Helper `sh.iter_indices()` to wrap `ndindex` equivalent * Update `test_equal` with `sh.iter_indices()`

sh.fmt_idx() helper

a590f8d

Better values testing in test_not_equal

d7e5e63

Better values testing for bitwise op/elwise tests

1a54bd4

Context objects for unary/binary params

2f8492b

Apply iter_indices() logic to binary op/elwise tests

4623214

Update test_remainder

4b2c41e

Move broadcast_shapes() to shape_helpers.py

1927c10

Skip sh.iter_indices() generation for 0-sided shapes

bb836b7

Also updates `test_logical_and`

Values testing for test_sign

f11a6d0

Values testing for test_add and test_subtract

47424e8

Rudimentary values testing refactor, updates to logical elwise tests

2077986

Favour lists compared to tuples for ph.assert_dtypes()

66a1fd4

Tuples give the impression of `in_dtype` being hetereogenous

Favour lists for ph.assert_result_shape()

b6d05da

honno force-pushed the values-testing branch from 06a1944 to b6d05da Compare January 28, 2022 11:21

honno added 15 commits January 28, 2022 13:08

Remove lru_cache use in sh.fmt_idx()

af6d150

Slices are not hashable!

Refactor parametrized unary tests

799b4e6

Also moves `ah.int_to_dtype()` and renames it `mock_int_dtype()`

Op/elwise fixes and improvements

e2b69df

- Fix old usage of `mock_int_dtype` - Infer `in_stype` - Allow scalar `right` for `binary_assert_against_refimpl()` - Use util in `test_add`

binary_param_assert_against_refimpl() to refactor elwise+op tests

3dfd665

Refactor remaining parametrized elwise+op tests

a4a7e04

Also `ignorer` -> `filter_`

Finish elwise TODOs

4d849f1

Fix typing issues with refimpl utils

5a82a33

Remove redundant in_stype arg in refimpl utils

7386615

Skip when refimpl overflows

80d2909

Values testing for remaining tests for elwise funcs starting with a

9521f6b

Defaults for expr_template in refimpl utils

e50fc1a

Refactor majority of elwise tests with refimpl utils

4a364a5

strict_check kwarg for refiml utils for testing integrals

56aa06d

Pass but filter out-of-range values for trig function tests

dfda4f5

Extend note on refimpl utils

9d1f4da

asmeurer reviewed Feb 1, 2022

View reviewed changes

honno added 6 commits February 2, 2022 10:02

Refactor remaining elwise/op tests

e72184e

Favour use of operator for refimpl

9edcfcc

Filter undefined dtypes in hh.two_mutual_arrays()

6e8cda6

Generic type hint for refimpl args

493f669

Introduce right_scalar_assert_against_refimpl()

d924ce4

Keeps all refimpl logic near eachother

Note why you'd want to not strictly check int outputs

3c85cae

honno marked this pull request as ready for review February 2, 2022 11:54

honno mentioned this pull request Feb 2, 2022

Inplace shapes #91

Merged

honno merged commit 0f63fab into data-apis:master Feb 3, 2022

honno deleted the values-testing branch February 8, 2022 10:05

honno mentioned this pull request Feb 9, 2022

Error messages should print arrays #69

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increase coverage and other niceties for the operator/elementwise tests#89

Increase coverage and other niceties for the operator/elementwise tests#89
honno merged 35 commits intodata-apis:masterfrom
honno:values-testing

honno commented Jan 26, 2022 •

edited

Loading

Uh oh!

honno commented Feb 1, 2022

Uh oh!

asmeurer commented Feb 1, 2022

Uh oh!

asmeurer commented Feb 1, 2022

Uh oh!

asmeurer Feb 1, 2022

Uh oh!

honno Feb 2, 2022 •

edited

Loading

Uh oh!

honno commented Feb 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		# when a function should return intergrals.


		def isclose(a: float, b: float, rel_tol: float = 0.25, abs_tol: float = 1) -> bool:

Conversation

honno commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

honno commented Feb 1, 2022

Uh oh!

asmeurer commented Feb 1, 2022

Uh oh!

asmeurer commented Feb 1, 2022

Uh oh!

asmeurer Feb 1, 2022

Choose a reason for hiding this comment

Uh oh!

honno Feb 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

honno commented Feb 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

honno commented Jan 26, 2022 •

edited

Loading

honno Feb 2, 2022 •

edited

Loading