Which versions of numpy do we need to test

[d-v-b]

Thanks @dstansby! We should decide on a testing strategy for different numpy versions, but that shouldn't stop this PR.

Originally posted by @d-v-b in #3636 (review)

Now that we don't support numpy 1.x, do we still need 2 versions of numpy in our test matrix, or can we just test against the oldest supported version of numpy 2 in our main tests, and add numpy to our upstream tests (if it's not there already)