What needs to be done before merging the stubs into NumPy

[person142]

A couple of months ago I argued against merging the stubs into NumPy right away so that we could keep a tighter iteration loop with SciPy. Now that we are close to getting an initial version of ArrayLike over in #66 I think the situation becomes different.

As I see it, the next big challenge will be making ndarray generic over dtype. After a lot of good discussion with @seberg over in #48 it seems like the path forward there likely involves

• Waiting for NEP 41 to be implemented
• Writing a mypy plugin that handles the new dynamic dtype class creation

If that is the case, then the stubs being merged into NumPy becomes an advantage because we will be relying on unreleased features.

What do people think? Should we try to merge the stubs into NumPy soon? It might behoove me to finish writing the prototype dtype plugin to see if it will actually work like I think it will.

[charris]

I'd like to branch 1.19 next weekend, so time is short. If you feel it would be pushing things to be ready at that point it would be best to wait for 1.20. @rgommers Thoughts?

[person142]

Oh yeah, I definitely wasn't thinking of doing it for 1.19. I was thinking of early in 1.20 so that we have lots of time to try and get it right before a release.

[rgommers]

SGTM!

[seberg]

Do we need to mark/make it preliminary for a while? And if, how should that look like?

[person142]

Do we need to mark/make it preliminary for a while? And if, how should that look like?

Yeah that's a good point. I think that it will need to be preliminary certainly in the first release and perhaps for much longer-it might depend on whether we can think of a good back-compatibility story for going from ndarray[dtype] to ndarray[dtype, shape]. Since it's uncertain what the latter will look like (I think everyone believes it will rely on new typing features in Python itself), a future back-compat break seems like a very real possibility.

I can't think of a way to mark the types as experimental beyond loud warnings in the documentation/release notes though.

[seberg]

The only true thing we could do is make it from numpy import _typing as typing for now. That shouts out that it will change (since the underscore must be dropped). the question is whether that would be acceptable for you in SciPy I think.

[person142]

Ok, I'm going to tentatively say I'll try to start the migration to the main NumPy repo next weekend. If it's ok I think I'll punt on the typing vs. _typing question for now; since we're starting very far from a release we can take a beat to see how stable things feel closer to the release date.

In the meantime I'll try to wrap up a some open issues and PRs in this repo.

[person142]

I've created a milestone:

https://github.com/numpy/numpy-stubs/milestone/1

tracking things I want to resolve before the migration, in case anybody wants to see the more detailed plan there.

[emmatyping]

it might depend on whether we can think of a good back-compatibility story for going from ndarray[dtype] to ndarray[dtype, shape]

I think this should be achievable by just marking the shape somewhat like Any once shape support is added. That should allow shape support to be added without breaking existing code.

[person142]

That should allow shape support to be added without breaking existing code

@ethanhs can you say more about this? The reason I thought we would have problems is that something like this doesn't work:

from typing import Generic, TypeVar

Dtype = TypeVar('Dtype')
Shape = TypeVar('Shape')


class ndarray(Generic[Dtype, Shape]): pass

a: ndarray[int] = ndarray()

We get the error:

error: "ndarray" expects 2 type arguments, but 1 given

My understanding was that we could work around the above issue with a plugin though.

But hopefully I'm just missing something!

[emmatyping]

@person142 I imagine that a plugin would simply insert Shape[Any, ...] or something for ndarray[int].

[person142]

Going to close this and open a new issue with concrete steps planned for the migration.

[shoyer]

@person142 I imagine that a plugin would simply insert Shape[Any, ...] or something for ndarray[int].

Exactly, you can define even convenient aliases that eliminate the need for any boilerplate, e.g.,

Array = ndarray[Dtype, Any]

Array[int] is then equivalent to ndarray[int, Any].