ENH: New proposed deprecation policy

[Aloqeely]

xref #54970

Use DeprecationWarning until the last minor version before a major release then we switch to FutureWarning

[Aloqeely]

One thing to note is that when using pytest.mark.filterwarnings I have kept the ignore string as "ignore::DeprecationWarning" which we will need to change to FutureWarning once we change Pandas40DeprecationWarning, my reasoning for this is that you have to pass the module path (so ignore::pandas.util._exceptions.Pandas40DeprecationWarning) for aliased warnings, which didn't look pretty to me

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[Aloqeely]

Gentle ping @lithomas1. I would like this to move forward so I don't have to update it every time a new deprecation is added

[lithomas1]

Gentle ping @lithomas1. I would like this to move forward so I don't have to update it every time a new deprecation is added

Thanks for the reminder (and sorry that this slipped off my radar).
Will take a look this evening.

[lithomas1]

I left some more comments.

I wouldn't be comfortable merging this in by myself. Let's see what some other core devs think.

cc @pandas-dev/pandas-core for other thoughts on this.

[lithomas1]

It might be good to add some reasoning to this
(e.g. This was done to give downstream packages more time to adjust to pandas deprecations, which should reducing the amount of warnings that an user gets from code that isn't theirs.)

[Dr-Irv]

I think we'd have to also say that there would always be a minimal amount of time (6 months, maybe more??) between the last minor version before the next major release.

[Aloqeely]

a minimal amount of time (6 months, maybe more??) between the last minor version before the next major release.

Is this consistent? 3.0 was planned to be released within 6 months of releasing 2.2.0 (#57064)

[Dr-Irv]

a minimal amount of time (6 months, maybe more??) between the last minor version before the next major release.

Is this consistent? 3.0 was planned to be released within 6 months of releasing 2.2.0 (#57064)

In your last commit, you wrote "This was done to give downstream packages more time to adjust to pandas deprecations". So if we are going to have a policy about this, then we have to decide how much time is sufficient for "more time".

[lithomas1]

One thing we also could consider doing is decoupling the test changes from this PR, and just introducing Pandas40DeprecationWarning, and basic tests that it works as expected.

That way there's less chances of this creating conflicts and getting out of sync.

[bashtage]

I think switching to Future warning only one release before the release that will break is too short. My motion on this is that it would be only 6 months or so, and sometimes less. Often end users report issues to projects through future warnings. This then requires downstream to make the changes and make a release before the next pandas, or face a break (unless strictly capping, which is not common).

Producing a FutureWarning 2 releases prior to breaking things seems like a more balanced approach.

[Aloqeely]

Producing a FutureWarning 2 releases prior to breaking things seems like a more balanced approach.

That would mean any deprecations released in the last 2 minor releases will immediately be FutureWarnings which means downstream libraries won't have any time to handle/suppress these warnings. Had we implemented your idea earlier, v2.1 deprecations would immediately be FutureWarning (there are a lot of them, see #50578), one of them being the cause of #54970 being opened in the first place.

I guess if we think some deprecation needs attention by end users, we can leave it as a FutureWarning and postpone enforcing the deprecation to a later major release.

[Dr-Irv]

I think this should be implemented as a Generic class with a parameter corresponding to the version number. So you'd have PandasDeprecationWarning[4] and PandasDeprecationWarning[5], etc. This would allow people to write code that would catch a PandasDeprecationWarning without having to worry about the particular version number.

[Aloqeely]

How will the class decide whether to inherit from FutureWarning or DeprecationWarning?

[Dr-Irv]

How will the class decide whether to inherit from FutureWarning or DeprecationWarning?

You could make the base class a parameter of the Generic declaration.

[rhshadrach]

@Dr-Irv - I don't follow what you're requesting here. Could you post a small PoC? I would like to get this implemented.

[Dr-Irv]

Something like this:

_WarningTypeT = TypeVar("_WarningTypeT", DeprecationWarning, FutureWarning)
_WarningVersionT = TypeVar("_WarningVersionT", bound=int)

class PandasWarning(_WarningTypeT, Generic[_WarningTypeT, _WarningVersionT]):
    pass

CurrentDeprecationWarning: TypeAlias = PandasWarning[DeprecationWarning, 3]
CurrentFutureWarning: TypeAlias = PandasWarning[FutureWarning, 4]

Then if you want to warn about a specific version in code, you just use PandasWarning[DeprecationWarning, 4] instead of the Pandas4DeprecationWarning proposed here.

[Dr-Irv]

Well, it appears I misunderstood how the Generic stuff works and got ahead of myself.

You can't parameterize a class via a TypeVar, nor can you parameterize it via a specified integer.

So I withdraw my suggestion to use Generic. Having said that, I do think we should create classes PandasDeprecationWarning and PandasFutureWarning that are just covers over DeprecationWarning and FutureWarning respectively, so that it is easy to catch those warnings without worrying about the version number.

[rhshadrach]

I think the following fits your original intent. Comments below are the warning raised; lack of a comment means no warning seen.

class PandasWarning(Warning):
    pass

class Pandas4Warning(PandasWarning, FutureWarning):
    pass

class Pandas5Warning(PandasWarning, DeprecationWarning):
    pass

warnings.warn("Future", category=Pandas4Warning)
# Pandas4Warning: Future
warnings.warn("Deprecation", category=Pandas5Warning)
# Pandas5Warning: Deprecation

with warnings.catch_warnings():
    warnings.filterwarnings('ignore', '', PandasWarning)
    warnings.warn("Future", category=Pandas4Warning)
    warnings.warn("Deprecation", category=Pandas5Warning)

with warnings.catch_warnings():
    warnings.filterwarnings('ignore', '', FutureWarning)
    warnings.warn("Future", category=Pandas4Warning)
    warnings.warn("Deprecation", category=Pandas5Warning)
    # Pandas5Warning: Deprecation

with warnings.catch_warnings():
    warnings.filterwarnings('ignore', '', DeprecationWarning)
    warnings.warn("Future", category=Pandas4Warning)
    # Pandas4Warning: Future
    warnings.warn("Deprecation", category=Pandas5Warning)

[Dr-Irv]

Instead of

class Pandas4Warning(PandasWarning, FutureWarning):
    pass

class Pandas5Warning(PandasWarning, DeprecationWarning):
    pass

why not do either
OPTION A (leave out the versions):

class PandasFutureWarning(PandasWarning, FutureWarning):
    pass

class PandasDeprecationWarning(PandasWarning, DeprecationWarning):
    pass

OR
OPTION B (include the versions and a more "generic" way)

class Pandas4Warning(PandasWarning, FutureWarning):
    pass

class Pandas5Warning(PandasWarning, DeprecationWarning):
    pass

class PandasFutureWarning(Pandas4Warning):
    pass

class PandasDeprecationWarning(Pandas5Warning):
    pass

Then people can just trap PandasFutureWarning or PandasDeprecationWarning and not worry about version numbers. Or, with option B, they can do it either way.

[rhshadrach]

OPTION A (leave out the versions):

Part of the proposal here is to reduce the maintenance of tracking the promotion of a warning from DeprecationWarning to FutureWarning to removal. Leaving out the version is not an option in my opinion.

I'm fine with adding PandasFutureWarning / PandasDeprecationWarning.

[Dr-Irv]

Part of the proposal here is to reduce the maintenance of tracking the promotion of a warning from DeprecationWarning to FutureWarning to removal. Leaving out the version is not an option in my opinion.

OK, that makes sense.

I'm fine with adding PandasFutureWarning / PandasDeprecationWarning.

I think we have converged.

[Dr-Irv]

You would also need to update https://github.com/pandas-dev/pandas/blob/main/doc/source/development/policies.rst as part of this PR

[Dr-Irv]

@Aloqeely we had a discussion about our deprecation policy at the bi-weekly developers meeting, and we think that a PDEP should be created that would formalize the policy and allow for greater discussion. Are you willing to write up a PDEP with a proposed policy?

[Aloqeely]

Sure, I can give it a go.
Going to close this for now