deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) (#337)

* feat: Drop support for Python 3.7 and 3.8

Removes support for Python 3.7 and 3.8, establishing Python 3.9 as the new minimum supported version.

This change involves:
- Updating `python_requires` and classifiers in `setup.py`.
- Modifying Python versions in `noxfile.py` (default, unit tests, system tests) and ensuring constraint file logic remains correct.
- Updating the GitHub Actions workflow (`unittest.yml`) matrix, runner, and coverage job version.
- Deleting constraint files for Python 3.7 and 3.8 (`testing/constraints-3.7.txt`, `testing/constraints-3.8.txt`).
- Removing Kokoro sample configuration directories (`.kokoro/samples/python3.7/`, `.kokoro/samples/python3.8/`).
- Updating supported version mentions in `README.rst`.
- Removing 3.7 and 3.8 from the `ALL_VERSIONS` list in `samples/snippets/noxfile.py`.

* Updates python version in lint.yml

* Updates owlbot, removing reference to 3.8

* Updates CONTRIBUTING.rst

* updates pytest warnings

* Removes test_samples-impl ref to older virtualenv package

* Removes references to pandas older than 1.5.0

* Removes pandas older than 1.5 and misc changes

* updates pandas in setup.py

* more updates related to pandas

* still broken

* Updates FutureWarning tests to account for unittest coverage

* Updates json array type tests to account for unittest coverage

* updates python version checks to ensure coverage

* update json test for unittest coverage

* Update pandas_backports unittests to ensure coverage

* Updates per review comments

* moves class from version specific compliance file to generic file

* Removes weird cut and paste error

* fix linting errors

* updates import statement to ensure import of JSONArrowType

* Revise required github status checks

* update linting

* temporarily marking a class as no cover

* more updates

* marked several snippets as pragma no cover

* updates linting

* Updates constraints and setup.py

* migrates class from one time compliance file to another

* updating pyarrow version

* Updates linting

* removes determine all and module reload tests

* updates re: ndarrybackedextensionarray

* testing blacken as part of owlbot processing using 3.8

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates blacken to 3.10

* update python version in lint.yml

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates owlbot.py

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates owlbot.py

* testing lint.yml

* testing linting issue

* testing linting issue

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates lint_setup session

* Update noxfile.py

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Author: 3 people

.github/sync-repo-settings.yaml

@@ -11,7 +11,6 @@ branchProtectionRules:
     - 'cla/google'
     - 'docs'
     - 'lint'
-    - 'unit (3.8)'
     - 'unit (3.9)'
     - 'unit (3.10)'
     - 'unit (3.11)'

.github/workflows/unittest.yml

@@ -5,13 +5,11 @@ on:
 name: unittest
 jobs:
   unit:
-    # TODO(https://github.com/googleapis/gapic-generator-python/issues/2303): use `ubuntu-latest` once this bug is fixed.
-    # Use ubuntu-22.04 until Python 3.7 is removed from the test matrix
-    # https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
-    runs-on: ubuntu-22.04
+    # Use `ubuntu-latest` runner.
+    runs-on: ubuntu-latest
     strategy:
       matrix:
-        python: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
+        python: ['3.9', '3.10', '3.11', '3.12', '3.13']
     steps:
     - name: Checkout
       uses: actions/checkout@v4
@@ -103,7 +101,7 @@ jobs:
     - name: Setup Python
       uses: actions/setup-python@v5
       with:
-        python-version: "3.8"
+        python-version: "3.9"
     - name: Install coverage
       run: |
         python -m pip install --upgrade setuptools pip wheel

CONTRIBUTING.rst

@@ -143,12 +143,12 @@ Running System Tests
    $ nox -s system
 
    # Run a single system test
-   $ nox -s system-3.8 -- -k <name of test>
+   $ nox -s system-3.9 -- -k <name of test>
 
 
   .. note::
 
-      System tests are only configured to run under Python 3.8.
+      System tests are only configured to run under Python 3.9.
       For expediency, we do not run them in older versions of Python 3.
 
   This alone will not run the tests. You'll need to change some local

README.rst

@@ -34,11 +34,11 @@ dependencies.
 
 Supported Python Versions
 ^^^^^^^^^^^^^^^^^^^^^^^^^
-Python >= 3.7
+Python >= 3.9
 
 Unsupported Python Versions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Python <= 3.6.
+Python <= 3.8.
 
 
 Mac/Linux

db_dtypes/__init__.py

@@ -21,15 +21,14 @@
 import warnings
 
 import numpy
-import packaging.version
 import pandas
 import pandas.api.extensions
 from pandas.errors import OutOfBoundsDatetime
 import pyarrow
 import pyarrow.compute
 
 from db_dtypes import core
-from db_dtypes.version import __version__
+from db_dtypes.json import JSONArray, JSONDtype, JSONArrowType  # noqa: F401
 
 from . import _versions_helpers
 
@@ -47,15 +46,6 @@
 _NP_BOX_DTYPE = "datetime64[us]"
 
 
-# To use JSONArray and JSONDtype, you'll need Pandas 1.5.0 or later. With the removal
-# of Python 3.7 compatibility, the minimum Pandas version will be updated to 1.5.0.
-if packaging.version.Version(pandas.__version__) >= packaging.version.Version("1.5.0"):
-    from db_dtypes.json import JSONArray, JSONArrowType, JSONDtype
-else:
-    JSONArray = None
-    JSONDtype = None
-
-
 @pandas.api.extensions.register_extension_dtype
 class TimeDtype(core.BaseDatetimeDtype):
     """
@@ -364,23 +354,13 @@ def _check_python_version():
 
 _check_python_version()
 
-
-if not JSONArray or not JSONDtype:
-    __all__ = [
-        "__version__",
-        "DateArray",
-        "DateDtype",
-        "TimeArray",
-        "TimeDtype",
-    ]
-else:
-    __all__ = [
-        "__version__",
-        "DateArray",
-        "DateDtype",
-        "JSONDtype",
-        "JSONArray",
-        "JSONArrowType",
-        "TimeArray",
-        "TimeDtype",
-    ]
+__all__ = [
+    "__version__",
+    "DateArray",
+    "DateDtype",
+    "TimeArray",
+    "TimeDtype",
+    "JSONDtype",
+    "JSONArray",
+    "JSONArrowType",
+]

db_dtypes/core.py

@@ -18,6 +18,7 @@
 import pandas
 import pandas.api.extensions
 from pandas.api.types import is_dtype_equal, is_list_like, is_scalar, pandas_dtype
+from pandas.core.arrays import _mixins
 
 from db_dtypes import pandas_backports
 
@@ -42,9 +43,7 @@ def construct_from_string(cls, name: str):
         return cls()
 
 
-class BaseDatetimeArray(
-    pandas_backports.OpsMixin, pandas_backports.NDArrayBackedExtensionArray
-):
+class BaseDatetimeArray(pandas_backports.OpsMixin, _mixins.NDArrayBackedExtensionArray):
     # scalar used to denote NA value inside our self._ndarray, e.g. -1 for
     # Categorical, iNaT for Period. Outside of object dtype, self.isna() should
     # be exactly locations in self._ndarray with _internal_fill_value. See:
@@ -186,9 +185,6 @@ def median(
         keepdims: bool = False,
         skipna: bool = True,
     ):
-        if not hasattr(pandas_backports, "numpy_validate_median"):
-            raise NotImplementedError("Need pandas 1.3 or later to calculate median.")
-
         pandas_backports.numpy_validate_median(
             (),
             {"out": out, "overwrite_input": overwrite_input, "keepdims": keepdims},

db_dtypes/pandas_backports.py

@@ -19,18 +19,13 @@
 the versions in the later versions of pandas.
 """
 
-from typing import Any
-
-import numpy
 import packaging.version
 import pandas
-from pandas.api.types import is_integer
 import pandas.compat.numpy.function
-import pandas.core.nanops
 
 pandas_release = packaging.version.parse(pandas.__version__).release
 
-# Create aliases for private methods in case they move in a future version.
+# # Create aliases for private methods in case they move in a future version.
 nanall = pandas.core.nanops.nanall
 nanany = pandas.core.nanops.nanany
 nanmax = pandas.core.nanops.nanmax
@@ -40,9 +35,8 @@
 numpy_validate_max = pandas.compat.numpy.function.validate_max
 numpy_validate_min = pandas.compat.numpy.function.validate_min
 
-if pandas_release >= (1, 3):
-    nanmedian = pandas.core.nanops.nanmedian
-    numpy_validate_median = pandas.compat.numpy.function.validate_median
+nanmedian = pandas.core.nanops.nanmedian
+numpy_validate_median = pandas.compat.numpy.function.validate_median
 
 
 def import_default(module_name, force=False, default=None):
@@ -78,83 +72,3 @@ def import_default(module_name, force=False, default=None):
 class OpsMixin:
     def _cmp_method(self, other, op):  # pragma: NO COVER
         return NotImplemented
-
-
-# TODO: use public API once pandas 1.5 / 2.x is released.
-# See: https://github.com/pandas-dev/pandas/pull/45544
-@import_default("pandas.core.arrays._mixins", pandas_release < (1, 3))
-class NDArrayBackedExtensionArray(pandas.core.arrays.base.ExtensionArray):
-    def __init__(self, values, dtype):
-        assert isinstance(values, numpy.ndarray)
-        self._ndarray = values
-        self._dtype = dtype
-
-    @classmethod
-    def _from_backing_data(cls, data):
-        return cls(data, data.dtype)
-
-    def __getitem__(self, index):
-        value = self._ndarray[index]
-        if is_integer(index):
-            return self._box_func(value)
-        return self.__class__(value, self._dtype)
-
-    def __setitem__(self, index, value):
-        self._ndarray[index] = self._validate_setitem_value(value)
-
-    def __len__(self):
-        return len(self._ndarray)
-
-    @property
-    def shape(self):
-        return self._ndarray.shape
-
-    @property
-    def ndim(self) -> int:
-        return self._ndarray.ndim
-
-    @property
-    def size(self) -> int:
-        return self._ndarray.size
-
-    @property
-    def nbytes(self) -> int:
-        return self._ndarray.nbytes
-
-    def copy(self):
-        return self[:]
-
-    def repeat(self, n):
-        return self.__class__(self._ndarray.repeat(n), self._dtype)
-
-    def take(
-        self,
-        indices,
-        *,
-        allow_fill: bool = False,
-        fill_value: Any = None,
-        axis: int = 0,
-    ):
-        from pandas.core.algorithms import take
-
-        if allow_fill:
-            fill_value = self._validate_scalar(fill_value)
-
-        new_data = take(
-            self._ndarray,
-            indices,
-            allow_fill=allow_fill,
-            fill_value=fill_value,
-            axis=axis,
-        )
-        return self._from_backing_data(new_data)
-
-    @classmethod
-    def _concat_same_type(cls, to_concat, axis=0):
-        dtypes = {str(x.dtype) for x in to_concat}
-        if len(dtypes) != 1:
-            raise ValueError("to_concat must have the same dtype (tz)", dtypes)
-
-        new_values = [x._ndarray for x in to_concat]
-        new_values = numpy.concatenate(new_values, axis=axis)
-        return to_concat[0]._from_backing_data(new_values)  # type: ignore[arg-type]

db_dtypes/version.py

@@ -12,4 +12,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-__version__ = "1.4.2"
+__version__ = "1.4.2"  # pragma: NO COVER

noxfile.py

@@ -32,11 +32,9 @@
 ISORT_VERSION = "isort==5.11.0"
 LINT_PATHS = ["docs", "db_dtypes", "tests", "noxfile.py", "setup.py"]
 
-DEFAULT_PYTHON_VERSION = "3.8"
+DEFAULT_PYTHON_VERSION = "3.9"
 
 UNIT_TEST_PYTHON_VERSIONS: List[str] = [
-    "3.7",
-    "3.8",
     "3.9",
     "3.10",
     "3.11",
@@ -56,7 +54,7 @@
 UNIT_TEST_EXTRAS: List[str] = []
 UNIT_TEST_EXTRAS_BY_PYTHON: Dict[str, List[str]] = {}
 
-SYSTEM_TEST_PYTHON_VERSIONS: List[str] = ["3.8"]
+SYSTEM_TEST_PYTHON_VERSIONS: List[str] = ["3.9"]
 SYSTEM_TEST_STANDARD_DEPENDENCIES: List[str] = [
     "mock",
     "pytest",
@@ -88,7 +86,10 @@
 nox.options.error_on_missing_interpreters = True
 
 
-@nox.session(python=DEFAULT_PYTHON_VERSION)
+# TODO: the linting process still uses python 3.8.
+# As soon as that gets upgraded, we should be able to revert this session
+# to using the DEFAULT_PYTHON_VERSION.
+@nox.session(python="3.8")
 def lint(session):
     """Run linters.
 
@@ -105,7 +106,11 @@ def lint(session):
     session.run("flake8", "db_dtypes", "tests")
 
 
-@nox.session(python=DEFAULT_PYTHON_VERSION)
+# TODO: the owlbot-python docker image still has python 3.8 installed (
+# and only 3.8).
+# As soon as that gets upgraded, we should be able to revert this session
+# to using the DEFAULT_PYTHON_VERSION.
+@nox.session(python="3.8")
 def blacken(session):
     """Run black. Format code to uniform standard."""
     session.install(BLACK_VERSION)
@@ -137,7 +142,10 @@ def format(session):
     )
 
 
-@nox.session(python=DEFAULT_PYTHON_VERSION)
+# TODO: the linting process still uses python 3.8.
+# As soon as that gets upgraded, we should be able to revert this session
+# to using the DEFAULT_PYTHON_VERSION.
+@nox.session(python="3.8")
 def lint_setup_py(session):
     """Verify that setup.py is valid (including RST check)."""
     session.install("docutils", "pygments")

owlbot.py

@@ -28,7 +28,7 @@
 # Add templated files
 # ----------------------------------------------------------------------------
 templated_files = common.py_library(
-    system_test_python_versions=["3.8"],
+    system_test_python_versions=["3.9"],
     cov_level=100,
     intersphinx_dependencies={
         "pandas": "https://pandas.pydata.org/pandas-docs/stable/"

pytest.ini

@@ -2,13 +2,6 @@
 filterwarnings =
     # treat all warnings as errors
     error
-    # Remove once support for python 3.7 and 3.8 is dropped
-    # Ignore warnings from older versions of pandas which still have python 3.7/3.8 support
-    ignore:.*distutils Version classes are deprecated:DeprecationWarning
-    ignore:.*resolve package from __spec__ or __package__, falling back on __name__ and __path__:ImportWarning
-    # Remove once https://github.com/dateutil/dateutil/issues/1314 is fixed
-    # dateutil is a dependency of pandas
-    ignore:datetime.datetime.utcfromtimestamp\(\) is deprecated:DeprecationWarning:dateutil.tz.tz
     # Remove once https://github.com/googleapis/python-db-dtypes-pandas/issues/227 is fixed
     ignore:.*any.*with datetime64 dtypes is deprecated and will raise in a future version:FutureWarning
     ignore:.*all.*with datetime64 dtypes is deprecated and will raise in a future version:FutureWarning 

samples/snippets/requirements-test.txt

@@ -1,2 +1 @@
-pytest===7.4.4; python_version == '3.7'  # prevents dependabot from upgrading it
-pytest==8.3.3; python_version > '3.7'
+pytest==8.3.5

setup.py

@@ -30,10 +30,10 @@
 release_status = "Development Status :: 5 - Production/Stable"
 
 dependencies = [
-    "packaging >= 17.0",
-    "pandas >= 1.2.0",
-    "pyarrow>=3.0.0",
-    "numpy >= 1.16.6",
+    "numpy >= 1.24.0",
+    "packaging >= 24.2.0",
+    "pandas >= 1.5.3",
+    "pyarrow >= 13.0.0",
 ]
 
 package_root = os.path.abspath(os.path.dirname(__file__))
@@ -63,8 +63,6 @@ def readme():
         "License :: OSI Approved :: Apache Software License",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
         "Programming Language :: Python :: 3.11",
@@ -75,6 +73,6 @@ def readme():
     ],
     platforms="Posix; MacOS X; Windows",
     install_requires=dependencies,
-    python_requires=">=3.7",
+    python_requires=">=3.9",
     tests_require=["pytest"],
 )