Can no longer slice DatetimeIndex with datetime.date values outside the index in 1.0.0 #31501

davidia · 2020-01-31T16:39:29Z

Code Sample

import pandas as pd
import datetime as dt
s = pd.Series([0,1],pd.DatetimeIndex([dt.date(2019,1,1),dt.date(2019,1,2)]))
s[dt.date(2019,1,1):]  # works 
s[dt.date(2018,1,1):]  # error

Problem description

As of 1.0.0 you can no longer slice a DatetimeIndex with dt.date values outside the index.

Error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()

KeyError: datetime.date(2018, 1, 1)

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in get_loc(self, key, method, tolerance)
    714         try:
--> 715             return Index.get_loc(self, key, method, tolerance)
    716         except (KeyError, ValueError, TypeError):

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()

KeyError: datetime.date(2018, 1, 1)

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1514764800000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2018-01-01 00:00:00')

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1514764800000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in get_loc(self, key, method, tolerance)
    727                     stamp = stamp.tz_localize(self.tz)
--> 728                 return Index.get_loc(self, stamp, method, tolerance)
    729             except KeyError:

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2018-01-01 00:00:00')

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_slice_bound(self, label, side, kind)
   4841         try:
-> 4842             slc = self.get_loc(label)
   4843         except KeyError as err:

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in get_loc(self, key, method, tolerance)
    729             except KeyError:
--> 730                 raise KeyError(key)
    731             except ValueError as e:

KeyError: datetime.date(2018, 1, 1)

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-47-a23c04333556> in <module>
      3 s = pd.Series([0,1],pd.DatetimeIndex([dt.date(2019,1,1),dt.date(2019,1,2)]))
      4 s[dt.date(2019,1,1):]  # works
----> 5 s[dt.date(2018,1,1):]  # error

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\series.py in __getitem__(self, key)
    908             key = check_bool_indexer(self.index, key)
    909 
--> 910         return self._get_with(key)
    911 
    912     def _get_with(self, key):

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\series.py in _get_with(self, key)
    913         # other: fancy integer or otherwise
    914         if isinstance(key, slice):
--> 915             return self._slice(key)
    916         elif isinstance(key, ABCDataFrame):
    917             raise TypeError(

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\series.py in _slice(self, slobj, axis, kind)
    863 
    864     def _slice(self, slobj: slice, axis: int = 0, kind=None):
--> 865         slobj = self.index._convert_slice_indexer(slobj, kind=kind or "getitem")
    866         return self._get_values(slobj)
    867 

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in _convert_slice_indexer(self, key, kind)
   2961             indexer = key
   2962         else:
-> 2963             indexer = self.slice_indexer(start, stop, step, kind=kind)
   2964 
   2965         return indexer

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in slice_indexer(self, start, end, step, kind)
    806 
    807         try:
--> 808             return Index.slice_indexer(self, start, end, step, kind=kind)
    809         except KeyError:
    810             # For historical reasons DatetimeIndex by default supports

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in slice_indexer(self, start, end, step, kind)
   4711         slice(1, 3)
   4712         """
-> 4713         start_slice, end_slice = self.slice_locs(start, end, step=step, kind=kind)
   4714 
   4715         # return a slice

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in slice_locs(self, start, end, step, kind)
   4924         start_slice = None
   4925         if start is not None:
-> 4926             start_slice = self.get_slice_bound(start, "left", kind)
   4927         if start_slice is None:
   4928             start_slice = 0

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_slice_bound(self, label, side, kind)
   4843         except KeyError as err:
   4844             try:
-> 4845                 return self._searchsorted_monotonic(label, side)
   4846             except ValueError:
   4847                 # raise the original KeyError

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in _searchsorted_monotonic(self, label, side)
   4794     def _searchsorted_monotonic(self, label, side="left"):
   4795         if self.is_monotonic_increasing:
-> 4796             return self.searchsorted(label, side=side)
   4797         elif self.is_monotonic_decreasing:
   4798             # np.searchsorted expects ascending sort order, have to reverse

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in searchsorted(self, value, side, sorter)
    851         elif not isinstance(value, DatetimeArray):
    852             raise TypeError(
--> 853                 "searchsorted requires compatible dtype or scalar, "
    854                 f"not {type(value).__name__}"
    855             )

TypeError: searchsorted requires compatible dtype or scalar, not date

Output of `pd.show_versions()`

INSTALLED VERSIONS

commit : None
python : 3.7.1.final.0
python-bits : 64
OS : Windows
OS-release : 7
machine : AMD64
processor : Intel64 Family 6 Model 79 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.0.0
numpy : 1.17.3
pytz : 2018.9
dateutil : 2.8.1
pip : 20.0.2
setuptools : 41.6.0.post20191030
Cython : 0.29.5
pytest : 5.1.2
hypothesis : None
sphinx : None
blosc : 1.7.0
feather : None
xlsxwriter : None
lxml.etree : 4.3.2
html5lib : None
pymysql : None
psycopg2 : 2.7.7 (dt dec pq3 ext lo64)
jinja2 : 2.10.3
IPython : 7.5.0
pandas_datareader: None
bs4 : 4.7.1
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : 4.3.2
matplotlib : 3.0.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.14.0
pytables : None
pytest : 5.1.2
pyxlsb : None
s3fs : None
scipy : 1.2.1
sqlalchemy : 1.2.17
tables : None
tabulate : None
xarray : 0.12.1
xlrd : 1.2.0
xlwt : None
xlsxwriter : None
numba : 0.45.1

The text was updated successfully, but these errors were encountered:

jorisvandenbossche · 2020-01-31T16:58:59Z

@davidia Thanks for the report! That's indeed a regression, tagged it for the 1.0.1 milestone.

davidia · 2020-01-31T17:14:01Z

My pleasure, thanks for all the awesome work!

seth-p · 2020-01-31T20:26:26Z

Note that slicing with datetime objects does work. See pydata/xarray#3736 (comment).

TomAugspurger · 2020-01-31T21:54:34Z

Looking into this now.

Closes pandas-dev#31501

jorisvandenbossche added Indexing Related to indexing on series/frames, not to indexes themselves Regression Functionality that used to work in a prior pandas version Datetime Datetime data dtype labels Jan 31, 2020

jorisvandenbossche added this to the 1.0.1 milestone Jan 31, 2020

seth-p mentioned this issue Jan 31, 2020

BUG: datetime.date slicing doesn't work with Pandas 1.0.0 pydata/xarray#3736

Closed

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Jan 31, 2020

REGR: Fixed slicing DatetimeIndex with date

ddef9d9

Closes pandas-dev#31501

TomAugspurger mentioned this issue Jan 31, 2020

REGR: Fixed slicing DatetimeIndex with date #31521

Merged

jreback closed this as completed in #31521 Feb 1, 2020

seth-p mentioned this issue May 8, 2020

BUG: DatetimeIndex.slice_locs() doesn't hande date objects not in index #34077

Closed

mroeschke mentioned this issue Aug 25, 2020

DatetimeIndex.get_slice_bound(...) raises TypeErrors for unexpected YYYY-MM-DD/datetime.date/Timestamp combinations #35690

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Can no longer slice DatetimeIndex with datetime.date values outside the index in 1.0.0 #31501

Can no longer slice DatetimeIndex with datetime.date values outside the index in 1.0.0 #31501

davidia commented Jan 31, 2020 •

edited

Loading

INSTALLED VERSIONS

jorisvandenbossche commented Jan 31, 2020

Uh oh!

davidia commented Jan 31, 2020

Uh oh!

seth-p commented Jan 31, 2020

Uh oh!

TomAugspurger commented Jan 31, 2020

Uh oh!

Uh oh!

Can no longer slice DatetimeIndex with datetime.date values outside the index in 1.0.0 #31501

Can no longer slice DatetimeIndex with datetime.date values outside the index in 1.0.0 #31501

Comments

davidia commented Jan 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Sample

Problem description

Error

Output of pd.show_versions()

INSTALLED VERSIONS

jorisvandenbossche commented Jan 31, 2020

Uh oh!

davidia commented Jan 31, 2020

Uh oh!

seth-p commented Jan 31, 2020

Uh oh!

TomAugspurger commented Jan 31, 2020

Uh oh!

davidia commented Jan 31, 2020 •

edited

Loading

Output of `pd.show_versions()`