Different behaviour for '==' and 'equals' on DatetimeIndex

[mgeens]

Code Sample

>>> import pandas as pd
>>> import pytz
>>> x = pd.DataFrame(data=[10,20,30], index=pd.date_range(start='2018-06-01T00:00:00Z', periods=3, freq='1h', tz=pytz.UTC))
>>> y = pd.DataFrame(data=[10,20,30], index=pd.date_range(start='2018-06-01T02:00:00', periods=3, freq='1h', tz=pytz.timezone('Europe/Brussels')))
>>> x.index[0]
Timestamp('2018-06-01 00:00:00+0000', tz='UTC', freq='H')
>>> y.index[0]
Timestamp('2018-06-01 02:00:00+0200', tz='Europe/Brussels', freq='H')
>>> x.index[0] == y.index[0]
True
>>> x.index == y.index
array([ True,  True,  True])
>>> x.equals(y)
False
>>> x.index.equals(y.index)
False

Problem description

When handling two dataframes which had the same datetimes in their index but different timezones, I was caught off-guard when they turned out to be considered equal in some cases but not in others, like in the example above.

Is it intentional or a bug that the two equals calls return False?

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.3.final.0 python-bits: 64 OS: Linux OS-release: 4.13.0-45-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.23.1
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: None
numpy: 1.14.5
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[mgeens]

Not sure to which degree this is related to #18435 or not.

[mroeschke]

equal checks if two objects are the same while == check if two objects are equal. In you example, the times are equal although they are in different timezones, but they aren't the same object

In [1]: import pandas as pd

In [2]: utc = pd.Timestamp('2018-06-01 00:00:00+0000', tz='UTC', freq='H')

In [3]: brus = pd.Timestamp('2018-06-01 02:00:00+0200', tz='Europe/Brussels', freq='H')

In [4]: brus
Out[4]: Timestamp('2018-06-01 02:00:00+0200', tz='Europe/Brussels', freq='H')

In [5]: utc
Out[5]: Timestamp('2018-06-01 00:00:00+0000', tz='UTC', freq='H')

# what .equals() checks, different objects
In [6]: brus is utc
Out[6]: False

# the same time although in different timezones.
In [7]: brus == utc
Out[7]: True

[mgeens]

But if equals checks if objects are the same with is, why is equals true in the examples below? The definition at https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.DataFrame.equals.html seems to leave some room for interpretation. I suppose that the term 'same elements' there implies every simple value that isn't wrapped in some container structure?

>>> import pandas as pd
>>> import pytz
>>> d1 = pd.date_range(start='2018-06-01T00:00:00Z', periods=3, freq='1h', tz=pytz.UTC)
>>> d2 = pd.date_range(start='2018-06-01T00:00:00Z', periods=3, freq='1h', tz=pytz.UTC)
>>> d1 is d2
False
>>> d1 == d2
array([ True,  True,  True])
>>> d1.equals(d2)
True
>>> x = pd.DataFrame(data=[10,20,30], index=d1)
>>> y = pd.DataFrame(data=[10,20,30], index=d2)
>>> x is y
False
>>> x == y
                              0
2018-06-01 00:00:00+00:00  True
2018-06-01 01:00:00+00:00  True
2018-06-01 02:00:00+00:00  True
>>> x.equals(y)
True

In your timestamp example, if the timezones are the same, equals will be true, but is will still be false.

>>> utc1 = pd.Timestamp('2018-06-01 00:00:00+0000', tz='UTC', freq='H')
>>> utc2 = pd.Timestamp('2018-06-01 00:00:00+0000', tz='UTC', freq='H')
>>> utc1 is utc2
False
>>> utc1 == utc2
True