Fixes timezone drop when encoding it in msgpack #25195

cgangwar11 · 2019-02-06T19:56:58Z

closes msgpack drops timezone info #25148
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

After

In [1]: import numpy as np
   ...: import pandas as pd
   ...:
   ...: idx = pd.date_range(start='2018-12-02 14:50:00-07:00', end='2018-12-03 03:11:00-07:00', freq='1min')
   ...:
   ...:
   ...: df = pd.DataFrame(np.random.randn(len(idx), 1), index=idx, columns=['A'])

In [2]: df.index[0]
Out[2]: Timestamp('2018-12-02 14:50:00-0700', tz='pytz.FixedOffset(-420)', freq='T')

In [3]: df.to_msgpack('~/foo.msgpack')

In [4]: df = pd.read_msgpack('~/foo.msgpack')

In [5]: df.index[0]
Out[5]: Timestamp('2018-12-02 14:50:00-0700', tz='pytz.FixedOffset(-420)', freq='T')

pep8speaks · 2019-02-06T19:57:05Z

Hello @cgangwar11! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on February 09, 2019 at 18:27 Hours UTC

codecov · 2019-02-06T20:11:39Z

Codecov Report

Merging #25195 into master will decrease coverage by 49.5%.
The diff coverage is 33.33%.

@@             Coverage Diff             @@
##           master   #25195       +/-   ##
===========================================
- Coverage   92.35%   42.84%   -49.51%     
===========================================
  Files         166      166               
  Lines       52314    52315        +1     
===========================================
- Hits        48312    22414    -25898     
- Misses       4002    29901    +25899

Flag	Coverage Δ
#multiple	`?`
#single	`42.84% <33.33%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/io/packers.py	`14.89% <33.33%> (-73.32%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/core/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
pandas/io/formats/html.py	`0% <0%> (-99.35%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-95.46%)`	⬇️
pandas/io/sas/sas7bdat.py	`0% <0%> (-91.17%)`	⬇️
pandas/io/sas/sas_xport.py	`0% <0%> (-90.15%)`	⬇️
... and 124 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 638ddeb...99cab5a. Read the comment docs.

codecov · 2019-02-06T20:11:41Z

Codecov Report

❗ No coverage uploaded for pull request base (master@2448e52). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master   #25195   +/-   ##
=========================================
  Coverage          ?   92.16%           
=========================================
  Files             ?      166           
  Lines             ?    52314           
  Branches          ?        0           
=========================================
  Hits              ?    48214           
  Misses            ?     4100           
  Partials          ?        0

Flag	Coverage Δ
#multiple	`90.62% <100%> (?)`
#single	`42.25% <9.52%> (?)`

Impacted Files	Coverage Δ
pandas/io/packers.py	`88.85% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2448e52...b1347ae. Read the comment docs.

mroeschke · 2019-02-06T22:37:20Z

doc/source/whatsnew/v0.24.2.rst

@@ -52,7 +52,7 @@ Bug Fixes
 **I/O**

 - Bug in reading a HDF5 table-format ``DataFrame`` created in Python 2, in Python 3 (:issue:`24925`)
-
+- Bug in retaining timezone in DateTimeIndex when saving and loading the DataFrame from msgpack. (:issue:`25148`)


Double back ticks around msgpack. Single backticks around DatetimeIndex and DataFrame with :class: before each

mroeschke · 2019-02-06T22:39:06Z

pandas/tests/io/test_packers.py

@@ -393,6 +393,12 @@ def categorical_index(self):
        result = self.encode_decode(df)
        tm.assert_frame_equal(result, df)

+    def test_timezone_encode_decode(self):


Can you parameterize using the tz_aware_fixture?

mroeschke

Overall, I suspect the fix is too narrowly focused on FixedOffets.

mroeschke · 2019-02-06T22:41:18Z

pandas/io/packers.py

@@ -390,7 +390,8 @@ def encode(obj):

            # store tz info and data as UTC
            if tz is not None:
-                tz = u(tz.zone)
+                tz = u(tz.zone) if not isinstance(tz, _FixedOffset) \


What happens if the timezone is not a FixedOffset? e.g. pytz.UTC or dateutil.tz.tzutc?

@mroeschke
Encoding process of TimeStamp is using zone attribute and normalize the datetime to utc.
Now there is no zone attribute in FixedOffset hence we will not be able to encode it and loosing the offset value in the process.
So i extracted offset value and encode that.

>>> Timestamp('2018-12-02 14:50:00',tz='UTC').tzinfo.zone 'UTC' >>> Timestamp('2018-12-02 14:50:00',tz=pytz.FixedOffset(-420)).tzinfo.zone is None True

Got it. Thanks for the explanation.

I'd change this to a try/except block instead of an if/else.

Another option would be to str the timezone otherwise?

In [3]: str(tz) Out[3]: 'pytz.FixedOffset(60)' In [4]: import dateutil In [5]: str(dateutil.tz.tzutc()) Out[5]: 'tzutc()' In [7]: dateutil.tz.tzutc().zone --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-7-d8163524e913> in <module>() ----> 1 dateutil.tz.tzutc().zone AttributeError: 'tzutc' object has no attribute 'zone'

@mroeschke
Converting the timezone object into string to encode is a great idea.
But I will have to use something like this in decoding side.

tz = eval('pytz.FixedOffset(60)' )

Using eval doesn't seem like a good idea.

Agreed. I'd prefer your minutes extraction method then.

I imagine there will need to be separate handling for dateutil timezones all together.

jreback

there must be something going else wrong here. The data is stored as utc and the tz is stored as well. what exactly is the problem?

jreback · 2019-02-09T17:45:51Z

doc/source/whatsnew/v0.24.2.rst

@@ -52,7 +52,7 @@ Bug Fixes
 **I/O**

 - Bug in reading a HDF5 table-format ``DataFrame`` created in Python 2, in Python 3 (:issue:`24925`)
-
+- Bug in retaining timezone in :class:`DateTimeIndex` when saving and loading the :class"`DataFrame` from ``msgpack``. (:issue:`25148`)


DatetimeIndex, can you use :meth:read_msgpack here

@jreback
Yes we were encoding tz using two value (zone as a str) and (Time value in utc format).
As per pandas DateTimeIndex documentation tz could be

tz : pytz.timezone or dateutil.tz.tzfile

The current encoding version is capable of encoding pytz.timezone but not the dateutil timezones.
Infact code will throw this exception when we try that.

>>> df1 = pd.DataFrame({'A':pd.date_range(start="2018-08-01",end="2018-08-30",tz=dateutil.tz.tzutc())}) >>> df1.to_msgpack("~/xyz.msg") >>> a = pd.read_msgpack("~/xyz.msg") Traceback (most recent call last): File "/home/chandan/Softwares/pandas/pandas/core/dtypes/common.py", line 2037, in pandas_dtype npdtype = np.dtype(dtype) TypeError: Invalid datetime unit in metadata string "[ns, tzutc()]" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/chandan/Softwares/pandas/pandas/io/packers.py", line 207, in read_msgpack return read(fh) File "/home/chandan/Softwares/pandas/pandas/io/packers.py", line 187, in read unpacked_obj = list(unpack(fh, encoding=encoding, **kwargs)) File "pandas/io/msgpack/_unpacker.pyx", line 483, in pandas.io.msgpack._unpacker.Unpacker.__next__ return self._unpack(unpack_construct, None, 1) File "pandas/io/msgpack/_unpacker.pyx", line 410, in pandas.io.msgpack._unpacker.Unpacker._unpack ret = execute(&self.ctx, self.buf, self.buf_tail, &self.buf_head) File "/home/chandan/Softwares/pandas/pandas/io/packers.py", line 667, in decode blocks = [create_block(b) for b in obj[u'blocks']] File "/home/chandan/Softwares/pandas/pandas/io/packers.py", line 667, in <listcomp> blocks = [create_block(b) for b in obj[u'blocks']] File "/home/chandan/Softwares/pandas/pandas/io/packers.py", line 648, in create_block b[u'compress']), b[u'shape']) File "/home/chandan/Softwares/pandas/pandas/io/packers.py", line 322, in unconvert dtype = pandas_dtype(dtype).base File "/home/chandan/Softwares/pandas/pandas/core/dtypes/common.py", line 2043, in pandas_dtype dtype)) TypeError: data type 'datetime64[ns, tzutc()]' not understood

jreback · 2019-03-20T02:06:21Z

an you merge master and update

jreback · 2019-05-12T21:27:34Z

good idea, but closing as stale

BUG: pandas-dev#25148

702d04f

cgangwar11 changed the title ~~BUG: #25148~~ Fixes timezone drop when encoding it in msgpack Feb 6, 2019

cgangwar11 added 2 commits February 7, 2019 01:32

flake8 warning fix

e5d971b

whatsnew added

99cab5a

mroeschke reviewed Feb 6, 2019

View reviewed changes

mroeschke requested changes Feb 6, 2019

View reviewed changes

mroeschke reviewed Feb 6, 2019

View reviewed changes

mroeschke added IO Msgpack Timezones Timezone data dtype labels Feb 6, 2019

cgangwar11 added 3 commits February 9, 2019 03:07

whatsnew updated

58e04fc

fix added to support dateutil timezones and offset

ef07efa

tz_aware_fixture_added

fbcf6f7

jreback requested changes Feb 9, 2019

View reviewed changes

cgangwar11 added 2 commits February 9, 2019 23:39

minore_changes_for_windows

2433884

resolved confict

b1347ae

jreback closed this May 12, 2019

Uh oh!

Fixes timezone drop when encoding it in msgpack #25195

Fixes timezone drop when encoding it in msgpack #25195

Uh oh!

Conversation

cgangwar11 commented Feb 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pep8speaks commented Feb 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated on February 09, 2019 at 18:27 Hours UTC

Uh oh!

codecov bot commented Feb 6, 2019

Codecov Report

Uh oh!

codecov bot commented Feb 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mroeschke left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgangwar11 Feb 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jreback commented Mar 20, 2019

Uh oh!

jreback commented May 12, 2019

Uh oh!

Uh oh!

cgangwar11 commented Feb 6, 2019 •

edited

Loading

pep8speaks commented Feb 6, 2019 •

edited

Loading

codecov bot commented Feb 6, 2019 •

edited

Loading

cgangwar11 Feb 9, 2019 •

edited

Loading