TST: errors in test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals) #12338

dhomeier · 2016-02-16T00:26:27Z

Getting these errors in the 0.18.0rc1 test suite on Mac OS X 10.10 (numpy 0.11.0b3):

with Python 2.7.11:

ERROR: test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 290, in test_constructor_compound_dtypes
    f('M8[ns]')
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 276, in f
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 262, in __init__
    arrays, columns = _to_arrays(data, columns, dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5349, in _to_arrays
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5428, in _list_to_arrays
    coerce_float=coerce_float)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5495, in _convert_object_array
    arrays = [convert(arr) for arr in content]
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5492, in convert
    arr = com._possibly_cast_to_datetime(arr, dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/common.py", line 1675, in _possibly_cast_to_datetime
    value = to_datetime(value, errors=errors)._values
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 280, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 407, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 381, in _convert_listlike
    require_iso8601=require_iso8601
  File "pandas/tslib.pyx", line 1956, in pandas.tslib.array_to_datetime (pandas/tslib.c:39318)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2145, in pandas.tslib.array_to_datetime (pandas/tslib.c:38913)
    raise
  File "pandas/tslib.pyx", line 2139, in pandas.tslib.array_to_datetime (pandas/tslib.c:38803)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1728, in pandas.tslib.parse_datetime_string (pandas/tslib.c:31663)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/sw/lib/python2.7/site-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/sw/lib/python2.7/site-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

with Python 3.4.4 and 3.5.1:

ERROR: test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pandas/tslib.pyx", line 2055, in pandas.tslib.array_to_datetime (pandas/tslib.c:37286)
    _string_to_dts(val, &dts, &out_local, &out_tzoffset)
  File "pandas/src/datetime.pxd", line 141, in datetime._string_to_dts (pandas/tslib.c:83021)
    raise ValueError('Unable to parse %s' % str(val))
ValueError: Unable to parse b'aa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas/tslib.pyx", line 2075, in pandas.tslib.array_to_datetime (pandas/tslib.c:37657)
    py_dt = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1728, in pandas.tslib.parse_datetime_string (pandas/tslib.c:31663)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas/tslib.pyx", line 2081, in pandas.tslib.array_to_datetime (pandas/tslib.c:37756)
    raise TypeError("invalid string coercion to datetime")
TypeError: invalid string coercion to datetime

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 290, in test_constructor_compound_dtypes
    f('M8[ns]')
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 276, in f
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 262, in __init__
    arrays, columns = _to_arrays(data, columns, dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5349, in _to_arrays
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5428, in _list_to_arrays
    coerce_float=coerce_float)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5495, in _convert_object_array
    arrays = [convert(arr) for arr in content]
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5495, in <listcomp>
    arrays = [convert(arr) for arr in content]
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5492, in convert
    arr = com._possibly_cast_to_datetime(arr, dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/common.py", line 1675, in _possibly_cast_to_datetime
    value = to_datetime(value, errors=errors)._values
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 280, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 407, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 381, in _convert_listlike
    require_iso8601=require_iso8601
  File "pandas/tslib.pyx", line 1956, in pandas.tslib.array_to_datetime (pandas/tslib.c:39318)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2145, in pandas.tslib.array_to_datetime (pandas/tslib.c:38913)
    raise
  File "pandas/tslib.pyx", line 2139, in pandas.tslib.array_to_datetime (pandas/tslib.c:38803)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1728, in pandas.tslib.parse_datetime_string (pandas/tslib.c:31663)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

The text was updated successfully, but these errors were encountered:

jreback · 2016-02-16T00:52:50Z

I cannot repro this (with the exact versions you are using)

I have PYTHONIOENCODING=UTF-8 set. (which is typical for a PY3 env)

actually this var doesn't matter

jreback · 2016-02-16T00:56:56Z

I have seen this happen on windows, IOW the dateparsing routines fail and raise a different error than we are expecting.

can you show versions of pytz, dateutil

(numpy_dev3)bash-3.2$ conda list pytz
# packages in environment at /Users/jreback/miniconda/envs/numpy_dev3:
#
pytz                      2015.7                   py35_0  
(numpy_dev3)bash-3.2$ conda list dateutil       
# packages in environment at /Users/jreback/miniconda/envs/numpy_dev3:
#
python-dateutil           2.4.2                    py35_0

dhomeier · 2016-02-16T01:03:36Z

A bit behind, yes:

> fink list pytz dateutil
Information about 9337 packages read in 4 seconds.
 i   dateutil-py27                    2.2-3                    Extended date and time calculations
 i   dateutil-py34                    2.2-3                    Extended date and time calculations
 i   dateutil-py35                    2.2-3                    Extended date and time calculations
 i   pytz-py27                        2014.2-1                 World timezone definitions database
 i   pytz-py34                        2014.2-1                 World timezone definitions database
 i   pytz-py35                        2014.2-1                 World timezone definitions database

jreback · 2016-02-16T01:08:34Z

ok, that is it. dateutil==2.2 (via pip) reproduces.

jreback · 2016-02-16T01:16:17Z

Wish I had a better test, but this fixes. We are not testing this. As have to modify the ci to explicity pip install dateutil=2.2 (as its not on conda, as IIRC it was a completely buggy/bogus version :<). Its not even tagged in the repo, really weird

diff --git a/pandas/core/common.py b/pandas/core/common.py
index 70c02c5..0a6a309 100644
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -1681,7 +1681,7 @@ def _possibly_cast_to_datetime(value, dtype, errors='raise'):
                                 errors=errors).tz_localize(dtype.tz)
                         elif is_timedelta64:
                             value = to_timedelta(value, errors=errors)._values
-                    except (AttributeError, ValueError):
+                    except (AttributeError, ValueError, TypeError):
                         pass

         # coerce datetimelike to object

closes pandas-dev#12338

dhomeier · 2016-02-16T01:44:30Z

Tested the patch here as well, and also the original source against dateutil-2.4.2, both passing.

jreback · 2016-02-16T01:45:58Z

@dhomeier gr8 thanks!

jreback added this to the 0.18.0 milestone Feb 16, 2016

jreback added Testing pandas testing functions or related to the test suite Can't Repro labels Feb 16, 2016

jreback mentioned this issue Feb 16, 2016

COMPAT: dateutil=2.2 compat #12339

Closed

jreback added a commit to jreback/pandas that referenced this issue Feb 16, 2016

COMPAT: dateutil=2.2 compat

d61653f

closes pandas-dev#12338

jreback closed this as completed in 0813a33 Feb 16, 2016

jreback mentioned this issue Mar 12, 2016

BUG: np.round does not work on a series in 0.18.0 #12600

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

TST: errors in test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals) #12338

TST: errors in test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals) #12338

dhomeier commented Feb 16, 2016

jreback commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!

dhomeier commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!

dhomeier commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!

Uh oh!

TST: errors in test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals) #12338

TST: errors in test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals) #12338

Comments

dhomeier commented Feb 16, 2016

jreback commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!

dhomeier commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!

dhomeier commented Feb 16, 2016

Uh oh!

jreback commented Feb 16, 2016

Uh oh!