Subclassing improvements #12308

cldy · 2016-02-12T09:30:57Z

Addresses indexing issues related to subclassing (#11559).

Also made some changes to base, groupby, reshape, strings.
So for example, if df was a SubclassedDataFrame and s was a SubclassedSeries
df.groupby('A').apply(lambda x: x)
and
s.value_counts()
would return SubclassedDataFrame and SubclassedSeries instances, respectively.

jreback · 2016-02-12T14:13:43Z

pandas/core/common.py

@@ -279,7 +279,12 @@ def _isnull_ndarraylike(obj):
    # box
    if isinstance(obj, ABCSeries):
        from pandas import Series
-        result = Series(result, index=obj.index, name=obj.name, copy=False)
+        if isinstance(obj, Series):
+            result = obj._constructor_sliced(


lots of repeatitive code here. I don't think you need anything except replace Series with obj._constructor_sliced no?

Yes, changed that.

cldy · 2016-02-18T15:10:15Z

So I have made most of the changes you suggested (also see comments above).

removed an unnecessary constructor in generic.py and an unnecessary if/then in common.py
constructors in groupby and merge are now properties
unstack logic is handled in unstack instead of series.py

I do not really understand what you mean by your comment on the change in ops.py, could you explain?

jreback · 2016-02-18T15:13:04Z

@cldy thanks for this. I will have a closer look prob next week.

jreback · 2016-03-12T17:56:29Z

pls rebase/update

jreback · 2016-03-16T13:52:46Z

btw, you will want to squash this down and simply rebase on master to fast-forward yourself.

codecov-io · 2016-03-16T15:01:11Z

Current coverage is 88.69%

Merging #12308 into master will increase coverage by 4.46%

@@             master   #12308   diff @@
========================================
  Files           138      276     +138   
  Lines         50712   134409   +83697   
  Methods           0        0            
  Branches          0        0            
========================================
+ Hits          42714   119218   +76504   
- Misses         7998    15191    +7193   
  Partials          0        0

Powered by Codecov. Last updated by 2e3c82e...297ffe0

cldy · 2016-03-18T10:23:13Z

Sorrry this is taking so long. The rebase and squash turned out to be more complicated than expected, but is now done (locally).

I am still at a bit of a loss as to why the Travis build fails. The error is apparently due to the locale setting zh_CN.GB18030 (No such file or directory). I will have a closer look at this at the beginning of next week, but would be grateful for a hint.

jreback · 2016-03-18T15:31:23Z

@cldy you need to rebase on master. you should only have your commits here.

jreback · 2016-03-23T12:49:39Z

pandas/core/common.py

@@ -333,7 +333,7 @@ def notnull(obj):
    pandas.isnull : boolean inverse of pandas.notnull
    """
    res = isnull(obj)
-    if lib.isscalar(res):
+    if np.isscalar(res):


you are reversing changes previously made.

Yes, thanks, fixed that.

cldy · 2016-03-23T14:17:36Z

Rebase done, still haven't figured out the locale error ...

cldy · 2016-03-24T15:58:55Z

The locale error occurs only in my Travis, but the build seems to pass here. Also, I have squashed the commits.

sinhrks · 2016-04-15T20:53:43Z

pandas/core/panel.py

@@ -130,6 +130,10 @@ def _constructor(self):

    _constructor_sliced = DataFrame

+    _constructor_expanddim = _constructor


Dupe, and is this tested somewhere? The definition looks inappropriate.

sinhrks · 2016-04-15T21:08:00Z

I think it's better to split the change to 2 parts which can be done with current _constructor_... mechanism and other methods which can't be covered.

If subclassing mechanism is being changed, internal.rst must be updated.

cldy · 2016-06-01T14:36:54Z

Sorry that this is taking so long ...

I am now working on it, but am currently having some problems with the (recent?) changes to sparse data types.

cldy · 2016-06-02T08:59:05Z

I have fixed the issues with Sparse data types, but I still get a number of these errors in Travis:

ERROR: pandas.io.tests.test_pickle.TestPickle.test_pickles('0.11.0',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/miniconda/envs/pandas/lib/python3.4/site-packages/nose/case.py", line 382, in setUp
    try_run(self.inst, ('setup', 'setUp'))
  File "/home/travis/miniconda/envs/pandas/lib/python3.4/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/home/travis/build/cldy/pandas/pandas/io/tests/test_pickle.py", line 38, in setUp
    self.data = create_pickle_data()
  File "/home/travis/build/cldy/pandas/pandas/io/tests/generate_legacy_storage_files.py", line 154, in create_pickle_data
    if _loose_version < '0.14.1':
  File "/home/travis/miniconda/envs/pandas/lib/python3.4/distutils/version.py", line 58, in __lt__
    c = self._cmp(other)
  File "/home/travis/miniconda/envs/pandas/lib/python3.4/distutils/version.py", line 343, in _cmp
    if self.version < other.version:
TypeError: unorderable types: str() < int()

I don't quite understand how this is related to the changes I made. Any help would be very much appreciated.

jreback · 2016-06-02T10:27:30Z

make sure you rebased on master
Travis hit a limit with got clone depth the other day
I bumped it

jreback · 2016-06-02T10:27:39Z

git

bruno-fs · 2016-06-02T16:27:11Z

yay! finally! will the pypi package get this update soon? HUGE thanks @cldy @jreback ;)

jreback · 2016-06-02T18:05:09Z

pandas/core/base.py

        result = value_counts(self, sort=sort, ascending=ascending,
                              normalize=normalize, bins=bins, dropna=dropna)
-        return result
+
+        if isinstance(self, Index):


use gt.ABCIndexClass (its already imported)

jreback · 2016-09-09T22:41:15Z

can you rebase / update?

jreback · 2016-11-22T11:44:44Z

@cldy closing; I would like to see this merged, but can we do this in pieces (e.g. smaller discrete changes). Certainly open an issue with a list of todo's that we can then check off though.

jreback added Indexing Related to indexing on series/frames, not to indexes themselves Compat pandas objects compatability with Numpy or Python functions labels Feb 12, 2016

jreback reviewed Feb 12, 2016
View reviewed changes

jreback added this to the 0.18.1 milestone Feb 18, 2016

cldy force-pushed the subclassing_fixes2 branch 2 times, most recently from 6008ae0 to cbea137 Compare March 23, 2016 10:58

jreback reviewed Mar 23, 2016
View reviewed changes

cldy force-pushed the subclassing_fixes2 branch from 4f792c3 to f8e46ab Compare March 24, 2016 15:27

sinhrks reviewed Apr 15, 2016
View reviewed changes

cldy added 3 commits June 2, 2016 08:44

don't take sparse as sample for concat

5afd6a2

make _isnull_ndarraylike with sparse series input

a41cd31

return dataframe if input is sparsedataframe

87d5e86

cldy added 6 commits June 2, 2016 13:30

linting fix

f515b27

linting fix

d7310be

linting fix

910c17e

linting fix

c784311

linting fix

7896d55

linting fix

04aed78

jreback reviewed Jun 2, 2016
View reviewed changes

jreback closed this Nov 22, 2016

		@@ -130,6 +130,10 @@ def _constructor(self):

		_constructor_sliced = DataFrame

		_constructor_expanddim = _constructor

Uh oh!

Subclassing improvements #12308

Subclassing improvements #12308

Uh oh!

Conversation

cldy commented Feb 12, 2016

Uh oh!

jreback Feb 12, 2016

Choose a reason for hiding this comment

Uh oh!

cldy Feb 16, 2016

Choose a reason for hiding this comment

Uh oh!

cldy commented Feb 18, 2016

Uh oh!

jreback commented Feb 18, 2016

Uh oh!

jreback commented Mar 12, 2016

Uh oh!

jreback commented Mar 16, 2016

Uh oh!

codecov-io commented Mar 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current coverage is 88.69%

Uh oh!

cldy commented Mar 18, 2016

Uh oh!

jreback commented Mar 18, 2016

Uh oh!

jreback Mar 23, 2016

Choose a reason for hiding this comment

Uh oh!

cldy Mar 23, 2016

Choose a reason for hiding this comment

Uh oh!

cldy commented Mar 23, 2016

Uh oh!

cldy commented Mar 24, 2016

Uh oh!

sinhrks Apr 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sinhrks commented Apr 15, 2016

Uh oh!

cldy commented Jun 1, 2016

Uh oh!

cldy commented Jun 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback commented Jun 2, 2016

Uh oh!

jreback commented Jun 2, 2016

Uh oh!

bruno-fs commented Jun 2, 2016

Uh oh!

jreback Jun 2, 2016

Choose a reason for hiding this comment

Uh oh!

jreback commented Sep 9, 2016

Uh oh!

jreback commented Nov 22, 2016

Uh oh!

Uh oh!

codecov-io commented Mar 16, 2016 •

edited

Loading

sinhrks Apr 15, 2016 •

edited

Loading

cldy commented Jun 2, 2016 •

edited

Loading