Skip to content

PERF: concat along axis 1 unnecessarily materializes RangeIndex->Int64Index #47501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 of 3 tasks
vyasr opened this issue Jun 24, 2022 · 3 comments · Fixed by #47508
Closed
2 of 3 tasks

PERF: concat along axis 1 unnecessarily materializes RangeIndex->Int64Index #47501

vyasr opened this issue Jun 24, 2022 · 3 comments · Fixed by #47508
Assignees
@phofl
Labels
Performance Memory or execution speed performance Regression Functionality that used to work in a prior pandas version Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Milestone
1.4.4

Comments

@vyasr
Copy link
Contributor

vyasr commented Jun 24, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this issue exists on the latest version of pandas.

  • I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

This is as of pandas 1.4.3

>>> import pandas as pd                                                                                                                                                                                                                                                                                                                                                
>>> pd.concat([pd.DataFrame({'a': range(10)}), pd.DataFrame({'b': range(10)})], sort=True, axis=1).index                                                                                                   Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')

Even though both inputs have identical RangeIndex inputs, the output index is an Int64Index. This behavior appears to be triggered by the sort=True parameter, since removing that gives a RangeIndex.

>>> pd.concat([pd.DataFrame({'a': range(10)}), pd.DataFrame({'b': range(10)})], axis=1).index
RangeIndex(start=0, stop=10, step=1)

My naive guess is that there is a missing check somewhere that the sort is a no-op on a RangeIndex .

This issue definitely seems related to #46675, but it is not identical since it has specifically appeared in 1.4.3, whereas that issue was already present in 1.4.2.

Installed Versions

``` commit : e8093ba python : 3.8.13.final.0 python-bits : 64 OS : Linux OS-release : 4.15.0-76-generic Version : #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : None LOCALE : en_US.UTF-8 pandas : 1.4.3 numpy : 1.22.4 pytz : 2022.1 dateutil : 2.8.2 setuptools : 62.6.0 pip : 22.1.2 Cython : 0.29.30 pytest : 7.1.2 hypothesis : 6.47.1 sphinx : 5.0.2 blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.0 html5lib : None pymysql : None psycopg2 : None jinja2 : 3.0.3 IPython : 8.4.0 pandas_datareader: None bs4 : 4.11.1 bottleneck : None brotli : fastparquet : None fsspec : 2022.5.0 gcsfs : None markupsafe : 2.1.1 matplotlib : 3.5.2 numba : 0.55.2 numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 8.0.0 pyreadstat : None pyxlsb : None s3fs : 2022.5.0 scipy : 1.8.1 snappy : sqlalchemy : 1.4.38 tables : None tabulate : 0.8.10 xarray : None xlrd : None xlwt : None zstandard : None ```

Prior Performance

This is as of pandas 1.4.2

>>> pd.concat([pd.DataFrame({'a': range(10)}), pd.DataFrame({'b': range(10)})], sort=True, axis=1).index
RangeIndex(start=0, stop=10, step=1)
@vyasr vyasr added Needs Triage Issue that has not been reviewed by a pandas team member Performance Memory or execution speed performance labels Jun 24, 2022
@vyasr vyasr mentioned this issue Jun 25, 2022
Merged
@jreback
Copy link
Contributor

jreback commented Jun 25, 2022

can u show how much this actually matters at scale eg 100 columns 100k rows and if it matters

rapids-bot bot pushed a commit to rapidsai/cudf that referenced this issue Jun 25, 2022
@vyasr
Fix compatibility issues with pandas 1.4.3 (#11152)
236a0ad 
This PR fixes the two main issues that crop up in pandas 1.4.3 relative to 1.4.2, both around `pd.concat`:
- Columns are now sorted such that integer values come before string values. That is a behavior change that we mimic.
- When multiple objects with identical RangeIndexes are concatenated along axis 1 and sorting is requested, pandas now creates an integer index instead of a RangeIndex. This is not what we want since it increases memory pressure, so those tests have been modified to stop checking the index type and a [pandas issue has been raised](pandas-dev/pandas#47501).

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - Nghia Truong (https://github.com/ttnghia)

URL: #11152
@phofl phofl self-assigned this Jun 25, 2022
@phofl phofl mentioned this issue Jun 25, 2022
Merged
5 tasks
@phofl phofl added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Regression Functionality that used to work in a prior pandas version labels Jun 25, 2022
@phofl phofl added this to the 1.4.4 milestone Jun 25, 2022
simonjayhawkins added a commit to simonjayhawkins/pandas that referenced this issue Jun 27, 2022
@simonjayhawkins
code sample for pandas-dev#47501
568d90a
@simonjayhawkins
Copy link
Member

This issue definitely seems related to #46675, but it is not identical since it has specifically appeared in 1.4.3, whereas that issue was already present in 1.4.2.

first bad commit: [75a799c] Backport PR #47206 on branch 1.4.x (REGR: concat not sorting columns for mixed column names) (#47251)

@phofl it appears that #47508 does not fix #46675, are they not related?

@simonjayhawkins simonjayhawkins removed the Needs Triage Issue that has not been reviewed by a pandas team member label Jun 27, 2022
@phofl
Copy link
Member

phofl commented Jun 27, 2022 •
edited
Loading

No I don't think so. Will take a look there too

This was referenced Mar 1, 2024
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phofl phofl

Labels
Performance Memory or execution speed performance Regression Functionality that used to work in a prior pandas version Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Milestone
1.4.4
Development

Successfully merging a pull request may close this issue.

REGR: concat materializing index even if sort is not necessary phofl/pandas
4 participants
@jreback @vyasr @simonjayhawkins @phofl