Skip to content

BUG:Dtypes changed from Int to Float when merging for different colunm names with Left_on Right_on outter join #34017

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
1 of 3 tasks
luzhideng opened this issue May 6, 2020 · 2 comments
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@luzhideng
Copy link

luzhideng commented May 6, 2020

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

#Different label names 'A' and 'a'
testa = pd.DataFrame([{"A":1,"b":2}])
testb = pd.DataFrame([{"a":2,"c":2}])
result = pd.merge(testa, testb, left_on='A', right_on='a', how="outer")
result.dtypes

Out[1]: 
A    float64
b    float64
a    float64
c    float64
dtype: object

#Same lable name with on[''] methond looks fine

testa = pd.DataFrame([{"a":1,"b":2}])
testb = pd.DataFrame([{"a":2,"c":2}])
result1 = pd.merge(testa, testb, on=["a"], how="outer")

result1.dtypes
Out[2]: 
a      int64
b    float64
c    float64
dtype: object

Problem description

I was merging some df with the zipcode, so I found this might be a problem. Maybe I am wrong about this, and probably pandas designed this way for some reasons? I don't know. If anyone knows the reasons why it had built this way, please share.
I also came across with this post: #9958

Expected Output

I expecte there's no data type changes after merging.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en
LOCALE : None.None

pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0.post20200210
Cython : 0.29.15
pytest : 5.3.5
hypothesis : 5.5.4
sphinx : 2.4.0
blosc : None
feather : None
xlsxwriter : 1.2.7
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.12.0
pandas_datareader: None
bs4 : 4.8.2
bottleneck : 1.3.2
fastparquet : None
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.1.3
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : 1.3.13
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.7
numba : 0.48.0

@luzhideng luzhideng added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels May 6, 2020
@TomAugspurger
Copy link
Contributor

You might try pandas' nullable integer dtype, which would avoid casting to floats when there aren't overlapping keys.

@peterpanmj
Copy link
Contributor

Silently casting from int64 to float64 is very common, and may cause some problem (e.g people might try to downcast float64 to int64 again), I think we should raise a warning or document it somewhere

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Development

No branches or pull requests

3 participants