BUG: value_counts(observed=True) inconsistent between SeriesGroupBy and DataFrameGroupBy

[LucasG0]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> s = pd.Series(["a", "b", "c"], dtype="category").iloc[0:2]
>>> s
0    a
1    b
dtype: category
Categories (3, object): ['a', 'b', 'c']
>>> s.groupby(level=0).value_counts()
0  a    1
   b    0
   c    0
1  b    1
   a    0
   c    0
dtype: int64
>>> s.groupby(level=0, observed=True).value_counts()
0  a    1
   b    0
   c    0
1  b    1
   a    0
   c    0
dtype: int64

Issue Description

The result contains 0 values for unused categories in each group with observed=True , unlike DataFrameGroupBy:

>>> df = pd.DataFrame(s)
>>> df
	0
0	a
1	b
>>> df.groupby(level=0, observed=True).value_counts()
0  a    1
1  b    1
dtype: int64

Expected Behavior

The behavior should be the same for SeriesGroupBy and DataFrameGroupBy. I am not sure what should be the expected behavior here.

The groupby documentation specifies that observed parameter has effect on groupers. Considering by and level parameters are the groupers, it might make sense that in this case GroupBy.value_counts is not affected by the observed keyword and it follows the behavior of Series.value_counts which does keep unused categories. It would mean that GroupBy.value_counts (and probably Series/DataFrame.value_counts too) would require an additional observed parameter, like the suggestion in #43498.

On the other hand, if we consider that the observed parameter makes sense in this case, SeriesGroupBy.value_counts(observed=True) should not return 0 values for unused categories as for DataFrameGroupBy.

Related: performance issue on GroupBy.value_counts on categoricals #46202

Installed Versions

INSTALLED VERSIONS

commit : 06d2301
python : 3.9.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 11, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : fr_FR.cp1252

pandas : 1.4.1
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 20.3.3
setuptools : 52.0.0.post20210125
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 8.1.1
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[rhshadrach]

Thanks for opening this @LucasG0. I was incorrect in #46202; as you stated above, observed should only impact the groupers. Moreover, we should be consistent with Series.value_counts and DataFrame.value_counts:

s = pd.Series(["a", "b", "c"], dtype="category").iloc[0:2]
df = pd.DataFrame(s)
print(s.value_counts())
print(df.value_counts())

gives

a    1
b    1
c    0
dtype: int64
a    1
b    1
c    0
dtype: int64

Thus the SeriesGroupBy.value_counts behavior is correct here, the DataFrameGroupBy.value_counts has a bug.

[LucasG0]

take