API: User-control of result keys in GroupBy.apply (#34998)

Author: TomAugspurger

doc/source/user_guide/groupby.rst

@@ -1052,7 +1052,14 @@ Some operations on the grouped data might not fit into either the aggregate or
 transform categories. Or, you may simply want GroupBy to infer how to combine
 the results. For these, use the ``apply`` function, which can be substituted
 for both ``aggregate`` and ``transform`` in many standard use cases. However,
-``apply`` can handle some exceptional use cases, for example:
+``apply`` can handle some exceptional use cases.
+
+.. note::
+
+   ``apply`` can act as a reducer, transformer, *or* filter function, depending
+   on exactly what is passed to it. It can depend on the passed function and
+   exactly what you are grouping. Thus the grouped column(s) may be included in
+   the output as well as set the indices.
 
 .. ipython:: python
 
@@ -1064,16 +1071,14 @@ for both ``aggregate`` and ``transform`` in many standard use cases. However,
 
 The dimension of the returned result can also change:
 
-.. ipython::
-
-    In [8]: grouped = df.groupby('A')['C']
+.. ipython:: python
 
-    In [10]: def f(group):
-       ....:     return pd.DataFrame({'original': group,
-       ....:                          'demeaned': group - group.mean()})
-       ....:
+    grouped = df.groupby('A')['C']
 
-    In [11]: grouped.apply(f)
+    def f(group):
+        return pd.DataFrame({'original': group,
+                             'demeaned': group - group.mean()})
+    grouped.apply(f)
 
 ``apply`` on a Series can operate on a returned value from the applied function,
 that is itself a series, and possibly upcast the result to a DataFrame:
@@ -1088,11 +1093,33 @@ that is itself a series, and possibly upcast the result to a DataFrame:
     s
     s.apply(f)
 
+Control grouped column(s) placement with ``group_keys``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 .. note::
 
-   ``apply`` can act as a reducer, transformer, *or* filter function, depending on exactly what is passed to it.
-   So depending on the path taken, and exactly what you are grouping. Thus the grouped columns(s) may be included in
-   the output as well as set the indices.
+   If ``group_keys=True`` is specified when calling :meth:`~DataFrame.groupby`,
+   functions passed to ``apply`` that return like-indexed outputs will have the
+   group keys added to the result index. Previous versions of pandas would add
+   the group keys only when the result from the applied function had a different
+   index than the input. If ``group_keys`` is not specified, the group keys will
+   not be added for like-indexed outputs. In the future this behavior
+   will change to always respect ``group_keys``, which defaults to ``True``.
+
+   .. versionchanged:: 1.5.0
+
+To control whether the grouped column(s) are included in the indices, you can use
+the argument ``group_keys``. Compare
+
+.. ipython:: python
+
+    df.groupby("A", group_keys=True).apply(lambda x: x)
+
+with
+
+.. ipython:: python
+
+    df.groupby("A", group_keys=False).apply(lambda x: x)
 
 Similar to :ref:`groupby.aggregate.udfs`, the resulting dtype will reflect that of the
 apply function. If the results from different groups have different dtypes, then

doc/source/whatsnew/v0.25.0.rst

@@ -342,10 +342,15 @@ Now every group is evaluated only a single time.
 
 *New behavior*:
 
-.. ipython:: python
-
-    df.groupby("a").apply(func)
+.. code-block:: python
 
+   In [3]: df.groupby('a').apply(func)
+   x
+   y
+   Out[3]:
+      a  b
+   0  x  1
+   1  y  2
 
 Concatenating sparse values
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/source/whatsnew/v1.4.0.rst

@@ -455,10 +455,20 @@ result's index is not the same as the input's.
 
 *New behavior*:
 
-.. ipython:: python
+.. code-block:: ipython
 
-    df.groupby(['a']).apply(func)
-    df.set_index(['a', 'b']).groupby(['a']).apply(func)
+    In [5]: df.groupby(['a']).apply(func)
+    Out[5]:
+       a  b  c
+    0  1  3  5
+    1  2  4  6
+
+    In [6]: df.set_index(['a', 'b']).groupby(['a']).apply(func)
+    Out[6]:
+         c
+    a b
+    1 3  5
+    2 4  6
 
 Now in both cases it is determined that ``func`` is a transform. In each case,
 the result has the same index as the input.

doc/source/whatsnew/v1.5.0.rst

@@ -24,10 +24,56 @@ Styler
   - Added a new method :meth:`.Styler.concat` which allows adding customised footer rows to visualise additional calculations on the data, e.g. totals and counts etc. (:issue:`43875`, :issue:`46186`)
   - :meth:`.Styler.highlight_null` now accepts ``color`` consistently with other builtin methods and deprecates ``null_color`` although this remains backwards compatible (:issue:`45907`)
 
-.. _whatsnew_150.enhancements.enhancement2:
+.. _whatsnew_150.enhancements.resample_group_keys:
 
-enhancement2
-^^^^^^^^^^^^
+Control of index with ``group_keys`` in :meth:`DataFrame.resample`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The argument ``group_keys`` has been added to the method :meth:`DataFrame.resample`.
+As with :meth:`DataFrame.groupby`, this argument controls the whether each group is added
+to the index in the resample when :meth:`.Resampler.apply` is used.
+
+.. warning::
+   Not specifying the ``group_keys`` argument will retain the
+   previous behavior and emit a warning if the result will change
+   by specifying ``group_keys=False``. In a future version
+   of pandas, not specifying ``group_keys`` will default to
+   the same behavior as ``group_keys=False``.
+
+.. ipython:: python
+
+    df = pd.DataFrame(
+        {'a': range(6)},
+        index=pd.date_range("2021-01-01", periods=6, freq="8H")
+    )
+    df.resample("D", group_keys=True).apply(lambda x: x)
+    df.resample("D", group_keys=False).apply(lambda x: x)
+
+Previously, the resulting index would depend upon the values returned by ``apply``,
+as seen in the following example.
+
+.. code-block:: ipython
+
+    In [1]: # pandas 1.3
+    In [2]: df.resample("D").apply(lambda x: x)
+    Out[2]:
+                         a
+    2021-01-01 00:00:00  0
+    2021-01-01 08:00:00  1
+    2021-01-01 16:00:00  2
+    2021-01-02 00:00:00  3
+    2021-01-02 08:00:00  4
+    2021-01-02 16:00:00  5
+
+    In [3]: df.resample("D").apply(lambda x: x.reset_index())
+    Out[3]:
+                               index  a
+    2021-01-01 0 2021-01-01 00:00:00  0
+               1 2021-01-01 08:00:00  1
+               2 2021-01-01 16:00:00  2
+    2021-01-02 0 2021-01-02 00:00:00  3
+               1 2021-01-02 08:00:00  4
+               2 2021-01-02 16:00:00  5
 
 .. _whatsnew_150.enhancements.other:
 
@@ -345,6 +391,23 @@ that their usage is considered unsafe, and can lead to unexpected results.
 
 See the documentation of :class:`ExcelWriter` for further details.
 
+.. _whatsnew_150.deprecations.group_keys_in_apply:
+
+Using ``group_keys`` with transformers in :meth:`.GroupBy.apply`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In previous versions of pandas, if it was inferred that the function passed to
+:meth:`.GroupBy.apply` was a transformer (i.e. the resulting index was equal to
+the input index), the ``group_keys`` argument of :meth:`DataFrame.groupby` and
+:meth:`Series.groupby` was ignored and the group keys would never be added to
+the index of the result. In the future, the group keys will be added to the index
+when the user specifies ``group_keys=True``.
+
+As ``group_keys=True`` is the default value of :meth:`DataFrame.groupby` and
+:meth:`Series.groupby`, not specifying ``group_keys`` with a transformer will
+raise a ``FutureWarning``. This can be silenced and the previous behavior
+retained by specifying ``group_keys=False``.
+
 .. _whatsnew_150.deprecations.other:
 
 Other Deprecations

pandas/core/frame.py

@@ -7885,6 +7885,27 @@ def update(
 a   13.0   13.0
 b   12.3  123.0
 NaN 12.3   33.0
+
+When using ``.apply()``, use ``group_keys`` to include or exclude the group keys.
+The ``group_keys`` argument defaults to ``True`` (include).
+
+>>> df = pd.DataFrame({'Animal': ['Falcon', 'Falcon',
+...                               'Parrot', 'Parrot'],
+...                    'Max Speed': [380., 370., 24., 26.]})
+>>> df.groupby("Animal", group_keys=True).apply(lambda x: x)
+          Animal  Max Speed
+Animal
+Falcon 0  Falcon      380.0
+       1  Falcon      370.0
+Parrot 2  Parrot       24.0
+       3  Parrot       26.0
+
+>>> df.groupby("Animal", group_keys=False).apply(lambda x: x)
+   Animal  Max Speed
+0  Falcon      380.0
+1  Falcon      370.0
+2  Parrot       24.0
+3  Parrot       26.0
 """
     )
     @Appender(_shared_docs["groupby"] % _shared_doc_kwargs)
@@ -7895,7 +7916,7 @@ def groupby(
         level: Level | None = None,
         as_index: bool = True,
         sort: bool = True,
-        group_keys: bool = True,
+        group_keys: bool | lib.NoDefault = no_default,
         squeeze: bool | lib.NoDefault = no_default,
         observed: bool = False,
         dropna: bool = True,
@@ -10840,6 +10861,7 @@ def resample(
         level=None,
         origin: str | TimestampConvertibleTypes = "start_day",
         offset: TimedeltaConvertibleTypes | None = None,
+        group_keys: bool | lib.NoDefault = no_default,
     ) -> Resampler:
         return super().resample(
             rule=rule,
@@ -10854,6 +10876,7 @@ def resample(
             level=level,
             origin=origin,
             offset=offset,
+            group_keys=group_keys,
         )
 
     def to_timestamp(

pandas/core/generic.py

@@ -8041,6 +8041,7 @@ def resample(
         level=None,
         origin: str | TimestampConvertibleTypes = "start_day",
         offset: TimedeltaConvertibleTypes | None = None,
+        group_keys: bool_t | lib.NoDefault = lib.no_default,
     ) -> Resampler:
         """
         Resample time-series data.
@@ -8115,6 +8116,17 @@ def resample(
 
             .. versionadded:: 1.1.0
 
+        group_keys : bool, optional
+            Whether to include the group keys in the result index when using
+            ``.apply()`` on the resampled object. Not specifying ``group_keys``
+            will retain values-dependent behavior from pandas 1.4
+            and earlier (see :ref:`pandas 1.5.0 Release notes
+            <whatsnew_150.enhancements.resample_group_keys>`
+            for examples). In a future version of pandas, the behavior will
+            default to the same as specifying ``group_keys=False``.
+
+            .. versionadded:: 1.5.0
+
         Returns
         -------
         pandas.core.Resampler
@@ -8454,6 +8466,7 @@ def resample(
             level=level,
             origin=origin,
             offset=offset,
+            group_keys=group_keys,
         )
 
     @final

pandas/core/groupby/generic.py

@@ -357,6 +357,7 @@ def _wrap_applied_output(
         data: Series,
         values: list[Any],
         not_indexed_same: bool = False,
+        override_group_keys: bool = False,
     ) -> DataFrame | Series:
         """
         Wrap the output of SeriesGroupBy.apply into the expected result.
@@ -395,7 +396,11 @@ def _wrap_applied_output(
             res_ser.name = self.obj.name
             return res_ser
         elif isinstance(values[0], (Series, DataFrame)):
-            return self._concat_objects(values, not_indexed_same=not_indexed_same)
+            return self._concat_objects(
+                values,
+                not_indexed_same=not_indexed_same,
+                override_group_keys=override_group_keys,
+            )
         else:
             # GH #6265 #24880
             result = self.obj._constructor(
@@ -983,7 +988,11 @@ def _aggregate_item_by_item(self, func, *args, **kwargs) -> DataFrame:
         return res_df
 
     def _wrap_applied_output(
-        self, data: DataFrame, values: list, not_indexed_same: bool = False
+        self,
+        data: DataFrame,
+        values: list,
+        not_indexed_same: bool = False,
+        override_group_keys: bool = False,
     ):
 
         if len(values) == 0:
@@ -1000,7 +1009,11 @@ def _wrap_applied_output(
             # GH9684 - All values are None, return an empty frame.
             return self.obj._constructor()
         elif isinstance(first_not_none, DataFrame):
-            return self._concat_objects(values, not_indexed_same=not_indexed_same)
+            return self._concat_objects(
+                values,
+                not_indexed_same=not_indexed_same,
+                override_group_keys=override_group_keys,
+            )
 
         key_index = self.grouper.result_index if self.as_index else None
 
@@ -1026,7 +1039,11 @@ def _wrap_applied_output(
         else:
             # values are Series
             return self._wrap_applied_output_series(
-                values, not_indexed_same, first_not_none, key_index
+                values,
+                not_indexed_same,
+                first_not_none,
+                key_index,
+                override_group_keys,
             )
 
     def _wrap_applied_output_series(
@@ -1035,6 +1052,7 @@ def _wrap_applied_output_series(
         not_indexed_same: bool,
         first_not_none,
         key_index,
+        override_group_keys: bool,
     ) -> DataFrame | Series:
         # this is to silence a DeprecationWarning
         # TODO(2.0): Remove when default dtype of empty Series is object
@@ -1058,7 +1076,11 @@ def _wrap_applied_output_series(
                 # if any of the sub-series are not indexed the same
                 # OR we don't have a multi-index and we have only a
                 # single values
-                return self._concat_objects(values, not_indexed_same=not_indexed_same)
+                return self._concat_objects(
+                    values,
+                    not_indexed_same=not_indexed_same,
+                    override_group_keys=override_group_keys,
+                )
 
             # still a series
             # path added as of GH 5545
@@ -1069,7 +1091,11 @@ def _wrap_applied_output_series(
 
         if not all_indexed_same:
             # GH 8467
-            return self._concat_objects(values, not_indexed_same=True)
+            return self._concat_objects(
+                values,
+                not_indexed_same=True,
+                override_group_keys=override_group_keys,
+            )
 
         # Combine values
         # vstack+constructor is faster than concat and handles MI-columns