DOC: Add details to DataFrame groupby transform

Add requirements for user function in groupby transform

closes #13543
[skip ci]

Author: bashtage

doc/source/groupby.rst

@@ -521,9 +521,17 @@ Transformation
 --------------
 
 The ``transform`` method returns an object that is indexed the same (same size)
-as the one being grouped. Thus, the passed transform function should return a
-result that is the same size as the group chunk. For example, suppose we wished
-to standardize the data within each group:
+as the one being grouped. The transform function must:
+
+* Return a result that is either the same size as the group chunk or
+  broadcastable to the size of the group chunk.
+* Operate column-by-column on the group chunk.  A fast path is used if the
+  transform function also operates on the entire group chunk as a DataFrame.
+* Does not perform in-place operations on the group chunk. Group chunks should
+  be treated as immutable, and changes to a group chunk may produce unexpected
+  results.
+
+For example, suppose we wished to standardize the data within each group:
 
 .. ipython:: python
 
@@ -605,8 +613,9 @@ and that the transformed data contains no NAs.
 
 .. note::
 
-   Some functions when applied to a groupby object will automatically transform the input, returning
-   an object of the same shape as the original. Passing ``as_index=False`` will not affect these transformation methods.
+   Some functions when applied to a groupby object will automatically transform
+   the input, returning an object of the same shape as the original. Passing
+   ``as_index=False`` will not affect these transformation methods.
 
    For example: ``fillna, ffill, bfill, shift``.
 

pandas/core/groupby.py

@@ -3507,6 +3507,16 @@ def transform(self, func, *args, **kwargs):
         Each subframe is endowed the attribute 'name' in case you need to know
         which group you are working on.
 
+        The current implementation imposes three requirements on f:
+
+        * f must return a value that either has the same shape as the input
+          subframe or can be broadcast to the shape of the input subframe.
+        * f must support application column-by-column in the subframe. An
+          optional fast path is used if f also supports application to the
+          entire subframe.
+        * f must not mutate subframes. Mutation is not supported and may
+          produce unexpected results.
+
         Examples
         --------
         >>> grouped = df.groupby(lambda x: mapping[x])