Merge branch master into master-slp

Author: Anselm Kruis

Doc/c-api/arg.rst

@@ -30,7 +30,7 @@ variable(s) whose address should be passed.
 Strings and buffers
 -------------------
 
-These formats allow to access an object as a contiguous chunk of memory.
+These formats allow accessing an object as a contiguous chunk of memory.
 You don't have to provide raw storage for the returned unicode or bytes
 area.  Also, you won't have to release any memory yourself, except with the
 ``es``, ``es#``, ``et`` and ``et#`` formats.

Doc/c-api/exceptions.rst

@@ -74,8 +74,8 @@ Printing and clearing
    :meth:`__del__` method.
 
    The function is called with a single argument *obj* that identifies the context
-   in which the unraisable exception occurred. The repr of *obj* will be printed in
-   the warning message.
+   in which the unraisable exception occurred. If possible,
+   the repr of *obj* will be printed in the warning message.
 
 
 Raising exceptions
@@ -334,6 +334,14 @@ an error value).
    .. versionadded:: 3.2
 
 
+.. c:function:: int PyErr_ResourceWarning(PyObject *source, Py_ssize_t stack_level, const char *format, ...)
+
+   Function similar to :c:func:`PyErr_WarnFormat`, but *category* is
+   :exc:`ResourceWarning` and pass *source* to :func:`warnings.WarningMessage`.
+
+   .. versionadded:: 3.6
+
+
 Querying the error indicator
 ============================
 

Doc/c-api/function.rst

@@ -40,7 +40,7 @@ There are a few functions specific to Python functions.
 
 .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
 
-   As :c:func:`PyFunction_New`, but also allows to set the function object's
+   As :c:func:`PyFunction_New`, but also allows setting the function object's
    ``__qualname__`` attribute.  *qualname* should be a unicode object or NULL;
    if NULL, the ``__qualname__`` attribute is set to the same value as its
    ``__name__`` attribute.

Doc/c-api/intro.rst

@@ -64,9 +64,10 @@ The header files are typically installed with Python.  On Unix, these  are
 located in the directories :file:`{prefix}/include/pythonversion/` and
 :file:`{exec_prefix}/include/pythonversion/`, where :envvar:`prefix` and
 :envvar:`exec_prefix` are defined by the corresponding parameters to Python's
-:program:`configure` script and *version* is ``sys.version[:3]``.  On Windows,
-the headers are installed in :file:`{prefix}/include`, where :envvar:`prefix` is
-the installation directory specified to the installer.
+:program:`configure` script and *version* is
+``'%d.%d' % sys.version_info[:2]``.  On Windows, the headers are installed
+in :file:`{prefix}/include`, where :envvar:`prefix` is the installation
+directory specified to the installer.
 
 To include the headers, place both directories (if different) on your compiler's
 search path for includes.  Do *not* place the parent directories on the search

Doc/c-api/memory.rst

@@ -83,6 +83,15 @@ collection, memory compaction or other preventive procedures. Note that by using
 the C library allocator as shown in the previous example, the allocated memory
 for the I/O buffer escapes completely the Python memory manager.
 
+.. seealso::
+
+   The :envvar:`PYTHONMALLOC` environment variable can be used to configure
+   the memory allocators used by Python.
+
+   The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print
+   statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a
+   new pymalloc object arena is created, and on shutdown.
+
 
 Raw Memory Interface
 ====================
@@ -100,41 +109,51 @@ The default raw memory block allocator uses the following functions:
 .. c:function:: void* PyMem_RawMalloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails. Requesting zero bytes
-   returns a distinct non-*NULL* pointer if possible, as if
-   ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
+   allocated memory, or *NULL* if the request fails.
+
+   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
 
 .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
    a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
-   request fails. The memory is initialized to zeros. Requesting zero elements
-   or elements of size zero bytes returns a distinct non-*NULL* pointer if
-   possible, as if ``PyMem_RawCalloc(1, 1)`` had been called instead.
+   request fails. The memory is initialized to zeros.
+
+   Requesting zero elements or elements of size zero bytes returns a distinct
+   non-*NULL* pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been
+   called instead.
 
    .. versionadded:: 3.5
 
 
 .. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
 
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will
-   be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
-   the call is equivalent to ``PyMem_RawMalloc(n)``; else if *n* is equal to
-   zero, the memory block is resized but is not freed, and the returned pointer
-   is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a
-   previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. If
-   the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* remains
-   a valid pointer to the previous memory area.
+   be unchanged to the minimum of the old and the new sizes.
+
+   If *p* is *NULL*, the call is equivalent to ``PyMem_RawMalloc(n)``; else if
+   *n* is equal to zero, the memory block is resized but is not freed, and the
+   returned pointer is non-*NULL*.
+
+   Unless *p* is *NULL*, it must have been returned by a previous call to
+   :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or
+   :c:func:`PyMem_RawCalloc`.
+
+   If the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p*
+   remains a valid pointer to the previous memory area.
 
 
 .. c:function:: void PyMem_RawFree(void *p)
 
    Frees the memory block pointed to by *p*, which must have been returned by a
-   previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
-   Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
-   behavior occurs. If *p* is *NULL*, no operation is performed.
+   previous call to :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or
+   :c:func:`PyMem_RawCalloc`.  Otherwise, or if ``PyMem_Free(p)`` has been
+   called before, undefined behavior occurs.
+
+   If *p* is *NULL*, no operation is performed.
 
 
 .. _memoryinterface:
@@ -158,40 +177,50 @@ The default memory block allocator uses the following functions:
 .. c:function:: void* PyMem_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails. Requesting zero bytes returns
-   a distinct non-*NULL* pointer if possible, as if ``PyMem_Malloc(1)`` had
-   been called instead. The memory will not have been initialized in any way.
+   allocated memory, or *NULL* if the request fails.
+
+   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   if ``PyMem_Malloc(1)`` had been called instead. The memory will not have
+   been initialized in any way.
 
 
 .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
    a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
-   request fails. The memory is initialized to zeros. Requesting zero elements
-   or elements of size zero bytes returns a distinct non-*NULL* pointer if
-   possible, as if ``PyMem_Calloc(1, 1)`` had been called instead.
+   request fails. The memory is initialized to zeros.
+
+   Requesting zero elements or elements of size zero bytes returns a distinct
+   non-*NULL* pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called
+   instead.
 
    .. versionadded:: 3.5
 
 
 .. c:function:: void* PyMem_Realloc(void *p, size_t n)
 
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
-   unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, the
-   call is equivalent to ``PyMem_Malloc(n)``; else if *n* is equal to zero,
-   the memory block is resized but is not freed, and the returned pointer is
-   non-*NULL*.  Unless *p* is *NULL*, it must have been returned by a previous call
-   to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. If the request fails,
-   :c:func:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the
-   previous memory area.
+   unchanged to the minimum of the old and the new sizes.
+
+   If *p* is *NULL*, the call is equivalent to ``PyMem_Malloc(n)``; else if *n*
+   is equal to zero, the memory block is resized but is not freed, and the
+   returned pointer is non-*NULL*.
+
+   Unless *p* is *NULL*, it must have been returned by a previous call to
+   :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or :c:func:`PyMem_Calloc`.
+
+   If the request fails, :c:func:`PyMem_Realloc` returns *NULL* and *p* remains
+   a valid pointer to the previous memory area.
 
 
 .. c:function:: void PyMem_Free(void *p)
 
    Frees the memory block pointed to by *p*, which must have been returned by a
-   previous call to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`.  Otherwise, or
-   if ``PyMem_Free(p)`` has been called before, undefined behavior occurs. If
-   *p* is *NULL*, no operation is performed.
+   previous call to :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or
+   :c:func:`PyMem_Calloc`.  Otherwise, or if ``PyMem_Free(p)`` has been called
+   before, undefined behavior occurs.
+
+   If *p* is *NULL*, no operation is performed.
 
 The following type-oriented macros are provided for convenience.  Note  that
 *TYPE* refers to any C type.
@@ -209,8 +238,10 @@ The following type-oriented macros are provided for convenience.  Note  that
    Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n *
    sizeof(TYPE))`` bytes.  Returns a pointer cast to :c:type:`TYPE\*`. On return,
    *p* will be a pointer to the new memory area, or *NULL* in the event of
-   failure.  This is a C preprocessor macro; p is always reassigned.  Save
-   the original value of p to avoid losing memory when handling errors.
+   failure.
+
+   This is a C preprocessor macro; *p* is always reassigned.  Save the original
+   value of *p* to avoid losing memory when handling errors.
 
 
 .. c:function:: void PyMem_Del(void *p)
@@ -222,9 +253,12 @@ allocator directly, without involving the C API functions listed above. However,
 note that their use does not preserve binary compatibility across Python
 versions and is therefore deprecated in extension modules.
 
-:c:func:`PyMem_MALLOC`, :c:func:`PyMem_REALLOC`, :c:func:`PyMem_FREE`.
-
-:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
+* ``PyMem_MALLOC(size)``
+* ``PyMem_NEW(type, size)``
+* ``PyMem_REALLOC(ptr, size)``
+* ``PyMem_RESIZE(ptr, type, size)``
+* ``PyMem_FREE(ptr)``
+* ``PyMem_DEL(ptr)``
 
 
 Customize Memory Allocators
@@ -262,11 +296,13 @@ Customize Memory Allocators
    Enum used to identify an allocator domain. Domains:
 
    * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
-     :c:func:`PyMem_RawRealloc` and :c:func:`PyMem_RawFree`
+     :c:func:`PyMem_RawRealloc`, :c:func:`PyMem_RawCalloc` and
+     :c:func:`PyMem_RawFree`
    * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
-     :c:func:`PyMem_Realloc` and :c:func:`PyMem_Free`
+     :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc` and :c:func:`PyMem_Free`
    * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
-     :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free`
+     :c:func:`PyObject_Realloc`, :c:func:`PyObject_Calloc` and
+     :c:func:`PyObject_Free`
 
 
 .. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)
@@ -296,38 +332,63 @@ Customize Memory Allocators
    functions:
 
    - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
-     :c:func:`PyMem_RawFree`
-   - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
+     :c:func:`PyMem_RawCalloc`, :c:func:`PyMem_RawFree`
+   - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc`,
+     :c:func:`PyMem_Free`
    - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
-     :c:func:`PyObject_Free`
+     :c:func:`PyObject_Calloc`, :c:func:`PyObject_Free`
 
    Newly allocated memory is filled with the byte ``0xCB``, freed memory is
    filled with the byte ``0xDB``. Additional checks:
 
-   - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
+   - Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
      allocated by :c:func:`PyMem_Malloc`
-   - detect write before the start of the buffer (buffer underflow)
-   - detect write after the end of the buffer (buffer overflow)
+   - Detect write before the start of the buffer (buffer underflow)
+   - Detect write after the end of the buffer (buffer overflow)
+   - Check that the :term:`GIL <global interpreter lock>` is held when
+     allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex:
+     :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex:
+     :c:func:`PyMem_Malloc`) domains are called
+
+   On error, the debug hooks use the :mod:`tracemalloc` module to get the
+   traceback where a memory block was allocated. The traceback is only
+   displayed if :mod:`tracemalloc` is tracing Python memory allocations and the
+   memory block was traced.
+
+   These hooks are installed by default if Python is compiled in debug
+   mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install
+   debug hooks on a Python compiled in release mode.
 
-   The function does nothing if Python is not compiled is debug mode.
+   .. versionchanged:: 3.6
+      This function now also works on Python compiled in release mode.
+      On error, the debug hooks now use :mod:`tracemalloc` to get the traceback
+      where a memory block was allocated. The debug hooks now also check
+      if the GIL is held when functions of :c:data:`PYMEM_DOMAIN_OBJ` and
+      :c:data:`PYMEM_DOMAIN_MEM` domains are called.
 
 
-Customize PyObject Arena Allocator
-==================================
+.. _pymalloc:
 
-Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
-allocator is optimized for small objects with a short lifetime. It uses memory
-mappings called "arenas" with a fixed size of 256 KB. It falls back to
-:c:func:`PyMem_RawMalloc` and :c:func:`PyMem_RawRealloc` for allocations larger
-than 512 bytes.  *pymalloc* is the default allocator used by
-:c:func:`PyObject_Malloc`.
+The pymalloc allocator
+======================
 
-The default arena allocator uses the following functions:
+Python has a *pymalloc* allocator optimized for small objects (smaller or equal
+to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
+with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and
+:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
+
+*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_OBJ` domain
+(ex: :c:func:`PyObject_Malloc`).
+
+The arena allocator uses the following functions:
 
 * :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
 * :c:func:`mmap` and :c:func:`munmap` if available,
 * :c:func:`malloc` and :c:func:`free` otherwise.
 
+Customize pymalloc Arena Allocator
+----------------------------------
+
 .. versionadded:: 3.4
 
 .. c:type:: PyObjectArenaAllocator

Doc/c-api/stable.rst

@@ -34,5 +34,5 @@ will work on all subsequent Python releases, but fail to load (because of
 missing symbols) on the older releases.
 
 As of Python 3.2, the set of functions available to the limited API is
-documented in PEP 384.  In the C API documentation, API elements that are not
+documented in :pep:`384`.  In the C API documentation, API elements that are not
 part of the limited API are marked as "Not part of the limited API."

Doc/distributing/index.rst

@@ -35,7 +35,7 @@ Key terms
   repository of open source licensed packages made available for use by
   other Python users
 * the `Python Packaging Authority
-  <https://packaging.python.org/en/latest/future.html>`__ are the group of
+  <https://www.pypa.io/>`__ are the group of
   developers and documentation authors responsible for the maintenance and
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation
@@ -62,7 +62,7 @@ Key terms
   locally.
 
 .. _setuptools: https://setuptools.pypa.io/en/latest/setuptools.html
-.. _wheel: http://wheel.readthedocs.org
+.. _wheel: https://wheel.readthedocs.org
 
 Open source licensing and collaboration
 =======================================
@@ -111,7 +111,7 @@ by invoking the ``pip`` module at the command line::
 The Python Packaging User Guide includes more details on the `currently
 recommended tools`_.
 
-.. _currently recommended tools: https://packaging.python.org/en/latest/current.html#packaging-tool-recommendations
+.. _currently recommended tools: https://packaging.python.org/en/latest/current/#packaging-tool-recommendations
 
 Reading the guide
 =================
@@ -124,11 +124,11 @@ involved in creating a project:
 * `Uploading the project to the Python Packaging Index`_
 
 .. _Project structure: \
-   https://packaging.python.org/en/latest/distributing.html#creating-your-own-project
+   https://packaging.python.org/en/latest/distributing/
 .. _Building and packaging the project: \
-   https://packaging.python.org/en/latest/distributing.html#packaging-your-project
+   https://packaging.python.org/en/latest/distributing/#packaging-your-project
 .. _Uploading the project to the Python Packaging Index: \
-   https://packaging.python.org/en/latest/distributing.html#uploading-your-project-to-pypi
+   https://packaging.python.org/en/latest/distributing/#uploading-your-project-to-pypi
 
 
 How do I...?
@@ -160,11 +160,11 @@ Python Packaging User Guide for more information and recommendations.
 .. seealso::
 
    `Python Packaging User Guide: Binary Extensions
-   <https://packaging.python.org/en/latest/extensions.html>`__
+   <https://packaging.python.org/en/latest/extensions>`__
 
 .. other topics:
 
    Once the Development & Deployment part of PPUG is fleshed out, some of
    those sections should be linked from new questions here (most notably,
    we should have a question about avoiding depending on PyPI that links to
-   https://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches)
+   https://packaging.python.org/en/latest/mirrors/)