Skip to content
This repository was archived by the owner on Feb 13, 2025. It is now read-only.

Separate pickling of generators, coroutines and asynchronous generators #139

Closed
akruis opened this issue Oct 27, 2017 · 4 comments
Closed

Separate pickling of generators, coroutines and asynchronous generators #139

akruis opened this issue Oct 27, 2017 · 4 comments
Labels
3.7-slp blocker enhancement main-slp stackless python task
Milestone
v3.7.2-slp

Comments

@akruis
Copy link

akruis commented Oct 27, 2017

Commit 3ea77a9 merges Asynchronous Generators into branch master-slp. Perhaps Stackless should learn how to pickle them and related types. This would be in line with the pickling support for generators and coroutines.
@akruis akruis added this to the v3.6.x-slp milestone Oct 27, 2017
@akruis akruis mentioned this issue Oct 27, 2017
Closed
7 tasks
@akruis akruis removed this from the v3.6.x-slp milestone Dec 22, 2017
@akruis
Copy link
Author

akruis commented Dec 22, 2017

Currently I have no time and I don't want to delay the first Stackless 3.6 release any further.

@akruis
Copy link
Author

akruis commented Nov 2, 2018

Meantime I'm working on Stackless 3.7. Upstream commit fc2f407 (bpo-32591) adds an additional field to PyCoroObject. Therefore

stackless/Stackless/pickling/prickelpit.c

Line 1560 in 8f908ec

Py_BUILD_ASSERT(sizeof(PyGenObject) == sizeof(PyCoroObject));
does not compile any more.

It's now finally time to separate the pickling code for generators, coroutines and asynchronous generators.

@akruis akruis changed the title Pickling of asynchronous generators Separate pickling of generators, coroutines and asynchronous generators Nov 2, 2018
@akruis akruis added this to the v3.7.x-slp milestone Nov 2, 2018
@akruis akruis mentioned this issue Nov 9, 2018
Closed
@akruis
Copy link
Author

akruis commented Nov 10, 2018

The pickling code is now separated (commit 94d5fad).

Pickling of asynchronous generators turned out to be not trivial, because they have - at C-level - the attribute "ag_finalizer". This attribute is more or less a cache for a global variable of the async-framework.
Usually one pickles global variables by reference, but it is not obvious how to do it. I decided to make the
details configurable at run time. Therefore I invented two new functions stackless.pickle_flags() and
stackless.pickle_flags_default() to control pickling behaviour. Using these functions I can deprecate stackless.pickle_with_tracing_state (issue #180).

akruis pushed a commit that referenced this issue Nov 11, 2018
Merge commit 'fc2f407829d9' into master-slp
08a8a01 
The outcome of this merge does not compile.
Stackless issue #139 will fix this.
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: separate pickling of generators and coroutines
94d5fad 
Since upstream commit fc2f407 (bpo-32591) the memory layout of
generators and coroutines differs. Therefore it is no longer feasible to
share the pickling code.
This refactor the common parts of the reduce/setstate implementation
into support methods and support the new coroutine attribute cr_origin.
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: adapt compute_cr_origin() to Stackless
b3a7011 
The new function compute_cr_origin(int origin_depth) inspects stack
frames. Modify it to ignore C-frames.
This fixes a segfault in test.test_asyncio.test_events.EPollEventLoopTests.test_sock_client_ops
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: update a coroutine test case
0ee1366 
Add the new attribute "cr_origin" to the copy-test.
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: refactor PyStacklessState and PyStacklessInterp…
413fabd 
…reterState

Resize flags and reorder them to avoid padding. This is a preparation
for adding another flag field for pickle related flags.
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: add pickle-flags to PyStacklessState
99ac643 
Add the flags and methods stackless.pickle_flags() and
stackless.pickle_flags_default() to get or set them. Currently only one
flag is defined in prickelpit.h. It will be used by the next commit.
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: documentation update
d6d23ea 
Update the documentation regarding pickling. More to come.
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: enable pickling of asynchronous generators
80b3f97 
Stackless now pickles asynchronous generators. Read the documentation
about Stackless pickling for details.
akruis pushed a commit that referenced this issue Nov 11, 2018
Stackless issue #139: pickle PyAsyncGenASend or PyAsyncGenAThrow
52ee18f 
Stackless now pickles awaitable objects of type PyAsyncGenASend or
PyAsyncGenAThrow (see PEP 525).
@akruis
Copy link
Author

akruis commented Nov 11, 2018

A short overview over the commits:

  • Pickle-flags infrastructure: 99ac643
  • Enable pickling of asynchronous generators: 80b3f97
  • Enable pickling of auxiliary types PyAsyncGenASend and PyAsyncGenAThrow: 52ee18f

There's another auxiliary type _PyAsyncGenWrappedValue. I looked at the code and I'm fairly sure that objects of this type can't be seen by Python code.
Therefore this issue is now completed.

@akruis akruis closed this as completed Nov 11, 2018
akruis pushed a commit that referenced this issue Jan 5, 2019
Stackless issue #139: ignore global gen_exhausted_asyncgen
cb04beb 
Add the global variable gen_exhausted_asyncgen to the list of ignored
globals. It is a constant singleton.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
3.7-slp blocker enhancement main-slp stackless python task
Projects
None yet
Milestone
v3.7.2-slp
Development

No branches or pull requests
1 participant
@akruis