Higher level: does check_garbage() really serve a purpose anymore? It's doing 2/3rds of the work done by the new function just to see whether at least one thing has become directly reachable. Which the new function does anyway. Since the amount of trash is usually small, the relative expense of running the new function all the time is minor (it's not crawling over the entire generation, just the trash objects - which is usually an empty list!).

I very much agree with this! I did not have removed check_garbage because I wanted to make sure everything else looks fine before discussing about just doing the work every time (calling deduce_unreachable over unreachable).

On the other hand check_garbage is setting clearing the flags (it calls gc_clear_collecting) so someone else needs to do that if we remove the call. Where do you think is the best place to do this if we remove the function?

To make sure our assumption is correct, I will run the pyperformance test suite :) (Although I think is not really necesary, but checking does not hurt),

On the other hand check_garbage is setting clearing the flags (it calls gc_clear_collecting) so someone else needs to do that if we remove the call. Where do you think is the best place to do this if we remove the function?

The bit-fiddling obfuscation in this module is depressing ☹️ There are no principled answers: the only scheme at work appears to be "delay undoing a trick as long as possible, regardless of how long that leaves things in a variety of semi-broken states, or how obscure they are".

So repair all the remaining damage before delete_garbage() is called? That's the last non-trivial thing collection does.

The bit-fiddling obfuscation in this module is depressing ☹️ There are no principled answers: the only scheme at work appears to be "delay undoing a trick as long as possible, regardless of how long that leaves things in a variety of semi-broken states, or how obscure they are".

Yep, modifying this code is surprisingly tricky as there are lots of contracts not precisely well documented and many optimization details make the thing even more obscure.

So repair all the remaining damage before delete_garbage() is called? That's the last non-trivial thing collection does.

Unless we somehow introduce the call to gc_clear_collecting into some existing function, we will end implementing 90% of the work that check_garbage is doing if we add another function that iterates over the lists to call gc_clear_collecting, as the main overhead is the iteration over these lists.

I'm not looking to do less work than check_garbage, just looking to do it once instead of twice. More, I think it's a Bad Idea to follow this module's lead of being as "clever" as possible with no regard for how brittle that makes the code. Do the simplest & most straightforward thing instead?

Looking at older versions of this code is good for the soul 😃 Neil's original was a joy to work with, subtle algorithms but coded in completely straightforward ways. I dare say I improved some on that, by introducing a concrete notion of "state", with well defined transitions that were trivial to implement and query. Pointers were always pointers, ints were always ints ...

But the gc header struct had 3 members then (next, prev, and refcount copy). All the pain was introduced to get rid of the header's refcount member, and almost nothing was straightforward anymore. It's not a trade-off I would have made, but it's certainly defensible.

In any case, the ubiquitous bit fiddling now also slowed the code. I don't at all mind making it a bit slower still to preserve some sanity 😜

@tim-one Ok, after some design iterations, I am more or less happy with the shape of
7ccc564 . This commit removes the check_garbage function as per our discussion and encapsulates all the work for handling resurrecting objects in a new function. Better names are welcomed 😜. I think is now a bit more clear and makes things a bit more sane :)

Note: I have rebased to pick the latest GC_DEBUG changes that we made in master.

Thank you very much for the thorough review! :)