smart_holder trampoline shared_from_this support

[rwgk]

Before this PR the smart_holder type_caster implementation did not take std::enable_shared_from_this into account at all. As long as no trampolines are involved, that was fine, but the "inheritance slicing" issue (#1333) resurfaces

• if shared_from_this uses its supporting weak_ptr as a backdoor to the shared_ptr inside smart_holder,
• which does not, and cannot, keep the trampoline Python object alive,
• because that would be a reference cycle, undetectable by gc.

Unfortunately the approach pioneered under PR #2839, and adopted in the smart_holder code, does not play nicely with shared_from_this. Here it gets complicated:

• Before C++17, the shared_from_this behavior was not even fully defined for a situation in which multiple shared_ptr objects exist simultaneously (which is valid if custom deleters are involved, and exactly what the smart_holder code does).
• C++17 made the definition more rigorous: shared_from_this will retrieve the first shared_ptr object created (until it expires):
  • https://en.cppreference.com/w/cpp/memory/enable_shared_from_this
  • "... and assign the newly created std::shared_ptr to the internally stored weak reference [if not already owned by a live std::shared_ptr (since C++17)]"

How can this be made to play nicely with pybind11 trampolines?

Omitting a very long story of failed experiments in the commit history of this PR, the key trick is:

• Make the shared_ptr inside smart_holder invisible to the shared_from_this mechanism, by casting the raw pointer to void * before passing it to the shared_ptr constructor (or shared_ptr::reset, to be precise wrt to the smart_holder implementation).

With that, the PR #2839 approach works again, although it also introduces a subtle but important requirement:

• shared_from_this will only succeed (vs throwing a bad_weak_ptr exception) while C++ code, outside of the smart_holder, owns a shared_ptr object.

That means, a situation that boils down to the following can (if no other non-smart_holder shared_ptr exists at call time) trigger a bad_weak_ptr RuntimeError:

long foo(const Bar &obj) { return obj.shared_from_this().use_count(); }
...
m.def("foo", foo);

An obvious way to solve the problem is to redefine foo():

long foo(const std::shared_ptr<const Bar> &obj) { return obj.use_count(); }

But if that's not an option (e.g. because it is an external dependency that cannot easily be modified), there is an easy alternative trick:

m.def("foo", [](const std::shared_ptr<const Bar>& obj) { return foo(*obj); });

The lambda function keeps the shared_ptr alive for the duration of the foo() function call. Problem solved.

This PR goes one step further, although that's mostly a nicety. The smart_holder now also keeps a weak_ptr to the shared_ptr that keeps the trampoline Python object alive, and uses the weak_ptr to generate shared_ptr objects in repeated calls. Therefore the use_counts of the returned shared_ptr objects are in line with naive expectations.

It is possible that this PR does not solve all possible corner cases of shared_from_this interacting with the smart_holder implementation, but at least it goes from "clearly incorrect behavior" to "mostly correct". Systematic test coverage for all corner cases, with fixes if any, is very involved and will be done separately.

[rwgk]

Closing/Opening to work around flake (see #2995).

[rwgk]

Merging comment:

• This PR was run through the Google internal global testing.
• Ignoring the valgrind errors observed in the last iteration here, assuming they are false positives. Google-internally this PR was tested with ASAN.
• This PR was reviewed by two Googlers and will be reviewed again when it is merged internally.

[virtuald]

It would be good if the description of the various subtle pieces involved with using enable_shared_from_this were put into the README.