Add RecoverInterceptor as alternative to WithRecover

[jhump]

This adds things to the handler signature (the other properties of AnyRequest, such as Peer and HTTP method) and also provides the interceptor separately, so it can be more easily combined with other interceptors. Finally, the value of ignoring http.ErrAbortHandler is unclear and adds a potential footgun in that a handler that panics with that value may inadvertently prevent capturing metrics or other side effects that are desirable for error-tracking in the face of a panic.

WithRecover remains backwards-compatible. But it is deprecated; users should prefer the new RecoverInterceptor method and use it with WithInterceptors.

The most controversial parts of this change:

1. Using AnyRequest as the way to pass request properties, even for streaming RPCs. Does this seem like a bad idea? This was done to avoid adding a new exported type and to avoid a long parameter list. One awkwardness with this approach is having a req.Any() method that isn't useful for client and bidi streams and is awkward to populate for server streams.
2. Can be used in clients, too, to recover from panics caused by client interceptors. (It could also technically recover from panics in the framework, though hopefully those kinds of issues don't exist!) This is more complicated for streaming calls since it means needing to wrap a stream in order recover from panics in various methods (send, receive, close...).

Thoughts?

Resolves #816.

[emcfarlane]

atomic.Pointer I think should be just atomic.Value here as you are using any anyway.

[jhump]

That's fair. I'm generally allergic to atomic.Value because it panics if the value is already set and a later attempt tries to CompareAndSwap with a different concrete type (example). In many cases (particularly for error values), the concrete type can trivially differ, resulting in a panic instead of a failed CAS.

But in this case, there is a generic type parameter for the request type requiring it to always be a particular concrete type, so there should be no such risk. So it should be safe to just use sync.Value.

There is a slight risk that a caller could call stream.Conn(), which provides untyped Send(any) error and Receive(any) error methods. If they tried to call these with mismatched concrete types, it would panic. But it's possible it would already panic elsewhere due to a failed type conversion to the strongly-typed stream wrapper's concrete request type. I'll try it out and see.

[emcfarlane]

Suggested change

	var msg any
	if msgPtr := r.req.Load(); msgPtr != nil {
	msg = *msgPtr
	}
	msg := r.req.Load()

[emcfarlane]

Passing the AnyRequest does feel odd for the streaming handlers. Would your solution in #851 (comment) solve this case nicely too and we can pass RequestInfo to the handler?

[jhump]

Would your solution in #851 (comment) solve this case nicely too and we can pass RequestInfo to the handler?

Oh, good idea. That would also completely eliminate any atomic.Pointer/atomic.Value shenanigans since that type doesn't include a request message. (The current signature does not get any request messages; I had only added it as an admittedly awkward component of re-using AnyRequest...)

I can update this PR to include a tentative RequestInfo interface type and then hold off merging until the shape of that interface is completely settled over in that other PR.

It might be slightly confusing for the handler to get a value that ostensibly includes response headers and trailers, but that is probably less awkward/confusing than using AnyRequest.