Support structlog within libraries, redux

[lemon24]

Hi @hynek,

This is a follow-up to #179 – I have a recipe for library support, but it could be made less brittle with support from structlog.

To summarize the requirements:

1. the library MUST NOT configure logging nor structlog (that's for the user to do)
2. the library MUST be able to use structlog internally
3. if the user configures logging, they MUST get logging output from the library (there's a sensible event k=v ... default formatter; the library MAY offer a way to configure that, but it seems YAGNI)
4. if the user opts in, they MUST get structlog output from the library
5. it is acceptable for the user opt in to work only before the library is used, threads are started, structlog is configured, etc. (the library can document this accordingly)

The code is here; in short:

• the library keeps a registry of its loggers
• by default, it wraps a logging logger
• during the opt-in, existing loggers are reset (details below)
• after the opt-in, loggers are created with structlog.get_logger

Usage looks this:

# library.py

_registry = WrappedLoggerFactory(... wrap_logger() kwargs ...)
# library uses this exclusively to create its loggers
get_logger = _registry.get_logger  
# users call this to opt in
enable_native_structlog = _registry.enable_native_structlog

# user code

import library, structlog
library.enable_native_structlog()
structlog.configure(...)

library.do_stuff(...)

Minimal support from structlog would require:

1. A (public) way to "reset" loggers created by wrap_logger and get_logger.

   My current code assumes both return the same type of object, which holds the entire state in its __dict__, so I can do:

   logger.__dict__.update(structlog.get_logger(name).__dict__)

   This is currently true (BoundLoggerLazyProxy), but it's obviously brittle.

   (More complicated alternatives that do not require structlog cooperation: wrap the returned object in another proxy; record the call site, and during opt in go in sys.modules and replace the loggers.)

2. A (public) way to fix the stacklevel per logging logger.

   In WLF.get_logger, I'm temporarily doing logging.setLoggerClass(_FixedFindCallerLogger). This is still brittle, since stdlib loggers for my library could be created before the library calls WLF.get_logger (e.g. by logging.config.dictConfig).

   I guess an alternative would be for WLF.get_logger to patch the findCaller of the stdlib logger, instead of bothering with logging.setLoggerClass. Update: yup, seems to work fine.

   (This is optional, a library should be able to implement this without making assumptions about structlog internals.)

structlog could also provide something higher level like WrappedLoggerFactory; in this case, the stuff above doesn't need to be public API anymore, just possible.

What do you think? (If the overall direction is acceptable, I can draft a PR.)

[lemon24]

For clarity, as far as I understand, this issue is orthogonal to Standard Library Logging / #395 – those are about an application that wants to use both structlog and stdlib, this is about a library that cannot mandate one or the other.

Presumably, structlog would only provide WrappedLoggerFactory, and it would be up to the library to configure it with the appropriate processors and renderer.

[hynek]

Given you seem to have found a solution to 2), there's only 1) left IIUC. So what exactly doing you mean/want by "reset logger"?

---

Generally speaking, my policy is trying my best to help people help themselves. I would love to be more library-friendly, but I suspect solving it in a general way that could live in structlog is too much complexity so I hope to be able to help you enough.

[lemon24]

So what exactly doing you mean/want by "reset logger"?

Making a logger that was created with wrap_logger() behave as if it was re-created with get_logger(). That is:

logger = wrap_logger(...)
reset_logger(logger)

has the same semantics as:

logger = wrap_logger(...)
logger = get_logger()

...except it happens in-place (logger remains the same object). I guess a more general term would be "reconfigure".

The return type of either function is a BoundLoggerLazyProxy with various private attributes and possibly a bind attribute, but this is not documented explicitly, so my understanding is that I should treat it as an opaque proxy.

In practice, I am updating the BoundLoggerLazyProxy's __dict__ in place (snippet above), but this makes assumptions about its type (being the same for either function) and state (being entirely in __dict__), which is at odds with it being opaque.

Generally speaking, my policy is trying my best to help people help themselves.

Technically, I have a complete solution, and tests to prevent breakage due to relying on internals. Aside from (1), I could also vendor the bits of code I'm using, but I don't expect any issues given how rarely they changed in the past.

The minimum API surface that would make it fully stable would be:

1. document that wrap_logger() and get_logger() return the same type and that the state is entirely in __dict__

   • OR, add a reset_logger(logger) function
   • OR, add a more general configure_logger(logger, **get_logger_kwargs) function

2. make _FixedFindCallerLogger public (optional due to the possibility of vendoring)

I suspect solving it in a general way that could live in structlog is too much complexity

Understood, no worries (I see how even baking in the assumptions above may restrict future evolution).

Thank you!