Filter PII

Python Logging is a centralized place to filter out sensitive Personally Identifiable Information.

The flow chart shows how a LogRecord propagates through the logging frameworks' layers. We can leverage the Filter layer (over the formatter) to redact information.

You can use the convenient Loggingredactor library to do so. The redaction acts on

• record.msg which is the format str line that's match across regex patterns

• record.args are any parameters that's passed to the log which are interpolated against the format string msg. This checks any dictionary for keys that needs to be redacted.

def filter(self, record):
    try:
        record.msg, record.args = (
            self.redact(record.msg), self.redact(deepcopy(record.args)
        )
    except Exception:
        pass
    return True

deepcopy to not mutate the shared (by reference) args object

Always logs content by returning True

Wesley's Page provides some nice example of the popular python-json-logger

References

• https://docs.python.org/3/howto/logging.html

• https://docs.python.org/3/library/logging.html#logging.LogRecord

• https://docs.python.org/3/library/logging.html#filter-objects

• https://relaxdiego.com/2014/07/logging-in-python.html

• https://betterstack.com/community/guides/logging/sensitive-data/

• https://pypi.org/project/loggingredactor/

• https://web.archive.org/web/20201130054012/https://wtanaka.com/node/8201

• https://github.com/madzak/python-json-logger