logfmt1

logfmt1 is an implementation and scheme for generic log parsing. It prescribes a .fmt descriptor adjacent to each log file. And a global database to resolve format string %placeholder%s and turn them into regular expressions.

Overview

• A .log.fmt for each log file
• logopen()
• update() and regex()
• Field .fmt definitions in /usr/share/logfmt
• update-logfmt to create *.log.fmt files (for Apache, Nginx, some static logs)

Sample .log.fmt

Currently the format for a *.log.fmt descriptor is:

{
   "class": "apache custom2",
   "record": "%a %u %l [%t] %V \"%r\" %O %>s %D %{SSL_CIPHER}e %M"
}

Usage

In the simplest of cases you can use logopen() to process a log file (and its adjacent .fmt descriptor) at once:

log = logfmt1.logopen("/var/log/apache2/access.log")
for row in log:
    print(row["request_line"])
#print(log.names())

 
There's a few options (debug=True) and an .alias dict on the logopen iterator.

About

See also the online documentation or pypi listing.
This project originated as part of modseccfg.