D 2020-12-16T10:36:46.146
L logfmt1
N text/x-markdown
U mario
W 1751
**logfmt1** handles `*.log.fmt` files to transform LogFormat / placeholder
strings to regular expressions (named capture groups). Currently just comes
with rules for Apache definitions. It bundles a `logex` and `update-logfmt`
to create/rewrite `*.log.fmt` files globally.
{
"class": "apache combined",
"record": "%h %l %u %t \"%r\" %>s %b",
}
It's basically meant for universal log parsing, whilst reducing manual
configuration or the restrain on basic log variants. It originated in
[modseccfg](https://fossil.include-once.org/modseccfg/). This Python
package is mostly a stub. You should preferrably install the
[system package](https://apt.include-once.org/):
apt install python3-logfmt1
This will yield the proper `/usr/share/logfmt/` structure and the run-parts
wrapper `update-logfmt`. The grok placeholders are supported, but remain
untested.
### logfmt1
To craft a regex:
import logfmt1, json
fmt = json.load(open("/.../access.log.fmt", "r"))
rx = logfmt1.regex(fmt)
rx = logfmt1.rx2re(rx) # turn into Python regex
Or with plain old guesswork / presuming a standard log format:
rx = logfmt1.regex({"class": "apache combined"})
Though that's of course not the intended use case, and hinges on
predefined formats in /usr/share/logfmt/.
### logex
Very crudementary extractor for log files:
logex .../access.log --tab @host @date +id
Which of course handles the `.fmt` implicitly.
### update-logfmt
The Python package does bundle a run-parts wrapper, but just the apache
collector, and a local Python copy of the format database. It should discover
all `*.log` files nonetheless and pair them with `.fmt` declarations.
Z afa47af06b3e73021945eb8760ba6c22