patch for PluginMeta() wrapper required in last pluginconf.gui.window() | ||
---|---|---|
mario authored 346 days ago last checkin 4f8b060ed ⎘ | ||
๐ docs | Fix mkdocs/RTD table styling per sedโนโบ | 1378 days ago |
๐ html | Typos fixed in logfmt1 docsโนโบ | 1374 days ago |
๐ manpage | Updated man pages for logfmt1โนโบ | 1363 days ago |
๐ share | logfmt1: Add update/nginx support (untested), fmt2md, #doc and #src cโนโบ | 1389 days ago |
๐ README.md | Comment updates, fixed script wrappers, unify update-logfmt to pythonโนโบ | 1390 days ago |
๐ __init__.py | Bundle logfmt1 into sub project. Support for /usr/share/logfmt/ databโนโบ | 1390 days ago |
๐ fmt2md | logfmt1: Add update/nginx support (untested), fmt2md, #doc and #src cโนโบ | 1389 days ago |
๐ grok2fmt1 | Bundle logfmt1 into sub project. Support for /usr/share/logfmt/ databโนโบ | 1390 days ago |
๐ logex.py | Use dateutil.parser fuzzy=Trueโนโบ | 1342 days ago |
๐ logfmt1.py | Stub manpage for logfmt(5)โนโบ | 1372 days ago |
๐ mkdocs.yml | Typos fixed in logfmt1 docsโนโบ | 1374 days ago |
๐ setup.py | Updated man pages for logfmt1โนโบ | 1363 days ago |
๐ update_logfmt.py | Add "type": classifiers for some logfmt fields. Support $1$2$3 for exโนโบ | 1388 days ago |
logfmt1
See also docs/
logfmt1 is meant for universal log parsing, whilst reducing manual
configuration or restricting to basic log variants. It handles *.log.fmt
files to transform LogFormat / placeholder strings to regular expressions
(with named capture groups).
{
"class": "apache combined",
"record": "%h %l %u %t \"%r\" %>s %b",
}
For instance would resolve to:
(?<remote_host>[\\w\\-.:]+) (?<remote_logname>[\\w\\-.:]+) (?<remote_user>[\\-\\w@.]+)
\\[?(?<request_time>\\d[\\d:\\w\\s:./\\-+,;]+)\\]? "(?<request_line>(?<request_method>\\w+)
(?<request_path>\\S+) (?<request_protocol>[\\w/\\d.]+))" (?<status>-|\\d\\d\\d)
(?<bytes_sent>\\d+|-)'
This python package currently just comes with:
.fmt
definitions for apache + strftime + grok placeholders.logex
- a basic log extractor- And
update-logfmt
to create/rewrite*.log.fmt
files globally.
It originated in modseccfg. You should ideally install the system package however:
apt install python3-logfmt1
This will yield the proper /usr/share/logfmt/
structure and the run-parts
wrapper update-logfmt
.
logfmt1
To manually craft a regex:
import logfmt1, json
fmt = json.load(open("/.../access.log.fmt", "r"))
rx = logfmt1.regex(fmt)
rx = logfmt1.rx2re(rx) # turn into Python regex
Or with plain old guesswork / presuming a standard log format:
rx = logfmt1.regex({"class": "apache combined"})
Though that's of course not the intended use case, and hinges on predefined formats in /usr/share/logfmt/.
logfmt1.logopen()
logopen(fn=โฆ)
is basically a file-like iterator that yields
dictionaries rather than text strings.
for row in logfmt1.logopen(".../access.log"):
print(row["request_time"])
And it provides a basic regex/formatstring debugging feature (via
debug=True
parameter or with logex -D
):
logex
Very crudementary extractor for log files:
logex .../access.log --tab @host @date +id
Which also handles the .fmt
implicitly. (Kinda the whole point of
this project.)
update-logfmt
The Python package does bundle a run-parts wrapper, but just the apache
collector, and a local Python copy of the format database. It should discover
all (Apache) *.log
files nonetheless and pair them with .fmt
declarations.
And that's sort of the main aspect of this project. Establish .log.fmt files until application vendors come around to making logs parseable. The rules database structure is subject to change, and only one possible implementation. There might also be simpler approaches (grok mapping) to generate regexps for format strings.