GUI editor to tame mod_security rules

โŒˆโŒ‹ โŽ‡ branch:  modseccfg

patch for PluginMeta() wrapper required in last pluginconf.gui.window()
mario authored 346 days ago last checkin 4f8b060ed
๐Ÿ“‚ docs Fix mkdocs/RTD table styling per sedโ€นโ€บ 1378 days ago
๐Ÿ“‚ html Typos fixed in logfmt1 docsโ€นโ€บ 1374 days ago
๐Ÿ“‚ manpage Updated man pages for logfmt1โ€นโ€บ 1363 days ago
๐Ÿ“‚ share logfmt1: Add update/nginx support (untested), fmt2md, #doc and #src cโ€นโ€บ 1389 days ago
๐Ÿ“„ README.md Comment updates, fixed script wrappers, unify update-logfmt to pythonโ€นโ€บ 1390 days ago
๐Ÿ“„ __init__.py Bundle logfmt1 into sub project. Support for /usr/share/logfmt/ databโ€นโ€บ 1390 days ago
๐Ÿ“„ fmt2md logfmt1: Add update/nginx support (untested), fmt2md, #doc and #src cโ€นโ€บ 1389 days ago
๐Ÿ“„ grok2fmt1 Bundle logfmt1 into sub project. Support for /usr/share/logfmt/ databโ€นโ€บ 1390 days ago
๐Ÿ“„ logex.py Use dateutil.parser fuzzy=Trueโ€นโ€บ 1342 days ago
๐Ÿ“„ logfmt1.py Stub manpage for logfmt(5)โ€นโ€บ 1372 days ago
๐Ÿ“„ mkdocs.yml Typos fixed in logfmt1 docsโ€นโ€บ 1374 days ago
๐Ÿ“„ setup.py Updated man pages for logfmt1โ€นโ€บ 1363 days ago
๐Ÿ“„ update_logfmt.py Add "type": classifiers for some logfmt fields. Support $1$2$3 for exโ€นโ€บ 1388 days ago

logfmt1

See also docs/


logfmt1 is meant for universal log parsing, whilst reducing manual configuration or restricting to basic log variants. It handles *.log.fmt files to transform LogFormat / placeholder strings to regular expressions (with named capture groups).

{
   "class": "apache combined",
   "record": "%h %l %u %t \"%r\" %>s %b",
}

For instance would resolve to:

(?<remote_host>[\\w\\-.:]+) (?<remote_logname>[\\w\\-.:]+) (?<remote_user>[\\-\\w@.]+)
\\[?(?<request_time>\\d[\\d:\\w\\s:./\\-+,;]+)\\]? "(?<request_line>(?<request_method>\\w+)
(?<request_path>\\S+) (?<request_protocol>[\\w/\\d.]+))" (?<status>-|\\d\\d\\d)
(?<bytes_sent>\\d+|-)'

This python package currently just comes with:

  • .fmt definitions for apache + strftime + grok placeholders.
  • logex - a basic log extractor
  • And update-logfmt to create/rewrite *.log.fmt files globally.

It originated in modseccfg. You should ideally install the system package however:

apt install python3-logfmt1

This will yield the proper /usr/share/logfmt/ structure and the run-parts wrapper update-logfmt.

logfmt1

To manually craft a regex:

import logfmt1, json
fmt = json.load(open("/.../access.log.fmt", "r"))
rx = logfmt1.regex(fmt)
rx = logfmt1.rx2re(rx)   # turn into Python regex

Or with plain old guesswork / presuming a standard log format:

rx = logfmt1.regex({"class": "apache combined"})

Though that's of course not the intended use case, and hinges on predefined formats in /usr/share/logfmt/.

logfmt1.logopen()

logopen(fn=โ€ฆ) is basically a file-like iterator that yields dictionaries rather than text strings.

for row in logfmt1.logopen(".../access.log"):
print(row["request_time"])

And it provides a basic regex/formatstring debugging feature (via debug=True parameter or with logex -D):

failed regex section

logex

Very crudementary extractor for log files:

logex .../access.log --tab @host @date +id

Which also handles the .fmt implicitly. (Kinda the whole point of this project.)

update-logfmt

The Python package does bundle a run-parts wrapper, but just the apache collector, and a local Python copy of the format database. It should discover all (Apache) *.log files nonetheless and pair them with .fmt declarations.

And that's sort of the main aspect of this project. Establish .log.fmt files until application vendors come around to making logs parseable. The rules database structure is subject to change, and only one possible implementation. There might also be simpler approaches (grok mapping) to generate regexps for format strings.