**logfmt1** handles `*.log.fmt` files to transform LogFormat / placeholder
strings to regular expressions (named capture groups). Currently just comes
with rules for Apache definitions. It bundles a `logex` and `update-logfmt`
to create/rewrite `*.log.fmt` files globally.

    {
       "class": "apache combined",
       "record": "%h %l %u %t \"%r\" %>s %b",
    }

It's basically meant for universal log parsing, whilst reducing manual
configuration or the restrain on basic log variants. It originated in












[modseccfg](https://fossil.include-once.org/modseccfg/). This Python
package is mostly a stub. You should preferrably install the
[system package](https://apt.include-once.org/):


    apt install python3-logfmt1

This will yield the proper `/usr/share/logfmt/` structure and the run-parts
wrapper `update-logfmt`. The grok placeholders are supported, but remain
untested.


### logfmt1

To craft a regex:

    import logfmt1, json
    fmt = json.load(open("/.../access.log.fmt", "r"))
    rx = logfmt1.regex(fmt)
    rx = logfmt1.rx2re(rx)   # turn into Python regex

Or with plain old guesswork / presuming a standard log format:

    rx = logfmt1.regex({"class": "apache combined"})

Though that's of course not the intended use case, and hinges on
predefined formats in /usr/share/logfmt/.
















### logex

Very crudementary extractor for log files:

    logex .../access.log --tab @host @date +id

Which of course handles the `.fmt` implicitly.



### update-logfmt

The Python package does bundle a run-parts wrapper, but just the apache
collector, and a local Python copy of the format database. It should discover
all `*.log` files nonetheless and pair them with `.fmt` declarations.

**logfmt1** is meant for universal log parsing, whilst reducing manual
configuration or restricting to basic log variants. It handles `*.log.fmt`
files to transform LogFormat / placeholder strings to regular expressions
(with named capture groups).

    {
       "class": "apache combined",
       "record": "%h %l %u %t \"%r\" %>s %b",
    }

For instance would resolve to:


    (?<remote_host>[\\w\\-.:]+) (?<remote_logname>[\\w\\-.:]+) (?<remote_user>[\\-\\w@.]+)
    \\[?(?<request_time>\\d[\\d:\\w\\s:./\\-+,;]+)\\]? "(?<request_line>(?<request_method>\\w+)
    (?<request_path>\\S+) (?<request_protocol>[\\w/\\d.]+))" (?<status>-|\\d\\d\\d)
    (?<bytes_sent>\\d+|-)'
    
This python package currently just comes with:

  * `.fmt` definitions for apache + strftime + grok placeholders.
  * `logex` - a basic log extractor
  * And `update-logfmt` to create/rewrite `*.log.fmt` files globally.

It originated in [modseccfg](https://fossil.include-once.org/modseccfg/).

You should ideally install the [system package](https://apt.include-once.org/)
however:

    apt install python3-logfmt1

This will yield the proper `/usr/share/logfmt/` structure and the run-parts
wrapper `update-logfmt`.



### logfmt1

To manually craft a regex:

    import logfmt1, json
    fmt = json.load(open("/.../access.log.fmt", "r"))
    rx = logfmt1.regex(fmt)
    rx = logfmt1.rx2re(rx)   # turn into Python regex

Or with plain old guesswork / presuming a standard log format:

    rx = logfmt1.regex({"class": "apache combined"})

Though that's of course not the intended use case, and hinges on
predefined formats in /usr/share/logfmt/.


### logfmt1.logopen()

`logopen(fn=…)` is basically a file-like iterator that yields
dictionaries rather than text strings.

    for row in logfmt1.logopen(".../access.log"):
	print(row["request_time"])

And it provides a basic regex/formatstring debugging feature (via
`debug=True` parameter or with `logex -D`):

![failed regex section](https://imgur.com/QBKzDsK.png)


### logex

Very crudementary extractor for log files:

    logex .../access.log --tab @host @date +id

Which also handles the `.fmt` implicitly. (Kinda the whole point of
this project.)


### update-logfmt

The Python package does bundle a run-parts wrapper, but just the apache
collector, and a local Python copy of the format database. It should discover
all (Apache) `*.log` files nonetheless and pair them with `.fmt` declarations.

And that's sort of the main aspect of this project. Establish .log.fmt files
until application vendors come around to making logs parseable. The rules
database structure is subject to change, and only one possible implementation.
There might also be simpler approaches (grok mapping) to generate regexps
for format strings.

import sys, re, json
import traceback, dateutil.parser
import logfmt1





#-- args
argv = sys.argv
space = " "
if "--tab" in argv:
    space = "\t"
if "--csv" in argv:
    space = "," 

import sys, re, json
import traceback, dateutil.parser
import logfmt1


def main():
    pass
    
#-- args
argv = sys.argv
space = " "
if "--tab" in argv:
    space = "\t"
if "--csv" in argv:
    space = "," 

# encoding: utf-8
# api: python
# title: python3-logfmt1
# description: handle *.log.fmt specifiers and regex conversion
# type: transform
# category: io
# version: 0.4
# license: Apache-2.0
# pack:
#    logfmt1.py=/usr/lib/python3/dist-packages/
#    update-logfmt=/usr/bin/
#    ./logex.py=/usr/bin/logex
#    share=/usr/share/logfmt
# architecture: all
# depends: python (>= 3.6)
# url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
#
# Logging format strings to regex conversion.
................................................................................

        #"record": "%h %l %u %t \"%r\" %>s %b",

        #"regex": "(?<remote_host>\S+) …",

        "separator": " ",
        "rewrite": {
            "%[\d!,]+": "%",      # strip Apache placehoder conditions
            "(?<!\\\\)([\[\]])": r"\\$1",  # escape meta chars
            "%%": "%",
        },
        "placeholder": "%[<>]?(?:\w*\{[^\}]+\})?\^?\w+",

        # placeholder definitions to build regex: from
        "fields": {
................................................................................
            "%a": { "id": "remote_addr", "rx": "[\d.:a-f]+" },
            "%{c}a": { "id": "remote_addr", "rx": "[\d.:a-f]+" },
            "%h": { "id": "remote_host", "rx": "[\w\-.:]+" },
            "%{c}h": { "id": "remote_host", "rx": "[\w\-.:]+" },
            "%A": { "id": "local_address", "rx": "[\d.:a-f]+" },
            "%u": { "id": "remote_user", "rx": "[\-\w@.]+" },
            "%l": { "id": "remote_logname", "rx": "[\w\-.:]+" },   # %alias `loglevel` (errlog)
            "%t": { "id": "request_time", "rx": "\[(\d[\d:\w\s:./\-+,;]+)\]" }, # might be "local" formatting, e.g. [01/Mnt/2020:11:22:33 +0100], %alias `ctime`
            "%{u}t": { "id": "request_time", "rx": "\d+/\w+/\d+:\d+:\d+:\d+\.\d+\s\+\d+" },  # 01/Mnt/2020:11:22:33.12345 +0100 no implicit brackets
            "%{cu}t": { "id": "request_time", "rx": "\d+-\w+-\d+\s\d+:\d+:\d+\.\d+" },  # error.log-only, 2020-01-31 11:22:33.901234, compact ISO 8601 format, no implicit brackets
            "%{msec_frac}t": { "id": "msec_frac", "rx": "[\d.]+" },
            "%{usec_frac}t": { "id": "usec_frac", "rx": "[\d.]+" },
            "%f": { "id": "request_file", "rx": "[^\s\"]+" },
            "%b": { "id": "bytes_sent", "rx": "\d+|-" },
            "%B": { "id": "bytes_sent", "rx": "\d+|-" },
................................................................................
    def __next__(self):
        line = self.f.readline()
        if not line:  # should be implied really
            raise StopIteration()
        m = self.rx.match(line)
        if m:
            d = m.groupdict()

            self.container_expand(d)
            if self.duplicate:
                for trg,src in self.alias.items():
                    if src in d and not trg in d:
                        d[trg] = d[src]
            return d
        elif self.debug:
            self.debug_rx(line)
            if self.fail:
                raise StopIteration()
        elif self.fail:
            raise StopIteration()
        else:
            pass # jusst try next line
    
    # pass .close() and similar to file object
    def __getattr__(self, name):
        return getattr(self.f, name)

    # add [key "value"] fields
    def container_expand(self, d):
        for k,opt in self.container.items():
            if k in d:
                for id,val in re.findall(opt["rx"], d[k]):
                    if not id in d:
                        d[id] = val
                    elif not isinstance(d[id], "list"):
                        d[id] = [d[id], val]
                    else:
                        d[id].append(val)

    # ANSI output for debugging regex/fmt string
    def debug_rx(self, line):
        rx = self.rx.pattern

# encoding: utf-8
# api: python
# title: python3-logfmt1
# description: handle *.log.fmt specifiers and regex conversion
# type: transform
# category: io
# version: 0.4-p2
# license: Apache-2.0
# pack:
#    logfmt1.py=/usr/lib/python3/dist-packages/
#    update_logfmt.py=/usr/bin/update-logfmt
#    ./logex.py=/usr/bin/logex
#    share=/usr/share/logfmt
# architecture: all
# depends: python (>= 3.6)
# url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
#
# Logging format strings to regex conversion.
................................................................................

        #"record": "%h %l %u %t \"%r\" %>s %b",

        #"regex": "(?<remote_host>\S+) …",

        "separator": " ",
        "rewrite": {
            "%[\d!,+\-]+": "%",      # strip Apache placehoder conditions
            "(?<!\\\\)([\[\]])": r"\\$1",  # escape meta chars
            "%%": "%",
        },
        "placeholder": "%[<>]?(?:\w*\{[^\}]+\})?\^?\w+",

        # placeholder definitions to build regex: from
        "fields": {
................................................................................
            "%a": { "id": "remote_addr", "rx": "[\d.:a-f]+" },
            "%{c}a": { "id": "remote_addr", "rx": "[\d.:a-f]+" },
            "%h": { "id": "remote_host", "rx": "[\w\-.:]+" },
            "%{c}h": { "id": "remote_host", "rx": "[\w\-.:]+" },
            "%A": { "id": "local_address", "rx": "[\d.:a-f]+" },
            "%u": { "id": "remote_user", "rx": "[\-\w@.]+" },
            "%l": { "id": "remote_logname", "rx": "[\w\-.:]+" },   # %alias `loglevel` (errlog)
            "%t": { "id": "request_time", "rx": "\[?(\d[\d:\w\s:./\-+,;]+)\]?" }, # might be "local" formatting, e.g. [01/Mnt/2020:11:22:33 +0100], %alias `ctime`
            "%{u}t": { "id": "request_time", "rx": "\d+/\w+/\d+:\d+:\d+:\d+\.\d+\s\+\d+" },  # 01/Mnt/2020:11:22:33.12345 +0100 no implicit brackets
            "%{cu}t": { "id": "request_time", "rx": "\d+-\w+-\d+\s\d+:\d+:\d+\.\d+" },  # error.log-only, 2020-01-31 11:22:33.901234, compact ISO 8601 format, no implicit brackets
            "%{msec_frac}t": { "id": "msec_frac", "rx": "[\d.]+" },
            "%{usec_frac}t": { "id": "usec_frac", "rx": "[\d.]+" },
            "%f": { "id": "request_file", "rx": "[^\s\"]+" },
            "%b": { "id": "bytes_sent", "rx": "\d+|-" },
            "%B": { "id": "bytes_sent", "rx": "\d+|-" },
................................................................................
    def __next__(self):
        line = self.f.readline()
        if not line:  # should be implied really
            raise StopIteration()
        m = self.rx.match(line)
        if m:
            d = m.groupdict()
            if self.container:
                self.container_expand(d)
            if self.duplicate:
                for trg,src in self.alias.items():
                    if src in d and not trg in d:
                        d[trg] = d[src]
            return d
        elif self.debug:
            self.debug_rx(line)
            if self.fail:
                raise StopIteration()
        elif self.fail:
            raise StopIteration()
        else:
            pass # just try next line
    
    # pass .close() and similar to file object
    def __getattr__(self, name):
        return getattr(self.f, name)

    # add [key "value"] fields
    def container_expand(self, d):
        for k,opt in self.container.items():
            if k in d:
                for id,val in re.findall(opt["rx"], d[k]):
                    if not id in d:
                        d[id] = val
                    elif not isinstance(d[id], list):
                        d[id] = [d[id], val]
                    else:
                        d[id].append(val)

    # ANSI output for debugging regex/fmt string
    def debug_rx(self, line):
        rx = self.rx.pattern

           "./share/*",
           "./share/update/*"
        ],
    },
    #data_files=[],
    entry_points={
        "console_scripts": [
            "logex=logfmt1.logex",
            "update-logfmt=logfmt1.update_logfmt_all",
        ]
    }
)

           "./share/*",
           "./share/update/*"
        ],
    },
    #data_files=[],
    entry_points={
        "console_scripts": [
            "logex=logfmt1.logex:main",
            "update-logfmt=logfmt1.update_logfmt:main",
        ]
    }
)

{
    "class": "apache combined",
    "record": "%h %l %u %t \"%r\" %>s %b",
    "glob": [
        "*.access.log"
    ]
}

{
    "class": "apache combined",
    "record": "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"",
    "glob": [
        "*.access.log"
    ]
}

{
    "class": "apache generic",
    "separator": " ",
    "rewrite": {
        "%[\\d!,]+": "%",
        "(?<!\\\\)([\\[\\]])": "\\\\$1",
        "%%": "%"
    },
    "placeholder": "%[<>]?(?:\\w*\\{[^\\}]+\\})?\\^?\\w+",
    "fields": {
        "%a": {
            "id": "remote_addr",
................................................................................
        },
        "%l": {
            "id": "remote_logname",
            "rx": "[\\w\\-.:]+"
        },
        "%t": {
            "id": "request_time",
            "rx": "\\[(\\d[\\d:\\w\\s:./\\-+,;]+)\\]"
        },
        "%{u}t": {
            "id": "request_time",
            "rx": "\\d+/\\w+/\\d+:\\d+:\\d+:\\d+\\.\\d+\\s\\+\\d+"
        },
        "%{cu}t": {
            "id": "request_time",

{
    "class": "apache generic",
    "separator": " ",
    "rewrite": {
        "%[\\d!,+\\-]+": "%",
        "(?<!\\\\)([\\[\\]])": "\\\\$1",
        "%%": "%"
    },
    "placeholder": "%[<>]?(?:\\w*\\{[^\\}]+\\})?\\^?\\w+",
    "fields": {
        "%a": {
            "id": "remote_addr",
................................................................................
        },
        "%l": {
            "id": "remote_logname",
            "rx": "[\\w\\-.:]+"
        },
        "%t": {
            "id": "request_time",
            "rx": "\\[?(\\d[\\d:\\w\\s:./\\-+,;]+)\\]?"
        },
        "%{u}t": {
            "id": "request_time",
            "rx": "\\d+/\\w+/\\d+:\\d+:\\d+:\\d+\\.\\d+\\s\\+\\d+"
        },
        "%{cu}t": {
            "id": "request_time",

#!/usr/bin/env python3





import os, re, sys, random
import subprocess
import traceback
import json
from pprint import pprint
try:
    import logfmt1
except:
    from modseccfg import logfmt1


# extraction patterns
class rx:
    # a conf file '(*) /etc/apache2/main.conf'
    dump_includes = re.compile("^\s*\([\d*]+\)\s+(.+)$", re.M)
    # directives we care about (to detect relevant .conf files)
    interesting = re.compile("""
        ^ \s*
         ( (Error|Custom|Global|Forensic|Transfer)Log | (Error)?LogFormat )           # log directivess
        """,
        re.M|re.I|re.X
    )
    # extract directive line including line continuations (<\><NL>)
    configline = re.compile(
        """ ^
        [\ \\t]*                          # whitespace \h*
................................................................................
    def transferlog(self, args):
        self.customlog([args[0], "transfer"])
    def logformat(self, args):
        if len(args) == 1: args[1] = "transfer"
        tmp.log_formats[args[1]] = args[0].replace('\\"', '"')
    def errorlogformat(self, args):
        self.logformat([args[0], "error"])





# scan for APACHE_ENV= vars
def read_env_vars():
    for fn in tmp.env_locations:
        if os.path.exists(fn):
            src = open(fn, "r", encoding="utf-8").read()

#!/usr/bin/env python3
# description: extract *Log options from all apache *.conf to create .log.fmt files
# 
# This is a simpler version of the modseccfg vhost reader.
#

import os, re, sys, random
import subprocess
import traceback
import json
from pprint import pprint

import logfmt1




# extraction patterns
class rx:
    # a conf file '(*) /etc/apache2/main.conf'
    dump_includes = re.compile("^\s*\([\d*]+\)\s+(.+)$", re.M)
    # directives we care about (to detect relevant .conf files)
    interesting = re.compile("""
        ^ \s*
         ( (Error|Custom|Global|Forensic|Transfer)Log | (Error)?LogFormat )           # log directives
        """,
        re.M|re.I|re.X
    )
    # extract directive line including line continuations (<\><NL>)
    configline = re.compile(
        """ ^
        [\ \\t]*                          # whitespace \h*
................................................................................
    def transferlog(self, args):
        self.customlog([args[0], "transfer"])
    def logformat(self, args):
        if len(args) == 1: args[1] = "transfer"
        tmp.log_formats[args[1]] = args[0].replace('\\"', '"')
    def errorlogformat(self, args):
        self.logformat([args[0], "error"])
        
    # could look into LoadModule directives to determine errorlogformat
    # from e.g. mpm_prefork being present


# scan for APACHE_ENV= vars
def read_env_vars():
    for fn in tmp.env_locations:
        if os.path.exists(fn):
            src = open(fn, "r", encoding="utf-8").read()

#!/bin/sh
run-parts /usr/share/logfmt/update/

# description: invoke ./share/update/* scripts
# type: virtual
#
# Stub that reimplements run-parts

import os, re




for dir in [re.sub("[.\w]+$", "share/update", __file__), "/usr/share/logfmt/update"]:
    if os.path.exists(dir):
        os.system(f"run-parts {dir}")
        break

# description: invoke ./share/update/* scripts
# type: virtual
#
# Stub that reimplements run-parts

import os, re

def main():
    pass

for dir in ["/usr/share/logfmt/update", re.sub("[.\w]+$", "share/update", __file__)]:
    if os.path.exists(dir):
        os.system(f"run-parts {dir}")
        break