scp python3-logfmt1* io:apt/
whl:
	pandoc logfmt1/README.md -o logfmt1/README.rst
	cd logfmt1 && ./setup.py bdist_wheel
whl_up:
	twine upload dist/logfmt*

docs:
	cd logfmt1 ; PYTHONPATH=. mkdocs build

body {
    font-size: 115% !important;
}

.doc-object {
    margin-top: 2.5rem;
}

blockquote {
    border-left: 5px #efa solid;
    padding-left: 2rem;
    background: #f7fcf0 !important;
}

> | ❮❗❯ | This is all very provisional. (First draft. Names might still change.) |
> | ---- | ----------------------------------- |


## Global .fmt database

While each log file should be accompanied by a [.fmt descriptor](log.fmt.md),
the global database in `/usr/share/logfmt/` contains a full .fmt field
definition for each class.

Most notably the `"fields":` and `"placeholder":` are used to construct the
regex for a `"record":` string definition.


### sample

For example the Apache format definition (apache.fmt) contains:

    {
        "class": "apache generic",
        "separator": " ",
        "rewrite": {
            "%[\\d!,+\\-]+": "%",
            "%%": "%"
        },
        "placeholder": "%[<>]?(?:\\w*\\{[^\\}]+\\})?\\^?\\w+",
        "fields": {
            "%a": {  "id": "remote_addr",    "rx": "[\\d.:a-f]+"  },
            "%h": {  "id": "remote_host",    "rx": "[\\w\\-.:]+"  },
            "%{c}h": { "id": "remote_host",  "rx": "[\\w\\-.:]+"  },
            "%A": {  "id": "local_address",  "rx": "[\\d.:a-f]+"  },
            "%u": {  "id": "remote_user",    "rx": "[\\-\\w@.]+"  },
            "%t": {  "id": "request_time",   "rx": "\\[?(\\d[\\d:\\w\\s:./\\-+,;]+)\\]?"  },
            …
        },
        "alias": {
            "remote_address": "remote_addr",
            "ip": "remote_addr",
            "file": "request_file",
            "size": "bytes_sent",
            …
        },
        "expand": {
            "%\\{([^{}]+)\\}t": {
                "id": "request_time",
                "class": "strftime",
                "record": "$1"
            }
        },
        "container": {
            "message": {
                "id": "$1",
                "value": "$2",
                "rx": "\\[(\\w+) \"(.*?)\"\\]",
                "class": "apache mod_security"
            }
        },
        "glob": ["/var/log/apache*/*acc*.log"]
    }

It usually does not describe a default "record" format (like the local .log.fmt descriptors do).


## class

The class in the global database is largely decorative.  The filenames
instead define the heritage of rules/fields.  The "class" as declared by
a .log.fmt is mapped onto `/usr/share/logfmt/application.variant.fmt`.

 * Usually there's just one variant level per log type. But the lookup is
   supposed to be mildly recursive.
 * Essentially it should merge `*.log.fmt` with `appclass.variant.fmt` and
   `appclass.fmt` applied last, so the most specific definitions are retained.
 * There's also a generic "grok" class. But the patterns therein are largely
   static (not build from variable format strings).
 * Some special classes like "json" might exist. (Not supported by logfmt1)


### record

The "record" entry is not usually present in the global .fmt definition. 
Some super specific variant definitions (for example apache.error.fmt) or
static formats (syslog.fmt) might however.


### separator

Most log formats use spaces for separating %placeholder fields.  And simpler
implementations might just split up the "record" declaration on this.


## placeholder

While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields. It should account for prefixes/suffixes, unless
those got cleared by the `rewrite` map.

Not all formatstrings use `%\w+` to signal placeholders. In nginx for instance
the sigil `$\w+` introduces placeholders (variable names, really).


### rewrite

A list/map of regex to apply before any transformations or field lookups. 
Which can be used to mask or simplify placeholder definitions (for instance
clean up the Apache conditional prefixes) or regex meta characters.

 * The `record` field starts as a static string, but is meant to be turned
   into a regex.
 * Therefore meta characters (such as `|` or `[]`) have to be
   taken care of.  Which is what the `rewrite` map is lazily used for.
 * Better implementations might look up the placeholders, and automatically
   escape the rest of the the "record" format string.


## fields

The core of the global .fmt definitions are the field lists.  Each defines a
static %F placeholder and associaties it with a default field name (id:) and
regex (rx:) or even a grok definition (grok:).

| key |	purpose |
|-----|---------|
| %F |	JSON key: static placeholder string (not a regex itself) |
| id |	field identifier, as specified by the application (internal name) |
| rx |	regex which %F placeholder gets replaced with |
| grok | alternatively to regex, %F might be turned into %PATTERN:id |
| type | "int" and "float" could designate strictly numeric fields |

 * As part of the regex transformation, a `%F` could be turned into
   `(?<id>\S+)` for instance.
 * If there's any unnamed capture group `(…)`, it should be augmented
   into a named capture group - instead of the whole match. (To account
   for implicit wrapping.)
 * The `rx` itself might however specify named subgroups (like request_line
   in Apache logs, itself comprised of _method, _path, _protocol, or the
   datetime made up of tm_wday, tm_year, tm_whatever).
 * `\S+` is also used as fallback for entirely undefined placeholders
   (no expand definition matched) in logfmt1.
 * `grok` isn't currently used, but might allow for simpler transformations
   (indirectly into a grok pattern, and later a regex).


## expand

The expand declarations are used to construct unknown fields/placeholders. 
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.  Thus it simply can be applied before
separator/placeholder are looked at, to augment the known `fields` list.

| key |	purpose |
|-----|---------|
| `%\{(\w+)\}t` | JSON key: a regex to detect mutable placeholders |
| id |	name for newly created fields entry, might use captures $1 |
| rx |	for static definitions (often just \S+)|
| if_quoted |	alternative regex, if placeholder was enclosed in "%\w+" quotes |
| class | recurse into other .fmt types|
| record | can be set to $2 if class: recursion is defined |

 * Typically it suffices to specify the `id` and `rx` field.
 * If no `id` is given, then the regex capture is normalized into
   an identifier (non-alphanumerics stripped, all lowercased).
 * But the `id` or `record` value might be set with regex captures
   (e.g. `$1` or `$2`) or compound values (`"id": "newfield_$1"`).
 * And logfmt1 allows to recurse into other format types per `class`
   (which is used to expand the captured `"record": "$1"` into regex
   tokens).


### alias

Maps alternative/more common field names onto the declared field `id`s.

To get to some state of standardization, the field ids usually refer
to application-internal names. (For instance `log_pfn_register(…,…,cb_id)`
names in Apache). And those aren't always the more commonly used identifiers.

Thus aliases makes sense not just for convenience, but also to be compatible
to other common names (e.g. w3c extend log format names like `cs-time`).


### container

Is utilized by logopen() to extract additional fields (lists even) from one
of the existing fields.  This is usually done at row traversal.  And makes
sense for application-specific subformats in logs.  Such as any `key=value`
lists in the main message field.

> Still not sure if automatic list conversion is a good idea.

### glob

Might be used by log processors to look up a log class, based on file names,
if no .log.fmt is declared.


### "#comment": fields

Documentation entries in the .fmt files have keys starting with `#`. For example
`"#license":` or `"#origin":`. Which is simpler than using JSON with
comments (JSOL).

-----

## Other format files

> This section is about fictional features.


### .grok definitions

> Not implemented yet.

The logfmt/ directory might also contain .grok files, which get transformed
into .fmt structures. (Probably with the grok: parameter for fields, and
a grok: pattern table alongside regular fields:).

There's already a pretransformed `grok.fmt`, which however requires
`%{GROK:%{PATTERN:id}}` references currently.


### .lnav formats

> Not implemented yet.

Likewise could we use lnav .json format definitions. Those are static
too, however.

# logfmt1

**logfmt1** is an implementation and scheme for generic log parsing.  It
prescribes a [.fmt](log.fmt.md) descriptor adjacent to each log file.  And a
global database to resolve format string %placeholder%s and turn them into
regular expressions.

## Overview

* A [.log.fmt](log.fmt.md) for each log file
* [logopen()](logopen.md)
* [update() and regex()](update-regex.md)
* Field [.fmt definitions](fmt.md) in `/usr/share/logfmt`
* [update-logfmt](update-logfmt.md) to create *.log.fmt files (for Apache, Nginx, some static logs)

## Sample .log.fmt

Currently the format for a *.log.fmt descriptor is:

    {
       "class": "apache custom2",
       "record": "%a %u %l [%t] %V \"%r\" %O %>s %D %{SSL_CIPHER}e %M"
    }

Might still change, of course. "record" might be better named "formatstring"
for example.

## Usage

In the simplest of cases you can use [logopen](logopen.md) to process
a log file (and its adjacent .fmt descriptor) at once:

    log = logfmt1.logopen("/var/log/apache2/access.log")
    for row in log:
        print(row["request_line"])
    #print(log.names())

## About

Project originated as part of [modseccfg](https://fossil.include-once.org/modseccfg/).

## A .log.fmt for each log file

Log parsing is a curse, because each application has its own format, and
oftentimes configurable fields at that.  Various attempts at standardizing
logs have failed, or are bound to.  Logging services and database storage
are largely just symptomatic kludges, with JSON logs and not-quite-JSON
formats held back by inertia.

Instead logfmt1 aims to have descriptors for each log file, in order to
make them parseable. You can't attempt anything but guesswork until you
know what's in a file.

So the idea is to have a `*.fmt` next to each `*.log` file, with a
descriptor such as:

    {
       "class": "apache combined",
       "record": "%h %l %u %t \"%r\" %>s %b"
    }

Notably the "record" field should be the most current format string that
the application itself uses. In order to resolve the placeholders, an
application reference is kept in "class". Which allows combining the
format string with placeholder field definitions from the global
[.fmt database (`/usr/share/logfmt`)](fmt.md) database.


### common classes

If the *.log itself is just JSON, then the .log.fmt would specify it
simply as:

    {
       "class": "json appmoniker"
    }

For widely-known formats such as the "combined" Apache logs, the class
might also suffice. But you can't rely on the user not having modified
the LogFormat within the Apache settings then.


### additional fields

The *.log.fmt itself might declare definitions such as aliases and
more specific/custom placeholders.

    {
       "class": "apache cust3",
       "record": "%a %h %{iso}t '%r' %s",
       "fields": {
           "%{iso}t": { "id": "datetime", "rx": "..." }
       },
       "alias": {
           "iso8601": "datetime",
       }
    }

Which ought to be joined and override any global [fmt](fmt) definitions.
Though such user customizations are more likely to be applied there anyway.
Care should be taken by `update-logfmt` or applications to not jettison
user-customized *.log.fmt options.

### logex

`logex` is a little command line tool around [`logopen()`](logopen.md).
It basically allows fetching individual fields from a log, and/or
restructuring it into a specific format (e.g. tab or comma-delimited).


    logex access.log @request_uri @datetime @user_agent

Where the first param is always the log file, and then an arbitrary
number of field names - either individually or as part of a string
argument:

    logex access.log --tab "uri,date,status" user_agent

(Would delimit the first three per comma, the last with a tab.)


## Flags

| param | desc |
|-------|------|
| --json | output each row as JSON |
| --csv  | join all fields with comma |
| --tab  | tab-separated output |
| --iso8601 | fix any datetime fields |
| --regex | just output regex for log |
| --debug | debug_rx in case the regex fails |


## Pefixes

The argument list allows field name prefixes.  Which are only relevant to
container-fields (expanded key:value pairs for some application formats), in
case they're holding lists.

| prefix | output |
| -------| ------ |
| @name  | will just show the first entry |
| %name  | space-separated list |
| *name  | comma-separated list |
| +name  | plus-joined list |
| #name  | as json array |
| name   | whatever |

### logopen()

::: logfmt1.logopen
    handler: python
    selection:
      members:
        - __init__
        - names
        - debug_rx

## update-logfmt

The `update-logfmt` script (should be in in /usr/bin/ when installed via
system package) will create [.log.fmt](log.fmt.md) files for known system
logs.

It's basically just a wrapper script that invokes handlers in
`/usr/share/logfmt/update/*`. Each of which scans one application
config to locate and decorate its according log files.


### args

| param | desc |
|-------|------|
|--test |print script names which would run, but don't run them.|
|--verbose|print script names before running them.|


## scripts

Currently just supports:

 * Apache (access, global, error and transfer logs, no forensic logs)
 * Nginx (only accesss logs)

More static system logs (klog/syslog) might be supported in the next
version.

[logopen()](logopen) does this internally, but you might want to manually
craft the `"regex":` for a .fmt descriptor.  In such cases,
you can use `regex()` or `update()` to combine the log.fmt with definitions
from the global fmt database.


### regex()

Takes a .fmt json/dict, and generates the regex with named capture groups
from it.

::: logfmt1.regex


----


### rx2re()

::: logfmt1.rx2re


----

### update()

Updates the `regex:` property in a .fmt dict. Also injects any used
`fields:` or `alias:` lists. (For now this is basically just a wrapper
of the `regex()` builder.)

::: logfmt1.update

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        
        
        
        <link rel="shortcut icon" href="/img/favicon.ico">
        <title>logfmt1</title>
        <link href="/css/bootstrap.min.css" rel="stylesheet">
        <link href="/css/font-awesome.min.css" rel="stylesheet">
        <link href="/css/base.css" rel="stylesheet">
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css">
        <link href="/custom.css" rel="stylesheet">

        <script src="/js/jquery-1.10.2.min.js" defer></script>
        <script src="/js/bootstrap.min.js" defer></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
        <script>hljs.initHighlightingOnLoad();</script> 
    </head>

    <body>
        <div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
            <div class="container">
                <a class="navbar-brand" href="/.">logfmt1</a>
                <!-- Expander button -->
                <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbar-collapse">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <!-- Expanded navigation -->
                <div id="navbar-collapse" class="navbar-collapse collapse">
                        <!-- Main navigation -->
                        <ul class="nav navbar-nav">
                            <li class="navitem">
                                <a href="/index.html" class="nav-link">Home</a>
                            </li>
                            <li class="navitem">
                                <a href="/log.fmt.html" class="nav-link">.log.fmt</a>
                            </li>
                            <li class="navitem">
                                <a href="/logopen.html" class="nav-link">logopen</a>
                            </li>
                            <li class="navitem">
                                <a href="/update-regex.html" class="nav-link">regex</a>
                            </li>
                            <li class="navitem">
                                <a href="/fmt.html" class="nav-link">global .fmt db</a>
                            </li>
                            <li class="navitem">
                                <a href="/update-logfmt.html" class="nav-link">update-logmt</a>
                            </li>
                            <li class="navitem">
                                <a href="/logex.html" class="nav-link">logex</a>
                            </li>
                        </ul>

                    <ul class="nav navbar-nav ml-auto">
                            <li class="nav-item">
                                <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1" class="nav-link">Fossil</a>
                            </li>
                    </ul>
                </div>
            </div>
        </div>

        <div class="container">
            <div class="row">

    <div class="row-fluid">
      <div id="main-content" class="span12">
        <h1 id="404-page-not-found" style="text-align: center">404</h1>
        <p style="text-align: center"><strong>Page not found</strong></p>
      </div>
    </div>


            </div>
        </div>

        <footer class="col-md-12">
            <hr>
            <p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
        </footer>
        <script>
            var base_url = "/",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
        <script src="/js/base.js" defer></script>

        <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
    <div class="modal-dialog">
        <div class="modal-content">
            <div class="modal-header">
                <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
            </div>
            <div class="modal-body">
              <table class="table">
                <thead>
                  <tr>
                    <th style="width: 20%;">Keys</th>
                    <th>Action</th>
                  </tr>
                </thead>
                <tbody>
                  <tr>
                    <td class="help shortcut"><kbd>?</kbd></td>
                    <td>Open this help</td>
                  </tr>
                  <tr>
                    <td class="next shortcut"><kbd>n</kbd></td>
                    <td>Next page</td>
                  </tr>
                  <tr>
                    <td class="prev shortcut"><kbd>p</kbd></td>
                    <td>Previous page</td>
                  </tr>
                  <tr>
                    <td class="search shortcut"><kbd>s</kbd></td>
                    <td>Search</td>
                  </tr>
                </tbody>
              </table>
            </div>
            <div class="modal-footer">
            </div>
        </div>
    </div>
</div>

    </body>
</html>

html {
    /* The nav header is 5.01875rem high, plus 20px for the margin-top of the
       main container. */
    scroll-padding-top: calc(5.01875rem + 20px);
}

body > .container {
    margin-top: 20px;
}

.navbar.fixed-top {
    position: -webkit-sticky;
    position: sticky;
}

.source-links {
    float: right;
}

.col-md-9 img {
    max-width: 100%;
}

pre, code {
    color: #000;
    border: none;
    border-radius: 0.25rem;
    box-shadow: 0 1px 4px rgba(0,0,0,0.4);
}

pre {
    padding: 1em;
}

code {
    padding: 3px;
}

pre code {
    box-shadow: none;
    /* Override styles from hljs theme */
    background: transparent !important;
    padding: 0 !important;
}

a code {
    color: #2196f3;
}

a:hover code {
    color: #0a6ebd;
}

kbd {
    padding: 2px 4px;
    font-size: 90%;
    color: #fff;
    background-color: #333;
    border-radius: 3px;
    -webkit-box-shadow: inset 0 -1px 0 rgba(0,0,0,.25);
    box-shadow: inset 0 -1px 0 rgba(0,0,0,.25);
}

footer {
    margin-top: 30px;
    margin-bottom: 10px;
    text-align: center;
    font-weight: 200;
}

.modal-dialog {
    margin-top: 60px;
}

/* Style the admonitions. */

.admonition {
    margin-bottom: 1rem;
    border-radius: 0.25rem;
    box-shadow: 0 1px 4px rgba(0,0,0,0.4);
}

.admonition > .admonition-title {
    margin: 0;
    padding: 0.75rem 1rem;
    background: rgba(0,0,0,0.03);
    border-radius: calc(0.25rem - 1px) calc(0.25rem - 1px) 0 0;
    font-size: 1rem;
}

.admonition > * {
    margin-left: 1rem;
    margin-right: 1rem;
}

.admonition > *:not(.admonition-title):first-of-type,
.admonition > .admonition-title + * {
    margin-top: 1rem;
}

.admonition > *:last-child {
    padding-bottom: 1rem;
}

/* Style each kind of admonition. */

.admonition.note > .admonition-title {
    background: #9c27b0;
    color: #fff;
}

.admonition.warning > .admonition-title {
    background: #ff9800;
    color: #fff;
}

.admonition.danger > .admonition-title {
    background: #de1b22;
    color: #fff;
}

/*
 * Side navigation
 *
 * Scrollspy and affixed enhanced navigation to highlight sections and secondary
 * sections of docs content.
 */

.bs-sidebar.affix {
    position: -webkit-sticky;
    position: sticky;
    /* The nav header is 5.01875rem high, plus 20px for the margin-top of the
       main container. */
    top: calc(5.01875rem + 20px);
}

.bs-sidebar.card {
    padding: 0;
}

/* Toggle (vertically flip) sidebar collapse icon */
.bs-sidebar .navbar-toggler span {
    -moz-transform: scale(1, -1);
    -webkit-transform: scale(1, -1);
    -o-transform: scale(1, -1);
    -ms-transform: scale(1, -1);
    transform: scale(1, -1);
}

.bs-sidebar .navbar-toggler.collapsed span {
    -moz-transform: scale(1, 1);
    -webkit-transform: scale(1, 1);
    -o-transform: scale(1, 1);
    -ms-transform: scale(1, 1);
    transform: scale(1, 1);
}

/* First level of nav */
.bs-sidebar > .navbar-collapse > .nav {
    padding-top:    10px;
    padding-bottom: 10px;
    border-radius: 5px;
    width: 100%;
}

/* All levels of nav */
.bs-sidebar .nav > li > a {
    display: block;
    padding: 5px 20px;
    z-index: 1;
}
.bs-sidebar .nav > li > a:hover,
.bs-sidebar .nav > li > a:focus {
    text-decoration: none;
    border-right: 1px solid;
}
.bs-sidebar .nav > li > a.active,
.bs-sidebar .nav > li > a.active:hover,
.bs-sidebar .nav > li > a.active:focus {
    font-weight: bold;
    background-color: transparent;
    border-right: 1px solid;
}

.bs-sidebar .nav .nav .nav {
    margin-left: 1em;
}

.bs-sidebar .nav > li > a {
    font-weight: bold;
}

.bs-sidebar .nav .nav > li > a {
    font-weight: normal;
}

.headerlink {
    display: none;
    padding-left: .5em;
}

h1:hover .headerlink, h2:hover .headerlink, h3:hover .headerlink, h4:hover .headerlink, h5:hover .headerlink, h6:hover .headerlink{
    display:inline-block;
}

@media (max-width: 991.98px) {
    .navbar-collapse.show {
        overflow-y: auto;
        max-height: calc(100vh - 5.01875rem);
    }
}

.dropdown-item.open {
    color: #141414;
    background: #f8f9fa -webkit-gradient(linear, left top, left bottom,
                                         from(#f9fafb), to(#f8f9fa)) repeat-x;
    background: #f8f9fa linear-gradient(180deg, #f9fafb, #f8f9fa) repeat-x;
}

.dropdown-submenu > .dropdown-menu {
    margin: 0 0 0 1.5rem;
    padding: 0;
    border-width: 0;
}

.dropdown-submenu > a::after {
    display: block;
    content: " ";
    float: right;
    width: 0;
    height: 0;
    border-color: transparent;
    border-style: solid;
    border-width: 5px 0 5px 5px;
    border-left-color: #ccc;
    margin-top: 5px;
    margin-right: -10px;
}

.dropdown-submenu:hover > a::after {
    border-left-color: #404040;
}

@media (min-width: 992px) {
    .dropdown-menu {
        overflow-y: auto;
        max-height: calc(100vh - 5.01875rem);
    }

    .dropdown-submenu {
        position: relative;
    }

    .dropdown-submenu > .dropdown-menu {
        position: fixed !important;
        margin-top: -9px;
        margin-left: -2px;
        border-width: 1px;
        padding: 0.5rem 0;
    }
}

/*!
 *  Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome
 *  License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)
 */@font-face{font-family:'FontAwesome';src:url('../fonts/fontawesome-webfont.eot?v=4.7.0');src:url('../fonts/fontawesome-webfont.eot?#iefix&v=4.7.0') format('embedded-opentype'),url('../fonts/fontawesome-webfont.woff2?v=4.7.0') format('woff2'),url('../fonts/fontawesome-webfont.woff?v=4.7.0') format('woff'),url('../fonts/fontawesome-webfont.ttf?v=4.7.0') format('truetype'),url('../fonts/fontawesome-webfont.svg?v=4.7.0#fontawesomeregular') format('svg');font-weight:normal;font-style:normal}.fa{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571429em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14285714em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14285714em;width:2.14285714em;top:.14285714em;text-align:center}.fa-li.fa-lg{left:-1.85714286em}.fa-border{padding:.2em .25em .15em;border:solid .08em #eee;border-radius:.1em}.fa-pull-left{float:left}.fa-pull-right{float:right}.fa.fa-pull-left{margin-right:.3em}.fa.fa-pull-right{margin-left:.3em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left{margin-right:.3em}.fa.pull-right{margin-left:.3em}.fa-spin{-webkit-animation:fa-spin 2s infinite linear;animation:fa-spin 2s infinite linear}.fa-pulse{-webkit-animation:fa-spin 1s infinite steps(8);animation:fa-spin 1s infinite steps(8)}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";-webkit-transform:scale(-1, 1);-ms-transform:scale(-1, 1);transform:scale(-1, 1)}.fa-flip-vertical{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";-webkit-transform:scale(1, -1);-ms-transform:scale(1, -1);transform:scale(1, -1)}:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270,:root .fa-flip-horizontal,:root .fa-flip-vertical{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:"\f000"}.fa-music:before{content:"\f001"}.fa-search:before{content:"\f002"}.fa-envelope-o:before{content:"\f003"}.fa-heart:before{content:"\f004"}.fa-star:before{content:"\f005"}.fa-star-o:before{content:"\f006"}.fa-user:before{content:"\f007"}.fa-film:before{content:"\f008"}.fa-th-large:before{content:"\f009"}.fa-th:before{content:"\f00a"}.fa-th-list:before{content:"\f00b"}.fa-check:before{content:"\f00c"}.fa-remove:before,.fa-close:before,.fa-times:before{content:"\f00d"}.fa-search-plus:before{content:"\f00e"}.fa-search-minus:before{content:"\f010"}.fa-power-off:before{content:"\f011"}.fa-signal:before{content:"\f012"}.fa-gear:before,.fa-cog:before{content:"\f013"}.fa-trash-o:before{content:"\f014"}.fa-home:before{content:"\f015"}.fa-file-o:before{content:"\f016"}.fa-clock-o:before{content:"\f017"}.fa-road:before{content:"\f018"}.fa-download:before{content:"\f019"}.fa-arrow-circle-o-down:before{content:"\f01a"}.fa-arrow-circle-o-up:before{content:"\f01b"}.fa-inbox:before{content:"\f01c"}.fa-play-circle-o:before{content:"\f01d"}.fa-rotate-right:before,.fa-repeat:before{content:"\f01e"}.fa-refresh:before{content:"\f021"}.fa-list-alt:before{content:"\f022"}.fa-lock:before{content:"\f023"}.fa-flag:before{content:"\f024"}.fa-headphones:before{content:"\f025"}.fa-volume-off:before{content:"\f026"}.fa-volume-down:before{content:"\f027"}.fa-volume-up:before{content:"\f028"}.fa-qrcode:before{content:"\f029"}.fa-barcode:before{content:"\f02a"}.fa-tag:before{content:"\f02b"}.fa-tags:before{content:"\f02c"}.fa-book:before{content:"\f02d"}.fa-bookmark:before{content:"\f02e"}.fa-print:before{content:"\f02f"}.fa-camera:before{content:"\f030"}.fa-font:before{content:"\f031"}.fa-bold:before{content:"\f032"}.fa-italic:before{content:"\f033"}.fa-text-height:before{content:"\f034"}.fa-text-width:before{content:"\f035"}.fa-align-left:before{content:"\f036"}.fa-align-center:before{content:"\f037"}.fa-align-right:before{content:"\f038"}.fa-align-justify:before{content:"\f039"}.fa-list:before{content:"\f03a"}.fa-dedent:before,.fa-outdent:before{content:"\f03b"}.fa-indent:before{content:"\f03c"}.fa-video-camera:before{content:"\f03d"}.fa-photo:before,.fa-image:before,.fa-picture-o:before{content:"\f03e"}.fa-pencil:before{content:"\f040"}.fa-map-marker:before{content:"\f041"}.fa-adjust:before{content:"\f042"}.fa-tint:before{content:"\f043"}.fa-edit:before,.fa-pencil-square-o:before{content:"\f044"}.fa-share-square-o:before{content:"\f045"}.fa-check-square-o:before{content:"\f046"}.fa-arrows:before{content:"\f047"}.fa-step-backward:before{content:"\f048"}.fa-fast-backward:before{content:"\f049"}.fa-backward:before{content:"\f04a"}.fa-play:before{content:"\f04b"}.fa-pause:before{content:"\f04c"}.fa-stop:before{content:"\f04d"}.fa-forward:before{content:"\f04e"}.fa-fast-forward:before{content:"\f050"}.fa-step-forward:before{content:"\f051"}.fa-eject:before{content:"\f052"}.fa-chevron-left:before{content:"\f053"}.fa-chevron-right:before{content:"\f054"}.fa-plus-circle:before{content:"\f055"}.fa-minus-circle:before{content:"\f056"}.fa-times-circle:before{content:"\f057"}.fa-check-circle:before{content:"\f058"}.fa-question-circle:before{content:"\f059"}.fa-info-circle:before{content:"\f05a"}.fa-crosshairs:before{content:"\f05b"}.fa-times-circle-o:before{content:"\f05c"}.fa-check-circle-o:before{content:"\f05d"}.fa-ban:before{content:"\f05e"}.fa-arrow-left:before{content:"\f060"}.fa-arrow-right:before{content:"\f061"}.fa-arrow-up:before{content:"\f062"}.fa-arrow-down:before{content:"\f063"}.fa-mail-forward:before,.fa-share:before{content:"\f064"}.fa-expand:before{content:"\f065"}.fa-compress:before{content:"\f066"}.fa-plus:before{content:"\f067"}.fa-minus:before{content:"\f068"}.fa-asterisk:before{content:"\f069"}.fa-exclamation-circle:before{content:"\f06a"}.fa-gift:before{content:"\f06b"}.fa-leaf:before{content:"\f06c"}.fa-fire:before{content:"\f06d"}.fa-eye:before{content:"\f06e"}.fa-eye-slash:before{content:"\f070"}.fa-warning:before,.fa-exclamation-triangle:before{content:"\f071"}.fa-plane:before{content:"\f072"}.fa-calendar:before{content:"\f073"}.fa-random:before{content:"\f074"}.fa-comment:before{content:"\f075"}.fa-magnet:before{content:"\f076"}.fa-chevron-up:before{content:"\f077"}.fa-chevron-down:before{content:"\f078"}.fa-retweet:before{content:"\f079"}.fa-shopping-cart:before{content:"\f07a"}.fa-folder:before{content:"\f07b"}.fa-folder-open:before{content:"\f07c"}.fa-arrows-v:before{content:"\f07d"}.fa-arrows-h:before{content:"\f07e"}.fa-bar-chart-o:before,.fa-bar-chart:before{content:"\f080"}.fa-twitter-square:before{content:"\f081"}.fa-facebook-square:before{content:"\f082"}.fa-camera-retro:before{content:"\f083"}.fa-key:before{content:"\f084"}.fa-gears:before,.fa-cogs:before{content:"\f085"}.fa-comments:before{content:"\f086"}.fa-thumbs-o-up:before{content:"\f087"}.fa-thumbs-o-down:before{content:"\f088"}.fa-star-half:before{content:"\f089"}.fa-heart-o:before{content:"\f08a"}.fa-sign-out:before{content:"\f08b"}.fa-linkedin-square:before{content:"\f08c"}.fa-thumb-tack:before{content:"\f08d"}.fa-external-link:before{content:"\f08e"}.fa-sign-in:before{content:"\f090"}.fa-trophy:before{content:"\f091"}.fa-github-square:before{content:"\f092"}.fa-upload:before{content:"\f093"}.fa-lemon-o:before{content:"\f094"}.fa-phone:before{content:"\f095"}.fa-square-o:before{content:"\f096"}.fa-bookmark-o:before{content:"\f097"}.fa-phone-square:before{content:"\f098"}.fa-twitter:before{content:"\f099"}.fa-facebook-f:before,.fa-facebook:before{content:"\f09a"}.fa-github:before{content:"\f09b"}.fa-unlock:before{content:"\f09c"}.fa-credit-card:before{content:"\f09d"}.fa-feed:before,.fa-rss:before{content:"\f09e"}.fa-hdd-o:before{content:"\f0a0"}.fa-bullhorn:before{content:"\f0a1"}.fa-bell:before{content:"\f0f3"}.fa-certificate:before{content:"\f0a3"}.fa-hand-o-right:before{content:"\f0a4"}.fa-hand-o-left:before{content:"\f0a5"}.fa-hand-o-up:before{content:"\f0a6"}.fa-hand-o-down:before{content:"\f0a7"}.fa-arrow-circle-left:before{content:"\f0a8"}.fa-arrow-circle-right:before{content:"\f0a9"}.fa-arrow-circle-up:before{content:"\f0aa"}.fa-arrow-circle-down:before{content:"\f0ab"}.fa-globe:before{content:"\f0ac"}.fa-wrench:before{content:"\f0ad"}.fa-tasks:before{content:"\f0ae"}.fa-filter:before{content:"\f0b0"}.fa-briefcase:before{content:"\f0b1"}.fa-arrows-alt:before{content:"\f0b2"}.fa-group:before,.fa-users:before{content:"\f0c0"}.fa-chain:before,.fa-link:before{content:"\f0c1"}.fa-cloud:before{content:"\f0c2"}.fa-flask:before{content:"\f0c3"}.fa-cut:before,.fa-scissors:before{content:"\f0c4"}.fa-copy:before,.fa-files-o:before{content:"\f0c5"}.fa-paperclip:before{content:"\f0c6"}.fa-save:before,.fa-floppy-o:before{content:"\f0c7"}.fa-square:before{content:"\f0c8"}.fa-navicon:before,.fa-reorder:before,.fa-bars:before{content:"\f0c9"}.fa-list-ul:before{content:"\f0ca"}.fa-list-ol:before{content:"\f0cb"}.fa-strikethrough:before{content:"\f0cc"}.fa-underline:before{content:"\f0cd"}.fa-table:before{content:"\f0ce"}.fa-magic:before{content:"\f0d0"}.fa-truck:before{content:"\f0d1"}.fa-pinterest:before{content:"\f0d2"}.fa-pinterest-square:before{content:"\f0d3"}.fa-google-plus-square:before{content:"\f0d4"}.fa-google-plus:before{content:"\f0d5"}.fa-money:before{content:"\f0d6"}.fa-caret-down:before{content:"\f0d7"}.fa-caret-up:before{content:"\f0d8"}.fa-caret-left:before{content:"\f0d9"}.fa-caret-right:before{content:"\f0da"}.fa-columns:before{content:"\f0db"}.fa-unsorted:before,.fa-sort:before{content:"\f0dc"}.fa-sort-down:before,.fa-sort-desc:before{content:"\f0dd"}.fa-sort-up:before,.fa-sort-asc:before{content:"\f0de"}.fa-envelope:before{content:"\f0e0"}.fa-linkedin:before{content:"\f0e1"}.fa-rotate-left:before,.fa-undo:before{content:"\f0e2"}.fa-legal:before,.fa-gavel:before{content:"\f0e3"}.fa-dashboard:before,.fa-tachometer:before{content:"\f0e4"}.fa-comment-o:before{content:"\f0e5"}.fa-comments-o:before{content:"\f0e6"}.fa-flash:before,.fa-bolt:before{content:"\f0e7"}.fa-sitemap:before{content:"\f0e8"}.fa-umbrella:before{content:"\f0e9"}.fa-paste:before,.fa-clipboard:before{content:"\f0ea"}.fa-lightbulb-o:before{content:"\f0eb"}.fa-exchange:before{content:"\f0ec"}.fa-cloud-download:before{content:"\f0ed"}.fa-cloud-upload:before{content:"\f0ee"}.fa-user-md:before{content:"\f0f0"}.fa-stethoscope:before{content:"\f0f1"}.fa-suitcase:before{content:"\f0f2"}.fa-bell-o:before{content:"\f0a2"}.fa-coffee:before{content:"\f0f4"}.fa-cutlery:before{content:"\f0f5"}.fa-file-text-o:before{content:"\f0f6"}.fa-building-o:before{content:"\f0f7"}.fa-hospital-o:before{content:"\f0f8"}.fa-ambulance:before{content:"\f0f9"}.fa-medkit:before{content:"\f0fa"}.fa-fighter-jet:before{content:"\f0fb"}.fa-beer:before{content:"\f0fc"}.fa-h-square:before{content:"\f0fd"}.fa-plus-square:before{content:"\f0fe"}.fa-angle-double-left:before{content:"\f100"}.fa-angle-double-right:before{content:"\f101"}.fa-angle-double-up:before{content:"\f102"}.fa-angle-double-down:before{content:"\f103"}.fa-angle-left:before{content:"\f104"}.fa-angle-right:before{content:"\f105"}.fa-angle-up:before{content:"\f106"}.fa-angle-down:before{content:"\f107"}.fa-desktop:before{content:"\f108"}.fa-laptop:before{content:"\f109"}.fa-tablet:before{content:"\f10a"}.fa-mobile-phone:before,.fa-mobile:before{content:"\f10b"}.fa-circle-o:before{content:"\f10c"}.fa-quote-left:before{content:"\f10d"}.fa-quote-right:before{content:"\f10e"}.fa-spinner:before{content:"\f110"}.fa-circle:before{content:"\f111"}.fa-mail-reply:before,.fa-reply:before{content:"\f112"}.fa-github-alt:before{content:"\f113"}.fa-folder-o:before{content:"\f114"}.fa-folder-open-o:before{content:"\f115"}.fa-smile-o:before{content:"\f118"}.fa-frown-o:before{content:"\f119"}.fa-meh-o:before{content:"\f11a"}.fa-gamepad:before{content:"\f11b"}.fa-keyboard-o:before{content:"\f11c"}.fa-flag-o:before{content:"\f11d"}.fa-flag-checkered:before{content:"\f11e"}.fa-terminal:before{content:"\f120"}.fa-code:before{content:"\f121"}.fa-mail-reply-all:before,.fa-reply-all:before{content:"\f122"}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:"\f123"}.fa-location-arrow:before{content:"\f124"}.fa-crop:before{content:"\f125"}.fa-code-fork:before{content:"\f126"}.fa-unlink:before,.fa-chain-broken:before{content:"\f127"}.fa-question:before{content:"\f128"}.fa-info:before{content:"\f129"}.fa-exclamation:before{content:"\f12a"}.fa-superscript:before{content:"\f12b"}.fa-subscript:before{content:"\f12c"}.fa-eraser:before{content:"\f12d"}.fa-puzzle-piece:before{content:"\f12e"}.fa-microphone:before{content:"\f130"}.fa-microphone-slash:before{content:"\f131"}.fa-shield:before{content:"\f132"}.fa-calendar-o:before{content:"\f133"}.fa-fire-extinguisher:before{content:"\f134"}.fa-rocket:before{content:"\f135"}.fa-maxcdn:before{content:"\f136"}.fa-chevron-circle-left:before{content:"\f137"}.fa-chevron-circle-right:before{content:"\f138"}.fa-chevron-circle-up:before{content:"\f139"}.fa-chevron-circle-down:before{content:"\f13a"}.fa-html5:before{content:"\f13b"}.fa-css3:before{content:"\f13c"}.fa-anchor:before{content:"\f13d"}.fa-unlock-alt:before{content:"\f13e"}.fa-bullseye:before{content:"\f140"}.fa-ellipsis-h:before{content:"\f141"}.fa-ellipsis-v:before{content:"\f142"}.fa-rss-square:before{content:"\f143"}.fa-play-circle:before{content:"\f144"}.fa-ticket:before{content:"\f145"}.fa-minus-square:before{content:"\f146"}.fa-minus-square-o:before{content:"\f147"}.fa-level-up:before{content:"\f148"}.fa-level-down:before{content:"\f149"}.fa-check-square:before{content:"\f14a"}.fa-pencil-square:before{content:"\f14b"}.fa-external-link-square:before{content:"\f14c"}.fa-share-square:before{content:"\f14d"}.fa-compass:before{content:"\f14e"}.fa-toggle-down:before,.fa-caret-square-o-down:before{content:"\f150"}.fa-toggle-up:before,.fa-caret-square-o-up:before{content:"\f151"}.fa-toggle-right:before,.fa-caret-square-o-right:before{content:"\f152"}.fa-euro:before,.fa-eur:before{content:"\f153"}.fa-gbp:before{content:"\f154"}.fa-dollar:before,.fa-usd:before{content:"\f155"}.fa-rupee:before,.fa-inr:before{content:"\f156"}.fa-cny:before,.fa-rmb:before,.fa-yen:before,.fa-jpy:before{content:"\f157"}.fa-ruble:before,.fa-rouble:before,.fa-rub:before{content:"\f158"}.fa-won:before,.fa-krw:before{content:"\f159"}.fa-bitcoin:before,.fa-btc:before{content:"\f15a"}.fa-file:before{content:"\f15b"}.fa-file-text:before{content:"\f15c"}.fa-sort-alpha-asc:before{content:"\f15d"}.fa-sort-alpha-desc:before{content:"\f15e"}.fa-sort-amount-asc:before{content:"\f160"}.fa-sort-amount-desc:before{content:"\f161"}.fa-sort-numeric-asc:before{content:"\f162"}.fa-sort-numeric-desc:before{content:"\f163"}.fa-thumbs-up:before{content:"\f164"}.fa-thumbs-down:before{content:"\f165"}.fa-youtube-square:before{content:"\f166"}.fa-youtube:before{content:"\f167"}.fa-xing:before{content:"\f168"}.fa-xing-square:before{content:"\f169"}.fa-youtube-play:before{content:"\f16a"}.fa-dropbox:before{content:"\f16b"}.fa-stack-overflow:before{content:"\f16c"}.fa-instagram:before{content:"\f16d"}.fa-flickr:before{content:"\f16e"}.fa-adn:before{content:"\f170"}.fa-bitbucket:before{content:"\f171"}.fa-bitbucket-square:before{content:"\f172"}.fa-tumblr:before{content:"\f173"}.fa-tumblr-square:before{content:"\f174"}.fa-long-arrow-down:before{content:"\f175"}.fa-long-arrow-up:before{content:"\f176"}.fa-long-arrow-left:before{content:"\f177"}.fa-long-arrow-right:before{content:"\f178"}.fa-apple:before{content:"\f179"}.fa-windows:before{content:"\f17a"}.fa-android:before{content:"\f17b"}.fa-linux:before{content:"\f17c"}.fa-dribbble:before{content:"\f17d"}.fa-skype:before{content:"\f17e"}.fa-foursquare:before{content:"\f180"}.fa-trello:before{content:"\f181"}.fa-female:before{content:"\f182"}.fa-male:before{content:"\f183"}.fa-gittip:before,.fa-gratipay:before{content:"\f184"}.fa-sun-o:before{content:"\f185"}.fa-moon-o:before{content:"\f186"}.fa-archive:before{content:"\f187"}.fa-bug:before{content:"\f188"}.fa-vk:before{content:"\f189"}.fa-weibo:before{content:"\f18a"}.fa-renren:before{content:"\f18b"}.fa-pagelines:before{content:"\f18c"}.fa-stack-exchange:before{content:"\f18d"}.fa-arrow-circle-o-right:before{content:"\f18e"}.fa-arrow-circle-o-left:before{content:"\f190"}.fa-toggle-left:before,.fa-caret-square-o-left:before{content:"\f191"}.fa-dot-circle-o:before{content:"\f192"}.fa-wheelchair:before{content:"\f193"}.fa-vimeo-square:before{content:"\f194"}.fa-turkish-lira:before,.fa-try:before{content:"\f195"}.fa-plus-square-o:before{content:"\f196"}.fa-space-shuttle:before{content:"\f197"}.fa-slack:before{content:"\f198"}.fa-envelope-square:before{content:"\f199"}.fa-wordpress:before{content:"\f19a"}.fa-openid:before{content:"\f19b"}.fa-institution:before,.fa-bank:before,.fa-university:before{content:"\f19c"}.fa-mortar-board:before,.fa-graduation-cap:before{content:"\f19d"}.fa-yahoo:before{content:"\f19e"}.fa-google:before{content:"\f1a0"}.fa-reddit:before{content:"\f1a1"}.fa-reddit-square:before{content:"\f1a2"}.fa-stumbleupon-circle:before{content:"\f1a3"}.fa-stumbleupon:before{content:"\f1a4"}.fa-delicious:before{content:"\f1a5"}.fa-digg:before{content:"\f1a6"}.fa-pied-piper-pp:before{content:"\f1a7"}.fa-pied-piper-alt:before{content:"\f1a8"}.fa-drupal:before{content:"\f1a9"}.fa-joomla:before{content:"\f1aa"}.fa-language:before{content:"\f1ab"}.fa-fax:before{content:"\f1ac"}.fa-building:before{content:"\f1ad"}.fa-child:before{content:"\f1ae"}.fa-paw:before{content:"\f1b0"}.fa-spoon:before{content:"\f1b1"}.fa-cube:before{content:"\f1b2"}.fa-cubes:before{content:"\f1b3"}.fa-behance:before{content:"\f1b4"}.fa-behance-square:before{content:"\f1b5"}.fa-steam:before{content:"\f1b6"}.fa-steam-square:before{content:"\f1b7"}.fa-recycle:before{content:"\f1b8"}.fa-automobile:before,.fa-car:before{content:"\f1b9"}.fa-cab:before,.fa-taxi:before{content:"\f1ba"}.fa-tree:before{content:"\f1bb"}.fa-spotify:before{content:"\f1bc"}.fa-deviantart:before{content:"\f1bd"}.fa-soundcloud:before{content:"\f1be"}.fa-database:before{content:"\f1c0"}.fa-file-pdf-o:before{content:"\f1c1"}.fa-file-word-o:before{content:"\f1c2"}.fa-file-excel-o:before{content:"\f1c3"}.fa-file-powerpoint-o:before{content:"\f1c4"}.fa-file-photo-o:before,.fa-file-picture-o:before,.fa-file-image-o:before{content:"\f1c5"}.fa-file-zip-o:before,.fa-file-archive-o:before{content:"\f1c6"}.fa-file-sound-o:before,.fa-file-audio-o:before{content:"\f1c7"}.fa-file-movie-o:before,.fa-file-video-o:before{content:"\f1c8"}.fa-file-code-o:before{content:"\f1c9"}.fa-vine:before{content:"\f1ca"}.fa-codepen:before{content:"\f1cb"}.fa-jsfiddle:before{content:"\f1cc"}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-saver:before,.fa-support:before,.fa-life-ring:before{content:"\f1cd"}.fa-circle-o-notch:before{content:"\f1ce"}.fa-ra:before,.fa-resistance:before,.fa-rebel:before{content:"\f1d0"}.fa-ge:before,.fa-empire:before{content:"\f1d1"}.fa-git-square:before{content:"\f1d2"}.fa-git:before{content:"\f1d3"}.fa-y-combinator-square:before,.fa-yc-square:before,.fa-hacker-news:before{content:"\f1d4"}.fa-tencent-weibo:before{content:"\f1d5"}.fa-qq:before{content:"\f1d6"}.fa-wechat:before,.fa-weixin:before{content:"\f1d7"}.fa-send:before,.fa-paper-plane:before{content:"\f1d8"}.fa-send-o:before,.fa-paper-plane-o:before{content:"\f1d9"}.fa-history:before{content:"\f1da"}.fa-circle-thin:before{content:"\f1db"}.fa-header:before{content:"\f1dc"}.fa-paragraph:before{content:"\f1dd"}.fa-sliders:before{content:"\f1de"}.fa-share-alt:before{content:"\f1e0"}.fa-share-alt-square:before{content:"\f1e1"}.fa-bomb:before{content:"\f1e2"}.fa-soccer-ball-o:before,.fa-futbol-o:before{content:"\f1e3"}.fa-tty:before{content:"\f1e4"}.fa-binoculars:before{content:"\f1e5"}.fa-plug:before{content:"\f1e6"}.fa-slideshare:before{content:"\f1e7"}.fa-twitch:before{content:"\f1e8"}.fa-yelp:before{content:"\f1e9"}.fa-newspaper-o:before{content:"\f1ea"}.fa-wifi:before{content:"\f1eb"}.fa-calculator:before{content:"\f1ec"}.fa-paypal:before{content:"\f1ed"}.fa-google-wallet:before{content:"\f1ee"}.fa-cc-visa:before{content:"\f1f0"}.fa-cc-mastercard:before{content:"\f1f1"}.fa-cc-discover:before{content:"\f1f2"}.fa-cc-amex:before{content:"\f1f3"}.fa-cc-paypal:before{content:"\f1f4"}.fa-cc-stripe:before{content:"\f1f5"}.fa-bell-slash:before{content:"\f1f6"}.fa-bell-slash-o:before{content:"\f1f7"}.fa-trash:before{content:"\f1f8"}.fa-copyright:before{content:"\f1f9"}.fa-at:before{content:"\f1fa"}.fa-eyedropper:before{content:"\f1fb"}.fa-paint-brush:before{content:"\f1fc"}.fa-birthday-cake:before{content:"\f1fd"}.fa-area-chart:before{content:"\f1fe"}.fa-pie-chart:before{content:"\f200"}.fa-line-chart:before{content:"\f201"}.fa-lastfm:before{content:"\f202"}.fa-lastfm-square:before{content:"\f203"}.fa-toggle-off:before{content:"\f204"}.fa-toggle-on:before{content:"\f205"}.fa-bicycle:before{content:"\f206"}.fa-bus:before{content:"\f207"}.fa-ioxhost:before{content:"\f208"}.fa-angellist:before{content:"\f209"}.fa-cc:before{content:"\f20a"}.fa-shekel:before,.fa-sheqel:before,.fa-ils:before{content:"\f20b"}.fa-meanpath:before{content:"\f20c"}.fa-buysellads:before{content:"\f20d"}.fa-connectdevelop:before{content:"\f20e"}.fa-dashcube:before{content:"\f210"}.fa-forumbee:before{content:"\f211"}.fa-leanpub:before{content:"\f212"}.fa-sellsy:before{content:"\f213"}.fa-shirtsinbulk:before{content:"\f214"}.fa-simplybuilt:before{content:"\f215"}.fa-skyatlas:before{content:"\f216"}.fa-cart-plus:before{content:"\f217"}.fa-cart-arrow-down:before{content:"\f218"}.fa-diamond:before{content:"\f219"}.fa-ship:before{content:"\f21a"}.fa-user-secret:before{content:"\f21b"}.fa-motorcycle:before{content:"\f21c"}.fa-street-view:before{content:"\f21d"}.fa-heartbeat:before{content:"\f21e"}.fa-venus:before{content:"\f221"}.fa-mars:before{content:"\f222"}.fa-mercury:before{content:"\f223"}.fa-intersex:before,.fa-transgender:before{content:"\f224"}.fa-transgender-alt:before{content:"\f225"}.fa-venus-double:before{content:"\f226"}.fa-mars-double:before{content:"\f227"}.fa-venus-mars:before{content:"\f228"}.fa-mars-stroke:before{content:"\f229"}.fa-mars-stroke-v:before{content:"\f22a"}.fa-mars-stroke-h:before{content:"\f22b"}.fa-neuter:before{content:"\f22c"}.fa-genderless:before{content:"\f22d"}.fa-facebook-official:before{content:"\f230"}.fa-pinterest-p:before{content:"\f231"}.fa-whatsapp:before{content:"\f232"}.fa-server:before{content:"\f233"}.fa-user-plus:before{content:"\f234"}.fa-user-times:before{content:"\f235"}.fa-hotel:before,.fa-bed:before{content:"\f236"}.fa-viacoin:before{content:"\f237"}.fa-train:before{content:"\f238"}.fa-subway:before{content:"\f239"}.fa-medium:before{content:"\f23a"}.fa-yc:before,.fa-y-combinator:before{content:"\f23b"}.fa-optin-monster:before{content:"\f23c"}.fa-opencart:before{content:"\f23d"}.fa-expeditedssl:before{content:"\f23e"}.fa-battery-4:before,.fa-battery:before,.fa-battery-full:before{content:"\f240"}.fa-battery-3:before,.fa-battery-three-quarters:before{content:"\f241"}.fa-battery-2:before,.fa-battery-half:before{content:"\f242"}.fa-battery-1:before,.fa-battery-quarter:before{content:"\f243"}.fa-battery-0:before,.fa-battery-empty:before{content:"\f244"}.fa-mouse-pointer:before{content:"\f245"}.fa-i-cursor:before{content:"\f246"}.fa-object-group:before{content:"\f247"}.fa-object-ungroup:before{content:"\f248"}.fa-sticky-note:before{content:"\f249"}.fa-sticky-note-o:before{content:"\f24a"}.fa-cc-jcb:before{content:"\f24b"}.fa-cc-diners-club:before{content:"\f24c"}.fa-clone:before{content:"\f24d"}.fa-balance-scale:before{content:"\f24e"}.fa-hourglass-o:before{content:"\f250"}.fa-hourglass-1:before,.fa-hourglass-start:before{content:"\f251"}.fa-hourglass-2:before,.fa-hourglass-half:before{content:"\f252"}.fa-hourglass-3:before,.fa-hourglass-end:before{content:"\f253"}.fa-hourglass:before{content:"\f254"}.fa-hand-grab-o:before,.fa-hand-rock-o:before{content:"\f255"}.fa-hand-stop-o:before,.fa-hand-paper-o:before{content:"\f256"}.fa-hand-scissors-o:before{content:"\f257"}.fa-hand-lizard-o:before{content:"\f258"}.fa-hand-spock-o:before{content:"\f259"}.fa-hand-pointer-o:before{content:"\f25a"}.fa-hand-peace-o:before{content:"\f25b"}.fa-trademark:before{content:"\f25c"}.fa-registered:before{content:"\f25d"}.fa-creative-commons:before{content:"\f25e"}.fa-gg:before{content:"\f260"}.fa-gg-circle:before{content:"\f261"}.fa-tripadvisor:before{content:"\f262"}.fa-odnoklassniki:before{content:"\f263"}.fa-odnoklassniki-square:before{content:"\f264"}.fa-get-pocket:before{content:"\f265"}.fa-wikipedia-w:before{content:"\f266"}.fa-safari:before{content:"\f267"}.fa-chrome:before{content:"\f268"}.fa-firefox:before{content:"\f269"}.fa-opera:before{content:"\f26a"}.fa-internet-explorer:before{content:"\f26b"}.fa-tv:before,.fa-television:before{content:"\f26c"}.fa-contao:before{content:"\f26d"}.fa-500px:before{content:"\f26e"}.fa-amazon:before{content:"\f270"}.fa-calendar-plus-o:before{content:"\f271"}.fa-calendar-minus-o:before{content:"\f272"}.fa-calendar-times-o:before{content:"\f273"}.fa-calendar-check-o:before{content:"\f274"}.fa-industry:before{content:"\f275"}.fa-map-pin:before{content:"\f276"}.fa-map-signs:before{content:"\f277"}.fa-map-o:before{content:"\f278"}.fa-map:before{content:"\f279"}.fa-commenting:before{content:"\f27a"}.fa-commenting-o:before{content:"\f27b"}.fa-houzz:before{content:"\f27c"}.fa-vimeo:before{content:"\f27d"}.fa-black-tie:before{content:"\f27e"}.fa-fonticons:before{content:"\f280"}.fa-reddit-alien:before{content:"\f281"}.fa-edge:before{content:"\f282"}.fa-credit-card-alt:before{content:"\f283"}.fa-codiepie:before{content:"\f284"}.fa-modx:before{content:"\f285"}.fa-fort-awesome:before{content:"\f286"}.fa-usb:before{content:"\f287"}.fa-product-hunt:before{content:"\f288"}.fa-mixcloud:before{content:"\f289"}.fa-scribd:before{content:"\f28a"}.fa-pause-circle:before{content:"\f28b"}.fa-pause-circle-o:before{content:"\f28c"}.fa-stop-circle:before{content:"\f28d"}.fa-stop-circle-o:before{content:"\f28e"}.fa-shopping-bag:before{content:"\f290"}.fa-shopping-basket:before{content:"\f291"}.fa-hashtag:before{content:"\f292"}.fa-bluetooth:before{content:"\f293"}.fa-bluetooth-b:before{content:"\f294"}.fa-percent:before{content:"\f295"}.fa-gitlab:before{content:"\f296"}.fa-wpbeginner:before{content:"\f297"}.fa-wpforms:before{content:"\f298"}.fa-envira:before{content:"\f299"}.fa-universal-access:before{content:"\f29a"}.fa-wheelchair-alt:before{content:"\f29b"}.fa-question-circle-o:before{content:"\f29c"}.fa-blind:before{content:"\f29d"}.fa-audio-description:before{content:"\f29e"}.fa-volume-control-phone:before{content:"\f2a0"}.fa-braille:before{content:"\f2a1"}.fa-assistive-listening-systems:before{content:"\f2a2"}.fa-asl-interpreting:before,.fa-american-sign-language-interpreting:before{content:"\f2a3"}.fa-deafness:before,.fa-hard-of-hearing:before,.fa-deaf:before{content:"\f2a4"}.fa-glide:before{content:"\f2a5"}.fa-glide-g:before{content:"\f2a6"}.fa-signing:before,.fa-sign-language:before{content:"\f2a7"}.fa-low-vision:before{content:"\f2a8"}.fa-viadeo:before{content:"\f2a9"}.fa-viadeo-square:before{content:"\f2aa"}.fa-snapchat:before{content:"\f2ab"}.fa-snapchat-ghost:before{content:"\f2ac"}.fa-snapchat-square:before{content:"\f2ad"}.fa-pied-piper:before{content:"\f2ae"}.fa-first-order:before{content:"\f2b0"}.fa-yoast:before{content:"\f2b1"}.fa-themeisle:before{content:"\f2b2"}.fa-google-plus-circle:before,.fa-google-plus-official:before{content:"\f2b3"}.fa-fa:before,.fa-font-awesome:before{content:"\f2b4"}.fa-handshake-o:before{content:"\f2b5"}.fa-envelope-open:before{content:"\f2b6"}.fa-envelope-open-o:before{content:"\f2b7"}.fa-linode:before{content:"\f2b8"}.fa-address-book:before{content:"\f2b9"}.fa-address-book-o:before{content:"\f2ba"}.fa-vcard:before,.fa-address-card:before{content:"\f2bb"}.fa-vcard-o:before,.fa-address-card-o:before{content:"\f2bc"}.fa-user-circle:before{content:"\f2bd"}.fa-user-circle-o:before{content:"\f2be"}.fa-user-o:before{content:"\f2c0"}.fa-id-badge:before{content:"\f2c1"}.fa-drivers-license:before,.fa-id-card:before{content:"\f2c2"}.fa-drivers-license-o:before,.fa-id-card-o:before{content:"\f2c3"}.fa-quora:before{content:"\f2c4"}.fa-free-code-camp:before{content:"\f2c5"}.fa-telegram:before{content:"\f2c6"}.fa-thermometer-4:before,.fa-thermometer:before,.fa-thermometer-full:before{content:"\f2c7"}.fa-thermometer-3:before,.fa-thermometer-three-quarters:before{content:"\f2c8"}.fa-thermometer-2:before,.fa-thermometer-half:before{content:"\f2c9"}.fa-thermometer-1:before,.fa-thermometer-quarter:before{content:"\f2ca"}.fa-thermometer-0:before,.fa-thermometer-empty:before{content:"\f2cb"}.fa-shower:before{content:"\f2cc"}.fa-bathtub:before,.fa-s15:before,.fa-bath:before{content:"\f2cd"}.fa-podcast:before{content:"\f2ce"}.fa-window-maximize:before{content:"\f2d0"}.fa-window-minimize:before{content:"\f2d1"}.fa-window-restore:before{content:"\f2d2"}.fa-times-rectangle:before,.fa-window-close:before{content:"\f2d3"}.fa-times-rectangle-o:before,.fa-window-close-o:before{content:"\f2d4"}.fa-bandcamp:before{content:"\f2d5"}.fa-grav:before{content:"\f2d6"}.fa-etsy:before{content:"\f2d7"}.fa-imdb:before{content:"\f2d8"}.fa-ravelry:before{content:"\f2d9"}.fa-eercast:before{content:"\f2da"}.fa-microchip:before{content:"\f2db"}.fa-snowflake-o:before{content:"\f2dc"}.fa-superpowers:before{content:"\f2dd"}.fa-wpexplorer:before{content:"\f2de"}.fa-meetup:before{content:"\f2e0"}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0, 0, 0, 0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}

body {
    font-size: 115% !important;
}

.doc-object {
    margin-top: 2.5rem;
}

blockquote {
    border-left: 5px #efa solid;
    padding-left: 2rem;
    background: #f7fcf0 !important;
}

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        
        
        
        <link rel="shortcut icon" href="img/favicon.ico">
        <title>global .fmt db - logfmt1</title>
        <link href="css/bootstrap.min.css" rel="stylesheet">
        <link href="css/font-awesome.min.css" rel="stylesheet">
        <link href="css/base.css" rel="stylesheet">
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css">
        <link href="custom.css" rel="stylesheet">

        <script src="js/jquery-1.10.2.min.js" defer></script>
        <script src="js/bootstrap.min.js" defer></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
        <script>hljs.initHighlightingOnLoad();</script> 
    </head>

    <body>
        <div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
            <div class="container">
                <a class="navbar-brand" href=".">logfmt1</a>
                <!-- Expander button -->
                <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbar-collapse">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <!-- Expanded navigation -->
                <div id="navbar-collapse" class="navbar-collapse collapse">
                        <!-- Main navigation -->
                        <ul class="nav navbar-nav">
                            <li class="navitem">
                                <a href="index.html" class="nav-link">Home</a>
                            </li>
                            <li class="navitem">
                                <a href="log.fmt.html" class="nav-link">.log.fmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logopen.html" class="nav-link">logopen</a>
                            </li>
                            <li class="navitem">
                                <a href="update-regex.html" class="nav-link">regex</a>
                            </li>
                            <li class="navitem active">
                                <a href="fmt.html" class="nav-link">global .fmt db</a>
                            </li>
                            <li class="navitem">
                                <a href="update-logfmt.html" class="nav-link">update-logmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logex.html" class="nav-link">logex</a>
                            </li>
                        </ul>

                    <ul class="nav navbar-nav ml-auto">
                            <li class="nav-item">
                                <a rel="prev" href="update-regex.html" class="nav-link">
                                    <i class="fa fa-arrow-left"></i> Previous
                                </a>
                            </li>
                            <li class="nav-item">
                                <a rel="next" href="update-logfmt.html" class="nav-link">
                                    Next <i class="fa fa-arrow-right"></i>
                                </a>
                            </li>
                            <li class="nav-item">
                                <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1" class="nav-link">Fossil</a>
                            </li>
                    </ul>
                </div>
            </div>
        </div>

        <div class="container">
            <div class="row">
                    <div class="col-md-3"><div class="navbar-light navbar-expand-md bs-sidebar hidden-print affix" role="complementary">
    <div class="navbar-header">
        <button type="button" class="navbar-toggler collapsed" data-toggle="collapse" data-target="#toc-collapse" title="Table of Contents">
            <span class="fa fa-angle-down"></span>
        </button>
    </div>

    
    <div id="toc-collapse" class="navbar-collapse collapse card bg-secondary">
        <ul class="nav flex-column">
            
            <li class="nav-item" data-level="2"><a href="#global-fmt-database" class="nav-link">Global .fmt database</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#class" class="nav-link">class</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#placeholder" class="nav-link">placeholder</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#fields" class="nav-link">fields</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#expand" class="nav-link">expand</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#other-format-files" class="nav-link">Other format files</a>
              <ul class="nav flex-column">
              </ul>
            </li>
        </ul>
    </div>
</div></div>
                    <div class="col-md-9" role="main">

<blockquote>
<table>
<thead>
<tr>
<th>❮❗❯</th>
<th>This is all very provisional. (First draft. Names might still change.)</th>
</tr>
</thead>
<tbody>
<tr>
<td></td>
<td></td>
</tr>
</tbody>
</table>
</blockquote>
<h2 id="global-fmt-database">Global .fmt database</h2>
<p>While each log file should be accompanied by a <a href="log.fmt.html">.fmt descriptor</a>,
the global database in <code>/usr/share/logfmt/</code> contains a full .fmt field
definition for each class.</p>
<p>Most notably the <code>"fields":</code> and <code>"placeholder":</code> are used to construct the
regex for a <code>"record":</code> string definition.</p>
<h3 id="sample">sample</h3>
<p>For example the Apache format definition (apache.fmt) contains:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
    <span class="s2">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache generic&quot;</span><span class="p">,</span>
    <span class="s2">&quot;separator&quot;</span><span class="p">:</span> <span class="s2">&quot; &quot;</span><span class="p">,</span>
    <span class="s2">&quot;rewrite&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s2">&quot;%[</span><span class="se">\\</span><span class="s2">d!,+</span><span class="se">\\</span><span class="s2">-]+&quot;</span><span class="p">:</span> <span class="s2">&quot;%&quot;</span><span class="p">,</span>
        <span class="s2">&quot;</span><span class="si">%%</span><span class="s2">&quot;</span><span class="p">:</span> <span class="s2">&quot;%&quot;</span>
    <span class="p">},</span>
    <span class="s2">&quot;placeholder&quot;</span><span class="p">:</span> <span class="s2">&quot;%[&lt;&gt;]?(?:</span><span class="se">\\</span><span class="s2">w*</span><span class="se">\\</span><span class="s2">{[^</span><span class="se">\\</span><span class="s2">}]+</span><span class="se">\\</span><span class="s2">})?</span><span class="se">\\</span><span class="s2">^?</span><span class="se">\\</span><span class="s2">w+&quot;</span><span class="p">,</span>
    <span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s2">&quot;%a&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_addr&quot;</span><span class="p">,</span>    <span class="s2">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[</span><span class="se">\\</span><span class="s2">d.:a-f]+&quot;</span>  <span class="p">},</span>
        <span class="s2">&quot;%h&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_host&quot;</span><span class="p">,</span>    <span class="s2">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[</span><span class="se">\\</span><span class="s2">w</span><span class="se">\\</span><span class="s2">-.:]+&quot;</span>  <span class="p">},</span>
        <span class="s2">&quot;%{c}h&quot;</span><span class="p">:</span> <span class="p">{</span> <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_host&quot;</span><span class="p">,</span>  <span class="s2">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[</span><span class="se">\\</span><span class="s2">w</span><span class="se">\\</span><span class="s2">-.:]+&quot;</span>  <span class="p">},</span>
        <span class="s2">&quot;%A&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;local_address&quot;</span><span class="p">,</span>  <span class="s2">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[</span><span class="se">\\</span><span class="s2">d.:a-f]+&quot;</span>  <span class="p">},</span>
        <span class="s2">&quot;</span><span class="si">%u</span><span class="s2">&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_user&quot;</span><span class="p">,</span>    <span class="s2">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[</span><span class="se">\\</span><span class="s2">-</span><span class="se">\\</span><span class="s2">w@.]+&quot;</span>  <span class="p">},</span>
        <span class="s2">&quot;%t&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;request_time&quot;</span><span class="p">,</span>   <span class="s2">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;</span><span class="se">\\</span><span class="s2">[?(</span><span class="se">\\</span><span class="s2">d[</span><span class="se">\\</span><span class="s2">d:</span><span class="se">\\</span><span class="s2">w</span><span class="se">\\</span><span class="s2">s:./</span><span class="se">\\</span><span class="s2">-+,;]+)</span><span class="se">\\</span><span class="s2">]?&quot;</span>  <span class="p">},</span>
        <span class="err">…</span>
    <span class="p">},</span>
    <span class="s2">&quot;alias&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s2">&quot;remote_address&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_addr&quot;</span><span class="p">,</span>
        <span class="s2">&quot;ip&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_addr&quot;</span><span class="p">,</span>
        <span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="s2">&quot;request_file&quot;</span><span class="p">,</span>
        <span class="s2">&quot;size&quot;</span><span class="p">:</span> <span class="s2">&quot;bytes_sent&quot;</span><span class="p">,</span>
        <span class="err">…</span>
    <span class="p">},</span>
    <span class="s2">&quot;expand&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s2">&quot;%</span><span class="se">\\</span><span class="s2">{([^{}]+)</span><span class="se">\\</span><span class="s2">}t&quot;</span><span class="p">:</span> <span class="p">{</span>
            <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;request_time&quot;</span><span class="p">,</span>
            <span class="s2">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;strftime&quot;</span><span class="p">,</span>
            <span class="s2">&quot;record&quot;</span><span class="p">:</span> <span class="s2">&quot;$1&quot;</span>
        <span class="p">}</span>
    <span class="p">},</span>
    <span class="s2">&quot;container&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s2">&quot;message&quot;</span><span class="p">:</span> <span class="p">{</span>
            <span class="s2">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;$1&quot;</span><span class="p">,</span>
            <span class="s2">&quot;value&quot;</span><span class="p">:</span> <span class="s2">&quot;$2&quot;</span><span class="p">,</span>
            <span class="s2">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;</span><span class="se">\\</span><span class="s2">[(</span><span class="se">\\</span><span class="s2">w+) </span><span class="se">\&quot;</span><span class="s2">(.*?)</span><span class="se">\&quot;\\</span><span class="s2">]&quot;</span><span class="p">,</span>
            <span class="s2">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache mod_security&quot;</span>
        <span class="p">}</span>
    <span class="p">},</span>
    <span class="s2">&quot;glob&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;/var/log/apache*/*acc*.log&quot;</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>

<p>It usually does not describe a default "record" format (like the local .log.fmt descriptors do).</p>
<h2 id="class">class</h2>
<p>The class in the global database is largely decorative.  The filenames
instead define the heritage of rules/fields.  The "class" as declared by
a .log.fmt is mapped onto <code>/usr/share/logfmt/application.variant.fmt</code>.</p>
<ul>
<li>Usually there's just one variant level per log type. But the lookup is
   supposed to be mildly recursive.</li>
<li>Essentially it should merge <code>*.log.fmt</code> with <code>appclass.variant.fmt</code> and
   <code>appclass.fmt</code> applied last, so the most specific definitions are retained.</li>
<li>There's also a generic "grok" class. But the patterns therein are largely
   static (not build from variable format strings).</li>
<li>Some special classes like "json" might exist. (Not supported by logfmt1)</li>
</ul>
<h3 id="record">record</h3>
<p>The "record" entry is not usually present in the global .fmt definition. 
Some super specific variant definitions (for example apache.error.fmt) or
static formats (syslog.fmt) might however.</p>
<h3 id="separator">separator</h3>
<p>Most log formats use spaces for separating %placeholder fields.  And simpler
implementations might just split up the "record" declaration on this.</p>
<h2 id="placeholder">placeholder</h2>
<p>While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields. It should account for prefixes/suffixes, unless
those got cleared by the <code>rewrite</code> map.</p>
<p>Not all formatstrings use <code>%\w+</code> to signal placeholders. In nginx for instance
the sigil <code>$\w+</code> introduces placeholders (variable names, really).</p>
<h3 id="rewrite">rewrite</h3>
<p>A list/map of regex to apply before any transformations or field lookups. 
Which can be used to mask or simplify placeholder definitions (for instance
clean up the Apache conditional prefixes) or regex meta characters.</p>
<ul>
<li>The <code>record</code> field starts as a static string, but is meant to be turned
   into a regex.</li>
<li>Therefore meta characters (such as <code>|</code> or <code>[]</code>) have to be
   taken care of.  Which is what the <code>rewrite</code> map is lazily used for.</li>
<li>Better implementations might look up the placeholders, and automatically
   escape the rest of the the "record" format string.</li>
</ul>
<h2 id="fields">fields</h2>
<p>The core of the global .fmt definitions are the field lists.  Each defines a
static %F placeholder and associaties it with a default field name (id:) and
regex (rx:) or even a grok definition (grok:).</p>
<table>
<thead>
<tr>
<th>key</th>
<th>purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td>%F</td>
<td>JSON key: static placeholder string (not a regex itself)</td>
</tr>
<tr>
<td>id</td>
<td>field identifier, as specified by the application (internal name)</td>
</tr>
<tr>
<td>rx</td>
<td>regex which %F placeholder gets replaced with</td>
</tr>
<tr>
<td>grok</td>
<td>alternatively to regex, %F might be turned into %PATTERN:id</td>
</tr>
<tr>
<td>type</td>
<td>"int" and "float" could designate strictly numeric fields</td>
</tr>
</tbody>
</table>
<ul>
<li>As part of the regex transformation, a <code>%F</code> could be turned into
   <code>(?&lt;id&gt;\S+)</code> for instance.</li>
<li>If there's any unnamed capture group <code>(…)</code>, it should be augmented
   into a named capture group - instead of the whole match. (To account
   for implicit wrapping.)</li>
<li>The <code>rx</code> itself might however specify named subgroups (like request_line
   in Apache logs, itself comprised of _method, _path, _protocol, or the
   datetime made up of tm_wday, tm_year, tm_whatever).</li>
<li><code>\S+</code> is also used as fallback for entirely undefined placeholders
   (no expand definition matched) in logfmt1.</li>
<li><code>grok</code> isn't currently used, but might allow for simpler transformations
   (indirectly into a grok pattern, and later a regex).</li>
</ul>
<h2 id="expand">expand</h2>
<p>The expand declarations are used to construct unknown fields/placeholders. 
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.  Thus it simply can be applied before
separator/placeholder are looked at, to augment the known <code>fields</code> list.</p>
<table>
<thead>
<tr>
<th>key</th>
<th>purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>%\{(\w+)\}t</code></td>
<td>JSON key: a regex to detect mutable placeholders</td>
</tr>
<tr>
<td>id</td>
<td>name for newly created fields entry, might use captures $1</td>
</tr>
<tr>
<td>rx</td>
<td>for static definitions (often just \S+)</td>
</tr>
<tr>
<td>if_quoted</td>
<td>alternative regex, if placeholder was enclosed in "%\w+" quotes</td>
</tr>
<tr>
<td>class</td>
<td>recurse into other .fmt types</td>
</tr>
<tr>
<td>record</td>
<td>can be set to $2 if class: recursion is defined</td>
</tr>
</tbody>
</table>
<ul>
<li>Typically it suffices to specify the <code>id</code> and <code>rx</code> field.</li>
<li>If no <code>id</code> is given, then the regex capture is normalized into
   an identifier (non-alphanumerics stripped, all lowercased).</li>
<li>But the <code>id</code> or <code>record</code> value might be set with regex captures
   (e.g. <code>$1</code> or <code>$2</code>) or compound values (<code>"id": "newfield_$1"</code>).</li>
<li>And logfmt1 allows to recurse into other format types per <code>class</code>
   (which is used to expand the captured <code>"record": "$1"</code> into regex
   tokens).</li>
</ul>
<h3 id="alias">alias</h3>
<p>Maps alternative/more common field names onto the declared field <code>id</code>s.</p>
<p>To get to some state of standardization, the field ids usually refer
to application-internal names. (For instance <code>log_pfn_register(…,…,cb_id)</code>
names in Apache). And those aren't always the more commonly used identifiers.</p>
<p>Thus aliases makes sense not just for convenience, but also to be compatible
to other common names (e.g. w3c extend log format names like <code>cs-time</code>).</p>
<h3 id="container">container</h3>
<p>Is utilized by logopen() to extract additional fields (lists even) from one
of the existing fields.  This is usually done at row traversal.  And makes
sense for application-specific subformats in logs.  Such as any <code>key=value</code>
lists in the main message field.</p>
<blockquote>
<p>Still not sure if automatic list conversion is a good idea.</p>
</blockquote>
<h3 id="glob">glob</h3>
<p>Might be used by log processors to look up a log class, based on file names,
if no .log.fmt is declared.</p>
<h3 id="comment-fields">"#comment": fields</h3>
<p>Documentation entries in the .fmt files have keys starting with <code>#</code>. For example
<code>"#license":</code> or <code>"#origin":</code>. Which is simpler than using JSON with
comments (JSOL).</p>
<hr />
<h2 id="other-format-files">Other format files</h2>
<blockquote>
<p>This section is about fictional features.</p>
</blockquote>
<h3 id="grok-definitions">.grok definitions</h3>
<blockquote>
<p>Not implemented yet.</p>
</blockquote>
<p>The logfmt/ directory might also contain .grok files, which get transformed
into .fmt structures. (Probably with the grok: parameter for fields, and
a grok: pattern table alongside regular fields:).</p>
<p>There's already a pretransformed <code>grok.fmt</code>, which however requires
<code>%{GROK:%{PATTERN:id}}</code> references currently.</p>
<h3 id="lnav-formats">.lnav formats</h3>
<blockquote>
<p>Not implemented yet.</p>
</blockquote>
<p>Likewise could we use lnav .json format definitions. Those are static
too, however.</p></div>
            </div>
        </div>

        <footer class="col-md-12">
            <hr>
            <p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
        </footer>
        <script>
            var base_url = ".",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
        <script src="js/base.js" defer></script>

        <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
    <div class="modal-dialog">
        <div class="modal-content">
            <div class="modal-header">
                <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
            </div>
            <div class="modal-body">
              <table class="table">
                <thead>
                  <tr>
                    <th style="width: 20%;">Keys</th>
                    <th>Action</th>
                  </tr>
                </thead>
                <tbody>
                  <tr>
                    <td class="help shortcut"><kbd>?</kbd></td>
                    <td>Open this help</td>
                  </tr>
                  <tr>
                    <td class="next shortcut"><kbd>n</kbd></td>
                    <td>Next page</td>
                  </tr>
                  <tr>
                    <td class="prev shortcut"><kbd>p</kbd></td>
                    <td>Previous page</td>
                  </tr>
                  <tr>
                    <td class="search shortcut"><kbd>s</kbd></td>
                    <td>Search</td>
                  </tr>
                </tbody>
              </table>
            </div>
            <div class="modal-footer">
            </div>
        </div>
    </div>
</div>

    </body>
</html>

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        
        
        
        <link rel="shortcut icon" href="img/favicon.ico">
        <title>Home - logfmt1</title>
        <link href="css/bootstrap.min.css" rel="stylesheet">
        <link href="css/font-awesome.min.css" rel="stylesheet">
        <link href="css/base.css" rel="stylesheet">
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css">
        <link href="custom.css" rel="stylesheet">

        <script src="js/jquery-1.10.2.min.js" defer></script>
        <script src="js/bootstrap.min.js" defer></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
        <script>hljs.initHighlightingOnLoad();</script> 
    </head>

    <body>
        <div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
            <div class="container">
                <a class="navbar-brand" href=".">logfmt1</a>
                <!-- Expander button -->
                <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbar-collapse">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <!-- Expanded navigation -->
                <div id="navbar-collapse" class="navbar-collapse collapse">
                        <!-- Main navigation -->
                        <ul class="nav navbar-nav">
                            <li class="navitem active">
                                <a href="index.html" class="nav-link">Home</a>
                            </li>
                            <li class="navitem">
                                <a href="log.fmt.html" class="nav-link">.log.fmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logopen.html" class="nav-link">logopen</a>
                            </li>
                            <li class="navitem">
                                <a href="update-regex.html" class="nav-link">regex</a>
                            </li>
                            <li class="navitem">
                                <a href="fmt.html" class="nav-link">global .fmt db</a>
                            </li>
                            <li class="navitem">
                                <a href="update-logfmt.html" class="nav-link">update-logmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logex.html" class="nav-link">logex</a>
                            </li>
                        </ul>

                    <ul class="nav navbar-nav ml-auto">
                            <li class="nav-item">
                                <a rel="prev" class="nav-link disabled">
                                    <i class="fa fa-arrow-left"></i> Previous
                                </a>
                            </li>
                            <li class="nav-item">
                                <a rel="next" href="log.fmt.html" class="nav-link">
                                    Next <i class="fa fa-arrow-right"></i>
                                </a>
                            </li>
                            <li class="nav-item">
                                <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1" class="nav-link">Fossil</a>
                            </li>
                    </ul>
                </div>
            </div>
        </div>

        <div class="container">
            <div class="row">
                    <div class="col-md-3"><div class="navbar-light navbar-expand-md bs-sidebar hidden-print affix" role="complementary">
    <div class="navbar-header">
        <button type="button" class="navbar-toggler collapsed" data-toggle="collapse" data-target="#toc-collapse" title="Table of Contents">
            <span class="fa fa-angle-down"></span>
        </button>
    </div>

    
    <div id="toc-collapse" class="navbar-collapse collapse card bg-secondary">
        <ul class="nav flex-column">
            
            <li class="nav-item" data-level="1"><a href="#logfmt1" class="nav-link">logfmt1</a>
              <ul class="nav flex-column">
            <li class="nav-item" data-level="2"><a href="#overview" class="nav-link">Overview</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            <li class="nav-item" data-level="2"><a href="#sample-logfmt" class="nav-link">Sample .log.fmt</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            <li class="nav-item" data-level="2"><a href="#usage" class="nav-link">Usage</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            <li class="nav-item" data-level="2"><a href="#about" class="nav-link">About</a>
              <ul class="nav flex-column">
              </ul>
            </li>
              </ul>
            </li>
        </ul>
    </div>
</div></div>
                    <div class="col-md-9" role="main">

<h1 id="logfmt1">logfmt1</h1>
<p><strong>logfmt1</strong> is an implementation and scheme for generic log parsing.  It
prescribes a <a href="log.fmt.html">.fmt</a> descriptor adjacent to each log file.  And a
global database to resolve format string %placeholder%s and turn them into
regular expressions.</p>
<h2 id="overview">Overview</h2>
<ul>
<li>A <a href="log.fmt.html">.log.fmt</a> for each log file</li>
<li><a href="logopen.html">logopen()</a></li>
<li><a href="update-regex.html">update() and regex()</a></li>
<li>Field <a href="fmt.html">.fmt definitions</a> in <code>/usr/share/logfmt</code></li>
<li><a href="update-logfmt.html">update-logfmt</a> to create *.log.fmt files (for Apache, Nginx, some static logs)</li>
</ul>
<h2 id="sample-logfmt">Sample .log.fmt</h2>
<p>Currently the format for a *.log.fmt descriptor is:</p>
<div class="codehilite"><pre><span></span><code><span class="err">{</span>
<span class="err">   &quot;class&quot;: &quot;apache custom2&quot;,</span>
<span class="err">   &quot;record&quot;: &quot;%a %u %l [%t] %V \&quot;%r\&quot; %O %&gt;s %D %{SSL_CIPHER}e %M&quot;</span>
<span class="err">}</span>
</code></pre></div>

<p>Might still change, of course. "record" might be better named "formatstring"
for example.</p>
<h2 id="usage">Usage</h2>
<p>In the simplest of cases you can use <a href="logopen.html">logopen</a> to process
a log file (and its adjacent .fmt descriptor) at once:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">log</span> <span class="o">=</span> <span class="n">logfmt1</span><span class="o">.</span><span class="n">logopen</span><span class="p">(</span><span class="s2">&quot;/var/log/apache2/access.log&quot;</span><span class="p">)</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="nb">log</span><span class="p">:</span>
    <span class="nb">print</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s2">&quot;request_line&quot;</span><span class="p">])</span>
<span class="c1">#print(log.names())</span>
</code></pre></div>

<h2 id="about">About</h2>
<p>Project originated as part of <a href="https://fossil.include-once.org/modseccfg/">modseccfg</a>.</p></div>
            </div>
        </div>

        <footer class="col-md-12">
            <hr>
            <p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
        </footer>
        <script>
            var base_url = ".",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
        <script src="js/base.js" defer></script>

        <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
    <div class="modal-dialog">
        <div class="modal-content">
            <div class="modal-header">
                <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
            </div>
            <div class="modal-body">
              <table class="table">
                <thead>
                  <tr>
                    <th style="width: 20%;">Keys</th>
                    <th>Action</th>
                  </tr>
                </thead>
                <tbody>
                  <tr>
                    <td class="help shortcut"><kbd>?</kbd></td>
                    <td>Open this help</td>
                  </tr>
                  <tr>
                    <td class="next shortcut"><kbd>n</kbd></td>
                    <td>Next page</td>
                  </tr>
                  <tr>
                    <td class="prev shortcut"><kbd>p</kbd></td>
                    <td>Previous page</td>
                  </tr>
                  <tr>
                    <td class="search shortcut"><kbd>s</kbd></td>
                    <td>Search</td>
                  </tr>
                </tbody>
              </table>
            </div>
            <div class="modal-footer">
            </div>
        </div>
    </div>
</div>

    </body>
</html>

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        
        
        
        <link rel="shortcut icon" href="img/favicon.ico">
        <title>.log.fmt - logfmt1</title>
        <link href="css/bootstrap.min.css" rel="stylesheet">
        <link href="css/font-awesome.min.css" rel="stylesheet">
        <link href="css/base.css" rel="stylesheet">
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css">
        <link href="custom.css" rel="stylesheet">

        <script src="js/jquery-1.10.2.min.js" defer></script>
        <script src="js/bootstrap.min.js" defer></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
        <script>hljs.initHighlightingOnLoad();</script> 
    </head>

    <body>
        <div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
            <div class="container">
                <a class="navbar-brand" href=".">logfmt1</a>
                <!-- Expander button -->
                <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbar-collapse">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <!-- Expanded navigation -->
                <div id="navbar-collapse" class="navbar-collapse collapse">
                        <!-- Main navigation -->
                        <ul class="nav navbar-nav">
                            <li class="navitem">
                                <a href="index.html" class="nav-link">Home</a>
                            </li>
                            <li class="navitem active">
                                <a href="log.fmt.html" class="nav-link">.log.fmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logopen.html" class="nav-link">logopen</a>
                            </li>
                            <li class="navitem">
                                <a href="update-regex.html" class="nav-link">regex</a>
                            </li>
                            <li class="navitem">
                                <a href="fmt.html" class="nav-link">global .fmt db</a>
                            </li>
                            <li class="navitem">
                                <a href="update-logfmt.html" class="nav-link">update-logmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logex.html" class="nav-link">logex</a>
                            </li>
                        </ul>

                    <ul class="nav navbar-nav ml-auto">
                            <li class="nav-item">
                                <a rel="prev" href="index.html" class="nav-link">
                                    <i class="fa fa-arrow-left"></i> Previous
                                </a>
                            </li>
                            <li class="nav-item">
                                <a rel="next" href="logopen.html" class="nav-link">
                                    Next <i class="fa fa-arrow-right"></i>
                                </a>
                            </li>
                            <li class="nav-item">
                                <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1" class="nav-link">Fossil</a>
                            </li>
                    </ul>
                </div>
            </div>
        </div>

        <div class="container">
            <div class="row">
                    <div class="col-md-3"><div class="navbar-light navbar-expand-md bs-sidebar hidden-print affix" role="complementary">
    <div class="navbar-header">
        <button type="button" class="navbar-toggler collapsed" data-toggle="collapse" data-target="#toc-collapse" title="Table of Contents">
            <span class="fa fa-angle-down"></span>
        </button>
    </div>

    
    <div id="toc-collapse" class="navbar-collapse collapse card bg-secondary">
        <ul class="nav flex-column">
            
            <li class="nav-item" data-level="2"><a href="#a-logfmt-for-each-log-file" class="nav-link">A .log.fmt for each log file</a>
              <ul class="nav flex-column">
              </ul>
            </li>
        </ul>
    </div>
</div></div>
                    <div class="col-md-9" role="main">

<h2 id="a-logfmt-for-each-log-file">A .log.fmt for each log file</h2>
<p>Log parsing is a curse, because each application has its own format, and
oftentimes configurable fields at that.  Various attempts at standardizing
logs have failed, or are bound to.  Logging services and database storage
are largely just symptomatic kludges, with JSON logs and not-quite-JSON
formats held back by inertia.</p>
<p>Instead logfmt1 aims to have descriptors for each log file, in order to
make them parseable. You can't attempt anything but guesswork until you
know what's in a file.</p>
<p>So the idea is to have a <code>*.fmt</code> next to each <code>*.log</code> file, with a
descriptor such as:</p>
<div class="codehilite"><pre><span></span><code><span class="err">{</span>
<span class="err">   &quot;class&quot;: &quot;apache combined&quot;,</span>
<span class="err">   &quot;record&quot;: &quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b&quot;</span>
<span class="err">}</span>
</code></pre></div>

<p>Notably the "record" field should be the most current format string that
the application itself uses. In order to resolve the placeholders, an
application reference is kept in "class". Which allows combining the
format string with placeholder field definitions from the global
<a href="fmt.html">.fmt database (<code>/usr/share/logfmt</code>)</a> database.</p>
<h3 id="common-classes">common classes</h3>
<p>If the *.log itself is just JSON, then the .log.fmt would specify it
simply as:</p>
<div class="codehilite"><pre><span></span><code><span class="err">{</span>
<span class="err">   &quot;class&quot;: &quot;json appmoniker&quot;</span>
<span class="err">}</span>
</code></pre></div>

<p>For widely-known formats such as the "combined" Apache logs, the class
might also suffice. But you can't rely on the user not having modified
the LogFormat within the Apache settings then.</p>
<h3 id="additional-fields">additional fields</h3>
<p>The *.log.fmt itself might declare definitions such as aliases and
more specific/custom placeholders.</p>
<div class="codehilite"><pre><span></span><code><span class="err">{</span>
<span class="err">   &quot;class&quot;: &quot;apache cust3&quot;,</span>
<span class="err">   &quot;record&quot;: &quot;%a %h %{iso}t &#39;%r&#39; %s&quot;,</span>
<span class="err">   &quot;fields&quot;: {</span>
<span class="err">       &quot;%{iso}t&quot;: { &quot;id&quot;: &quot;datetime&quot;, &quot;rx&quot;: &quot;...&quot; }</span>
<span class="err">   },</span>
<span class="err">   &quot;alias&quot;: {</span>
<span class="err">       &quot;iso8601&quot;: &quot;datetime&quot;,</span>
<span class="err">   }</span>
<span class="err">}</span>
</code></pre></div>

<p>Which ought to be joined and override any global <a href="fmt">fmt</a> definitions.
Though such user customizations are more likely to be applied there anyway.
Care should be taken by <code>update-logfmt</code> or applications to not jettison
user-customized *.log.fmt options.</p></div>
            </div>
        </div>

        <footer class="col-md-12">
            <hr>
            <p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
        </footer>
        <script>
            var base_url = ".",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
        <script src="js/base.js" defer></script>

        <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
    <div class="modal-dialog">
        <div class="modal-content">
            <div class="modal-header">
                <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
            </div>
            <div class="modal-body">
              <table class="table">
                <thead>
                  <tr>
                    <th style="width: 20%;">Keys</th>
                    <th>Action</th>
                  </tr>
                </thead>
                <tbody>
                  <tr>
                    <td class="help shortcut"><kbd>?</kbd></td>
                    <td>Open this help</td>
                  </tr>
                  <tr>
                    <td class="next shortcut"><kbd>n</kbd></td>
                    <td>Next page</td>
                  </tr>
                  <tr>
                    <td class="prev shortcut"><kbd>p</kbd></td>
                    <td>Previous page</td>
                  </tr>
                  <tr>
                    <td class="search shortcut"><kbd>s</kbd></td>
                    <td>Search</td>
                  </tr>
                </tbody>
              </table>
            </div>
            <div class="modal-footer">
            </div>
        </div>
    </div>
</div>

    </body>
</html>

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        
        
        
        <link rel="shortcut icon" href="img/favicon.ico">
        <title>logex - logfmt1</title>
        <link href="css/bootstrap.min.css" rel="stylesheet">
        <link href="css/font-awesome.min.css" rel="stylesheet">
        <link href="css/base.css" rel="stylesheet">
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css">
        <link href="custom.css" rel="stylesheet">

        <script src="js/jquery-1.10.2.min.js" defer></script>
        <script src="js/bootstrap.min.js" defer></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
        <script>hljs.initHighlightingOnLoad();</script> 
    </head>

    <body>
        <div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
            <div class="container">
                <a class="navbar-brand" href=".">logfmt1</a>
                <!-- Expander button -->
                <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbar-collapse">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <!-- Expanded navigation -->
                <div id="navbar-collapse" class="navbar-collapse collapse">
                        <!-- Main navigation -->
                        <ul class="nav navbar-nav">
                            <li class="navitem">
                                <a href="index.html" class="nav-link">Home</a>
                            </li>
                            <li class="navitem">
                                <a href="log.fmt.html" class="nav-link">.log.fmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logopen.html" class="nav-link">logopen</a>
                            </li>
                            <li class="navitem">
                                <a href="update-regex.html" class="nav-link">regex</a>
                            </li>
                            <li class="navitem">
                                <a href="fmt.html" class="nav-link">global .fmt db</a>
                            </li>
                            <li class="navitem">
                                <a href="update-logfmt.html" class="nav-link">update-logmt</a>
                            </li>
                            <li class="navitem active">
                                <a href="logex.html" class="nav-link">logex</a>
                            </li>
                        </ul>

                    <ul class="nav navbar-nav ml-auto">
                            <li class="nav-item">
                                <a rel="prev" href="update-logfmt.html" class="nav-link">
                                    <i class="fa fa-arrow-left"></i> Previous
                                </a>
                            </li>
                            <li class="nav-item">
                                <a rel="next" class="nav-link disabled">
                                    Next <i class="fa fa-arrow-right"></i>
                                </a>
                            </li>
                            <li class="nav-item">
                                <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1" class="nav-link">Fossil</a>
                            </li>
                    </ul>
                </div>
            </div>
        </div>

        <div class="container">
            <div class="row">
                    <div class="col-md-3"><div class="navbar-light navbar-expand-md bs-sidebar hidden-print affix" role="complementary">
    <div class="navbar-header">
        <button type="button" class="navbar-toggler collapsed" data-toggle="collapse" data-target="#toc-collapse" title="Table of Contents">
            <span class="fa fa-angle-down"></span>
        </button>
    </div>

    
    <div id="toc-collapse" class="navbar-collapse collapse card bg-secondary">
        <ul class="nav flex-column">
            
            
            <li class="nav-item" data-level="2"><a href="#flags" class="nav-link">Flags</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#pefixes" class="nav-link">Pefixes</a>
              <ul class="nav flex-column">
              </ul>
            </li>
        </ul>
    </div>
</div></div>
                    <div class="col-md-9" role="main">

<h3 id="logex">logex</h3>
<p><code>logex</code> is a little command line tool around <a href="logopen.html"><code>logopen()</code></a>.
It basically allows fetching individual fields from a log, and/or
restructuring it into a specific format (e.g. tab or comma-delimited).</p>
<div class="codehilite"><pre><span></span><code><span class="err">logex access.log @request_uri @datetime @user_agent</span>
</code></pre></div>

<p>Where the first param is always the log file, and then an arbitrary
number of field names - either individually or as part of a string
argument:</p>
<div class="codehilite"><pre><span></span><code><span class="err">logex access.log --tab &quot;uri,date,status&quot; user_agent</span>
</code></pre></div>

<p>(Would delimit the first three per comma, the last with a tab.)</p>
<h2 id="flags">Flags</h2>
<table>
<thead>
<tr>
<th>param</th>
<th>desc</th>
</tr>
</thead>
<tbody>
<tr>
<td>--json</td>
<td>output each row as JSON</td>
</tr>
<tr>
<td>--csv</td>
<td>join all fields with comma</td>
</tr>
<tr>
<td>--tab</td>
<td>tab-separated output</td>
</tr>
<tr>
<td>--iso8601</td>
<td>fix any datetime fields</td>
</tr>
<tr>
<td>--regex</td>
<td>just output regex for log</td>
</tr>
<tr>
<td>--debug</td>
<td>debug_rx in case the regex fails</td>
</tr>
</tbody>
</table>
<h2 id="pefixes">Pefixes</h2>
<p>The argument list allows field name prefixes.  Which are only relevant to
container-fields (expanded key:value pairs for some application formats), in
case they're holding lists.</p>
<table>
<thead>
<tr>
<th>prefix</th>
<th>output</th>
</tr>
</thead>
<tbody>
<tr>
<td>@name</td>
<td>will just show the first entry</td>
</tr>
<tr>
<td>%name</td>
<td>space-separated list</td>
</tr>
<tr>
<td>*name</td>
<td>comma-separated list</td>
</tr>
<tr>
<td>+name</td>
<td>plus-joined list</td>
</tr>
<tr>
<td>#name</td>
<td>as json array</td>
</tr>
<tr>
<td>name</td>
<td>whatever</td>
</tr>
</tbody>
</table></div>
            </div>
        </div>

        <footer class="col-md-12">
            <hr>
            <p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
        </footer>
        <script>
            var base_url = ".",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
        <script src="js/base.js" defer></script>

        <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
    <div class="modal-dialog">
        <div class="modal-content">
            <div class="modal-header">
                <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
            </div>
            <div class="modal-body">
              <table class="table">
                <thead>
                  <tr>
                    <th style="width: 20%;">Keys</th>
                    <th>Action</th>
                  </tr>
                </thead>
                <tbody>
                  <tr>
                    <td class="help shortcut"><kbd>?</kbd></td>
                    <td>Open this help</td>
                  </tr>
                  <tr>
                    <td class="next shortcut"><kbd>n</kbd></td>
                    <td>Next page</td>
                  </tr>
                  <tr>
                    <td class="prev shortcut"><kbd>p</kbd></td>
                    <td>Previous page</td>
                  </tr>
                  <tr>
                    <td class="search shortcut"><kbd>s</kbd></td>
                    <td>Search</td>
                  </tr>
                </tbody>
              </table>
            </div>
            <div class="modal-footer">
            </div>
        </div>
    </div>
</div>

    </body>
</html>

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        
        
        
        <link rel="shortcut icon" href="img/favicon.ico">
        <title>logopen - logfmt1</title>
        <link href="css/bootstrap.min.css" rel="stylesheet">
        <link href="css/font-awesome.min.css" rel="stylesheet">
        <link href="css/base.css" rel="stylesheet">
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css">
        <link href="custom.css" rel="stylesheet">

        <script src="js/jquery-1.10.2.min.js" defer></script>
        <script src="js/bootstrap.min.js" defer></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
        <script>hljs.initHighlightingOnLoad();</script> 
    </head>

    <body>
        <div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
            <div class="container">
                <a class="navbar-brand" href=".">logfmt1</a>
                <!-- Expander button -->
                <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbar-collapse">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <!-- Expanded navigation -->
                <div id="navbar-collapse" class="navbar-collapse collapse">
                        <!-- Main navigation -->
                        <ul class="nav navbar-nav">
                            <li class="navitem">
                                <a href="index.html" class="nav-link">Home</a>
                            </li>
                            <li class="navitem">
                                <a href="log.fmt.html" class="nav-link">.log.fmt</a>
                            </li>
                            <li class="navitem active">
                                <a href="logopen.html" class="nav-link">logopen</a>
                            </li>
                            <li class="navitem">
                                <a href="update-regex.html" class="nav-link">regex</a>
                            </li>
                            <li class="navitem">
                                <a href="fmt.html" class="nav-link">global .fmt db</a>
                            </li>
                            <li class="navitem">
                                <a href="update-logfmt.html" class="nav-link">update-logmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logex.html" class="nav-link">logex</a>
                            </li>
                        </ul>

                    <ul class="nav navbar-nav ml-auto">
                            <li class="nav-item">
                                <a rel="prev" href="log.fmt.html" class="nav-link">
                                    <i class="fa fa-arrow-left"></i> Previous
                                </a>
                            </li>
                            <li class="nav-item">
                                <a rel="next" href="update-regex.html" class="nav-link">
                                    Next <i class="fa fa-arrow-right"></i>
                                </a>
                            </li>
                            <li class="nav-item">
                                <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1" class="nav-link">Fossil</a>
                            </li>
                    </ul>
                </div>
            </div>
        </div>

        <div class="container">
            <div class="row">
                    <div class="col-md-3"><div class="navbar-light navbar-expand-md bs-sidebar hidden-print affix" role="complementary">
    <div class="navbar-header">
        <button type="button" class="navbar-toggler collapsed" data-toggle="collapse" data-target="#toc-collapse" title="Table of Contents">
            <span class="fa fa-angle-down"></span>
        </button>
    </div>

    
    <div id="toc-collapse" class="navbar-collapse collapse card bg-secondary">
        <ul class="nav flex-column">
            
            
            <li class="nav-item" data-level="2"><a href="#logfmt1.logopen" class="nav-link">logfmt1.logopen</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#logfmt1.logopen.__init__" class="nav-link">__init__()</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#logfmt1.logopen.debug_rx" class="nav-link">debug_rx()</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#logfmt1.logopen.names" class="nav-link">names()</a>
              <ul class="nav flex-column">
              </ul>
            </li>
        </ul>
    </div>
</div></div>
                    <div class="col-md-9" role="main">

<h3 id="logopen">logopen()</h3>
<div class="doc doc-object doc-class">
<h2 class="hidden-toc" href="#logfmt1.logopen" id="logfmt1.logopen" style="visibility: hidden; position: absolute;">
        </h2>
<div class="doc doc-contents first">
<p>Open log file and its associated .fmt specifier, to iterate over log lines
as dictionary.
File-style wrapper that yields parsed dictionaries instead of string lines.</p>
<div class="codehilite">
<pre><span></span><code><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">logfmt1</span><span class="o">.</span><span class="n">logopen</span><span class="p">(</span><span class="s2">"/var/log/apache2/access.log"</span><span class="p">,</span> <span class="n">debug</span><span class="o">=</span><span class="n">True</span><span class="p">):</span>
    <span class="nb">print</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s2">"remote_host"</span><span class="p">])</span>
</code></pre>
</div>
<p>Though you might want to keep a reference to the iterator to utilize
<code>.names()</code> and <code>.alias{}</code> manually.</p>
<div class="doc doc-children">
<div class="doc doc-object doc-method">
<h2 class="doc doc-heading" id="logfmt1.logopen.__init__">

        <code class="highlight language-python">
__init__<span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">logfn</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">fmt</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">debug</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">fail</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">duplicate</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>        </code>

  <span class="doc doc-properties">
      <small class="doc doc-property doc-property-special"><code>special</code></small>
  </span>

      </h2>
<div class="doc doc-contents ">
<p>Open log file and its associated .fmt specifier, to iterate over log lines
as dictionary.</p>
<p><strong>Parameters:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>logfn</code></td>
<td><code>str</code></td>
<td>
<p>Filename of .log file to open.  Which should be
accompanied by a .log.fmt declaration to allow unpacking
lines into dictionary.</p>
</td>
<td><code>''</code></td>
</tr>
<tr>
<td><code>fmt</code></td>
<td><code>dict</code></td>
<td>
<p>Alternatively to existing .log.fmt, a predefined class
might be given with <code>fmt={"class":"syslog"}</code>.  You might
even add a fixated <code>{"record":"%a %t %e %F"}</code> format string
this way.</p>
</td>
<td><code>None</code></td>
</tr>
<tr>
<td><code>debug</code></td>
<td><code>bool</code></td>
<td>
<p>In case of log extraction failures, prints (stdout)
some regex debugging.</p>
</td>
<td><code>False</code></td>
</tr>
<tr>
<td><code>fail</code></td>
<td><code>bool</code></td>
<td>
<p>In case of failure, just error out instead of continuing
the iterator.</p>
</td>
<td><code>False</code></td>
</tr>
<tr>
<td><code>duplicate</code></td>
<td><code>bool</code></td>
<td>
<p>Automatically expand aliases.  This effectively
copies row entries.</p>
</td>
<td><code>True</code></td>
</tr>
</tbody>
</table>
<p><strong>Attributes:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>f</code></td>
<td><code>file</code></td>
<td>
<p>Read handle onto log file</p>
</td>
</tr>
<tr>
<td><code>debug</code></td>
<td><code>bool</code></td>
<td>
<p>Debug flag</p>
</td>
</tr>
<tr>
<td><code>fail</code></td>
<td><code>bool</code></td>
<td>
<p>Exception flag</p>
</td>
</tr>
<tr>
<td><code>alias</code></td>
<td><code>dict</code></td>
<td>
<p>List of row aliases</p>
</td>
</tr>
<tr>
<td><code>container</code></td>
<td><code>dict</code></td>
<td>
<p>Rules for field expansion</p>
</td>
</tr>
<tr>
<td><code>rx</code></td>
<td><code>re.compile</code></td>
<td>
<p>Compiled regex</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Exceptions:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>StopIteration</code></td>
<td>
<p>For EOF or if the regex failed and fail=True</p>
</td>
</tr>
<tr>
<td><code>FileNotFound</code></td>
<td>
<p>If logfn doesn't exist etc.</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Returns:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>iterator</code></td>
<td>
<p>Traverses log file line-wise, but yields dictionaries.</p>
</td>
</tr>
</tbody>
</table>
<details class="quote">
          <summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
          <div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">logfn</span><span class="o">=</span><span class="s2">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">debug</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">fail</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">duplicate</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
    <span class="sd">"""</span>
<span class="sd">        Open log file and its associated .fmt specifier, to iterate over log lines</span>
<span class="sd">        as dictionary.</span>

<span class="sd">        Args:</span>
<span class="sd">            logfn (str):  Filename of .log file to open.  Which should be</span>
<span class="sd">                accompanied by a .log.fmt declaration to allow unpacking</span>
<span class="sd">                lines into dictionary.</span>
<span class="sd">            fmt (dict): Alternatively to existing .log.fmt, a predefined class</span>
<span class="sd">                might be given with `fmt={"class":"syslog"}`.  You might</span>
<span class="sd">                even add a fixated `{"record":"%a %t %e %F"}` format string</span>
<span class="sd">                this way.</span>
<span class="sd">            debug (bool): In case of log extraction failures, prints (stdout)</span>
<span class="sd">                some regex debugging.</span>
<span class="sd">            fail (bool): In case of failure, just error out instead of continuing</span>
<span class="sd">                the iterator.</span>
<span class="sd">            duplicate (bool): Automatically expand aliases.  This effectively</span>
<span class="sd">                copies row entries.</span>

<span class="sd">        Attributes:</span>
<span class="sd">            f (file): Read handle onto log file</span>
<span class="sd">            debug (bool): Debug flag</span>
<span class="sd">            fail (bool): Exception flag</span>
<span class="sd">            alias (dict): List of row aliases</span>
<span class="sd">            container (dict): Rules for field expansion</span>
<span class="sd">            rx (re.compile): Compiled regex</span>

<span class="sd">        Raises:</span>
<span class="sd">            StopIteration: For EOF or if the regex failed and fail=True</span>
<span class="sd">            FileNotFound: If logfn doesn't exist etc.</span>

<span class="sd">        Returns:</span>
<span class="sd">            iterator: Traverses log file line-wise, but yields dictionaries.</span>
<span class="sd">    """</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">debug</span> <span class="o">=</span> <span class="n">debug</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">fail</span> <span class="o">=</span> <span class="n">fail</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">duplicate</span> <span class="o">=</span> <span class="n">duplicate</span>
    <span class="c1"># try + nicer error....</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">logfn</span><span class="p">,</span> <span class="s2">"r"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"utf-8"</span><span class="p">)</span>
    <span class="k">if</span> <span class="ow">not</span> <span class="n">fmt</span><span class="p">:</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="n">fmt</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">logfn</span><span class="si">}</span><span class="s2">.fmt"</span><span class="p">,</span> <span class="s2">"r"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"utf-8"</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
        <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
            <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
            <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">"Use `update-logfmt` or `modseccfg`→File→Install→update_logfmt.py to generate a *.log.fmt descriptor.</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
            <span class="n">fmt</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"class"</span><span class="p">:</span><span class="s2">"apache combined"</span><span class="p">}</span>
            <span class="c1">#fmt = rulesdb.find_by_glob(logfn)</span>
    <span class="n">fmt</span> <span class="o">=</span> <span class="n">rulesdb</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span>
        <span class="n">fmt</span><span class="p">,</span>   <span class="c1"># this should be in regex/update</span>
        <span class="n">rulesdb</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">fmt</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"class"</span><span class="p">))</span>
    <span class="p">)</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">alias</span> <span class="o">=</span> <span class="n">fmt</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"alias"</span><span class="p">,</span> <span class="p">{})</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">container</span> <span class="o">=</span> <span class="n">fmt</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"container"</span><span class="p">,</span> <span class="p">{})</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">rx</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">rx2re</span><span class="p">(</span><span class="n">regex</span><span class="p">(</span><span class="n">fmt</span><span class="p">)))</span>
</code></pre>
</div>
</details>
</div>
</div>
<div class="doc doc-object doc-method">
<h2 class="doc doc-heading" id="logfmt1.logopen.debug_rx">

        <code class="highlight language-python">
debug_rx<span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>        </code>


      </h2>
<div class="doc doc-contents ">
<p>ANSI output for debugging regex/fmt string. Automatically
invoked for failing lines if <code>debug=True</code> was given.</p>
<p><img alt="failed regex + log line" src="https://imgur.com/QBKzDsK.png" /></p>
<p><strong>Parameters:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>line</code></td>
<td><code>str</code></td>
<td>
<p>Current raw line (string) from log file.</p>
</td>
<td><em>required</em></td>
</tr>
</tbody>
</table>
<p>Output:
  Prints directly to stdout using ANSI escape sequences to
  highlight where regex failed on input line. It's not very
  exact anymore, but reasonably speedy.</p>
<details class="quote">
          <summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
          <div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="nf">debug_rx</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">line</span><span class="p">:</span><span class="nb">str</span><span class="p">):</span>
    <span class="sd">"""</span>
<span class="sd">        ANSI output for debugging regex/fmt string. Automatically</span>
<span class="sd">        invoked for failing lines if `debug=True` was given.</span>

<span class="sd">        ![failed regex + log line](https://imgur.com/QBKzDsK.png)</span>

<span class="sd">        Args:</span>
<span class="sd">          line: Current raw line (string) from log file.</span>
<span class="sd">        Output:</span>
<span class="sd">          Prints directly to stdout using ANSI escape sequences to</span>
<span class="sd">          highlight where regex failed on input line. It's not very</span>
<span class="sd">          exact anymore, but reasonably speedy.</span>
<span class="sd">    """</span>

    <span class="n">rx</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rx</span><span class="o">.</span><span class="n">pattern</span>
    <span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span>
    <span class="c1">#rx_cut = re.compile("[^)]*  \(\?P&lt;\w+&gt;  ( [^()]+ | \([^()]+\) )+  \)  [^()]* \Z", re.X)</span>
    <span class="c1"># iteratively strip (?...) capture groups</span>
    <span class="k">while</span> <span class="nb">len</span><span class="p">(</span><span class="n">rx</span><span class="p">)</span> <span class="ow">and</span> <span class="n">rx</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s2">"(?P&lt;"</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="mi">0</span><span class="p">:</span>
        <span class="c1">#fail = rx_cut.search(rx)</span>
        <span class="c1">#if fail: fail = fail.group(0)</span>
        <span class="c1">#else: fail = "&lt;unknown-last-capture&gt;"; break</span>
        <span class="n">last</span> <span class="o">=</span> <span class="n">rx</span><span class="o">.</span><span class="n">rindex</span><span class="p">(</span><span class="s2">"(?P&lt;"</span><span class="p">)</span>
        <span class="k">if</span> <span class="n">last</span> <span class="o">&lt;</span> <span class="mi">1</span><span class="p">:</span>
            <span class="n">fail</span> <span class="o">=</span> <span class="s2">"&lt;unknown-last-capture&gt;"</span><span class="p">;</span> <span class="k">break</span>
        <span class="n">fail</span> <span class="o">=</span> <span class="n">rx</span><span class="p">[</span><span class="n">last</span><span class="p">:]</span>
        <span class="c1">#print(f"testfail: `{fail}`")</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="n">rx</span> <span class="o">=</span> <span class="n">rx</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">last</span><span class="p">]</span>
            <span class="n">rx</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s2">"[^)]*$"</span><span class="p">,</span> <span class="s2">""</span><span class="p">,</span> <span class="n">rx</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span>
                <span class="k">break</span> <span class="c1"># works now, so `fail` was the culprit</span>
        <span class="k">except</span><span class="p">:</span>
            <span class="c1"># likely broke regex nesting, try removing next (?...)</span>
            <span class="k">pass</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="n">matched</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
        <span class="n">matched</span> <span class="o">=</span> <span class="n">matched</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
    <span class="k">except</span><span class="p">:</span>
        <span class="n">matched</span> <span class="o">=</span> <span class="s2">""</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">"</span><span class="se">\033</span><span class="s2">[36m"</span> <span class="o">+</span> <span class="s2">"failed regex section: </span><span class="se">\033</span><span class="s2">[1;33;41m"</span> <span class="o">+</span> <span class="n">fail</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\033</span><span class="s2">[40;0m"</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">"</span><span class="se">\033</span><span class="s2">[42m"</span> <span class="o">+</span> <span class="n">matched</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\033</span><span class="s2">[41m"</span> <span class="o">+</span> <span class="n">line</span><span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">matched</span><span class="p">):]</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\033</span><span class="s2">[40;0m"</span><span class="p">)</span>
</code></pre>
</div>
</details>
</div>
</div>
<div class="doc doc-object doc-method">
<h2 class="doc doc-heading" id="logfmt1.logopen.names">

        <code class="highlight language-python">
names<span class="p">(</span><span class="bp">self</span><span class="p">)</span>        </code>


      </h2>
<div class="doc doc-contents ">
<p>Get column names from generated .fmt regex.</p>
<p><strong>Returns:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>list</code></td>
<td>
<p>dictionary keys of row (without aliases).</p>
</td>
</tr>
</tbody>
</table>
<details class="quote">
          <summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
          <div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="nf">names</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
    <span class="sd">"""</span>
<span class="sd">        Get column names from generated .fmt regex.</span>

<span class="sd">        Returns:</span>
<span class="sd">          list: dictionary keys of row (without aliases).</span>
<span class="sd">    """</span>
    <span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s2">"\(\?P?&lt;(\w+)&gt;"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">rx</span><span class="o">.</span><span class="n">pattern</span><span class="p">)</span>
</code></pre>
</div>
</details>
</div>
</div>
</div>
</div>
</div></div>
            </div>
        </div>

        <footer class="col-md-12">
            <hr>
            <p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
        </footer>
        <script>
            var base_url = ".",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
        <script src="js/base.js" defer></script>

        <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
    <div class="modal-dialog">
        <div class="modal-content">
            <div class="modal-header">
                <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
            </div>
            <div class="modal-body">
              <table class="table">
                <thead>
                  <tr>
                    <th style="width: 20%;">Keys</th>
                    <th>Action</th>
                  </tr>
                </thead>
                <tbody>
                  <tr>
                    <td class="help shortcut"><kbd>?</kbd></td>
                    <td>Open this help</td>
                  </tr>
                  <tr>
                    <td class="next shortcut"><kbd>n</kbd></td>
                    <td>Next page</td>
                  </tr>
                  <tr>
                    <td class="prev shortcut"><kbd>p</kbd></td>
                    <td>Previous page</td>
                  </tr>
                  <tr>
                    <td class="search shortcut"><kbd>s</kbd></td>
                    <td>Search</td>
                  </tr>
                </tbody>
              </table>
            </div>
            <div class="modal-footer">
            </div>
        </div>
    </div>
</div>

    </body>
</html>

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        
        
        
        <link rel="shortcut icon" href="img/favicon.ico">
        <title>update-logmt - logfmt1</title>
        <link href="css/bootstrap.min.css" rel="stylesheet">
        <link href="css/font-awesome.min.css" rel="stylesheet">
        <link href="css/base.css" rel="stylesheet">
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css">
        <link href="custom.css" rel="stylesheet">

        <script src="js/jquery-1.10.2.min.js" defer></script>
        <script src="js/bootstrap.min.js" defer></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
        <script>hljs.initHighlightingOnLoad();</script> 
    </head>

    <body>
        <div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
            <div class="container">
                <a class="navbar-brand" href=".">logfmt1</a>
                <!-- Expander button -->
                <button type="button" class="navbar-toggler" data-toggle="collapse" data-target="#navbar-collapse">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <!-- Expanded navigation -->
                <div id="navbar-collapse" class="navbar-collapse collapse">
                        <!-- Main navigation -->
                        <ul class="nav navbar-nav">
                            <li class="navitem">
                                <a href="index.html" class="nav-link">Home</a>
                            </li>
                            <li class="navitem">
                                <a href="log.fmt.html" class="nav-link">.log.fmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logopen.html" class="nav-link">logopen</a>
                            </li>
                            <li class="navitem">
                                <a href="update-regex.html" class="nav-link">regex</a>
                            </li>
                            <li class="navitem">
                                <a href="fmt.html" class="nav-link">global .fmt db</a>
                            </li>
                            <li class="navitem active">
                                <a href="update-logfmt.html" class="nav-link">update-logmt</a>
                            </li>
                            <li class="navitem">
                                <a href="logex.html" class="nav-link">logex</a>
                            </li>
                        </ul>

                    <ul class="nav navbar-nav ml-auto">
                            <li class="nav-item">
                                <a rel="prev" href="fmt.html" class="nav-link">
                                    <i class="fa fa-arrow-left"></i> Previous
                                </a>
                            </li>
                            <li class="nav-item">
                                <a rel="next" href="logex.html" class="nav-link">
                                    Next <i class="fa fa-arrow-right"></i>
                                </a>
                            </li>
                            <li class="nav-item">
                                <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1" class="nav-link">Fossil</a>
                            </li>
                    </ul>
                </div>
            </div>
        </div>

        <div class="container">
            <div class="row">
                    <div class="col-md-3"><div class="navbar-light navbar-expand-md bs-sidebar hidden-print affix" role="complementary">
    <div class="navbar-header">
        <button type="button" class="navbar-toggler collapsed" data-toggle="collapse" data-target="#toc-collapse" title="Table of Contents">
            <span class="fa fa-angle-down"></span>
        </button>
    </div>

    
    <div id="toc-collapse" class="navbar-collapse collapse card bg-secondary">
        <ul class="nav flex-column">
            
            <li class="nav-item" data-level="2"><a href="#update-logfmt" class="nav-link">update-logfmt</a>
              <ul class="nav flex-column">
              </ul>
            </li>
            
            <li class="nav-item" data-level="2"><a href="#scripts" class="nav-link">scripts</a>
              <ul class="nav flex-column">
              </ul>
            </li>
        </ul>
    </div>
</div></div>
                    <div class="col-md-9" role="main">

<h2 id="update-logfmt">update-logfmt</h2>
<p>The <code>update-logfmt</code> script (should be in in /usr/bin/ when installed via
system package) will create <a href="log.fmt.html">.log.fmt</a> files for known system
logs.</p>
<p>It's basically just a wrapper script that invokes handlers in
<code>/usr/share/logfmt/update/*</code>. Each of which scans one application
config to locate and decorate its according log files.</p>
<h3 id="args">args</h3>
<table>
<thead>
<tr>
<th>param</th>
<th>desc</th>
</tr>
</thead>
<tbody>
<tr>
<td>--test</td>
<td>print script names which would run, but don't run them.</td>
</tr>
<tr>
<td>--verbose</td>
<td>print script names before running them.</td>
</tr>
</tbody>
</table>
<h2 id="scripts">scripts</h2>
<p>Currently just supports:</p>
<ul>
<li>Apache (access, global, error and transfer logs, no forensic logs)</li>
<li>Nginx (only accesss logs)</li>
</ul>
<p>More static system logs (klog/syslog) might be supported in the next
version.</p></div>
            </div>
        </div>

        <footer class="col-md-12">
            <hr>
            <p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
        </footer>
        <script>
            var base_url = ".",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
        <script src="js/base.js" defer></script>

        <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
    <div class="modal-dialog">
        <div class="modal-content">
            <div class="modal-header">
                <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
            </div>
            <div class="modal-body">
              <table class="table">
                <thead>
                  <tr>
                    <th style="width: 20%;">Keys</th>
                    <th>Action</th>
                  </tr>
                </thead>
                <tbody>
                  <tr>
                    <td class="help shortcut"><kbd>?</kbd></td>
                    <td>Open this help</td>
                  </tr>
                  <tr>
                    <td class="next shortcut"><kbd>n</kbd></td>
                    <td>Next page</td>
                  </tr>
                  <tr>
                    <td class="prev shortcut"><kbd>p</kbd></td>
                    <td>Previous page</td>
                  </tr>
                  <tr>
                    <td class="search shortcut"><kbd>s</kbd></td>
                    <td>Search</td>
                  </tr>
                </tbody>
              </table>
            </div>
            <div class="modal-footer">
            </div>
        </div>
    </div>
</div>

    </body>
</html>

<!DOCTYPE html>

<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1.0" name="viewport"/>
<link href="img/favicon.ico" rel="shortcut icon"/>
<title>regex - logfmt1</title>
<link href="css/bootstrap.min.css" rel="stylesheet"/>
<link href="css/font-awesome.min.css" rel="stylesheet"/>
<link href="css/base.css" rel="stylesheet"/>
<link href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/color-brewer.min.css" rel="stylesheet"/>
<link href="custom.css" rel="stylesheet"/>
<script defer="" src="js/jquery-1.10.2.min.js"></script>
<script defer="" src="js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
</head>
<body>
<div class="navbar fixed-top navbar-expand-lg navbar-dark bg-primary">
<div class="container">
<a class="navbar-brand" href=".">logfmt1</a>
<!-- Expander button -->
<button class="navbar-toggler" data-target="#navbar-collapse" data-toggle="collapse" type="button">
<span class="navbar-toggler-icon"></span>
</button>
<!-- Expanded navigation -->
<div class="navbar-collapse collapse" id="navbar-collapse">
<!-- Main navigation -->
<ul class="nav navbar-nav">
<li class="navitem">
<a class="nav-link" href="index.html">Home</a>
</li>
<li class="navitem">
<a class="nav-link" href="log.fmt.html">.log.fmt</a>
</li>
<li class="navitem">
<a class="nav-link" href="logopen.html">logopen</a>
</li>
<li class="navitem active">
<a class="nav-link" href="update-regex.html">regex</a>
</li>
<li class="navitem">
<a class="nav-link" href="fmt.html">global .fmt db</a>
</li>
<li class="navitem">
<a class="nav-link" href="update-logfmt.html">update-logmt</a>
</li>
<li class="navitem">
<a class="nav-link" href="logex.html">logex</a>
</li>
</ul>
<ul class="nav navbar-nav ml-auto">
<li class="nav-item">
<a class="nav-link" href="logopen.html" rel="prev">
<i class="fa fa-arrow-left"></i> Previous
                                </a>
</li>
<li class="nav-item">
<a class="nav-link" href="fmt.html" rel="next">
                                    Next <i class="fa fa-arrow-right"></i>
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="https://fossil.include-once.org/modseccfg/wiki/logfmt1">Fossil</a>
</li>
</ul>
</div>
</div>
</div>
<div class="container">
<div class="row">
<div class="col-md-3"><div class="navbar-light navbar-expand-md bs-sidebar hidden-print affix" role="complementary">
<div class="navbar-header">
<button class="navbar-toggler collapsed" data-target="#toc-collapse" data-toggle="collapse" title="Table of Contents" type="button">
<span class="fa fa-angle-down"></span>
</button>
</div>
<div class="navbar-collapse collapse card bg-secondary" id="toc-collapse">
<ul class="nav flex-column">
<li class="nav-item" data-level="2"><a class="nav-link" href="#logfmt1.regex">logfmt1.regex</a>
<ul class="nav flex-column">
</ul>
</li>
<li class="nav-item" data-level="2"><a class="nav-link" href="#logfmt1.rx2re">logfmt1.rx2re</a>
<ul class="nav flex-column">
</ul>
</li>
<li class="nav-item" data-level="2"><a class="nav-link" href="#logfmt1.update">logfmt1.update</a>
<ul class="nav flex-column">
</ul>
</li>
</ul>
</div>
</div></div>
<div class="col-md-9" role="main">
<p><a href="logopen">logopen()</a> does this internally, but you might want to manually
craft the <code>"regex":</code> for a .fmt descriptor.  In such cases,
you can use <code>regex()</code> or <code>update()</code> to combine the log.fmt with definitions
from the global fmt database.</p>
<h3 id="regex">regex()</h3>
<p>Takes a .fmt json/dict, and generates the regex with named capture groups
from it.</p>
<div class="doc doc-object doc-function">
<h2 class="hidden-toc" href="#logfmt1.regex" id="logfmt1.regex" style="visibility: hidden; position: absolute;">
</h2>
<div class="doc doc-contents first">
<p>Create regex for log fmt{}.</p>
<p><strong>Parameters:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>fmt</code></td>
<td><code>dict</code></td>
<td>
<p>Should contain record: and class:, but may define
custom fields or aliases.</p>
</td>
<td><em>required</em></td>
</tr>
<tr>
<td><code>update</code></td>
<td><code>bool</code></td>
<td>
<p>Inject fields and other declarations from shared
.fmt database into <code>fmt</code> dict.</p>
</td>
<td><code>False</code></td>
</tr>
</tbody>
</table>
<p><strong>Returns:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>str</code></td>
<td>
<p>Combined regex, for example <code>(?&lt;remote_host&gt;[\w\-.:]+)
     (?&lt;remote_logname&gt;[\w\-.:]+) (?&lt;remote_user&gt;[\-\w@.]+)
     \[?(?&lt;request_time&gt;\d[\d:\w\s:./\-+,;]+)\]? 
     "(?&lt;request_line&gt;(?&lt;request_method&gt;\w+) …␣…)"…</code></p>
</td>
</tr>
</tbody>
</table>
<details class="quote">
<summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
<div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="nf">regex</span><span class="p">(</span><span class="n">fmt</span><span class="p">,</span> <span class="n">update</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
    <span class="sd">"""</span>
<span class="sd">        Create regex for log fmt{}.</span>

<span class="sd">        Args:</span>
<span class="sd">            fmt (dict): Should contain record: and class:, but may define</span>
<span class="sd">                custom fields or aliases.</span>

<span class="sd">            update (bool: Inject fields and other declarations from shared</span>
<span class="sd">                .fmt database into `fmt` dict.</span>

<span class="sd">        Returns:</span>
<span class="sd">          str: Combined regex, for example `(?&lt;remote_host&gt;[\\w\\-.:]+)</span>
<span class="sd">               (?&lt;remote_logname&gt;[\\w\\-.:]+) (?&lt;remote_user&gt;[\\-\\w@.]+)</span>
<span class="sd">               \\[?(?&lt;request_time&gt;\\d[\\d:\\w\\s:./\\-+,;]+)\\]? </span>
<span class="sd">               "(?&lt;request_line&gt;(?&lt;request_method&gt;\\w+) …␣…)"…`</span>
<span class="sd">    """</span>

    <span class="n">rules</span> <span class="o">=</span> <span class="n">rulesdb</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span>
        <span class="n">fmt</span><span class="p">,</span>
        <span class="n">rulesdb</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">fmt</span><span class="p">[</span><span class="s2">"class"</span><span class="p">])</span>
    <span class="p">)</span>
    <span class="n">fields</span> <span class="o">=</span> <span class="n">rules</span><span class="p">[</span><span class="s2">"fields"</span><span class="p">]</span>
    <span class="n">record</span> <span class="o">=</span> <span class="n">fmt</span><span class="p">[</span><span class="s2">"record"</span><span class="p">]</span>
    <span class="k">if</span> <span class="n">update</span><span class="p">:</span>
        <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">"rewrite"</span><span class="p">,</span> <span class="s2">"fields"</span><span class="p">,</span> <span class="s2">"expand"</span><span class="p">,</span> <span class="s2">"alias"</span><span class="p">,</span> <span class="s2">"container"</span><span class="p">]:</span>
            <span class="k">if</span> <span class="ow">not</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">fmt</span><span class="p">:</span>
                <span class="n">fmt</span><span class="p">[</span><span class="n">field</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>

    <span class="c1"># pre-cleanup (for irrelevant format string `%!200,500&lt;s` control prefixes)</span>
    <span class="k">if</span> <span class="s2">"rewrite"</span> <span class="ow">in</span> <span class="n">rules</span><span class="p">:</span>
        <span class="k">for</span> <span class="n">rx</span><span class="p">,</span> <span class="n">repl</span> <span class="ow">in</span> <span class="n">rules</span><span class="p">[</span><span class="s2">"rewrite"</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
            <span class="n">record</span> <span class="o">=</span> <span class="n">rx_sub</span><span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">repl</span><span class="p">,</span> <span class="n">record</span><span class="p">)</span>

    <span class="c1"># create fields from variant placeholders</span>
    <span class="k">if</span> <span class="s2">"expand"</span> <span class="ow">in</span> <span class="n">rules</span><span class="p">:</span>
        <span class="n">rx_quote_alt</span> <span class="o">=</span> <span class="p">{</span> <span class="c1"># (is_quoted + rx) can map to alternative regex</span>
            <span class="p">(</span><span class="kc">False</span><span class="p">,</span>  <span class="s1">'[^"]*'</span><span class="p">):</span>  <span class="s2">"\S*"</span><span class="p">,</span>
            <span class="p">(</span><span class="kc">True</span><span class="p">,</span>   <span class="s2">"\S+"</span><span class="p">):</span>    <span class="s2">"(?:[^</span><span class="se">\"</span><span class="s2">]*|</span><span class="se">\\\\\"</span><span class="s2">)+"</span><span class="p">,</span>
        <span class="p">}</span>
        <span class="k">for</span> <span class="n">rx</span><span class="p">,</span> <span class="n">expand</span> <span class="ow">in</span> <span class="n">rules</span><span class="p">[</span><span class="s2">"expand"</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
            <span class="k">for</span> <span class="n">is_quoted</span><span class="p">,</span> <span class="n">match</span><span class="p">,</span> <span class="o">*</span><span class="n">uu</span> <span class="ow">in</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="sa">f</span><span class="s2">"(</span><span class="se">\"</span><span class="s2">?)(</span><span class="si">{</span><span class="n">rx</span><span class="si">}</span><span class="s2">)"</span><span class="p">,</span> <span class="n">record</span><span class="p">):</span>
                <span class="k">if</span> <span class="n">match</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">:</span>
                    <span class="k">continue</span>
                <span class="n">x</span> <span class="o">=</span> <span class="n">copy</span><span class="p">(</span><span class="n">expand</span><span class="p">)</span>
                <span class="c1"># id: is usually "$1", but might be "prefix_$2" or something</span>
                <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s2">"id"</span><span class="p">]</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'$'</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="mi">0</span><span class="p">:</span>
                    <span class="n">x</span><span class="p">[</span><span class="s2">"id"</span><span class="p">]</span> <span class="o">=</span> <span class="n">rx_sub</span><span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">x</span><span class="p">[</span><span class="s2">"id"</span><span class="p">],</span> <span class="n">match</span><span class="p">)</span>
                    <span class="n">x</span><span class="p">[</span><span class="s2">"id"</span><span class="p">]</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s2">"\W+"</span><span class="p">,</span> <span class="s2">""</span><span class="p">,</span> <span class="n">x</span><span class="p">[</span><span class="s2">"id"</span><span class="p">])</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
                <span class="c1"># recurse into other pattern types</span>
                <span class="k">if</span> <span class="ow">not</span> <span class="s2">"rx"</span> <span class="ow">in</span> <span class="n">x</span> <span class="ow">and</span> <span class="s2">"class"</span> <span class="ow">in</span> <span class="n">x</span><span class="p">:</span>
                    <span class="n">x</span><span class="p">[</span><span class="s2">"rx"</span><span class="p">]</span> <span class="o">=</span> <span class="n">regex</span><span class="p">({</span>
                        <span class="s2">"class"</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s2">"class"</span><span class="p">],</span>
                        <span class="s2">"record"</span><span class="p">:</span> <span class="n">rx_sub</span><span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"record"</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">"$1"</span><span class="p">,</span> <span class="n">match</span><span class="p">)</span>
                    <span class="p">})</span>
                <span class="c1"># regex alternatives, depending on quotes preceeding the placeholder</span>
                <span class="k">if</span> <span class="n">is_quoted</span> <span class="ow">and</span> <span class="s2">"if_quoted"</span> <span class="ow">in</span> <span class="n">x</span><span class="p">:</span>  <span class="c1"># maybe an `if_quoted` definition makes this block redundant</span>
                    <span class="n">x</span><span class="p">[</span><span class="s2">"rx"</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s2">"if_quoted"</span><span class="p">]</span>
                <span class="k">elif</span> <span class="p">(</span><span class="n">is_quoted</span><span class="p">,</span> <span class="n">x</span><span class="p">[</span><span class="s2">"rx"</span><span class="p">])</span> <span class="ow">in</span> <span class="n">rx_quote_alt</span><span class="p">:</span>
                    <span class="n">x</span><span class="p">[</span><span class="s2">"rx"</span><span class="p">]</span> <span class="o">=</span> <span class="n">rx_quote_alt</span><span class="p">[(</span><span class="n">is_quoted</span><span class="p">,</span> <span class="n">x</span><span class="p">[</span><span class="s2">"rx"</span><span class="p">])]</span>
                <span class="n">fields</span><span class="p">[</span><span class="n">match</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>

    <span class="c1"># catch-all \S+ for completely unknown placeholders</span>
    <span class="k">if</span> <span class="s2">"placeholder"</span> <span class="ow">in</span> <span class="n">rules</span><span class="p">:</span>
        <span class="k">for</span> <span class="n">ph</span> <span class="ow">in</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="n">rules</span><span class="p">[</span><span class="s2">"placeholder"</span><span class="p">],</span> <span class="n">record</span><span class="p">):</span>
            <span class="k">if</span> <span class="ow">not</span> <span class="n">ph</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">:</span>
                <span class="nb">id</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s2">"\W+"</span><span class="p">,</span> <span class="s2">""</span><span class="p">,</span> <span class="n">ph</span><span class="p">)</span>
                <span class="n">fields</span><span class="p">[</span><span class="n">ph</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span> <span class="s2">"id"</span><span class="p">:</span> <span class="nb">id</span><span class="p">,</span> <span class="s2">"rx"</span><span class="p">:</span> <span class="s2">"\S+"</span> <span class="p">}</span>

    <span class="c1"># do the actual replacement</span>
    <span class="k">def</span> <span class="nf">sub_placeholder</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
        <span class="n">ph</span> <span class="o">=</span> <span class="n">fields</span><span class="p">[</span><span class="n">m</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)]</span>
        <span class="k">if</span> <span class="n">update</span><span class="p">:</span>
            <span class="n">fmt</span><span class="p">[</span><span class="s2">"fields"</span><span class="p">][</span><span class="n">m</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)]</span> <span class="o">=</span> <span class="n">ph</span>  <span class="c1"># store used placeholders in fmt</span>
        <span class="n">rx</span> <span class="o">=</span> <span class="n">ph</span><span class="p">[</span><span class="s2">"rx"</span><span class="p">]</span>
        <span class="nb">id</span> <span class="o">=</span> <span class="n">ph</span><span class="p">[</span><span class="s2">"id"</span><span class="p">]</span>
        <span class="c1"># check for existing (…) capture group to mark up</span>
        <span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="s2">"(?&lt;!</span><span class="se">\\\\</span><span class="s2">)\((?!\?)"</span><span class="p">,</span> <span class="n">rx</span><span class="p">):</span>
            <span class="n">rx</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s2">"(?&lt;!</span><span class="se">\\\\</span><span class="s2">)\((?!\?)"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"(?&lt;</span><span class="si">{</span><span class="nb">id</span><span class="si">}</span><span class="s2">&gt;"</span><span class="p">,</span> <span class="n">rx</span><span class="p">,</span> <span class="n">re</span><span class="o">.</span><span class="n">M</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="n">rx</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"(?&lt;</span><span class="si">{</span><span class="nb">id</span><span class="si">}</span><span class="s2">&gt;</span><span class="si">{</span><span class="n">rx</span><span class="si">}</span><span class="s2">)"</span>
        <span class="k">return</span> <span class="n">rx</span>
    <span class="n">rx</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="n">rules</span><span class="p">[</span><span class="s2">"placeholder"</span><span class="p">],</span> <span class="n">sub_placeholder</span><span class="p">,</span> <span class="n">record</span><span class="p">)</span>
    <span class="n">rx</span> <span class="o">=</span> <span class="n">rename_duplicates</span><span class="p">(</span><span class="n">rx</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">rx</span>
</code></pre>
</div>
</details>
</div>
</div>
<hr/>
<h3 id="rx2re">rx2re()</h3>
<div class="doc doc-object doc-function">
<h2 class="hidden-toc" href="#logfmt1.rx2re" id="logfmt1.rx2re" style="visibility: hidden; position: absolute;">
</h2>
<div class="doc doc-contents first">
<p>Convert generic <code>(?&lt;name&gt;…)</code> to Python <code>(?P&lt;name&gt;…)</code> regex capture group.
(logfmt1 definitions use standard syntax per default.)</p>
<p><strong>Parameters:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>rx</code></td>
<td><code>str</code></td>
<td>
<p>Generic/PCRE regex syntax</p>
</td>
<td><em>required</em></td>
</tr>
</tbody>
</table>
<p><strong>Returns:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>str</code></td>
<td>
<p>Python re syntax</p>
</td>
</tr>
</tbody>
</table>
<details class="quote">
<summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
<div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="nf">rx2re</span><span class="p">(</span><span class="n">rx</span><span class="p">):</span>
    <span class="sd">"""</span>
<span class="sd">        Convert generic `(?&lt;name&gt;…)` to Python `(?P&lt;name&gt;…)` regex capture group.</span>
<span class="sd">        (logfmt1 definitions use standard syntax per default.)</span>

<span class="sd">        Args:</span>
<span class="sd">            rx (str): Generic/PCRE regex syntax</span>

<span class="sd">        Returns:</span>
<span class="sd">            str: Python re syntax</span>
<span class="sd">    """</span>
    <span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s2">"\(\?&lt;(?=\w+&gt;)"</span><span class="p">,</span> <span class="s2">"(?P&lt;"</span><span class="p">,</span> <span class="n">rx</span><span class="p">)</span>
</code></pre>
</div>
</details>
</div>
</div>
<hr/>
<h3 id="update">update()</h3>
<p>Updates the <code>regex:</code> property in a .fmt dict. Also injects any used
<code>fields:</code> or <code>alias:</code> lists. (For now this is basically just a wrapper
of the <code>regex()</code> builder.)</p>
<div class="doc doc-object doc-function">
<h2 class="hidden-toc" href="#logfmt1.update" id="logfmt1.update" style="visibility: hidden; position: absolute;">
</h2>
<div class="doc doc-contents first">
<p>should be the other way round: regex() is meant to be a subset of update() </p>
<details class="quote">
<summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
<div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="n">fmt</span><span class="p">):</span>
    <span class="sd">""" should be the other way round: regex() is meant to be a subset of update() """</span>
    <span class="n">fmt</span><span class="p">[</span><span class="s2">"regex"</span><span class="p">]</span> <span class="o">=</span> <span class="n">regex</span><span class="p">(</span><span class="n">fmt</span><span class="p">,</span> <span class="n">update</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</code></pre>
</div>
</details>
</div>
</div></div>
</div>
</div>
<footer class="col-md-12">
<hr/>
<p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a>.</p>
</footer>
<script>
            var base_url = ".",
                shortcuts = {"help": 191, "next": 78, "previous": 80, "search": 83};
        </script>
<script defer="" src="js/base.js"></script>
<div aria-hidden="true" aria-labelledby="keyboardModalLabel" class="modal" id="mkdocs_keyboard_modal" role="dialog" tabindex="-1">
<div class="modal-dialog">
<div class="modal-content">
<div class="modal-header">
<h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
<button class="close" data-dismiss="modal" type="button"><span aria-hidden="true">×</span><span class="sr-only">Close</span></button>
</div>
<div class="modal-body">
<table class="table">
<thead>
<tr>
<th style="width: 20%;">Keys</th>
<th>Action</th>
</tr>
</thead>
<tbody>
<tr>
<td class="help shortcut"><kbd>?</kbd></td>
<td>Open this help</td>
</tr>
<tr>
<td class="next shortcut"><kbd>n</kbd></td>
<td>Next page</td>
</tr>
<tr>
<td class="prev shortcut"><kbd>p</kbd></td>
<td>Previous page</td>
</tr>
<tr>
<td class="search shortcut"><kbd>s</kbd></td>
<td>Search</td>
</tr>
</tbody>
</table>
</div>
<div class="modal-footer">
</div>
</div>
</div>
</div>
</body>
</html>

# alongside (generate with `update-logfmt`).
#
# Syntax:
#   logex.py /var/log/apache2/access.log  request_path  request_time  @host
#
# Other args:
#   --json / --tab / --csv
#   --iso8601 / --debug / --rx
#
# Where any @'s are decoration, and fields can be supplied as individual
# arguments (become space-separated without --tab/--csv). Field names are
# application-type specific (internal) names. (E.g. @request_method, @host
# or @tm_wday for Apache logs. With some predefined aliases, e.g. the w3c
# extended log field names.)
#

if "--tab" in argv:
    space = "\t"
if "--csv" in argv:
    space = "," 
iso8601 = any(a in argv for a in ("--iso", "--iso8601", "--date", "--fixdates", "--die-apachedateformat-die"))
as_json = any(a in argv for a in ("--json", "--asjson", "--as-json"))
dodebug = any(a in argv for a in ("--debug", "-D"))
only_rx = any(a in argv for a in ("--regex", "--rx"))
# remove --params
argv = [a for a in argv if not re.match("^--\w+$|^-\w$", a)]
# filename and field list
try:
    log_fn = argv[1]
except:
    sys.stderr.write("logex: no filename given\n")
    sys.exit(errno.ENOENT)
output_fields = space.join(argv[2:])


#-- open log file
try:
    reader = logfmt1.logopen(log_fn, debug=dodebug, duplicate=False)
    #if dodebug:
    #    sys.stdout.write(json.dumps(reader.__dict__, indent=2, default=lambda x:str(x))+"\n")
    if only_rx:
        sys.stdout.write(reader.rx.pattern)
        exit(0)
except Exception as e:
    sys.stderr.write(traceback.format_exc()+"\n")
    sys.stderr.write("Use `update-logfmt-apache` or modseccfg→File→Install→update_logfmt to generate a .fmt descriptor\n")
    sys.exit(errno.ENODATA)


# extra aliases (for apache/httpd)

# encoding: utf-8
# api: python
# title: python3-logfmt1
# description: handle *.log.fmt specifiers and regex conversion
# type: transform
# category: io
# version: 0.5.3
# license: Apache-2.0
# pack:
#    logfmt1.py=/usr/lib/python3/dist-packages/
#    update_logfmt.py=/usr/bin/update-logfmt
#    ./logex.py=/usr/bin/logex
#    share=/usr/share/logfmt
# architecture: all

import re, json, os, sys
from copy import copy


class rulesdb:
    """
     Lookup table for known format strings and field identifiers.
     This class will also `.get("appname variant")` definitions from
     /usr/share/logfmt/*.fmt files, if they exist. Future versions
     might support prebuilt .grok or .lnav files as well.
     
     In the case of Apache logs, this mixes both accesslog and errorlog
     format strings, albeit they probably should be split out perhaps,
     as `%{cu}t` won't work in access.logs. It's both close enough
     to make one set of placeholders work, but requires aliasing at least.
    
    """
    # - "[client %s:%d]" : "[remote %s:%d]" only in errorlogdefault?

    apache = {
        "class": "apache generic",

        #"record": "%h %l %u %t \"%r\" %>s %b",

        #"regex": "(?<remote_host>\S+) …",

                "id": "$1",
                "value": "$2$3",
                "class": "inilog"
            }
        },
    }
    

    @staticmethod
    def get(cls):
        """
            Return builtin definitions or from /usr/share/logfmt/*.*.fmt
        """
        rules = {}
        cls = cls.split(" ")
        while cls:
            lookup = ".".join(cls)
            lookup_ = "_".join(cls)
            add = {}
            dir = "/usr/share/logfmt"

    def extract_all(self):
        for key,val in rulesdb.__dict__.items():
            if isinstance(val, dict):
                open(f"share/{key}.fmt", "w").write(json.dumps(val, indent=4))
#rulesdb().extract_all()



def update(fmt):
    """ should be the other way round: regex() is meant to be a subset of update() """
    fmt["regex"] = regex(fmt, update=True)


def regex(fmt, update=False):
    """
        Create regex for log fmt{}.
        
        Args:
            fmt (dict): Should contain record: and class:, but may define
                custom fields or aliases.
          
            update (bool: Inject fields and other declarations from shared
                .fmt database into `fmt` dict.


        Returns:
          str: Combined regex, for example `(?<remote_host>[\\w\\-.:]+)
               (?<remote_logname>[\\w\\-.:]+) (?<remote_user>[\\-\\w@.]+)
               \\[?(?<request_time>\\d[\\d:\\w\\s:./\\-+,;]+)\\]? 
               "(?<request_line>(?<request_method>\\w+) …␣…)"…`
    """

    rules = rulesdb.merge(
        fmt,
        rulesdb.get(fmt["class"])
    )
    fields = rules["fields"]

        else:
            rx = f"(?<{id}>{rx})"
        return rx
    rx = re.sub(rules["placeholder"], sub_placeholder, record)
    rx = rename_duplicates(rx)
    return rx


def rename_duplicates(rx):
    """ Rename duplicate regex <?P<capt_groups>…) """
    fields = []
    def count_ph(m):
        s, i = m.group(1), 1
        while s in fields:
            i += 1
            s = f"{m.group(1)}{i}"
        fields.append(s)
        return s
    return re.sub("(?<=\(\?\<)(\w+)(?=\>)", count_ph, rx)


def rx2re(rx):
    """
        Convert generic `(?<name>…)` to Python `(?P<name>…)` regex capture group.
        (logfmt1 definitions use standard syntax per default.)
        
        Args:
            rx (str): Generic/PCRE regex syntax
          
        Returns:
            str: Python re syntax
    """
    return re.sub("\(\?<(?=\w+>)", "(?P<", rx)


def rx_sub(pattern, replacement, source, flags=0):
    """ allow for $1, $2, $3 in re.sub() """
    if replacement.find('$') >= 0:
        replacement = re.sub(r'[\\\\](?=[0-9])', '$', replacement)
    return re.sub(pattern, replacement, source, flags)


def repl_sub_dict(s, row):
    """ replace $0 $1 $2 in string with entries from list """
    return re.sub("\$(\d+)", lambda m: row.get(int(m.group(1))), s)



# log extraction 
class parsy_parse:
    """
        Open log file and its associated .fmt specifier, to iterate over log lines
        as dictionary.
        File-style wrapper that yields parsed dictionaries instead of string lines.
        
            for row in logfmt1.logopen("/var/log/apache2/access.log", debug=True):
                print(row["remote_host"])
        
        Though you might want to keep a reference to the iterator to utilize
        `.names()` and `.alias{}` manually.

    """

    def __init__(self, logfn="", fmt=None, debug=False, fail=False, duplicate=True):
        """
            Open log file and its associated .fmt specifier, to iterate over log lines
            as dictionary.
            
            Args:
                logfn (str):  Filename of .log file to open.  Which should be
                    accompanied by a .log.fmt declaration to allow unpacking
                    lines into dictionary.
                fmt (dict): Alternatively to existing .log.fmt, a predefined class
                    might be given with `fmt={"class":"syslog"}`.  You might
                    even add a fixated `{"record":"%a %t %e %F"}` format string
                    this way.
                debug (bool): In case of log extraction failures, prints (stdout)
                    some regex debugging.
                fail (bool): In case of failure, just error out instead of continuing
                    the iterator.
                duplicate (bool): Automatically expand aliases.  This effectively
                    copies row entries.

            Attributes:
                f (file): Read handle onto log file
                debug (bool): Debug flag
                fail (bool): Exception flag
                alias (dict): List of row aliases
                container (dict): Rules for field expansion
                rx (re.compile): Compiled regex

            Raises:
                StopIteration: For EOF or if the regex failed and fail=True
                FileNotFound: If logfn doesn't exist etc.

            Returns:
                iterator: Traverses log file line-wise, but yields dictionaries.
        """
        self.debug = debug
        self.fail = fail
        self.duplicate = duplicate
        # try + nicer error....
        self.f = open(logfn, "r", encoding="utf-8")
        if not fmt:

            fmt,   # this should be in regex/update
            rulesdb.get(fmt.get("class"))
        )
        self.alias = fmt.get("alias", {})
        self.container = fmt.get("container", {})
        self.rx = re.compile(rx2re(regex(fmt)))


    def __iter__(self):
        """ allow iterating multiple times? """
        self.f.seek(0, os.SEEK_SET)
        return self


    def __next__(self):
        """ iterate over lines, and unpack with regex and aliases """
        line = self.f.readline()
        if not line:  # should be implied really
            raise StopIteration()
        m = self.rx.match(line)
        if m:
            d = m.groupdict()
            if self.container:

            if self.fail:
                raise StopIteration()
        elif self.fail:
            raise StopIteration()
        else:
            pass # just try next line
    

    def __getattr__(self, name):
        """ pass .close() and similar to file object """
        return getattr(self.f, name)


    def names(self):
        """
            Get column names from generated .fmt regex.













            


            Returns:
              list: dictionary keys of row (without aliases).
        """
        return re.findall("\(\?P?<(\w+)>", self.rx.pattern)

    def debug_rx(self, line:str):
        """
            ANSI output for debugging regex/fmt string. Automatically
            invoked for failing lines if `debug=True` was given.
            
            ![failed regex + log line](https://imgur.com/QBKzDsK.png)
            
            Args:
              line: Current raw line (string) from log file.
            Output:
              Prints directly to stdout using ANSI escape sequences to
              highlight where regex failed on input line. It's not very
              exact anymore, but reasonably speedy.
        """

        rx = self.rx.pattern
        line = line.rstrip()
        #rx_cut = re.compile("[^)]*  \(\?P<\w+>  ( [^()]+ | \([^()]+\) )+  \)  [^()]* \Z", re.X)
        # iteratively strip (?...) capture groups
        while len(rx) and rx.find("(?P<") >= 0:
            #fail = rx_cut.search(rx)
            #if fail: fail = fail.group(0)

        try:
            matched = re.match(rx, line)
            matched = matched.group(0)
        except:
            matched = ""
        print("\033[36m" + "failed regex section: \033[1;33;41m" + fail + "\033[40;0m")
        print("\033[42m" + matched + "\033[41m" + line[len(matched):] + "\033[40;0m")

    def container_expand(self, d:dict):
        """
            Internal. Unpacks e.g. `[key "value"]` fields, if any `container:`
            regex was defined in the .fmt defiiution.
            
            Arguments:
              d: current log row
        """
        for k,opt in self.container.items():
            if not k in d:
                continue
            # find `(key)……(val+)` pairs according to regex
            for row in re.findall(opt["rx"], d[k]):
                id = repl_sub_dict(opt.get("id", "$1"), row)
                val = repl_sub_dict(opt.get("val", "$2"), row)
                # pack into list, if key is duplicated
                if not id in d:
                    d[id] = val
                elif not isinstance(d[id], list):
                    d[id] = [d[id], val]
                else:
                    d[id].append(val)

# alias
logopen = parsy_parse

site_name: logfmt1
docs_dir: docs
site_dir: html
nav:
    - Home: index.md
    - .log.fmt: log.fmt.md
    - logopen: logopen.md
    - regex: update-regex.md 
    - global .fmt db: fmt.md
    - update-logmt: update-logfmt.md
    - logex: logex.md
theme:
  name: materia
#  name: bootstrap386
#  name: ivory
#  name: material
#  name: readthedocs
repo_url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
markdown_extensions:
  - admonition
  - codehilite
#  - mkautodoc
plugins:
  - mkdocstrings
extra_css:
    - custom.css
use_directory_urls: false