scp python3-logfmt1* io:apt/
whl:
	pandoc logfmt1/README.md -o logfmt1/README.rst
	cd logfmt1 && ./setup.py bdist_wheel
whl_up:
	twine upload dist/logfmt*
docs:
	pygmentize -S pastie -f html > logfmt1/docs/syntax.css
	cd logfmt1 ; PYTHONPATH=. mkdocs build -v -v -v
	sed -i 's/table\.docutils/table/g' logfmt1/html/css/theme.css 
%.1:	%.md
	pandoc --standalone -f markdown+pandoc_title_block -t man $< -o $@
man:	logfmt1/manpage/logex.1 logfmt1/manpage/update-logfmt.1

!!! Warning
    ❮❗❯ This is all very provisional. (First draft. Names might still change.)


## Global .fmt database

While each log file should be accompanied by a [.fmt descriptor](log.fmt.md),
the global database in `/usr/share/logfmt/` contains a full .fmt field
definition for each class. And the cross-section of both allows to construct
a regex.

Most notably the `"fields":` and `"placeholder":` are used to turn the
`"record":` string definition into a capture pattern.


### .fmt Example

The Apache format definition (apache.fmt) contains:

```json
{
    "class": "apache generic",
    "separator": " ",
    "rewrite": {
        "%[\\d!,+\\-]+": "%",
        "%%": "%"
    },
    "placeholder": "%[<>]?(?:\\w*\\{[^\\}]+\\})?\\^?\\w+",
    "fields": {
        "%a": {  "id": "remote_addr",    "rx": "[\\d.:a-f]+"  },
        "%h": {  "id": "remote_host",    "rx": "[\\w\\-.:]+"  },
        "%{c}h": { "id": "remote_host",  "rx": "[\\w\\-.:]+"  },
        "%A": {  "id": "local_address",  "rx": "[\\d.:a-f]+"  },
        "%u": {  "id": "remote_user",    "rx": "[\\-\\w@.]+"  },
        "%t": {  "id": "request_time",   "rx": "\\[?(\\d[\\d:\\w\\s:./\\-+,;]+)\\]?"  },
        …
    },
    "alias": {
        "remote_address": "remote_addr",
        "ip": "remote_addr",
        "file": "request_file",
        "size": "bytes_sent",
        …
    },
    "expand": {
        "%\\{([^{}]+)\\}t": {
            "id": "request_time",
            "class": "strftime",
            "record": "$1"
        }
    },
    "container": {
        "message": {
            "id": "$1",
            "value": "$2",
            "rx": "\\[(\\w+) \"(.*?)\"\\]",
            "class": "apache mod_security"
        }
    },
    "glob": ["/var/log/apache*/*acc*.log"]
}
```

It usually does not describe a default "record" format (like the local .log.fmt descriptors do).


### class:

The class in the global database is largely decorative.  The filenames
instead define the heritage of rules/fields.  The "class" as declared by
a .log.fmt is mapped onto `/usr/share/logfmt/application.variant.fmt`.

 * Usually there's just one variant level per log type. But the lookup is
   supposed to be mildly recursive.
 * Essentially it should merge `*.log.fmt` with `appclass.variant.fmt` and
   `appclass.fmt` applied last, so the most specific definitions are retained.
 * There's also a generic "grok" class. But the patterns therein are largely
   static (not build from variable format strings).
 * Some special classes like "json" might exist. (Not supported by logfmt1)


### record:

The "record" entry is not usually present in the global .fmt definition. 
Some super specific variant definitions (for example apache.error.fmt) or
static formats (syslog.fmt) might however.


### separator:

Most log formats use spaces for separating %placeholder fields.  And simpler
implementations might just split up the "record" declaration on this.


### placeholder:

While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields. It should account for prefixes/suffixes, unless
those got cleared by the `rewrite` map.

Not all formatstrings use `%\w+` to signal placeholders. In nginx for instance
the sigil `$\w+` introduces placeholders (variable names, really).


### rewrite:

A list/map of regex to apply before any transformations or field lookups. 
Which can be used to mask or simplify placeholder definitions (for instance
clean up the Apache conditional prefixes) or regex meta characters.

 * The `record` field starts as a static string, but is meant to be turned
   into a regex.
 * Therefore meta characters (such as `|` or `[]`) have to be
   taken care of.  Which is what the `rewrite` map is lazily used for.
 * Better implementations might look up the placeholders, and automatically
   escape the rest of the the "record" format string.


### fields:

The core of the global .fmt definitions are the field lists.  Each defines a
static %F placeholder and associaties it with a default field name (id:) and
regex (rx:) or even a grok definition (grok:).

| key | purpose |
|-----|---------|
|`%F`| **JSON key**: static placeholder string (not a regex itself) |
| id | field identifier, as specified by the application (internal name) |
| rx | regex which %F placeholder gets replaced with |
| grok | alternatively to regex, %F might be turned into %PATTERN:id |
| type |  "int" and "float" could designate strictly numeric fields |

!!! Notes
    * As part of the regex transformation, a `%F` could be turned into
      `(?<id>\S+)` for instance.
    * If there's any unnamed capture group `(…)`, it should be augmented
      into a named capture group - instead of the whole match. (To account
      for implicit wrapping.)
    * The `rx` itself might however specify named subgroups (like request_line
      in Apache logs, itself comprised of _method, _path, _protocol, or the
      datetime made up of tm_wday, tm_year, tm_whatever).
    * `\S+` is also used as fallback for entirely undefined placeholders
      (no expand definition matched) in logfmt1.
    * `grok` isn't currently used, but might allow for simpler transformations
      (indirectly into a grok pattern, and later a regex).


### expand:

The expand declarations are used to construct unknown fields/placeholders. 
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.  Thus it simply can be applied before
separator/placeholder are looked at, to augment the known `fields` list.

| key      |  purpose                                               |
|----------|--------------------------------------------------------|
| `%\{(\w+)\}t` | **JSON key**: a regex to detect mutable placeholders  |
| id     | name for newly created fields entry, might use captures´ $1|
| rx     | for static definitions (often just \S+)                  |
|if_quoted| alternative regex, if placeholder was enclosed in "%\w+" quotes|
| class  | recurse into other .fmt types                            |
| record | can be set to $2 if class: recursion is defined          |


!!! Notes
    * Typically it suffices to specify the `id` and `rx` field.
    * If no `id` is given, then the regex capture is normalized into
      an identifier (non-alphanumerics stripped, all lowercased).
    * But the `id` or `record` value might be set with regex captures
      (e.g. `$1` or `$2`) or compound values (`"id": "newfield_$1"`).
    * And logfmt1 allows to recurse into other format types per `class`
      (which is used to expand the captured `"record": "$1"` into regex
      tokens).


### alias:

Maps alternative/more common field names onto the declared field `id`s.

To get to some state of standardization, the field ids usually refer
to application-internal names. (For instance `log_pfn_register(…,…,cb_id)`
names in Apache). And those aren't always the more commonly used identifiers.

Thus aliases makes sense not just for convenience, but also to be compatible
to other common names (e.g. w3c extend log format names like `cs-time`).


### container:

Is utilized by logopen() to extract additional fields (lists even) from one
of the existing fields.  This is usually done at row traversal.  And makes
sense for application-specific subformats in logs.  Such as any `key=value`
lists in the main message field.

| key      |  purpose                                               |
|----------|--------------------------------------------------------|
| `message` | **JSON key**: from which field to extract  |
| rx     | regex to detect and capture (key)=(value) fields         |
| id     |unpacked field name (usually just `$1` from the rx capture|
| value  | value from capture (so `$2` typically)                   |
| class  | decorative description (no .fmt recursion supported in logfmt1) |


!!! Notes
    * The entries here might become lists, since commonly there's just one
      `message` field in logs, yet multiple key:value schemes might be
      utilized within.
    * Or the target field might become a `"extract_from":` property, and
      `container` a list itself.
    * Still not sure if automatic list conversion is a good idea.  -
      Standard fields get an enumaration suffix `(?<request_uri2>…)` if
      duplicated.


### glob:

Might be used by log processors to look up a log class, based on file names,
if no .log.fmt is declared.


### #comment: fields

Documentation entries in the .fmt files have keys starting with `#`. For example
`"#license":` or `"#origin":`. Which is simpler than using JSON with
comments (JSOL/JSON5).

-----

### Other format files

!!! Note
    This section is about fictional features.

* Field [.fmt definitions](fmt.md) in `/usr/share/logfmt`
* [update-logfmt](update-logfmt.md) to create *.log.fmt files (for Apache, Nginx, some static logs)

## Sample .log.fmt

Currently the format for a *.log.fmt descriptor is:

```json
{
   "class": "apache custom2",
   "record": "%a %u %l [%t] %V \"%r\" %O %>s %D %{SSL_CIPHER}e %M"
}
```

!!! Info "Preliminary"
    Might still change, of course. "record" might be better named "formatstring"
    for example.

## Usage

In the simplest of cases you can use [logopen()](logopen.md) to process
a log file (and its adjacent .fmt descriptor) at once:

    log = logfmt1.logopen("/var/log/apache2/access.log")
    for row in log:
        print(row["request_line"])
    #print(log.names())

   
There's a few options (`debug=True`) and an `.alias` dict on the
logopen iterator.


## About

See also the [online documentation](https://fossil.include-once.org/modseccfg/wiki/logfmt1)
or [pypi listing](https://pypi.org/project/logfmt1/).  
This project originated as part of [modseccfg](https://fossil.include-once.org/modseccfg/).

## A .log.fmt for each log file

!!! Info "Rationale"
    Log parsing is a curse, because each application has its own format, and
    oftentimes configurable fields at that.  Various attempts at standardizing
    logs have failed, or are bound to.  Logging services and database storage
    are largely just symptomatic kludges, with JSON logs and not-quite-JSON
    formats held back by inertia.

Instead logfmt1 aims to have descriptors for each log file, in order to
make them parseable. You can't attempt anything but guesswork until you
know what's in a file.

So the idea is to have a `*.fmt` next to each `*.log` file, with a
descriptor such as:

```json
{
   "class": "apache combined",
   "record": "%h %l %u %t \"%r\" %>s %b"
}
```

Notably the "record" field should be the most current format string that
the application itself uses. In order to resolve the placeholders, an
application reference is kept in "class". Which allows combining the
format string with placeholder field definitions from the global
[.fmt database (`/usr/share/logfmt`)](fmt.md) database.


### common classes

There aren't many predefined classes yet, but special values that could
work without a current `"record":` declaration might be:

`"class": "grok syslog"`
:  Reads the according definition from a .grok (or perhaps preconverted)
   pattern definition. Which are largely static patterns.

`"class": "inilog"`
:  For Heroku/Go "logfmt" style logs comprised of only key=value fields

`"class": "json appmoniker"`
:  For real JSON logs, with an application identifier here (for decoration)

`"class": "apache common"`
:  Reads a predefined/static record: definition from the global
   [apache.common.fmt](fmt.md). Which of course means it would fail to
   parse, if the user diverted the LogFormat declaration in Apache.

Note that predefined classes undermine the purpose of logfmt1, in that
they're only suitable for static/non-variant log formats.




### additional fields

The *.log.fmt itself might declare definitions such as aliases and
more specific/custom placeholders.

```json
{
   "class": "apache cust3",
   "record": "%a %h %{iso}t '%r' %s",
   "fields": {
       "%{iso}t": { "id": "datetime", "rx": "..." }
   },
   "alias": {
       "iso8601": "datetime",
   }
}
```

Which ought to be joined and override any global [fmt](fmt) definitions.
Though such user customizations are more likely to be applied there anyway.
Care should be taken by `update-logfmt` or applications to not jettison
user-customized *.log.fmt options.


### rationale

Having the .fmt files adjecent to log files seems the most convenient
option.

 * Appending a `.fmt` suffix to the `….log` filename doesn't obstruct tab
   completion as much as `.fmt` substituting `.log`.
 * Doesn't require a lookup table or directory, with additional permission
   or updating woes.
 * And (over time) enabled applications themselves to create a `.log.fmt`
   for each log file. (That's kinda the goal. The
   [`update-logfmt`](update-logfmt.md) scripts are a stop-gap workaround.)

[logopen()](logopen.md) does this internally, but you might want to manually
craft the `"regex":` for a .fmt descriptor.  In such cases,
you can use `regex()` or `update()` to combine the log.fmt with definitions
from the global fmt database.


### regex()

  
  <link rel="shortcut icon" href="/img/favicon.ico">
  <title>logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="/css/theme.css" />
  <link rel="stylesheet" href="/css/theme_extra.css" />

  <link href="/custom.css" rel="stylesheet" />
  <link href="/syntax.css" rel="stylesheet" />
  
  <script src="/js/jquery-2.1.1.min.js" defer></script>
  <script src="/js/modernizr-2.8.3.min.js" defer></script> 


  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="/log.fmt.html">.log.fmt</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="/logopen.html">logopen()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="/regex.html">regex()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="/fmt.html">global .fmt db</a>
                    </li>
                </ul>
                <ul>

  
  <link rel="shortcut icon" href="img/favicon.ico">
  <title>global .fmt db - logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="css/theme.css" />
  <link rel="stylesheet" href="css/theme_extra.css" />

  <link href="custom.css" rel="stylesheet" />
  <link href="syntax.css" rel="stylesheet" />
  
  <script>
    // Current page data
    var mkdocs_page_name = "global .fmt db";
    var mkdocs_page_input_path = "fmt.md";
    var mkdocs_page_url = null;
  </script>
  
  <script src="js/jquery-2.1.1.min.js" defer></script>
  <script src="js/modernizr-2.8.3.min.js" defer></script> 


  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="log.fmt.html">.log.fmt</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="logopen.html">logopen()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
                    </li>
                </ul>
                <ul class="current">
                    <li class="toctree-l1 current"><a class="reference internal current" href="fmt.html">global .fmt db</a>
    <ul class="current">
    <li class="toctree-l2"><a class="reference internal" href="#fmt-example">.fmt Example</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#class">class:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#record">record:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#separator">separator:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#placeholder">placeholder:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#rewrite">rewrite:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#fields">fields:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#expand">expand:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#alias">alias:</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#container">container:</a>


    </li>
    <li class="toctree-l2"><a class="reference internal" href="#glob">glob:</a>
    </li>


    <li class="toctree-l2"><a class="reference internal" href="#comment-fields">#comment: fields</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#other-format-files">Other format files</a>
        <ul>
    <li class="toctree-l3"><a class="reference internal" href="#grok-definitions">.grok definitions</a>
    </li>
    <li class="toctree-l3"><a class="reference internal" href="#lnav-formats">.lnav formats</a>
    </li>

                <div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>❮❗❯ This is all very provisional. (First draft. Names might still change.)</p>
</div>
<h2 id="global-fmt-database">Global .fmt database</h2>
<p>While each log file should be accompanied by a <a href="log.fmt.html">.fmt descriptor</a>,
the global database in <code>/usr/share/logfmt/</code> contains a full .fmt field
definition for each class. And the cross-section of both allows to construct
a regex.</p>
<p>Most notably the <code>"fields":</code> and <code>"placeholder":</code> are used to turn the
<code>"record":</code> string definition into a capture pattern.</p>
<h3 id="fmt-example">.fmt Example</h3>
<p>The Apache format definition (apache.fmt) contains:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
    <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache generic&quot;</span><span class="p">,</span>
    <span class="nt">&quot;separator&quot;</span><span class="p">:</span> <span class="s2">&quot; &quot;</span><span class="p">,</span>
    <span class="nt">&quot;rewrite&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="nt">&quot;%[\\d!,+\\-]+&quot;</span><span class="p">:</span> <span class="s2">&quot;%&quot;</span><span class="p">,</span>
        <span class="nt">&quot;%%&quot;</span><span class="p">:</span> <span class="s2">&quot;%&quot;</span>
    <span class="p">},</span>
    <span class="nt">&quot;placeholder&quot;</span><span class="p">:</span> <span class="s2">&quot;%[&lt;&gt;]?(?:\\w*\\{[^\\}]+\\})?\\^?\\w+&quot;</span><span class="p">,</span>
    <span class="nt">&quot;fields&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="nt">&quot;%a&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_addr&quot;</span><span class="p">,</span>    <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[\\d.:a-f]+&quot;</span>  <span class="p">},</span>
        <span class="nt">&quot;%h&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_host&quot;</span><span class="p">,</span>    <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[\\w\\-.:]+&quot;</span>  <span class="p">},</span>
        <span class="nt">&quot;%{c}h&quot;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_host&quot;</span><span class="p">,</span>  <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[\\w\\-.:]+&quot;</span>  <span class="p">},</span>
        <span class="nt">&quot;%A&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;local_address&quot;</span><span class="p">,</span>  <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[\\d.:a-f]+&quot;</span>  <span class="p">},</span>
        <span class="nt">&quot;%u&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_user&quot;</span><span class="p">,</span>    <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;[\\-\\w@.]+&quot;</span>  <span class="p">},</span>
        <span class="nt">&quot;%t&quot;</span><span class="p">:</span> <span class="p">{</span>  <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;request_time&quot;</span><span class="p">,</span>   <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;\\[?(\\d[\\d:\\w\\s:./\\-+,;]+)\\]?&quot;</span>  <span class="p">},</span>
        <span class="err">…</span>
    <span class="p">},</span>
    <span class="nt">&quot;alias&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="nt">&quot;remote_address&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_addr&quot;</span><span class="p">,</span>
        <span class="nt">&quot;ip&quot;</span><span class="p">:</span> <span class="s2">&quot;remote_addr&quot;</span><span class="p">,</span>
        <span class="nt">&quot;file&quot;</span><span class="p">:</span> <span class="s2">&quot;request_file&quot;</span><span class="p">,</span>
        <span class="nt">&quot;size&quot;</span><span class="p">:</span> <span class="s2">&quot;bytes_sent&quot;</span><span class="p">,</span>
        <span class="err">…</span>
    <span class="p">},</span>
    <span class="nt">&quot;expand&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="nt">&quot;%\\{([^{}]+)\\}t&quot;</span><span class="p">:</span> <span class="p">{</span>
            <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;request_time&quot;</span><span class="p">,</span>
            <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;strftime&quot;</span><span class="p">,</span>
            <span class="nt">&quot;record&quot;</span><span class="p">:</span> <span class="s2">&quot;$1&quot;</span>
        <span class="p">}</span>
    <span class="p">},</span>
    <span class="nt">&quot;container&quot;</span><span class="p">:</span> <span class="p">{</span>
        <span class="nt">&quot;message&quot;</span><span class="p">:</span> <span class="p">{</span>
            <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;$1&quot;</span><span class="p">,</span>
            <span class="nt">&quot;value&quot;</span><span class="p">:</span> <span class="s2">&quot;$2&quot;</span><span class="p">,</span>
            <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;\\[(\\w+) \&quot;(.*?)\&quot;\\]&quot;</span><span class="p">,</span>
            <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache mod_security&quot;</span>
        <span class="p">}</span>
    <span class="p">},</span>
    <span class="nt">&quot;glob&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;/var/log/apache*/*acc*.log&quot;</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>

<p>It usually does not describe a default "record" format (like the local .log.fmt descriptors do).</p>
<h3 id="class">class:</h3>
<p>The class in the global database is largely decorative.  The filenames
instead define the heritage of rules/fields.  The "class" as declared by
a .log.fmt is mapped onto <code>/usr/share/logfmt/application.variant.fmt</code>.</p>
<ul>
<li>Usually there's just one variant level per log type. But the lookup is
   supposed to be mildly recursive.</li>
<li>Essentially it should merge <code>*.log.fmt</code> with <code>appclass.variant.fmt</code> and
   <code>appclass.fmt</code> applied last, so the most specific definitions are retained.</li>
<li>There's also a generic "grok" class. But the patterns therein are largely
   static (not build from variable format strings).</li>
<li>Some special classes like "json" might exist. (Not supported by logfmt1)</li>
</ul>
<h3 id="record">record:</h3>
<p>The "record" entry is not usually present in the global .fmt definition. 
Some super specific variant definitions (for example apache.error.fmt) or
static formats (syslog.fmt) might however.</p>
<h3 id="separator">separator:</h3>
<p>Most log formats use spaces for separating %placeholder fields.  And simpler
implementations might just split up the "record" declaration on this.</p>
<h3 id="placeholder">placeholder:</h3>
<p>While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields. It should account for prefixes/suffixes, unless
those got cleared by the <code>rewrite</code> map.</p>
<p>Not all formatstrings use <code>%\w+</code> to signal placeholders. In nginx for instance
the sigil <code>$\w+</code> introduces placeholders (variable names, really).</p>
<h3 id="rewrite">rewrite:</h3>
<p>A list/map of regex to apply before any transformations or field lookups. 
Which can be used to mask or simplify placeholder definitions (for instance
clean up the Apache conditional prefixes) or regex meta characters.</p>
<ul>
<li>The <code>record</code> field starts as a static string, but is meant to be turned
   into a regex.</li>
<li>Therefore meta characters (such as <code>|</code> or <code>[]</code>) have to be
   taken care of.  Which is what the <code>rewrite</code> map is lazily used for.</li>
<li>Better implementations might look up the placeholders, and automatically
   escape the rest of the the "record" format string.</li>
</ul>
<h3 id="fields">fields:</h3>
<p>The core of the global .fmt definitions are the field lists.  Each defines a
static %F placeholder and associaties it with a default field name (id:) and
regex (rx:) or even a grok definition (grok:).</p>
<table>
<thead>
<tr>
<th>key</th>

</tr>
<tr>
<td>type</td>
<td>"int" and "float" could designate strictly numeric fields</td>
</tr>
</tbody>
</table>
<div class="admonition notes">
<p class="admonition-title">Notes</p>
<ul>
<li>As part of the regex transformation, a <code>%F</code> could be turned into
  <code>(?&lt;id&gt;\S+)</code> for instance.</li>
<li>If there's any unnamed capture group <code>(…)</code>, it should be augmented
  into a named capture group - instead of the whole match. (To account
  for implicit wrapping.)</li>
<li>The <code>rx</code> itself might however specify named subgroups (like request_line
  in Apache logs, itself comprised of _method, _path, _protocol, or the
  datetime made up of tm_wday, tm_year, tm_whatever).</li>
<li><code>\S+</code> is also used as fallback for entirely undefined placeholders
  (no expand definition matched) in logfmt1.</li>
<li><code>grok</code> isn't currently used, but might allow for simpler transformations
  (indirectly into a grok pattern, and later a regex).</li>
</ul>
</div>
<h3 id="expand">expand:</h3>
<p>The expand declarations are used to construct unknown fields/placeholders. 
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.  Thus it simply can be applied before
separator/placeholder are looked at, to augment the known <code>fields</code> list.</p>
<table>
<thead>
<tr>

</tr>
<tr>
<td>record</td>
<td>can be set to $2 if class: recursion is defined</td>
</tr>
</tbody>
</table>
<div class="admonition notes">
<p class="admonition-title">Notes</p>
<ul>
<li>Typically it suffices to specify the <code>id</code> and <code>rx</code> field.</li>
<li>If no <code>id</code> is given, then the regex capture is normalized into
  an identifier (non-alphanumerics stripped, all lowercased).</li>
<li>But the <code>id</code> or <code>record</code> value might be set with regex captures
  (e.g. <code>$1</code> or <code>$2</code>) or compound values (<code>"id": "newfield_$1"</code>).</li>
<li>And logfmt1 allows to recurse into other format types per <code>class</code>
  (which is used to expand the captured <code>"record": "$1"</code> into regex
  tokens).</li>
</ul>
</div>
<h3 id="alias">alias:</h3>
<p>Maps alternative/more common field names onto the declared field <code>id</code>s.</p>
<p>To get to some state of standardization, the field ids usually refer
to application-internal names. (For instance <code>log_pfn_register(…,…,cb_id)</code>
names in Apache). And those aren't always the more commonly used identifiers.</p>
<p>Thus aliases makes sense not just for convenience, but also to be compatible
to other common names (e.g. w3c extend log format names like <code>cs-time</code>).</p>
<h3 id="container">container:</h3>
<p>Is utilized by logopen() to extract additional fields (lists even) from one
of the existing fields.  This is usually done at row traversal.  And makes
sense for application-specific subformats in logs.  Such as any <code>key=value</code>
lists in the main message field.</p>
<table>
<thead>
<tr>
<th>key</th>
<th>purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>message</code></td>
<td><strong>JSON key</strong>: from which field to extract</td>
</tr>
<tr>
<td>rx</td>
<td>regex to detect and capture (key)=(value) fields</td>
</tr>
<tr>
<td>id</td>
<td>unpacked field name (usually just <code>$1</code> from the rx capture</td>
</tr>
<tr>
<td>value</td>
<td>value from capture (so <code>$2</code> typically)</td>
</tr>
<tr>
<td>class</td>
<td>decorative description (no .fmt recursion supported in logfmt1)</td>
</tr>
</tbody>
</table>
<div class="admonition notes">
<p class="admonition-title">Notes</p>
<ul>
<li>The entries here might become lists, since commonly there's just one
  <code>message</code> field in logs, yet multiple key:value schemes might be
  utilized within.</li>
<li>Or the target field might become a <code>"extract_from":</code> property, and
  <code>container</code> a list itself.</li>
<li>Still not sure if automatic list conversion is a good idea.  -
  Standard fields get an enumaration suffix <code>(?&lt;request_uri2&gt;…)</code> if
  duplicated.</li>
</ul>
</div>
<h3 id="glob">glob:</h3>
<p>Might be used by log processors to look up a log class, based on file names,
if no .log.fmt is declared.</p>
<h3 id="comment-fields">#comment: fields</h3>
<p>Documentation entries in the .fmt files have keys starting with <code>#</code>. For example
<code>"#license":</code> or <code>"#origin":</code>. Which is simpler than using JSON with
comments (JSOL/JSON5).</p>
<hr />
<h3 id="other-format-files">Other format files</h3>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This section is about fictional features.</p>
</div>
<h4 id="grok-definitions">.grok definitions</h4>

          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="update-logfmt.html" class="btn btn-neutral float-right" title="update-logmt">Next <span class="icon icon-circle-arrow-right"></span></a>
      
      
        <a href="regex.html" class="btn btn-neutral" title="regex()"><span class="icon icon-circle-arrow-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">

  </div>

  <div class="rst-versions" role="note" aria-label="versions">
    <span class="rst-current-version" data-toggle="rst-current-version">
      
      
        <span><a href="regex.html" style="color: #fcfcfc;">&laquo; Previous</a></span>
      
      
        <span style="margin-left: 15px"><a href="update-logfmt.html" style="color: #fcfcfc">Next &raquo;</a></span>
      
    </span>
</div>
    <script>var base_url = '.';</script>

  
  <link rel="shortcut icon" href="img/favicon.ico">
  <title>Home - logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="css/theme.css" />
  <link rel="stylesheet" href="css/theme_extra.css" />

  <link href="custom.css" rel="stylesheet" />
  <link href="syntax.css" rel="stylesheet" />
  
  <script>
    // Current page data
    var mkdocs_page_name = "Home";
    var mkdocs_page_input_path = "index.md";
    var mkdocs_page_url = null;
  </script>
  
  <script src="js/jquery-2.1.1.min.js" defer></script>
  <script src="js/modernizr-2.8.3.min.js" defer></script> 


  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="log.fmt.html">.log.fmt</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="logopen.html">logopen()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
                    </li>
                </ul>
                <ul>

prescribes a <a href="log.fmt.html">.fmt</a> descriptor adjacent to each log file.  And a
global database to resolve format string %placeholder%s and turn them into
regular expressions.</p>
<h2 id="overview">Overview</h2>
<ul>
<li>A <a href="log.fmt.html">.log.fmt</a> for each log file</li>
<li><a href="logopen.html">logopen()</a></li>
<li><a href="update-regex.md">update() and regex()</a></li>
<li>Field <a href="fmt.html">.fmt definitions</a> in <code>/usr/share/logfmt</code></li>
<li><a href="update-logfmt.html">update-logfmt</a> to create *.log.fmt files (for Apache, Nginx, some static logs)</li>
</ul>
<h2 id="sample-logfmt">Sample .log.fmt</h2>
<p>Currently the format for a *.log.fmt descriptor is:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
   <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache custom2&quot;</span><span class="p">,</span>
   <span class="nt">&quot;record&quot;</span><span class="p">:</span> <span class="s2">&quot;%a %u %l [%t] %V \&quot;%r\&quot; %O %&gt;s %D %{SSL_CIPHER}e %M&quot;</span>
<span class="p">}</span>
</code></pre></div>
<div class="admonition info">
<p class="admonition-title">Preliminary</p>
<p>Might still change, of course. "record" might be better named "formatstring"
for example.</p>
</div>
<h2 id="usage">Usage</h2>
<p>In the simplest of cases you can use <a href="logopen.html">logopen()</a> to process
a log file (and its adjacent .fmt descriptor) at once:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">log</span> <span class="o">=</span> <span class="n">logfmt1</span><span class="o">.</span><span class="n">logopen</span><span class="p">(</span><span class="s2">&quot;/var/log/apache2/access.log&quot;</span><span class="p">)</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="nb">log</span><span class="p">:</span>
    <span class="nb">print</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s2">&quot;request_line&quot;</span><span class="p">])</span>
<span class="c1">#print(log.names())</span>
</code></pre></div>

<p>&nbsp;<br />
There's a few options (<code>debug=True</code>) and an <code>.alias</code> dict on the
logopen iterator.</p>
<h2 id="about">About</h2>
<p>See also the <a href="https://fossil.include-once.org/modseccfg/wiki/logfmt1">online documentation</a>
or <a href="https://pypi.org/project/logfmt1/">pypi listing</a>.<br />
This project originated as part of <a href="https://fossil.include-once.org/modseccfg/">modseccfg</a>.</p>
              
            </div>
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      

  
  <link rel="shortcut icon" href="img/favicon.ico">
  <title>.log.fmt - logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="css/theme.css" />
  <link rel="stylesheet" href="css/theme_extra.css" />

  <link href="custom.css" rel="stylesheet" />
  <link href="syntax.css" rel="stylesheet" />
  
  <script>
    // Current page data
    var mkdocs_page_name = ".log.fmt";
    var mkdocs_page_input_path = "log.fmt.md";
    var mkdocs_page_url = null;
  </script>
  
  <script src="js/jquery-2.1.1.min.js" defer></script>
  <script src="js/modernizr-2.8.3.min.js" defer></script> 


  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

                <ul class="current">
                    <li class="toctree-l1 current"><a class="reference internal current" href="log.fmt.html">.log.fmt</a>
    <ul class="current">
    <li class="toctree-l2"><a class="reference internal" href="#common-classes">common classes</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#additional-fields">additional fields</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#rationale">rationale</a>
    </li>
    </ul>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="logopen.html">logopen()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
                    </li>
                </ul>
                <ul>

  
  <hr/>
</div>
          <div role="main">
            <div class="section">
              
                <h2 id="a-logfmt-for-each-log-file">A .log.fmt for each log file</h2>
<div class="admonition info">
<p class="admonition-title">Rationale</p>
<p>Log parsing is a curse, because each application has its own format, and
oftentimes configurable fields at that.  Various attempts at standardizing
logs have failed, or are bound to.  Logging services and database storage
are largely just symptomatic kludges, with JSON logs and not-quite-JSON
formats held back by inertia.</p>
</div>
<p>Instead logfmt1 aims to have descriptors for each log file, in order to
make them parseable. You can't attempt anything but guesswork until you
know what's in a file.</p>
<p>So the idea is to have a <code>*.fmt</code> next to each <code>*.log</code> file, with a
descriptor such as:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
   <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache combined&quot;</span><span class="p">,</span>
   <span class="nt">&quot;record&quot;</span><span class="p">:</span> <span class="s2">&quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b&quot;</span>
<span class="p">}</span>
</code></pre></div>

<p>Notably the "record" field should be the most current format string that
the application itself uses. In order to resolve the placeholders, an
application reference is kept in "class". Which allows combining the
format string with placeholder field definitions from the global
<a href="fmt.html">.fmt database (<code>/usr/share/logfmt</code>)</a> database.</p>
<h3 id="common-classes">common classes</h3>
<p>There aren't many predefined classes yet, but special values that could
work without a current <code>"record":</code> declaration might be:</p>
<dl>
<dt><code>"class": "grok syslog"</code></dt>
<dd>Reads the according definition from a .grok (or perhaps preconverted)
   pattern definition. Which are largely static patterns.</dd>
<dt><code>"class": "inilog"</code></dt>
<dd>For Heroku/Go "logfmt" style logs comprised of only key=value fields</dd>
<dt><code>"class": "json appmoniker"</code></dt>
<dd>For real JSON logs, with an application identifier here (for decoration)</dd>
<dt><code>"class": "apache common"</code></dt>

<dd>Reads a predefined/static record: definition from the global
   <a href="fmt.html">apache.common.fmt</a>. Which of course means it would fail to
   parse, if the user diverted the LogFormat declaration in Apache.</dd>
</dl>
<p>Note that predefined classes undermine the purpose of logfmt1, in that
they're only suitable for static/non-variant log formats.</p>
<h3 id="additional-fields">additional fields</h3>
<p>The *.log.fmt itself might declare definitions such as aliases and
more specific/custom placeholders.</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
   <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache cust3&quot;</span><span class="p">,</span>
   <span class="nt">&quot;record&quot;</span><span class="p">:</span> <span class="s2">&quot;%a %h %{iso}t &#39;%r&#39; %s&quot;</span><span class="p">,</span>
   <span class="nt">&quot;fields&quot;</span><span class="p">:</span> <span class="p">{</span>
       <span class="nt">&quot;%{iso}t&quot;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;datetime&quot;</span><span class="p">,</span> <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;...&quot;</span> <span class="p">}</span>
   <span class="p">},</span>
   <span class="nt">&quot;alias&quot;</span><span class="p">:</span> <span class="p">{</span>
       <span class="nt">&quot;iso8601&quot;</span><span class="p">:</span> <span class="s2">&quot;datetime&quot;</span><span class="p">,</span>
   <span class="p">}</span>
<span class="p">}</span>
</code></pre></div>

<p>Which ought to be joined and override any global <a href="fmt">fmt</a> definitions.
Though such user customizations are more likely to be applied there anyway.
Care should be taken by <code>update-logfmt</code> or applications to not jettison
user-customized *.log.fmt options.</p>
<h3 id="rationale">rationale</h3>
<p>Having the .fmt files adjecent to log files seems the most convenient
option.</p>
<ul>
<li>Appending a <code>.fmt</code> suffix to the <code>….log</code> filename doesn't obstruct tab
   completion as much as <code>.fmt</code> substituting <code>.log</code>.</li>
<li>Doesn't require a lookup table or directory, with additional permission
   or updating woes.</li>
<li>And (over time) enabled applications themselves to create a <code>.log.fmt</code>
   for each log file. (That's kinda the goal. The
   <a href="update-logfmt.html"><code>update-logfmt</code></a> scripts are a stop-gap workaround.)</li>
</ul>
              
            </div>
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="logopen.html" class="btn btn-neutral float-right" title="logopen()">Next <span class="icon icon-circle-arrow-right"></span></a>
      
      
        <a href="index.html" class="btn btn-neutral" title="Home"><span class="icon icon-circle-arrow-left"></span> Previous</a>
      
    </div>
  

  
  <link rel="shortcut icon" href="img/favicon.ico">
  <title>logex - logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="css/theme.css" />
  <link rel="stylesheet" href="css/theme_extra.css" />

  <link href="custom.css" rel="stylesheet" />
  <link href="syntax.css" rel="stylesheet" />
  
  <script>
    // Current page data
    var mkdocs_page_name = "logex";
    var mkdocs_page_input_path = "logex.md";
    var mkdocs_page_url = null;
  </script>
  
  <script src="js/jquery-2.1.1.min.js" defer></script>
  <script src="js/modernizr-2.8.3.min.js" defer></script> 


  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="log.fmt.html">.log.fmt</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="logopen.html">logopen()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
                    </li>
                </ul>
                <ul>

<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  
  
  <link rel="shortcut icon" href="img/favicon.ico">
  <title>logopen() - logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="css/theme.css" />
  <link rel="stylesheet" href="css/theme_extra.css" />

  <link href="custom.css" rel="stylesheet" />
  <link href="syntax.css" rel="stylesheet" />
  
  <script>
    // Current page data
    var mkdocs_page_name = "logopen()";
    var mkdocs_page_input_path = "logopen.md";
    var mkdocs_page_url = null;
  </script>
  
  <script src="js/jquery-2.1.1.min.js" defer></script>
  <script src="js/modernizr-2.8.3.min.js" defer></script> 


  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="log.fmt.html">.log.fmt</a>
                    </li>
                </ul>
                <ul class="current">
                    <li class="toctree-l1 current"><a class="reference internal current" href="logopen.html">logopen()</a>
    <ul class="current">
    </ul>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
                    </li>
                </ul>
                <ul>

        <div class="rst-content">
          <div role="navigation" aria-label="breadcrumbs navigation">
  <ul class="wy-breadcrumbs">
    <li><a href=".">Docs</a> &raquo;</li>
    
      
    
    <li>logopen()</li>
    <li class="wy-breadcrumbs-aside">
      
    </li>
  </ul>
  
  <hr/>
</div>

              
            </div>
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="regex.html" class="btn btn-neutral float-right" title="regex()">Next <span class="icon icon-circle-arrow-right"></span></a>
      
      
        <a href="log.fmt.html" class="btn btn-neutral" title=".log.fmt"><span class="icon icon-circle-arrow-left"></span> Previous</a>
      
    </div>
  

  <div class="rst-versions" role="note" aria-label="versions">
    <span class="rst-current-version" data-toggle="rst-current-version">
      
      
        <span><a href="log.fmt.html" style="color: #fcfcfc;">&laquo; Previous</a></span>
      
      
        <span style="margin-left: 15px"><a href="regex.html" style="color: #fcfcfc">Next &raquo;</a></span>
      
    </span>
</div>
    <script>var base_url = '.';</script>
    <script src="js/theme.js" defer></script>
    <script defer>
        window.onload = function () {
            SphinxRtdTheme.Navigation.enable(true);
        };
    </script>

</body>
</html>

<!DOCTYPE html>

<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1.0" name="viewport"/>
<link href="img/favicon.ico" rel="shortcut icon"/>
<title>regex() - logfmt1</title>
<link href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" rel="stylesheet">
<link href="css/theme.css" rel="stylesheet"/>
<link href="css/theme_extra.css" rel="stylesheet"/>

<link href="custom.css" rel="stylesheet"/>
<link href="syntax.css" rel="stylesheet"/>
<script>
    // Current page data
    var mkdocs_page_name = "regex()";
    var mkdocs_page_input_path = "regex.md";
    var mkdocs_page_url = null;
  </script>
<script defer="" src="js/jquery-2.1.1.min.js"></script>
<script defer="" src="js/modernizr-2.8.3.min.js"></script>


</link></head>
<body class="wy-body-for-nav" role="document">
<div class="wy-grid-for-nav">
<nav class="wy-nav-side stickynav" data-toggle="wy-nav-shift">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a class="icon icon-home" href="."> logfmt1</a>
</div>
<div aria-label="main navigation" class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="index.html">Home</a>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="log.fmt.html">.log.fmt</a>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="logopen.html">logopen()</a>
</li>
</ul>
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal current" href="regex.html">regex()</a>
<ul class="current">
</ul>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
</li>

<a href=".">logfmt1</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div aria-label="breadcrumbs navigation" role="navigation">
<ul class="wy-breadcrumbs">
<li><a href=".">Docs</a> »</li>
<li>regex()</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main">
<div class="section">
<p><a href="logopen.html">logopen()</a> does this internally, but you might want to manually
craft the <code>"regex":</code> for a .fmt descriptor.  In such cases,
you can use <code>regex()</code> or <code>update()</code> to combine the log.fmt with definitions
from the global fmt database.</p>
<h3 id="regex">regex()</h3>
<p>Takes a .fmt json/dict, and generates the regex with named capture groups
from it.</p>
<div class="doc doc-object doc-function">

</div>
</div>
</div>
</div>
<footer>
<div aria-label="footer navigation" class="rst-footer-buttons" role="navigation">
<a class="btn btn-neutral float-right" href="fmt.html" title="global .fmt db">Next <span class="icon icon-circle-arrow-right"></span></a>
<a class="btn btn-neutral" href="logopen.html" title="logopen()"><span class="icon icon-circle-arrow-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<!-- Copyright etc -->
</div>

  Built with <a href="https://www.mkdocs.org/">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

  
  <link rel="shortcut icon" href="img/favicon.ico">
  <title>update-logmt - logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="css/theme.css" />
  <link rel="stylesheet" href="css/theme_extra.css" />

  <link href="custom.css" rel="stylesheet" />
  <link href="syntax.css" rel="stylesheet" />
  
  <script>
    // Current page data
    var mkdocs_page_name = "update-logmt";
    var mkdocs_page_input_path = "update-logfmt.md";
    var mkdocs_page_url = null;
  </script>
  
  <script src="js/jquery-2.1.1.min.js" defer></script>
  <script src="js/modernizr-2.8.3.min.js" defer></script> 


  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="log.fmt.html">.log.fmt</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="logopen.html">logopen()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
                    </li>
                </ul>
                <ul class="current">

site_name: logfmt1
docs_dir: docs
site_dir: html
nav:
    - Home: index.md
    - .log.fmt: log.fmt.md
    - logopen(): logopen.md
    - regex(): regex.md 
    - global .fmt db: fmt.md
    - update-logmt: update-logfmt.md
    - logex: logex.md
theme:
#  name: materia
#  name: bootstrap386
#  name: ivory
#  name: material
  name: readthedocs
  highlightjs: false
repo_url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
markdown_extensions:
  - admonition
  - codehilite
#      - use_pygments: True
  - attr_list
  - def_list
  - tables
  - markdown.extensions.codehilite:
      guess_lang: true
  - pymdownx.inlinehilite:
  - pymdownx.superfences:
plugins:
    - mkdocstrings
extra_css:
    - custom.css
    - syntax.css
use_directory_urls: false