whl:
	pandoc logfmt1/README.md -o logfmt1/README.rst
	cd logfmt1 && ./setup.py bdist_wheel
whl_up:
	twine upload dist/logfmt*

docs:
	cd logfmt1 ; PYTHONPATH=. mkdocs build -v -v -v
	sed -i 's/table\.docutils/table/g' logfmt1/html/css/theme.css 
	

body, * {
    font-size: 1.1rem !important;
}
code {
    font-size: 0.75rem !important;
}

.doc-object {
    margin-top: 2.5rem;
}

blockquote {

!!! Warning
    ❮❗❯ This is all very provisional. (First draft. Names might still change.)



## Global .fmt database

While each log file should be accompanied by a [.fmt descriptor](log.fmt.md),
the global database in `/usr/share/logfmt/` contains a full .fmt field
definition for each class.

        },
        "glob": ["/var/log/apache*/*acc*.log"]
    }

It usually does not describe a default "record" format (like the local .log.fmt descriptors do).


### class

The class in the global database is largely decorative.  The filenames
instead define the heritage of rules/fields.  The "class" as declared by
a .log.fmt is mapped onto `/usr/share/logfmt/application.variant.fmt`.

 * Usually there's just one variant level per log type. But the lookup is
   supposed to be mildly recursive.
 * Essentially it should merge `*.log.fmt` with `appclass.variant.fmt` and
   `appclass.fmt` applied last, so the most specific definitions are retained.
 * There's also a generic "grok" class. But the patterns therein are largely
   static (not build from variable format strings).
 * Some special classes like "json" might exist. (Not supported by logfmt1)


#### record

The "record" entry is not usually present in the global .fmt definition. 
Some super specific variant definitions (for example apache.error.fmt) or
static formats (syslog.fmt) might however.


#### separator

Most log formats use spaces for separating %placeholder fields.  And simpler
implementations might just split up the "record" declaration on this.


### placeholder

While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields. It should account for prefixes/suffixes, unless
those got cleared by the `rewrite` map.

Not all formatstrings use `%\w+` to signal placeholders. In nginx for instance
the sigil `$\w+` introduces placeholders (variable names, really).

   into a regex.
 * Therefore meta characters (such as `|` or `[]`) have to be
   taken care of.  Which is what the `rewrite` map is lazily used for.
 * Better implementations might look up the placeholders, and automatically
   escape the rest of the the "record" format string.


### fields

The core of the global .fmt definitions are the field lists.  Each defines a
static %F placeholder and associaties it with a default field name (id:) and
regex (rx:) or even a grok definition (grok:).

| key | purpose |
|-----|---------|
|`%F`| **JSON key**: static placeholder string (not a regex itself) |
| id | field identifier, as specified by the application (internal name) |
| rx | regex which %F placeholder gets replaced with |
| grok | alternatively to regex, %F might be turned into %PATTERN:id |
| type | "int" and "float" could designate strictly numeric fields |

 * As part of the regex transformation, a `%F` could be turned into
   `(?<id>\S+)` for instance.
 * If there's any unnamed capture group `(…)`, it should be augmented
   into a named capture group - instead of the whole match. (To account
   for implicit wrapping.)
 * The `rx` itself might however specify named subgroups (like request_line
   in Apache logs, itself comprised of _method, _path, _protocol, or the
   datetime made up of tm_wday, tm_year, tm_whatever).
 * `\S+` is also used as fallback for entirely undefined placeholders
   (no expand definition matched) in logfmt1.
 * `grok` isn't currently used, but might allow for simpler transformations
   (indirectly into a grok pattern, and later a regex).


### expand

The expand declarations are used to construct unknown fields/placeholders. 
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.  Thus it simply can be applied before
separator/placeholder are looked at, to augment the known `fields` list.

| key      |  purpose                                               |
|----------|--------------------------------------------------------|
| `%\{(\w+)\}t` | **JSON key**: a regex to detect mutable placeholders  |
| id     | name for newly created fields entry, might use captures´ $1|
| rx     | for static definitions (often just \S+)                  |
|if_quoted| alternative regex, if placeholder was enclosed in "%\w+" quotes|
| class  | recurse into other .fmt types                            |
| record | can be set to $2 if class: recursion is defined          |


 * Typically it suffices to specify the `id` and `rx` field.
 * If no `id` is given, then the regex capture is normalized into
   an identifier (non-alphanumerics stripped, all lowercased).
 * But the `id` or `record` value might be set with regex captures
   (e.g. `$1` or `$2`) or compound values (`"id": "newfield_$1"`).
 * And logfmt1 allows to recurse into other format types per `class`

to application-internal names. (For instance `log_pfn_register(…,…,cb_id)`
names in Apache). And those aren't always the more commonly used identifiers.

Thus aliases makes sense not just for convenience, but also to be compatible
to other common names (e.g. w3c extend log format names like `cs-time`).


#### container

Is utilized by logopen() to extract additional fields (lists even) from one
of the existing fields.  This is usually done at row traversal.  And makes
sense for application-specific subformats in logs.  Such as any `key=value`
lists in the main message field.

> Still not sure if automatic list conversion is a good idea.

#### glob

Might be used by log processors to look up a log class, based on file names,
if no .log.fmt is declared.


### "#comment": fields

Documentation entries in the .fmt files have keys starting with `#`. For example
`"#license":` or `"#origin":`. Which is simpler than using JSON with
comments (JSOL).

-----

### Other format files

!!! Note
    This section is about fictional features.


#### .grok definitions

> Not implemented yet.

The logfmt/ directory might also contain .grok files, which get transformed
into .fmt structures. (Probably with the grok: parameter for fields, and
a grok: pattern table alongside regular fields:).

There's already a pretransformed `grok.fmt`, which however requires
`%{GROK:%{PATTERN:id}}` references currently.


#### .lnav formats

> Not implemented yet.

Likewise could we use lnav .json format definitions. Those are static
too, however.

  white-space: pre;
  word-wrap: normal;
  display: block;
  padding: 12px;
  font-size: 12px;
}















/*
 * Fix link colors when the link text is inline code.
 *
 * https://github.com/mkdocs/mkdocs/issues/718
 */
a code {
    color: #2980B9;

body, * {
    font-size: 1.1rem !important;
}
code {
    font-size: 0.75rem !important;
}

.doc-object {
    margin-top: 2.5rem;
}

blockquote {

                    </li>
                </ul>
                <ul class="current">
                    <li class="toctree-l1 current"><a class="reference internal current" href="fmt.html">global .fmt db</a>
    <ul class="current">
    <li class="toctree-l2"><a class="reference internal" href="#sample">sample</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#class">class</a>
        <ul>
    <li class="toctree-l3"><a class="reference internal" href="#record">record</a>
    </li>
    <li class="toctree-l3"><a class="reference internal" href="#separator">separator</a>
    </li>
        </ul>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#placeholder">placeholder</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#rewrite">rewrite</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#fields">fields</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#expand">expand</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#alias">alias</a>
        <ul>
    <li class="toctree-l3"><a class="reference internal" href="#container">container</a>
    </li>
    <li class="toctree-l3"><a class="reference internal" href="#glob">glob</a>
    </li>
        </ul>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#comment-fields">"#comment": fields</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#other-format-files">Other format files</a>
        <ul>
    <li class="toctree-l3"><a class="reference internal" href="#grok-definitions">.grok definitions</a>
    </li>
    <li class="toctree-l3"><a class="reference internal" href="#lnav-formats">.lnav formats</a>
    </li>
        </ul>
    </li>
    </ul>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="update-logfmt.html">update-logmt</a>
                    </li>
                </ul>

  </ul>
  
  <hr/>
</div>
          <div role="main">
            <div class="section">
              
                <div class="admonition warning">
<p class="admonition-title">Warning</p>



<p>❮❗❯ This is all very provisional. (First draft. Names might still change.)</p>
</div>









<h2 id="global-fmt-database">Global .fmt database</h2>
<p>While each log file should be accompanied by a <a href="log.fmt.html">.fmt descriptor</a>,
the global database in <code>/usr/share/logfmt/</code> contains a full .fmt field
definition for each class.</p>
<p>Most notably the <code>"fields":</code> and <code>"placeholder":</code> are used to construct the
regex for a <code>"record":</code> string definition.</p>
<h3 id="sample">sample</h3>

        <span class="p">}</span>
    <span class="p">},</span>
    <span class="s2">&quot;glob&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;/var/log/apache*/*acc*.log&quot;</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>

<p>It usually does not describe a default "record" format (like the local .log.fmt descriptors do).</p>
<h3 id="class">class</h3>
<p>The class in the global database is largely decorative.  The filenames
instead define the heritage of rules/fields.  The "class" as declared by
a .log.fmt is mapped onto <code>/usr/share/logfmt/application.variant.fmt</code>.</p>
<ul>
<li>Usually there's just one variant level per log type. But the lookup is
   supposed to be mildly recursive.</li>
<li>Essentially it should merge <code>*.log.fmt</code> with <code>appclass.variant.fmt</code> and
   <code>appclass.fmt</code> applied last, so the most specific definitions are retained.</li>
<li>There's also a generic "grok" class. But the patterns therein are largely
   static (not build from variable format strings).</li>
<li>Some special classes like "json" might exist. (Not supported by logfmt1)</li>
</ul>
<h4 id="record">record</h4>
<p>The "record" entry is not usually present in the global .fmt definition. 
Some super specific variant definitions (for example apache.error.fmt) or
static formats (syslog.fmt) might however.</p>
<h4 id="separator">separator</h4>
<p>Most log formats use spaces for separating %placeholder fields.  And simpler
implementations might just split up the "record" declaration on this.</p>
<h3 id="placeholder">placeholder</h3>
<p>While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields. It should account for prefixes/suffixes, unless
those got cleared by the <code>rewrite</code> map.</p>
<p>Not all formatstrings use <code>%\w+</code> to signal placeholders. In nginx for instance
the sigil <code>$\w+</code> introduces placeholders (variable names, really).</p>
<h3 id="rewrite">rewrite</h3>
<p>A list/map of regex to apply before any transformations or field lookups. 
Which can be used to mask or simplify placeholder definitions (for instance
clean up the Apache conditional prefixes) or regex meta characters.</p>
<ul>
<li>The <code>record</code> field starts as a static string, but is meant to be turned
   into a regex.</li>
<li>Therefore meta characters (such as <code>|</code> or <code>[]</code>) have to be
   taken care of.  Which is what the <code>rewrite</code> map is lazily used for.</li>
<li>Better implementations might look up the placeholders, and automatically
   escape the rest of the the "record" format string.</li>
</ul>
<h3 id="fields">fields</h3>
<p>The core of the global .fmt definitions are the field lists.  Each defines a
static %F placeholder and associaties it with a default field name (id:) and
regex (rx:) or even a grok definition (grok:).</p>
<table>
<thead>
<tr>
<th>key</th>
<th>purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>%F</code></td>
<td><strong>JSON key</strong>: static placeholder string (not a regex itself)</td>
</tr>
<tr>
<td>id</td>
<td>field identifier, as specified by the application (internal name)</td>
</tr>
<tr>
<td>rx</td>

   in Apache logs, itself comprised of _method, _path, _protocol, or the
   datetime made up of tm_wday, tm_year, tm_whatever).</li>
<li><code>\S+</code> is also used as fallback for entirely undefined placeholders
   (no expand definition matched) in logfmt1.</li>
<li><code>grok</code> isn't currently used, but might allow for simpler transformations
   (indirectly into a grok pattern, and later a regex).</li>
</ul>
<h3 id="expand">expand</h3>
<p>The expand declarations are used to construct unknown fields/placeholders. 
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.  Thus it simply can be applied before
separator/placeholder are looked at, to augment the known <code>fields</code> list.</p>
<table>
<thead>
<tr>
<th>key</th>
<th>purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>%\{(\w+)\}t</code></td>
<td><strong>JSON key</strong>: a regex to detect mutable placeholders</td>
</tr>
<tr>
<td>id</td>
<td>name for newly created fields entry, might use captures´ $1</td>
</tr>
<tr>
<td>rx</td>
<td>for static definitions (often just \S+)</td>
</tr>
<tr>
<td>if_quoted</td>

<h3 id="alias">alias</h3>
<p>Maps alternative/more common field names onto the declared field <code>id</code>s.</p>
<p>To get to some state of standardization, the field ids usually refer
to application-internal names. (For instance <code>log_pfn_register(…,…,cb_id)</code>
names in Apache). And those aren't always the more commonly used identifiers.</p>
<p>Thus aliases makes sense not just for convenience, but also to be compatible
to other common names (e.g. w3c extend log format names like <code>cs-time</code>).</p>
<h4 id="container">container</h4>
<p>Is utilized by logopen() to extract additional fields (lists even) from one
of the existing fields.  This is usually done at row traversal.  And makes
sense for application-specific subformats in logs.  Such as any <code>key=value</code>
lists in the main message field.</p>
<blockquote>
<p>Still not sure if automatic list conversion is a good idea.</p>
</blockquote>
<h4 id="glob">glob</h4>
<p>Might be used by log processors to look up a log class, based on file names,
if no .log.fmt is declared.</p>
<h3 id="comment-fields">"#comment": fields</h3>
<p>Documentation entries in the .fmt files have keys starting with <code>#</code>. For example
<code>"#license":</code> or <code>"#origin":</code>. Which is simpler than using JSON with
comments (JSOL).</p>
<hr />
<h3 id="other-format-files">Other format files</h3>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This section is about fictional features.</p>
</div>
<h4 id="grok-definitions">.grok definitions</h4>
<blockquote>
<p>Not implemented yet.</p>
</blockquote>
<p>The logfmt/ directory might also contain .grok files, which get transformed
into .fmt structures. (Probably with the grok: parameter for fields, and
a grok: pattern table alongside regular fields:).</p>
<p>There's already a pretransformed <code>grok.fmt</code>, which however requires
<code>%{GROK:%{PATTERN:id}}</code> references currently.</p>
<h4 id="lnav-formats">.lnav formats</h4>
<blockquote>
<p>Not implemented yet.</p>
</blockquote>
<p>Likewise could we use lnav .json format definitions. Those are static
too, however.</p>
              
            </div>

    - global .fmt db: fmt.md
    - update-logmt: update-logfmt.md
    - logex: logex.md
theme:
#  name: materia
#  name: bootstrap386
#  name: ivory
  name: material
  name: readthedocs
repo_url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
markdown_extensions:
  - admonition
  - codehilite
  - tables
#  - mkautodoc
plugins:
    - mkdocstrings
extra_css:
    - custom.css
use_directory_urls: false