<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  
  
  <link rel="shortcut icon" href="img/favicon.ico">
  <title>.log.fmt - logfmt1</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />

  <link rel="stylesheet" href="css/theme.css" />
  <link rel="stylesheet" href="css/theme_extra.css" />
  <link href="custom.css" rel="stylesheet" />
  <link href="syntax.css" rel="stylesheet" />
  
  <script>
    // Current page data
    var mkdocs_page_name = ".log.fmt";
    var mkdocs_page_input_path = "log.fmt.md";
    var mkdocs_page_url = null;
  </script>
  
  <script src="js/jquery-2.1.1.min.js" defer></script>
  <script src="js/modernizr-2.8.3.min.js" defer></script> 
  
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">

    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav">
    <div class="wy-side-scroll">
      <div class="wy-side-nav-search">
        <a href="." class="icon icon-home"> logfmt1</a>
        
      </div>

      <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="index.html">Intro</a>
                    </li>
                </ul>
                <ul class="current">
                    <li class="toctree-l1 current"><a class="reference internal current" href="log.fmt.html">.log.fmt</a>
    <ul class="current">
    <li class="toctree-l2"><a class="reference internal" href="#common-classes">common classes</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#additional-fields">additional fields</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#rationale">rationale</a>
    </li>
    </ul>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="logopen.html">logopen()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="update-logfmt.html">update-logfmt</a>
                    </li>
                </ul>
                <ul>
                    <li class="toctree-l1"><a class="reference internal" href="logex.html">logex</a>
                    </li>
                </ul>
      </div>
    </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
        <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
        <a href=".">logfmt1</a>
      </nav>

      
      <div class="wy-nav-content">
        <div class="rst-content">
          <div role="navigation" aria-label="breadcrumbs navigation">
  <ul class="wy-breadcrumbs">
    <li><a href=".">Docs</a> &raquo;</li>
    
      
    
    <li>.log.fmt</li>
    <li class="wy-breadcrumbs-aside">
      
    </li>
  </ul>
  
  <hr/>
</div>
          <div role="main">
            <div class="section">
              
                <h2 id="a-logfmt-for-each-log-file">A .log.fmt for each log file</h2>
<div class="admonition info">
<p class="admonition-title">Rationale</p>
<p>Log parsing is a curse, because each application has its own format, and
oftentimes configurable fields at that.  Various attempts at standardizing
logs have failed, or are bound to.  Logging services and database storage
are largely just symptomatic kludges, with JSON logs and not-quite-JSON
formats held back by inertia.</p>
</div>
<p>Instead logfmt1 aims to have descriptors for each log file, in order to
make them parseable. You can't attempt anything but guesswork until you
know what's in a file.</p>
<p>So the idea is to have a <code>*.fmt</code> next to each <code>*.log</code> file, with a
descriptor such as:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
   <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache combined&quot;</span><span class="p">,</span>
   <span class="nt">&quot;record&quot;</span><span class="p">:</span> <span class="s2">&quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b&quot;</span>
<span class="p">}</span>
</code></pre></div>
<p>Notably the "record" field should be the most current format string that
the application itself uses. In order to resolve the placeholders, an
application reference is kept in "class". Which allows combining the
format string with placeholder field definitions from the global
<a href="fmt.html">.fmt database (<code>/usr/share/logfmt</code>)</a> database.</p>
<h3 id="common-classes">common classes</h3>
<p>There aren't many predefined classes yet, but special values that could
work without a current <code>"record":</code> declaration might be:</p>
<dl>
<dt><code>"class": "grok syslog"</code></dt>
<dd>Reads the according definition from a .grok (or perhaps preconverted)
   pattern definition. Which are largely static patterns.</dd>
<dt><code>"class": "inilog"</code></dt>
<dd>For Heroku/Go "logfmt" style logs comprised of only key=value fields</dd>
<dt><code>"class": "json appmoniker"</code></dt>
<dd>For real JSON logs, with an application identifier here (for decoration)</dd>
<dt><code>"class": "apache common"</code></dt>
<dd>Reads a predefined/static record: definition from the global
   <a href="fmt.html">apache.common.fmt</a>. Which of course means it would fail to
   parse, if the user diverted the LogFormat declaration in Apache.</dd>
</dl>
<p>Note that predefined classes undermine the purpose of logfmt1, in that
they're only suitable for static/non-variant log formats.</p>
<h3 id="additional-fields">additional fields</h3>
<p>The *.log.fmt itself might declare definitions such as aliases and
more specific/custom placeholders.</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
   <span class="nt">&quot;class&quot;</span><span class="p">:</span> <span class="s2">&quot;apache cust3&quot;</span><span class="p">,</span>
   <span class="nt">&quot;record&quot;</span><span class="p">:</span> <span class="s2">&quot;%a %h %{iso}t &#39;%r&#39; %s&quot;</span><span class="p">,</span>
   <span class="nt">&quot;fields&quot;</span><span class="p">:</span> <span class="p">{</span>
       <span class="nt">&quot;%{iso}t&quot;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&quot;id&quot;</span><span class="p">:</span> <span class="s2">&quot;datetime&quot;</span><span class="p">,</span> <span class="nt">&quot;rx&quot;</span><span class="p">:</span> <span class="s2">&quot;...&quot;</span> <span class="p">}</span>
   <span class="p">},</span>
   <span class="nt">&quot;alias&quot;</span><span class="p">:</span> <span class="p">{</span>
       <span class="nt">&quot;iso8601&quot;</span><span class="p">:</span> <span class="s2">&quot;datetime&quot;</span><span class="p">,</span>
   <span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Which ought to be joined and override any global <a href="fmt">fmt</a> definitions.
Though such user customizations are more likely to be applied there anyway.
Care should be taken by <code>update-logfmt</code> or applications to not jettison
user-customized *.log.fmt options.</p>
<h3 id="rationale">rationale</h3>
<p>Having the .fmt files adjecent to log files seems the most convenient
option.</p>
<ul>
<li>Appending a <code>.fmt</code> suffix to the <code>….log</code> filename doesn't obstruct tab
   completion as much as <code>.fmt</code> substituting <code>.log</code>.</li>
<li>Doesn't require a lookup table or directory, with additional permission
   or updating woes.</li>
<li>And (over time) enabled applications themselves to create a <code>.log.fmt</code>
   for each log file. (That's kinda the goal. The
   <a href="update-logfmt.html"><code>update-logfmt</code></a> scripts are a stop-gap workaround.)</li>
</ul>
              
            </div>
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="logopen.html" class="btn btn-neutral float-right" title="logopen()">Next <span class="icon icon-circle-arrow-right"></span></a>
      
      
        <a href="index.html" class="btn btn-neutral" title="Intro"><span class="icon icon-circle-arrow-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <!-- Copyright etc -->
    
  </div>

  Built with <a href="https://www.mkdocs.org/">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
      
        </div>
      </div>

    </section>

  </div>

  <div class="rst-versions" role="note" aria-label="versions">
    <span class="rst-current-version" data-toggle="rst-current-version">
      
      
        <span><a href="index.html" style="color: #fcfcfc;">&laquo; Previous</a></span>
      
      
        <span style="margin-left: 15px"><a href="logopen.html" style="color: #fcfcfc">Next &raquo;</a></span>
      
    </span>
</div>
    <script>var base_url = '.';</script>
    <script src="js/theme.js" defer></script>
    <script defer>
        window.onload = function () {
            SphinxRtdTheme.Navigation.enable(true);
        };
    </script>

</body>
</html>
