<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="shortcut icon" href="img/favicon.ico">
<title>logopen() - logfmt1</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700" />
<link rel="stylesheet" href="css/theme.css" />
<link rel="stylesheet" href="css/theme_extra.css" />
<link href="custom.css" rel="stylesheet" />
<link href="syntax.css" rel="stylesheet" />
<script>
// Current page data
var mkdocs_page_name = "logopen()";
var mkdocs_page_input_path = "logopen.md";
var mkdocs_page_url = null;
</script>
<script src="js/jquery-2.1.1.min.js" defer></script>
<script src="js/modernizr-2.8.3.min.js" defer></script>
</head>
<body class="wy-body-for-nav" role="document">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a href="." class="icon icon-home"> logfmt1</a>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="index.html">Intro</a>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="log.fmt.html">.log.fmt</a>
</li>
</ul>
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal current" href="logopen.html">logopen()</a>
<ul class="current">
</ul>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="regex.html">regex()</a>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="fmt.html">global .fmt db</a>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="update-logfmt.html">update-logfmt</a>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="logex.html">logex</a>
</li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href=".">logfmt1</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href=".">Docs</a> »</li>
<li>logopen()</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main">
<div class="section">
<h3 id="logopen">logopen()</h3>
<div class="doc doc-object doc-class">
<h2 class="hidden-toc" href="#logfmt1.logopen" id="logfmt1.logopen" style="visibility: hidden; position: absolute;">
</h2>
<div class="doc doc-contents first">
<p>Open log file and its associated .fmt specifier, to iterate over log lines
as dictionary.
File-style wrapper that yields parsed dictionaries instead of string lines.</p>
<div class="codehilite">
<pre><span></span><code><span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">logfmt1</span><span class="o">.</span><span class="n">logopen</span><span class="p">(</span><span class="s2">"/var/log/apache2/access.log"</span><span class="p">,</span> <span class="n">debug</span><span class="o">=</span><span class="n">True</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s2">"remote_host"</span><span class="p">])</span>
</code></pre>
</div>
<p>Though you might want to keep a reference to the iterator to utilize
<code>.names()</code> and <code>.alias{}</code> manually.</p>
<div class="doc doc-children">
<div class="doc doc-object doc-method">
<h2 class="doc doc-heading" id="logfmt1.logopen.__init__">
<code class="highlight language-python">
__init__<span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">logfn</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">fmt</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">debug</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">fail</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">duplicate</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> </code>
<span class="doc doc-properties">
<small class="doc doc-property doc-property-special"><code>special</code></small>
</span>
</h2>
<div class="doc doc-contents ">
<p>Open log file and its associated .fmt specifier, to iterate over log lines
as dictionary.</p>
<p><strong>Parameters:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>logfn</code></td>
<td><code>str</code></td>
<td>
<p>Filename of .log file to open. Which should be
accompanied by a .log.fmt declaration to allow unpacking
lines into dictionary.</p>
</td>
<td><code>''</code></td>
</tr>
<tr>
<td><code>fmt</code></td>
<td><code>dict</code></td>
<td>
<p>Alternatively to existing .log.fmt, a predefined class
might be given with <code>fmt={"class":"syslog"}</code>. You might
even add a fixated <code>{"record":"%a %t %e %F"}</code> format string
this way.</p>
</td>
<td><code>None</code></td>
</tr>
<tr>
<td><code>debug</code></td>
<td><code>bool</code></td>
<td>
<p>In case of log extraction failures, prints (stdout)
some regex debugging.</p>
</td>
<td><code>False</code></td>
</tr>
<tr>
<td><code>fail</code></td>
<td><code>bool</code></td>
<td>
<p>In case of failure, just error out instead of continuing
the iterator.</p>
</td>
<td><code>False</code></td>
</tr>
<tr>
<td><code>duplicate</code></td>
<td><code>bool</code></td>
<td>
<p>Automatically expand aliases. This effectively
copies row entries.</p>
</td>
<td><code>True</code></td>
</tr>
</tbody>
</table>
<p><strong>Attributes:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>f</code></td>
<td><code>file</code></td>
<td>
<p>Read handle onto log file</p>
</td>
</tr>
<tr>
<td><code>debug</code></td>
<td><code>bool</code></td>
<td>
<p>Debug flag</p>
</td>
</tr>
<tr>
<td><code>fail</code></td>
<td><code>bool</code></td>
<td>
<p>Exception flag</p>
</td>
</tr>
<tr>
<td><code>alias</code></td>
<td><code>dict</code></td>
<td>
<p>List of row aliases</p>
</td>
</tr>
<tr>
<td><code>container</code></td>
<td><code>dict</code></td>
<td>
<p>Rules for field expansion</p>
</td>
</tr>
<tr>
<td><code>rx</code></td>
<td><code>re.compile</code></td>
<td>
<p>Compiled regex</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Exceptions:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>StopIteration</code></td>
<td>
<p>For EOF or if the regex failed and fail=True</p>
</td>
</tr>
<tr>
<td><code>FileNotFound</code></td>
<td>
<p>If logfn doesn't exist etc.</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Returns:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>iterator</code></td>
<td>
<p>Traverses log file line-wise, but yields dictionaries.</p>
</td>
</tr>
</tbody>
</table>
<details class="quote">
<summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
<div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">logfn</span><span class="o">=</span><span class="s2">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">debug</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">fail</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">duplicate</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Open log file and its associated .fmt specifier, to iterate over log lines</span>
<span class="sd"> as dictionary.</span>
<span class="sd"> Args:</span>
<span class="sd"> logfn (str): Filename of .log file to open. Which should be</span>
<span class="sd"> accompanied by a .log.fmt declaration to allow unpacking</span>
<span class="sd"> lines into dictionary.</span>
<span class="sd"> fmt (dict): Alternatively to existing .log.fmt, a predefined class</span>
<span class="sd"> might be given with `fmt={"class":"syslog"}`. You might</span>
<span class="sd"> even add a fixated `{"record":"%a %t %e %F"}` format string</span>
<span class="sd"> this way.</span>
<span class="sd"> debug (bool): In case of log extraction failures, prints (stdout)</span>
<span class="sd"> some regex debugging.</span>
<span class="sd"> fail (bool): In case of failure, just error out instead of continuing</span>
<span class="sd"> the iterator.</span>
<span class="sd"> duplicate (bool): Automatically expand aliases. This effectively</span>
<span class="sd"> copies row entries.</span>
<span class="sd"> Attributes:</span>
<span class="sd"> f (file): Read handle onto log file</span>
<span class="sd"> debug (bool): Debug flag</span>
<span class="sd"> fail (bool): Exception flag</span>
<span class="sd"> alias (dict): List of row aliases</span>
<span class="sd"> container (dict): Rules for field expansion</span>
<span class="sd"> rx (re.compile): Compiled regex</span>
<span class="sd"> Raises:</span>
<span class="sd"> StopIteration: For EOF or if the regex failed and fail=True</span>
<span class="sd"> FileNotFound: If logfn doesn't exist etc.</span>
<span class="sd"> Returns:</span>
<span class="sd"> iterator: Traverses log file line-wise, but yields dictionaries.</span>
<span class="sd"> """</span>
<span class="bp">self</span><span class="o">.</span><span class="n">debug</span> <span class="o">=</span> <span class="n">debug</span>
<span class="bp">self</span><span class="o">.</span><span class="n">fail</span> <span class="o">=</span> <span class="n">fail</span>
<span class="bp">self</span><span class="o">.</span><span class="n">duplicate</span> <span class="o">=</span> <span class="n">duplicate</span>
<span class="c1"># try + nicer error....</span>
<span class="bp">self</span><span class="o">.</span><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">logfn</span><span class="p">,</span> <span class="s2">"r"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"utf-8"</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">fmt</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">fmt</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">logfn</span><span class="si">}</span><span class="s2">.fmt"</span><span class="p">,</span> <span class="s2">"r"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"utf-8"</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">"Use `update-logfmt` or `modseccfg`→File→Install→update_logfmt.py to generate a *.log.fmt descriptor.</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
<span class="n">fmt</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"class"</span><span class="p">:</span><span class="s2">"apache combined"</span><span class="p">}</span>
<span class="c1">#fmt = rulesdb.find_by_glob(logfn)</span>
<span class="n">fmt</span> <span class="o">=</span> <span class="n">rulesdb</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span>
<span class="n">fmt</span><span class="p">,</span> <span class="c1"># this should be in regex/update</span>
<span class="n">rulesdb</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">fmt</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"class"</span><span class="p">))</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">alias</span> <span class="o">=</span> <span class="n">fmt</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"alias"</span><span class="p">,</span> <span class="p">{})</span>
<span class="bp">self</span><span class="o">.</span><span class="n">container</span> <span class="o">=</span> <span class="n">fmt</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"container"</span><span class="p">,</span> <span class="p">{})</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rx</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">rx2re</span><span class="p">(</span><span class="n">regex</span><span class="p">(</span><span class="n">fmt</span><span class="p">)))</span>
</code></pre>
</div>
</details>
</div>
</div>
<div class="doc doc-object doc-method">
<h2 class="doc doc-heading" id="logfmt1.logopen.debug_rx">
<code class="highlight language-python">
debug_rx<span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span> </code>
</h2>
<div class="doc doc-contents ">
<p>ANSI output for debugging regex/fmt string. Automatically
invoked for failing lines if <code>debug=True</code> was given.</p>
<p><img alt="failed regex + log line" src="https://imgur.com/QBKzDsK.png" /></p>
<p><strong>Parameters:</strong></p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>line</code></td>
<td><code>str</code></td>
<td>
<p>Current raw line (string) from log file.</p>
</td>
<td><em>required</em></td>
</tr>
</tbody>
</table>
<p>Output:
Prints directly to stdout using ANSI escape sequences to
highlight where regex failed on input line. It's not very
exact anymore, but reasonably speedy.</p>
<details class="quote">
<summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
<div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="nf">debug_rx</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">line</span><span class="p">:</span><span class="nb">str</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> ANSI output for debugging regex/fmt string. Automatically</span>
<span class="sd"> invoked for failing lines if `debug=True` was given.</span>
<span class="sd"> </span>
<span class="sd"> Args:</span>
<span class="sd"> line: Current raw line (string) from log file.</span>
<span class="sd"> Output:</span>
<span class="sd"> Prints directly to stdout using ANSI escape sequences to</span>
<span class="sd"> highlight where regex failed on input line. It's not very</span>
<span class="sd"> exact anymore, but reasonably speedy.</span>
<span class="sd"> """</span>
<span class="n">rx</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rx</span><span class="o">.</span><span class="n">pattern</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span>
<span class="c1">#rx_cut = re.compile("[^)]* \(\?P<\w+> ( [^()]+ | \([^()]+\) )+ \) [^()]* \Z", re.X)</span>
<span class="c1"># iteratively strip (?...) capture groups</span>
<span class="k">while</span> <span class="nb">len</span><span class="p">(</span><span class="n">rx</span><span class="p">)</span> <span class="ow">and</span> <span class="n">rx</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s2">"(?P<"</span><span class="p">)</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1">#fail = rx_cut.search(rx)</span>
<span class="c1">#if fail: fail = fail.group(0)</span>
<span class="c1">#else: fail = "<unknown-last-capture>"; break</span>
<span class="n">last</span> <span class="o">=</span> <span class="n">rx</span><span class="o">.</span><span class="n">rindex</span><span class="p">(</span><span class="s2">"(?P<"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">last</span> <span class="o"><</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">fail</span> <span class="o">=</span> <span class="s2">"<unknown-last-capture>"</span><span class="p">;</span> <span class="k">break</span>
<span class="n">fail</span> <span class="o">=</span> <span class="n">rx</span><span class="p">[</span><span class="n">last</span><span class="p">:]</span>
<span class="c1">#print(f"testfail: `{fail}`")</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">rx</span> <span class="o">=</span> <span class="n">rx</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">last</span><span class="p">]</span>
<span class="n">rx</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s2">"[^)]*$"</span><span class="p">,</span> <span class="s2">""</span><span class="p">,</span> <span class="n">rx</span><span class="p">)</span>
<span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span>
<span class="k">break</span> <span class="c1"># works now, so `fail` was the culprit</span>
<span class="k">except</span><span class="p">:</span>
<span class="c1"># likely broke regex nesting, try removing next (?...)</span>
<span class="k">pass</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">matched</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
<span class="n">matched</span> <span class="o">=</span> <span class="n">matched</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">matched</span> <span class="o">=</span> <span class="s2">""</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"</span><span class="se">\033</span><span class="s2">[36m"</span> <span class="o">+</span> <span class="s2">"failed regex section: </span><span class="se">\033</span><span class="s2">[1;33;41m"</span> <span class="o">+</span> <span class="n">fail</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\033</span><span class="s2">[40;0m"</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"</span><span class="se">\033</span><span class="s2">[42m"</span> <span class="o">+</span> <span class="n">matched</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\033</span><span class="s2">[41m"</span> <span class="o">+</span> <span class="n">line</span><span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">matched</span><span class="p">):]</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\033</span><span class="s2">[40;0m"</span><span class="p">)</span>
</code></pre>
</div>
</details>
</div>
</div>
<div class="doc doc-object doc-method">
<h2 class="doc doc-heading" id="logfmt1.logopen.names">
<code class="highlight language-python">
names<span class="p">(</span><span class="bp">self</span><span class="p">)</span> </code>
</h2>
<div class="doc doc-contents ">
<p>Get column names from generated .fmt regex.</p>
<p><strong>Returns:</strong></p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>list</code></td>
<td>
<p>dictionary keys of row (without aliases).</p>
</td>
</tr>
</tbody>
</table>
<details class="quote">
<summary>Source code in <code>logfmt1/logfmt1.py</code></summary>
<div class="highlight">
<pre><span></span><code><span class="k">def</span> <span class="nf">names</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Get column names from generated .fmt regex.</span>
<span class="sd"> Returns:</span>
<span class="sd"> list: dictionary keys of row (without aliases).</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s2">"\(\?P?<(\w+)>"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">rx</span><span class="o">.</span><span class="n">pattern</span><span class="p">)</span>
</code></pre>
</div>
</details>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="regex.html" class="btn btn-neutral float-right" title="regex()">Next <span class="icon icon-circle-arrow-right"></span></a>
<a href="log.fmt.html" class="btn btn-neutral" title=".log.fmt"><span class="icon icon-circle-arrow-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<!-- Copyright etc -->
</div>
Built with <a href="https://www.mkdocs.org/">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<div class="rst-versions" role="note" aria-label="versions">
<span class="rst-current-version" data-toggle="rst-current-version">
<span><a href="log.fmt.html" style="color: #fcfcfc;">« Previous</a></span>
<span style="margin-left: 15px"><a href="regex.html" style="color: #fcfcfc">Next »</a></span>
</span>
</div>
<script>var base_url = '.';</script>
<script src="js/theme.js" defer></script>
<script defer>
window.onload = function () {
SphinxRtdTheme.Navigation.enable(true);
};
</script>
</body>
</html>