GUI editor to tame mod_security rules

βŒˆβŒ‹ βŽ‡ branch:  modseccfg


Check-in [9be300bfed]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Stub manpage for logfmt(5)
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 9be300bfedba17d63ecf245a04d8a624cbbf544d4ff7042e7acf115aa6c2dca3
User & Date: mario 2021-01-03 20:06:05
Context
2021-01-03
20:07
html2mallard split up rules for different templates, add --debug flag check-in: b7a9065f17 user: mario tags: trunk
20:06
Stub manpage for logfmt(5) check-in: 9be300bfed user: mario tags: trunk
20:04
Add minimal docs on modify dialog. check-in: 56ad63cc02 user: mario tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to logfmt1/logfmt1.py.

8
9
10
11
12
13
14

15
16
17
18
19
20
21
# license: Apache-2.0
# pack:
#    logfmt1.py=/usr/lib/python3/dist-packages/
#    update_logfmt.py=/usr/bin/update-logfmt
#    ./logex.py=/usr/bin/logex
#    share=/usr/share/logfmt
#    manpage/*.1=/usr/share/man/man1/

# architecture: all
# depends: python (>= 3.6)
# url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
# documentation: https://fossil.include-once.org/modseccfg/doc/trunk//logfmt1/html/index.html
#
# Logging format strings to regex conversion.
#







>







8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# license: Apache-2.0
# pack:
#    logfmt1.py=/usr/lib/python3/dist-packages/
#    update_logfmt.py=/usr/bin/update-logfmt
#    ./logex.py=/usr/bin/logex
#    share=/usr/share/logfmt
#    manpage/*.1=/usr/share/man/man1/
#    manpage/*.5=/usr/share/man/man5/
# architecture: all
# depends: python (>= 3.6)
# url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
# documentation: https://fossil.include-once.org/modseccfg/doc/trunk//logfmt1/html/index.html
#
# Logging format strings to regex conversion.
#

Changes to logfmt1/manpage/logex.md.

63
64
65
66
67
68
69
70
| #name  | as json array |
| name   | whatever |


SEE ALSO
========

**python**(1), **update-logfmt**(1)







|
63
64
65
66
67
68
69
70
| #name  | as json array |
| name   | whatever |


SEE ALSO
========

**python**(1), **update-logfmt**(1), **logfmt**(5)

Added logfmt1/manpage/logfmt.5.







































































































































































































































































































































































































































































































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
.\"t
.\" Automatically generated by Pandoc 2.5
.\"
.TH "log.fmt" "5" "" "logfmt1 tools" "Version 0.3"
.hy
.SH .LOG.FMT
.SS A .log.fmt for each log file
.PP
Instead logfmt1 aims to have descriptors for each log file, in order to
make them parseable.
You can\[cq]t attempt anything but guesswork until you know what\[cq]s
in a file.
.PP
So the idea is to have a \f[C]*.fmt\f[R] next to each \f[C]*.log\f[R]
file, with a descriptor such as:
.IP
.nf
\f[C]
{
   \[dq]class\[dq]: \[dq]apache combined\[dq],
   \[dq]record\[dq]: \[dq]%h %l %u %t \[rs]\[dq]%r\[rs]\[dq] %>s %b\[dq]
}
\f[R]
.fi
.PP
Notably the \[lq]record\[rq] field should be the most current format
string that the application itself uses.
In order to resolve the placeholders, an application reference is kept
in \[lq]class\[rq].
Which allows combining the format string with placeholder field
definitions from the global .fmt database
(\f[C]/usr/share/logfmt\f[R]) (fmt.md) database.
.SS common classes
.PP
There aren\[cq]t many predefined classes yet, but special values that
could work without a current \f[C]\[dq]record\[dq]:\f[R] declaration
might be:
.TP
.B \f[C]\[dq]class\[dq]: \[dq]grok syslog\[dq]\f[R]
Reads the according definition from a .grok (or perhaps preconverted)
pattern definition.
Which are largely static patterns.
.TP
.B \f[C]\[dq]class\[dq]: \[dq]inilog\[dq]\f[R]
For Heroku/Go \[lq]logfmt\[rq] style logs comprised of only key=value
fields
.TP
.B \f[C]\[dq]class\[dq]: \[dq]json appmoniker\[dq]\f[R]
For real JSON logs, with an application identifier here (for decoration)
.TP
.B \f[C]\[dq]class\[dq]: \[dq]apache common\[dq]\f[R]
Reads a predefined/static record: definition from the global
apache.common.fmt (fmt.md).
Which of course means it would fail to parse, if the user diverted the
LogFormat declaration in Apache.
.PP
Note that predefined classes undermine the purpose of logfmt1, in that
they\[cq]re only suitable for static/non\-variant log formats.
.SS additional fields
.PP
The *.log.fmt itself might declare definitions such as aliases and more
specific/custom placeholders.
.IP
.nf
\f[C]
{
   \[dq]class\[dq]: \[dq]apache cust3\[dq],
   \[dq]record\[dq]: \[dq]%a %h %{iso}t \[aq]%r\[aq] %s\[dq],
   \[dq]fields\[dq]: {
       \[dq]%{iso}t\[dq]: { \[dq]id\[dq]: \[dq]datetime\[dq], \[dq]rx\[dq]: \[dq]...\[dq] }
   },
   \[dq]alias\[dq]: {
       \[dq]iso8601\[dq]: \[dq]datetime\[dq],
   }
}
\f[R]
.fi
.PP
Which ought to be joined and override any global <fmt> definitions.
Though such user customizations are more likely to be applied there
anyway.
Care should be taken by \f[C]update\-logfmt\f[R] or applications to not
jettison user\-customized *.log.fmt options.
.SS rationale
.PP
Having the .fmt files adjecent to log files seems the most convenient
option.
.IP \[bu] 2
Appending a \f[C].fmt\f[R] suffix to the \f[C]\&....log\f[R] filename
doesn\[cq]t obstruct tab completion as much as \f[C].fmt\f[R]
substituting \f[C].log\f[R].
.IP \[bu] 2
Doesn\[cq]t require a lookup table or directory, with additional
permission or updating woes.
.IP \[bu] 2
And (over time) enabled applications themselves to create a
\f[C].log.fmt\f[R] for each log file.
(That\[cq]s kinda the goal.
The \f[C]update\-logfmt\f[R] (update-logfmt.md) scripts are a stop\-gap
workaround.)
.SH GLOBAL .fmt DATABASE
.PP
While each log file should be accompanied by a .fmt
descriptor (log.fmt.md), the global database in
\f[C]/usr/share/logfmt/\f[R] contains a full .fmt field definition for
each class.
And the cross\-section of both allows to construct a regex.
.PP
Most notably the \f[C]\[dq]fields\[dq]:\f[R] and
\f[C]\[dq]placeholder\[dq]:\f[R] are used to turn the
\f[C]\[dq]record\[dq]:\f[R] string definition into a capture pattern.
.SS .fmt Example
.PP
The Apache format definition (apache.fmt) contains:
.IP
.nf
\f[C]
{
    \[dq]class\[dq]: \[dq]apache generic\[dq],
    \[dq]separator\[dq]: \[dq] \[dq],
    \[dq]rewrite\[dq]: {
        \[dq]%[\[rs]\[rs]d!,+\[rs]\[rs]\-]+\[dq]: \[dq]%\[dq],
        \[dq]%%\[dq]: \[dq]%\[dq]
    },
    \[dq]placeholder\[dq]: \[dq]%[<>]?(?:\[rs]\[rs]w*\[rs]\[rs]{[\[ha]\[rs]\[rs]}]+\[rs]\[rs]})?\[rs]\[rs]\[ha]?\[rs]\[rs]w+\[dq],
    \[dq]fields\[dq]: {
        \[dq]%a\[dq]: {  \[dq]id\[dq]: \[dq]remote_addr\[dq],    \[dq]rx\[dq]: \[dq][\[rs]\[rs]d.:a\-f]+\[dq]  },
        \[dq]%h\[dq]: {  \[dq]id\[dq]: \[dq]remote_host\[dq],    \[dq]rx\[dq]: \[dq][\[rs]\[rs]w\[rs]\[rs]\-.:]+\[dq]  },
        \[dq]%{c}h\[dq]: { \[dq]id\[dq]: \[dq]remote_host\[dq],  \[dq]rx\[dq]: \[dq][\[rs]\[rs]w\[rs]\[rs]\-.:]+\[dq]  },
        \[dq]%A\[dq]: {  \[dq]id\[dq]: \[dq]local_address\[dq],  \[dq]rx\[dq]: \[dq][\[rs]\[rs]d.:a\-f]+\[dq]  },
        \[dq]%u\[dq]: {  \[dq]id\[dq]: \[dq]remote_user\[dq],    \[dq]rx\[dq]: \[dq][\[rs]\[rs]\-\[rs]\[rs]w\[at].]+\[dq]  },
        \[dq]%t\[dq]: {  \[dq]id\[dq]: \[dq]request_time\[dq],   \[dq]rx\[dq]: \[dq]\[rs]\[rs][?(\[rs]\[rs]d[\[rs]\[rs]d:\[rs]\[rs]w\[rs]\[rs]s:./\[rs]\[rs]\-+,;]+)\[rs]\[rs]]?\[dq]  },
        \&...
    },
    \[dq]alias\[dq]: {
        \[dq]remote_address\[dq]: \[dq]remote_addr\[dq],
        \[dq]ip\[dq]: \[dq]remote_addr\[dq],
        \[dq]file\[dq]: \[dq]request_file\[dq],
        \[dq]size\[dq]: \[dq]bytes_sent\[dq],
        \&...
    },
    \[dq]expand\[dq]: {
        \[dq]%\[rs]\[rs]{([\[ha]{}]+)\[rs]\[rs]}t\[dq]: {
            \[dq]id\[dq]: \[dq]request_time\[dq],
            \[dq]class\[dq]: \[dq]strftime\[dq],
            \[dq]record\[dq]: \[dq]$1\[dq]
        }
    },
    \[dq]container\[dq]: {
        \[dq]message\[dq]: {
            \[dq]id\[dq]: \[dq]$1\[dq],
            \[dq]value\[dq]: \[dq]$2\[dq],
            \[dq]rx\[dq]: \[dq]\[rs]\[rs][(\[rs]\[rs]w+) \[rs]\[dq](.*?)\[rs]\[dq]\[rs]\[rs]]\[dq],
            \[dq]class\[dq]: \[dq]apache mod_security\[dq]
        }
    },
    \[dq]glob\[dq]: [\[dq]/var/log/apache*/*acc*.log\[dq]]
}
\f[R]
.fi
.PP
It usually does not describe a default \[lq]record\[rq] format (like the
local .log.fmt descriptors do).
.SS class:
.PP
The class in the global database is largely decorative.
The filenames instead define the heritage of rules/fields.
The \[lq]class\[rq] as declared by a .log.fmt is mapped onto
\f[C]/usr/share/logfmt/application.variant.fmt\f[R].
.IP \[bu] 2
Usually there\[cq]s just one variant level per log type.
But the lookup is supposed to be mildly recursive.
.IP \[bu] 2
Essentially it should merge \f[C]*.log.fmt\f[R] with
\f[C]appclass.variant.fmt\f[R] and \f[C]appclass.fmt\f[R] applied last,
so the most specific definitions are retained.
.IP \[bu] 2
There\[cq]s also a generic \[lq]grok\[rq] class.
But the patterns therein are largely static (not build from variable
format strings).
.IP \[bu] 2
Some special classes like \[lq]json\[rq] might exist.
(Not supported by logfmt1)
.SS record:
.PP
The \[lq]record\[rq] entry is not usually present in the global .fmt
definition.
Some super specific variant definitions (for example apache.error.fmt)
or static formats (syslog.fmt) might however.
.SS separator:
.PP
Most log formats use spaces for separating %placeholder fields.
And simpler implementations might just split up the \[lq]record\[rq]
declaration on this.
.SS placeholder:
.PP
While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields.
It should account for prefixes/suffixes, unless those got cleared by the
\f[C]rewrite\f[R] map.
.PP
Not all formatstrings use \f[C]%\[rs]w+\f[R] to signal placeholders.
In nginx for instance the sigil \f[C]$\[rs]w+\f[R] introduces
placeholders (variable names, really).
.SS rewrite:
.PP
A list/map of regex to apply before any transformations or field
lookups.
Which can be used to mask or simplify placeholder definitions (for
instance clean up the Apache conditional prefixes) or regex meta
characters.
.IP \[bu] 2
The \f[C]record\f[R] field starts as a static string, but is meant to be
turned into a regex.
.IP \[bu] 2
Therefore meta characters (such as \f[C]|\f[R] or \f[C][]\f[R]) have to
be taken care of.
Which is what the \f[C]rewrite\f[R] map is lazily used for.
.IP \[bu] 2
Better implementations might look up the placeholders, and automatically
escape the rest of the the \[lq]record\[rq] format string.
.SS fields:
.PP
The core of the global .fmt definitions are the field lists.
Each defines a static %F placeholder and associaties it with a default
field name (id:) and regex (rx:) or even a grok definition (grok:).
.PP
.TS
tab(@);
l l.
T{
key
T}@T{
purpose
T}
_
T{
\f[C]%F\f[R]
T}@T{
\f[B]JSON key\f[R]: static placeholder string (not a regex itself)
T}
T{
id
T}@T{
field identifier, as specified by the application (internal name)
T}
T{
rx
T}@T{
regex which %F placeholder gets replaced with
T}
T{
grok
T}@T{
alternatively to regex, %F might be turned into %PATTERN:id
T}
T{
type
T}@T{
\[lq]int\[rq] and \[lq]float\[rq] could designate strictly numeric
fields
T}
.TE
.IP \[bu] 2
As part of the regex transformation, a \f[C]%F\f[R] could be turned into
\f[C](?<id>\[rs]S+)\f[R] for instance.
.IP \[bu] 2
If there\[cq]s any unnamed capture group \f[C](\&...)\f[R], it should be
augmented into a named capture group \- instead of the whole match.
(To account for implicit wrapping.)
.IP \[bu] 2
The \f[C]rx\f[R] itself might however specify named subgroups (like
request_line in Apache logs, itself comprised of _method, _path,
_protocol, or the datetime made up of tm_wday, tm_year, tm_whatever).
.IP \[bu] 2
\f[C]\[rs]S+\f[R] is also used as fallback for entirely undefined
placeholders (no expand definition matched) in logfmt1.
.IP \[bu] 2
\f[C]grok\f[R] isn\[cq]t currently used, but might allow for simpler
transformations (indirectly into a grok pattern, and later a regex).
.SS expand:
.PP
The expand declarations are used to construct unknown
fields/placeholders.
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.
Thus it simply can be applied before separator/placeholder are looked
at, to augment the known \f[C]fields\f[R] list.
.PP
.TS
tab(@);
l l.
T{
key
T}@T{
purpose
T}
_
T{
\f[C]%\[rs]{(\[rs]w+)\[rs]}t\f[R]
T}@T{
\f[B]JSON key\f[R]: a regex to detect mutable placeholders
T}
T{
id
T}@T{
name for newly created fields entry, might use captures\[aa] $1
T}
T{
rx
T}@T{
for static definitions (often just +)
T}
T{
if_quoted
T}@T{
alternative regex, if placeholder was enclosed in \[lq]%+\[rq] quotes
T}
T{
class
T}@T{
recurse into other .fmt types
T}
T{
record
T}@T{
can be set to $2 if class: recursion is defined
T}
.TE
.IP \[bu] 2
Typically it suffices to specify the \f[C]id\f[R] and \f[C]rx\f[R]
field.
.IP \[bu] 2
If no \f[C]id\f[R] is given, then the regex capture is normalized into
an identifier (non\-alphanumerics stripped, all lowercased).
.IP \[bu] 2
But the \f[C]id\f[R] or \f[C]record\f[R] value might be set with regex
captures (e.g.\ \f[C]$1\f[R] or \f[C]$2\f[R]) or compound values
(\f[C]\[dq]id\[dq]: \[dq]newfield_$1\[dq]\f[R]).
.IP \[bu] 2
And logfmt1 allows to recurse into other format types per
\f[C]class\f[R] (which is used to expand the captured
\f[C]\[dq]record\[dq]: \[dq]$1\[dq]\f[R] into regex tokens).
.SS alias:
.PP
Maps alternative/more common field names onto the declared field
\f[C]id\f[R]s.
.PP
To get to some state of standardization, the field ids usually refer to
application\-internal names.
(For instance \f[C]log_pfn_register(\&...,\&...,cb_id)\f[R] names in
Apache).
And those aren\[cq]t always the more commonly used identifiers.
.PP
Thus aliases makes sense not just for convenience, but also to be
compatible to other common names (e.g.\ w3c extend log format names like
\f[C]cs\-time\f[R]).
.SS container:
.PP
Is utilized by logopen() to extract additional fields (lists even) from
one of the existing fields.
This is usually done at row traversal.
And makes sense for application\-specific subformats in logs.
Such as any \f[C]key=value\f[R] lists in the main message field.
.PP
.TS
tab(@);
l l.
T{
key
T}@T{
purpose
T}
_
T{
\f[C]message\f[R]
T}@T{
\f[B]JSON key\f[R]: from which field to extract
T}
T{
rx
T}@T{
regex to detect and capture (key)=(value) fields
T}
T{
id
T}@T{
unpacked field name (usually just \f[C]$1\f[R] from the rx capture
T}
T{
value
T}@T{
value from capture (so \f[C]$2\f[R] typically)
T}
T{
class
T}@T{
decorative description (no .fmt recursion supported in logfmt1)
T}
.TE
.IP \[bu] 2
The entries here might become lists, since commonly there\[cq]s just one
\f[C]message\f[R] field in logs, yet multiple key:value schemes might be
utilized within.
.IP \[bu] 2
Or the target field might become a \f[C]\[dq]extract_from\[dq]:\f[R]
property, and \f[C]container\f[R] a list itself.
.IP \[bu] 2
Still not sure if automatic list conversion is a good idea.
\- Standard fields get an enumaration suffix
\f[C](?<request_uri2>\&...)\f[R] if duplicated.
.SS glob:
.PP
Might be used by log processors to look up a log class, based on file
names, if no .log.fmt is declared.
.SS #comment: fields
.PP
Documentation entries in the .fmt files have keys starting with
\f[C]#\f[R].
For example \f[C]\[dq]#license\[dq]:\f[R] or
\f[C]\[dq]#origin\[dq]:\f[R].
Which is simpler than using JSON with comments (JSOL/JSON5).
.PP
   *   *   *   *   *
.SS Other format files
.PP
!!! Note This section is about fictional features.
.SS .grok definitions
.RS
.PP
Not implemented yet.
.RE
.PP
The logfmt/ directory might also contain .grok files, which get
transformed into .fmt structures.
(Probably with the grok: parameter for fields, and a grok: pattern table
alongside regular fields:).
.PP
There\[cq]s already a pretransformed \f[C]grok.fmt\f[R], which however
requires \f[C]%{GROK:%{PATTERN:id}}\f[R] references currently.
.SS .lnav formats
.RS
.PP
Not implemented yet.
.RE
.PP
Likewise could we use lnav .json format definitions.
Those are static too, however.
.SH SEE ALSO
.PP
\f[B]update\-logfmt\f[R](1)

Added logfmt1/manpage/logfmt.md.













































































































































































































































































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
% log.fmt(5) logfmt1 tools | Version 0.3

.LOG.FMT
========

## A .log.fmt for each log file

Instead logfmt1 aims to have descriptors for each log file, in order to
make them parseable. You can't attempt anything but guesswork until you
know what's in a file.

So the idea is to have a `*.fmt` next to each `*.log` file, with a
descriptor such as:

```json
{
   "class": "apache combined",
   "record": "%h %l %u %t \"%r\" %>s %b"
}
```

Notably the "record" field should be the most current format string that
the application itself uses. In order to resolve the placeholders, an
application reference is kept in "class". Which allows combining the
format string with placeholder field definitions from the global
[.fmt database (`/usr/share/logfmt`)](fmt.md) database.


### common classes

There aren't many predefined classes yet, but special values that could
work without a current `"record":` declaration might be:

`"class": "grok syslog"`
:  Reads the according definition from a .grok (or perhaps preconverted)
   pattern definition. Which are largely static patterns.

`"class": "inilog"`
:  For Heroku/Go "logfmt" style logs comprised of only key=value fields

`"class": "json appmoniker"`
:  For real JSON logs, with an application identifier here (for decoration)

`"class": "apache common"`
:  Reads a predefined/static record: definition from the global
   [apache.common.fmt](fmt.md). Which of course means it would fail to
   parse, if the user diverted the LogFormat declaration in Apache.

Note that predefined classes undermine the purpose of logfmt1, in that
they're only suitable for static/non-variant log formats.


### additional fields

The *.log.fmt itself might declare definitions such as aliases and
more specific/custom placeholders.

```json
{
   "class": "apache cust3",
   "record": "%a %h %{iso}t '%r' %s",
   "fields": {
       "%{iso}t": { "id": "datetime", "rx": "..." }
   },
   "alias": {
       "iso8601": "datetime",
   }
}
```

Which ought to be joined and override any global [fmt](fmt) definitions.
Though such user customizations are more likely to be applied there anyway.
Care should be taken by `update-logfmt` or applications to not jettison
user-customized *.log.fmt options.


### rationale

Having the .fmt files adjecent to log files seems the most convenient
option.

 * Appending a `.fmt` suffix to the `….log` filename doesn't obstruct tab
   completion as much as `.fmt` substituting `.log`.
 * Doesn't require a lookup table or directory, with additional permission
   or updating woes.
 * And (over time) enabled applications themselves to create a `.log.fmt`
   for each log file. (That's kinda the goal. The
   [`update-logfmt`](update-logfmt.md) scripts are a stop-gap workaround.)


GLOBAL .fmt DATABASE
====================

While each log file should be accompanied by a [.fmt descriptor](log.fmt.md),
the global database in `/usr/share/logfmt/` contains a full .fmt field
definition for each class. And the cross-section of both allows to construct
a regex.

Most notably the `"fields":` and `"placeholder":` are used to turn the
`"record":` string definition into a capture pattern.


### .fmt Example

The Apache format definition (apache.fmt) contains:

```json
{
    "class": "apache generic",
    "separator": " ",
    "rewrite": {
        "%[\\d!,+\\-]+": "%",
        "%%": "%"
    },
    "placeholder": "%[<>]?(?:\\w*\\{[^\\}]+\\})?\\^?\\w+",
    "fields": {
        "%a": {  "id": "remote_addr",    "rx": "[\\d.:a-f]+"  },
        "%h": {  "id": "remote_host",    "rx": "[\\w\\-.:]+"  },
        "%{c}h": { "id": "remote_host",  "rx": "[\\w\\-.:]+"  },
        "%A": {  "id": "local_address",  "rx": "[\\d.:a-f]+"  },
        "%u": {  "id": "remote_user",    "rx": "[\\-\\w@.]+"  },
        "%t": {  "id": "request_time",   "rx": "\\[?(\\d[\\d:\\w\\s:./\\-+,;]+)\\]?"  },
        …
    },
    "alias": {
        "remote_address": "remote_addr",
        "ip": "remote_addr",
        "file": "request_file",
        "size": "bytes_sent",
        …
    },
    "expand": {
        "%\\{([^{}]+)\\}t": {
            "id": "request_time",
            "class": "strftime",
            "record": "$1"
        }
    },
    "container": {
        "message": {
            "id": "$1",
            "value": "$2",
            "rx": "\\[(\\w+) \"(.*?)\"\\]",
            "class": "apache mod_security"
        }
    },
    "glob": ["/var/log/apache*/*acc*.log"]
}
```

It usually does not describe a default "record" format (like the local .log.fmt descriptors do).


### class:

The class in the global database is largely decorative.  The filenames
instead define the heritage of rules/fields.  The "class" as declared by
a .log.fmt is mapped onto `/usr/share/logfmt/application.variant.fmt`.

 * Usually there's just one variant level per log type. But the lookup is
   supposed to be mildly recursive.
 * Essentially it should merge `*.log.fmt` with `appclass.variant.fmt` and
   `appclass.fmt` applied last, so the most specific definitions are retained.
 * There's also a generic "grok" class. But the patterns therein are largely
   static (not build from variable format strings).
 * Some special classes like "json" might exist. (Not supported by logfmt1)


### record:

The "record" entry is not usually present in the global .fmt definition. 
Some super specific variant definitions (for example apache.error.fmt) or
static formats (syslog.fmt) might however.


### separator:

Most log formats use spaces for separating %placeholder fields.  And simpler
implementations might just split up the "record" declaration on this.


### placeholder:

While logfmt1 instead uses a regex definition of possible %placeholder
strings to map onto fields. It should account for prefixes/suffixes, unless
those got cleared by the `rewrite` map.

Not all formatstrings use `%\w+` to signal placeholders. In nginx for instance
the sigil `$\w+` introduces placeholders (variable names, really).


### rewrite:

A list/map of regex to apply before any transformations or field lookups. 
Which can be used to mask or simplify placeholder definitions (for instance
clean up the Apache conditional prefixes) or regex meta characters.

 * The `record` field starts as a static string, but is meant to be turned
   into a regex.
 * Therefore meta characters (such as `|` or `[]`) have to be
   taken care of.  Which is what the `rewrite` map is lazily used for.
 * Better implementations might look up the placeholders, and automatically
   escape the rest of the the "record" format string.


### fields:

The core of the global .fmt definitions are the field lists.  Each defines a
static %F placeholder and associaties it with a default field name (id:) and
regex (rx:) or even a grok definition (grok:).

| key | purpose |
|-----|---------|
|`%F`| **JSON key**: static placeholder string (not a regex itself) |
| id | field identifier, as specified by the application (internal name) |
| rx | regex which %F placeholder gets replaced with |
| grok | alternatively to regex, %F might be turned into %PATTERN:id |
| type |  "int" and "float" could designate strictly numeric fields |

  * As part of the regex transformation, a `%F` could be turned into
    `(?<id>\S+)` for instance.
  * If there's any unnamed capture group `(…)`, it should be augmented
    into a named capture group - instead of the whole match. (To account
    for implicit wrapping.)
  * The `rx` itself might however specify named subgroups (like request_line
    in Apache logs, itself comprised of _method, _path, _protocol, or the
    datetime made up of tm_wday, tm_year, tm_whatever).
  * `\S+` is also used as fallback for entirely undefined placeholders
    (no expand definition matched) in logfmt1.
  * `grok` isn't currently used, but might allow for simpler transformations
    (indirectly into a grok pattern, and later a regex).


### expand:

The expand declarations are used to construct unknown fields/placeholders. 
Instead of static %placeholders, each entry describes a regex to detect
new/variant placeholders.  Thus it simply can be applied before
separator/placeholder are looked at, to augment the known `fields` list.

| key      |  purpose                                               |
|----------|--------------------------------------------------------|
| `%\{(\w+)\}t` | **JSON key**: a regex to detect mutable placeholders  |
| id     | name for newly created fields entry, might use capturesΒ΄ $1|
| rx     | for static definitions (often just \S+)                  |
|if_quoted| alternative regex, if placeholder was enclosed in "%\w+" quotes|
| class  | recurse into other .fmt types                            |
| record | can be set to $2 if class: recursion is defined          |


  * Typically it suffices to specify the `id` and `rx` field.
  * If no `id` is given, then the regex capture is normalized into
    an identifier (non-alphanumerics stripped, all lowercased).
  * But the `id` or `record` value might be set with regex captures
    (e.g. `$1` or `$2`) or compound values (`"id": "newfield_$1"`).
  * And logfmt1 allows to recurse into other format types per `class`
    (which is used to expand the captured `"record": "$1"` into regex
    tokens).


### alias:

Maps alternative/more common field names onto the declared field `id`s.

To get to some state of standardization, the field ids usually refer
to application-internal names. (For instance `log_pfn_register(…,…,cb_id)`
names in Apache). And those aren't always the more commonly used identifiers.

Thus aliases makes sense not just for convenience, but also to be compatible
to other common names (e.g. w3c extend log format names like `cs-time`).


### container:

Is utilized by logopen() to extract additional fields (lists even) from one
of the existing fields.  This is usually done at row traversal.  And makes
sense for application-specific subformats in logs.  Such as any `key=value`
lists in the main message field.

| key      |  purpose                                               |
|----------|--------------------------------------------------------|
| `message` | **JSON key**: from which field to extract  |
| rx     | regex to detect and capture (key)=(value) fields         |
| id     |unpacked field name (usually just `$1` from the rx capture|
| value  | value from capture (so `$2` typically)                   |
| class  | decorative description (no .fmt recursion supported in logfmt1) |


  * The entries here might become lists, since commonly there's just one
    `message` field in logs, yet multiple key:value schemes might be
    utilized within.
  * Or the target field might become a `"extract_from":` property, and
    `container` a list itself.
  * Still not sure if automatic list conversion is a good idea.  -
    Standard fields get an enumaration suffix `(?<request_uri2>…)` if
    duplicated.


### glob:

Might be used by log processors to look up a log class, based on file names,
if no .log.fmt is declared.


### #comment: fields

Documentation entries in the .fmt files have keys starting with `#`. For example
`"#license":` or `"#origin":`. Which is simpler than using JSON with
comments (JSOL/JSON5).

-----

### Other format files

!!! Note
    This section is about fictional features.


#### .grok definitions

> Not implemented yet.

The logfmt/ directory might also contain .grok files, which get transformed
into .fmt structures. (Probably with the grok: parameter for fields, and
a grok: pattern table alongside regular fields:).

There's already a pretransformed `grok.fmt`, which however requires
`%{GROK:%{PATTERN:id}}` references currently.


#### .lnav formats

> Not implemented yet.

Likewise could we use lnav .json format definitions. Those are static
too, however.


SEE ALSO
========

**update-logfmt**(1)

Changes to logfmt1/manpage/update-logfmt.1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
.\" Automatically generated by Pandoc 2.5
.\"
.TH "update\-logfmt" "1" "" "logfmt1 utilities" "Version 0.1"
.hy
.SH NAME
.PP
\f[B]update\-logfmt\f[R] \[em] crafts *.log.fmt files globally for known
application configurations
.SH SYNOPSIS
.PP
\f[B]update\-logfmt\f[R]
.PP
\f[B]update\-logfmt\f[R] [\f[B]\[en]test\f[R]|\f[B]\[en]verbose\f[R]]
.SH DESCRIPTION
.PP
The \f[C]update\-logfmt\f[R] script (should be in in /usr/bin/ when
installed via system package) will create .log.fmt (log.fmt.md) files
for known system logs.
.PP
It\[cq]s basically just a wrapper script that invokes handlers in
\f[C]/usr/share/logfmt/update/*\f[R].
Each of which scans one application config to locate and decorate its
according log files.
.SH SCRIPTS
.PP
Currently just supports:
.IP \[bu] 2
Apache (access, global, error and transfer logs, no forensic logs)












|



|
|


|







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
.\" Automatically generated by Pandoc 2.5
.\"
.TH "update\-logfmt" "1" "" "logfmt1 utilities" "Version 0.1"
.hy
.SH NAME
.PP
\f[B]update\-logfmt\f[R] \[em] crafts *.log.fmt files globally for known
application configurations
.SH SYNOPSIS
.PP
\f[B]update\-logfmt\f[R]
.PP
\f[B]update\-logfmt\f[R] [\f[I]\[en]test\f[R]|\f[I]\[en]verbose\f[R]]
.SH DESCRIPTION
.PP
The \f[C]update\-logfmt\f[R] script (should be in in /usr/bin/ when
installed via system package) will create
\f[I].log.fmt (log.fmt.md)\f[R] files for known system logs.
.PP
It\[cq]s basically just a wrapper script that invokes handlers in
\f[I]\f[CI]/usr/share/logfmt/update/*\f[I]\f[R].
Each of which scans one application config to locate and decorate its
according log files.
.SH SCRIPTS
.PP
Currently just supports:
.IP \[bu] 2
Apache (access, global, error and transfer logs, no forensic logs)

Changes to logfmt1/manpage/update-logfmt.md.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
% update-logfmt(1) logfmt1 utilities | Version 0.1


NAME
====

**update-logfmt** β€” crafts *.log.fmt files globally for known application configurations

SYNOPSIS
========

  **update-logfmt**

  **update-logfmt** \[**--test**|**--verbose**]


DESCRIPTION
===========

The `update-logfmt` script (should be in in /usr/bin/ when installed via
system package) will create [.log.fmt](log.fmt.md) files for known system
logs.

It's basically just a wrapper script that invokes handlers in
`/usr/share/logfmt/update/*`. Each of which scans one application
config to locate and decorate its according log files.


SCRIPTS
=======

Currently just supports:

 * Apache (access, global, error and transfer logs, no forensic logs)
 * Nginx (only accesss logs)

More static system logs (klog/syslog) might be supported in the next
version.


SEE ALSO
========

**python**(1), **logex**(1), **modseccfg**(1)













|






|



|


















|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
% update-logfmt(1) logfmt1 utilities | Version 0.1


NAME
====

**update-logfmt** β€” crafts *.log.fmt files globally for known application configurations

SYNOPSIS
========

  **update-logfmt**

  **update-logfmt** \[*--test*|*--verbose*]


DESCRIPTION
===========

The `update-logfmt` script (should be in in /usr/bin/ when installed via
system package) will create *[.log.fmt](log.fmt.md)* files for known system
logs.

It's basically just a wrapper script that invokes handlers in
*`/usr/share/logfmt/update/*`*. Each of which scans one application
config to locate and decorate its according log files.


SCRIPTS
=======

Currently just supports:

 * Apache (access, global, error and transfer logs, no forensic logs)
 * Nginx (only accesss logs)

More static system logs (klog/syslog) might be supported in the next
version.


SEE ALSO
========

**python**(1), **logex**(1), **modseccfg**(1), **logfmt**(5)

Changes to logfmt1/setup.py.

14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from glob import glob

setup(
    fn="./logfmt1.py",
    long_description="@README.rst",
    package_dir={"logfmt1": "./"},
    packages=["logfmt1"],
    data_files=[("man/man1", glob("manpage/*.1"))],
    package_data={
        "logfmt1": [
           "./share/*",
           "./share/update/*"
        ],
    },
    #data_files=[],







|







14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from glob import glob

setup(
    fn="./logfmt1.py",
    long_description="@README.rst",
    package_dir={"logfmt1": "./"},
    packages=["logfmt1"],
    data_files=[("man/man1", glob("manpage/*.1")), ("man/man5", glob("manpage/*.5"))],
    package_data={
        "logfmt1": [
           "./share/*",
           "./share/update/*"
        ],
    },
    #data_files=[],