GUI editor to tame mod_security rules

⌈⌋ ⎇ branch:  modseccfg


Check-in [28405ca3cf]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add manpages (seemingly setup(data_files=…) knows where to place them.)
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 28405ca3cf2434c136cfecb75c645c49558ab48e4c06518759dca2947ad8307a
User & Date: mario 2020-12-29 14:26:35
Context
2020-12-30
22:54
logfmt1 manual changes check-in: 9a5ae7b93b user: mario tags: trunk
2020-12-29
14:26
Add manpages (seemingly setup(data_files=…) knows where to place them.) check-in: 28405ca3cf user: mario tags: trunk
2020-12-28
22:12
Fix mkdocs/RTD table styling per sed check-in: 43e6843f24 user: mario tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to Makefile.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34



35


run:
	python3 -m modseccfg


setup:
	pandoc README.md -o README.rst
	version --read modseccfg/__init__.py --write modseccfg/mainwindow.py
	python3 setup.py bdist_wheel
	rm -r modseccfg.egg-info

upload: setup
	python3 setup.py bdist_wheel upload
	rm -r modseccfg.egg-info

t:
	pytest -v -v -v -v


logfmt1: deb whl
deb:
	cd logfmt1 ; xpm -f -s src -t deb  -n python3-logfmt1 logfmt1.py
	mv logfmt1/python3-logfmt1_*.deb .
	dpkg-deb -c python3-logfmt1_*.deb
deb_up:
	scp python3-logfmt1* io:apt/
whl:
	pandoc logfmt1/README.md -o logfmt1/README.rst
	cd logfmt1 && ./setup.py bdist_wheel
whl_up:
	twine upload dist/logfmt*

docs:
	cd logfmt1 ; PYTHONPATH=. mkdocs build -v -v -v
	sed -i 's/table\.docutils/table/g' logfmt1/html/css/theme.css 



	
>
>


<
<





<



<
















<



>
>
>
|
1
2
3
4


5
6
7
8
9

10
11
12

13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

29
30
31
32
33
34
35
#!/usr/bin/make

run:
	python3 -m modseccfg


setup:
	pandoc README.md -o README.rst
	version --read modseccfg/__init__.py --write modseccfg/mainwindow.py
	python3 setup.py bdist_wheel
	rm -r modseccfg.egg-info

upload: setup
	python3 setup.py bdist_wheel upload
	rm -r modseccfg.egg-info

t:
	pytest -v -v -v -v


logfmt1: deb whl
deb:
	cd logfmt1 ; xpm -f -s src -t deb  -n python3-logfmt1 logfmt1.py
	mv logfmt1/python3-logfmt1_*.deb .
	dpkg-deb -c python3-logfmt1_*.deb
deb_up:
	scp python3-logfmt1* io:apt/
whl:
	pandoc logfmt1/README.md -o logfmt1/README.rst
	cd logfmt1 && ./setup.py bdist_wheel
whl_up:
	twine upload dist/logfmt*

docs:
	cd logfmt1 ; PYTHONPATH=. mkdocs build -v -v -v
	sed -i 's/table\.docutils/table/g' logfmt1/html/css/theme.css 
%.1:	%.md
	pandoc --standalone -f markdown+pandoc_title_block -t man $< -o $@
man:	logfmt1/manpage/logex.1 logfmt1/manpage/update-logfmt.1

Changes to logfmt1/logex.py.

1
2
3
4
5
6
7
8
9
10
11
12
#!/usr/bin/env python3
# encoding: utf-8
# title: logex
# description: extract fields from log (with .fmt)
# version: 0.2
# type: cli
# category: extract
#
#
# First parameter should be the log file. And a .log.fmt must exist
# alongside (generate with `update-logfmt`).
#




|







1
2
3
4
5
6
7
8
9
10
11
12
#!/usr/bin/env python3
# encoding: utf-8
# title: logex
# description: extract fields from log (with .fmt)
# version: 0.3
# type: cli
# category: extract
#
#
# First parameter should be the log file. And a .log.fmt must exist
# alongside (generate with `update-logfmt`).
#

Changes to logfmt1/logfmt1.py.

1
2
3
4
5
6
7
8
9
10
11
12
13

14
15
16
17
18
19
20
# encoding: utf-8
# api: python
# title: python3-logfmt1
# description: handle *.log.fmt specifiers and regex conversion
# type: transform
# category: io
# version: 0.5.3
# license: Apache-2.0
# pack:
#    logfmt1.py=/usr/lib/python3/dist-packages/
#    update_logfmt.py=/usr/bin/update-logfmt
#    ./logex.py=/usr/bin/logex
#    share=/usr/share/logfmt

# architecture: all
# depends: python (>= 3.6)
# url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
# documentation: https://fossil.include-once.org/modseccfg/doc/trunk//logfmt1/html/index.html
#
# Logging format strings to regex conversion.
#













>







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# encoding: utf-8
# api: python
# title: python3-logfmt1
# description: handle *.log.fmt specifiers and regex conversion
# type: transform
# category: io
# version: 0.5.3
# license: Apache-2.0
# pack:
#    logfmt1.py=/usr/lib/python3/dist-packages/
#    update_logfmt.py=/usr/bin/update-logfmt
#    ./logex.py=/usr/bin/logex
#    share=/usr/share/logfmt
#    manpage/*.1=/usr/share/man/man1/
# architecture: all
# depends: python (>= 3.6)
# url: https://fossil.include-once.org/modseccfg/wiki/logfmt1
# documentation: https://fossil.include-once.org/modseccfg/doc/trunk//logfmt1/html/index.html
#
# Logging format strings to regex conversion.
#

Added logfmt1/manpage/logex.1.































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
.\"t
.\" Automatically generated by Pandoc 2.5
.\"
.TH "logex" "1" "" "logfmt1 tool" "Version 0.3"
.hy
.SH NAME
.PP
\f[B]logex\f[R] \[em] extracts fields from log files which have a
\&.log.fmt alongside \- as generated by \f[B]update\-logfmt\f[R](1)
.SH SYNOPSIS
.PP
\f[B]logex\f[R] [\f[I]file.log\f[R]] [\f[I]field field2 field3\f[R]]
.PP
\f[B]logex\f[R]
[\f[B]\[en]tab\f[R]|\f[B]\[en]csv\f[R]|\f[B]\[en]json\f[R]|\f[B]\[en]debug\f[R]]
[\f[I]file.log\f[R]] [\f[I]\[lq]fields\[rq]\f[R]]
.PP
\f[B]logex\f[R] [\f[B]\[en]regex\f[R]] [\f[I]file.log\f[R]]
.SH DESCRIPTION
.PP
\f[C]logex\f[R] is a little command line tool around
\f[C]logopen()\f[R] (logopen.md).
It basically allows fetching individual fields from a log, and/or
restructuring it into a specific format (e.g.\ tab or comma\-delimited).
.PP
\f[B]logex\f[R] \f[I]access.log\f[R] \f[I]request_uri\f[R]
\f[I]datetime\f[R] \f[I]user_agent\f[R]
.PP
Where the first param is always the log file, and then an arbitrary
number of field names \- either individually or as part of a string
argument:
.PP
\f[B]logex\f[R] \f[I]access.log\f[R] \f[B]\[en]tab\f[R]
\f[I]\[lq]uri,date,status\[rq]\f[R] \f[I]user_agent\f[R]
.PP
Would delimit the first three per comma, the last with a tab.
.SH Options
.PP
.TS
tab(@);
l l.
T{
param
T}@T{
desc
T}
_
T{
\[en]json
T}@T{
output each row as JSON
T}
T{
\[en]csv
T}@T{
join all fields with comma
T}
T{
\[en]tab
T}@T{
tab\-separated output
T}
T{
\[en]iso8601
T}@T{
fix any datetime fields
T}
T{
\[en]regex
T}@T{
just output regex for log
T}
T{
\[en]debug
T}@T{
debug_rx in case the regex fails
T}
.TE
.SH Prefixes
.PP
The argument list allows field name prefixes.
Which are only relevant to container\-fields (expanded key:value pairs
for some application formats), in case they\[cq]re holding lists.
.PP
.TS
tab(@);
l l.
T{
prefix
T}@T{
output
T}
_
T{
\[at]name
T}@T{
will just show the first entry
T}
T{
%name
T}@T{
space\-separated list
T}
T{
*name
T}@T{
comma\-separated list
T}
T{
+name
T}@T{
plus\-joined list
T}
T{
#name
T}@T{
as json array
T}
T{
name
T}@T{
whatever
T}
.TE
.SH SEE ALSO
.PP
\f[B]python\f[R](1), \f[B]update\-logfmt\f[R](1)

Added logfmt1/manpage/logex.md.













































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
% logex(1) logfmt1 tool | Version 0.3


NAME
====

**logex** — extracts fields from log files which have a .log.fmt alongside - as generated by **update-logfmt**(1)

SYNOPSIS
========

  **logex** \[_file.log_] \[_field field2 field3_]

  **logex** \[**--tab**|**--csv**|**--json**|**--debug**] \[_file.log_] \[_"fields"_]

  **logex** \[**--regex**] \[_file.log_]


DESCRIPTION
===========

`logex` is a little command line tool around [`logopen()`](logopen.md).
It basically allows fetching individual fields from a log, and/or
restructuring it into a specific format (e.g. tab or comma-delimited).

  **logex** _access.log_ _request_uri_ _datetime_ _user_agent_

Where the first param is always the log file, and then an arbitrary
number of field names - either individually or as part of a string
argument:

  **logex** _access.log_ **--tab** _"uri,date,status"_ _user_agent_

Would delimit the first three per comma, the last with a tab.


Options
=======

| param | desc |
|-------|------|
| --json | output each row as JSON |
| --csv  | join all fields with comma |
| --tab  | tab-separated output |
| --iso8601 | fix any datetime fields |
| --regex | just output regex for log |
| --debug | debug_rx in case the regex fails |


Prefixes
========

The argument list allows field name prefixes.  Which are only relevant to
container-fields (expanded key:value pairs for some application formats), in
case they're holding lists.

| prefix | output |
| -------| ------ |
| @name  | will just show the first entry |
| %name  | space-separated list |
| *name  | comma-separated list |
| +name  | plus-joined list |
| #name  | as json array |
| name   | whatever |


SEE ALSO
========

**python**(1), **update-logfmt**(1)

Added logfmt1/manpage/update-logfmt.1.









































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
.\" Automatically generated by Pandoc 2.5
.\"
.TH "update\-logfmt" "1" "" "logfmt1 utilities" "Version 0.1"
.hy
.SH NAME
.PP
\f[B]update\-logfmt\f[R] \[em] crafts *.log.fmt files globally for known
application configurations
.SH SYNOPSIS
.PP
\f[B]update\-logfmt\f[R]
.PP
\f[B]update\-logfmt\f[R] [\f[B]\[en]test\f[R]|\f[B]\[en]verbose\f[R]]
.SH DESCRIPTION
.PP
The \f[C]update\-logfmt\f[R] script (should be in in /usr/bin/ when
installed via system package) will create .log.fmt (log.fmt.md) files
for known system logs.
.PP
It\[cq]s basically just a wrapper script that invokes handlers in
\f[C]/usr/share/logfmt/update/*\f[R].
Each of which scans one application config to locate and decorate its
according log files.
.SH SCRIPTS
.PP
Currently just supports:
.IP \[bu] 2
Apache (access, global, error and transfer logs, no forensic logs)
.IP \[bu] 2
Nginx (only accesss logs)
.PP
More static system logs (klog/syslog) might be supported in the next
version.
.SH SEE ALSO
.PP
\f[B]python\f[R](1), \f[B]logex\f[R](1), \f[B]modseccfg\f[R](1)

Added logfmt1/manpage/update-logfmt.md.

























































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
% update-logfmt(1) logfmt1 utilities | Version 0.1


NAME
====

**update-logfmt** — crafts *.log.fmt files globally for known application configurations

SYNOPSIS
========

  **update-logfmt**

  **update-logfmt** \[**--test**|**--verbose**]


DESCRIPTION
===========

The `update-logfmt` script (should be in in /usr/bin/ when installed via
system package) will create [.log.fmt](log.fmt.md) files for known system
logs.

It's basically just a wrapper script that invokes handlers in
`/usr/share/logfmt/update/*`. Each of which scans one application
config to locate and decorate its according log files.


SCRIPTS
=======

Currently just supports:

 * Apache (access, global, error and transfer logs, no forensic logs)
 * Nginx (only accesss logs)

More static system logs (klog/syslog) might be supported in the next
version.


SEE ALSO
========

**python**(1), **logex**(1), **modseccfg**(1)

Changes to logfmt1/setup.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

21
22
23
24
25
26
27
#!/usr/bin/env python3
# encoding: utf-8
# api: pip
# type: build
# title: config for setuptools
#
# Notably the .deb will install as just dist-packages/logfmt1.py.
# Whereas the .whl creates a logfmt1/__init__.py wrapper and
# directory structure.
# - share/ files shouldn't really reside within the pkg.
#

from pluginconf.setup import setup


setup(
    fn="./logfmt1.py",
    long_description="@README.rst",
    package_dir={"logfmt1": "./"},
    packages=["logfmt1"],

    package_data={
        "logfmt1": [
           "./share/*",
           "./share/update/*"
        ],
    },
    #data_files=[],













|






>







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/usr/bin/env python3
# encoding: utf-8
# api: pip
# type: build
# title: config for setuptools
#
# Notably the .deb will install as just dist-packages/logfmt1.py.
# Whereas the .whl creates a logfmt1/__init__.py wrapper and
# directory structure.
# - share/ files shouldn't really reside within the pkg.
#

from pluginconf.setup import setup
from glob import glob

setup(
    fn="./logfmt1.py",
    long_description="@README.rst",
    package_dir={"logfmt1": "./"},
    packages=["logfmt1"],
    data_files=[("man/man1", glob("manpage/*.1"))],
    package_data={
        "logfmt1": [
           "./share/*",
           "./share/update/*"
        ],
    },
    #data_files=[],

Added manpage/modseccfg.1.





















































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
.\" Automatically generated by Pandoc 2.5
.\"
.TH "modseccfg" "1" "" "modseccfg and logfmt1 utilities" "Version 0.7.0"
.hy
.SH NAME
.PP
\f[B]modseccfg\f[R] \[em] GUI editor for mod_security rules on Apache
setups
.SH SYNOPSIS
.PP
\f[B]modseccfg\f[R]
.PP
\f[B]modseccfg\f[R] [\f[I]ssh\[at]remote:/\f[R]]
.SH DESCRIPTION
.PP
Is a GUI tool to adapt Apache vhost configuration files that pertain
mod_security / Core Rule Set options.
.PP
Can be used locally, or with a \f[I]ssh:/\f[R] parameter to connect to
remote filesystems.
Notably the ssh shorthand should be configured in
\f[B]ssh_config\f[R](5) with public key authorization beforehand.
.SH DOCS
.IP \[bu] 2
usage (https://fossil.include-once.org/modseccfg/wiki/usage)
.IP \[bu] 2
remoting (https://fossil.include-once.org/modseccfg/wiki/remoting)
.SH FILES
.TP
.B \f[B]\[ti]/.config/modseccfg/\f[R]
application config files
.TP
.B \f[B]\[ti]/mnt/\f[R]
default sshfs automount point for remoting
.TP
.B \f[B]\[ti]/backup\-config/\f[R]
automatically archived server configuration files (whenever something
gets edited, a copy is kept per default)
.SH SEE ALSO
.PP
\f[B]apache\f[R](1), \f[B]logex\f[R](1), \f[B]update\-logfmt\f[R](1),
\f[B]ssh_config\f[R](5)

Added manpage/modseccfg.md.









































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
% modseccfg(1) modseccfg and logfmt1 utilities | Version 0.7.0


NAME
====

**modseccfg** — GUI editor for mod_security rules on Apache setups

SYNOPSIS
========

  **modseccfg**

  **modseccfg** \[_ssh@remote:/_]


DESCRIPTION
===========

Is a GUI tool to adapt Apache vhost configuration files that pertain
mod_security / Core Rule Set options.

Can be used locally, or with a _ssh:/_ parameter to connect to remote
filesystems. Notably the ssh shorthand should be configured in
**ssh_config**(5) with public key authorization beforehand.


DOCS
====

 * [usage](https://fossil.include-once.org/modseccfg/wiki/usage)
 * [remoting](https://fossil.include-once.org/modseccfg/wiki/remoting)


FILES
=====

**~/.config/modseccfg/**
 : application config files

**~/mnt/**
 : default sshfs automount point for remoting

**~/backup-config/**
 : automatically archived server configuration files (whenever something
   gets edited, a copy is kept per default)


SEE ALSO
========

**apache**(1), **logex**(1), **update-logfmt**(1), **ssh_config**(5)

Changes to setup.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#!/usr/bin/env python3
# encoding: utf-8
# api: pip
# type: build
# title: config for setuptools
#
# Always prefer setuptools over distutils
#

from pluginconf.setup import setup

setup(
    debug=1,
    fn="modseccfg/__init__.py",
    name="modseccfg",
    long_description="README.rst",
    packages=["modseccfg"],
    package_dir={"": "."},
    package_data={
        "modseccfg": [
            "./data/*.*",
            "./install/*.*",
            "./scripts/*"
        ],
    },
    include_package_data=True,
    #data_files=[],
    entry_points={
        "console_scripts": [
            "modseccfg=modseccfg.mainwindow:main",
        ]
    }
)












|













|






1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#!/usr/bin/env python3
# encoding: utf-8
# api: pip
# type: build
# title: config for setuptools
#
# Always prefer setuptools over distutils
#

from pluginconf.setup import setup

setup(
    #debug=1,
    fn="modseccfg/__init__.py",
    name="modseccfg",
    long_description="README.rst",
    packages=["modseccfg"],
    package_dir={"": "."},
    package_data={
        "modseccfg": [
            "./data/*.*",
            "./install/*.*",
            "./scripts/*"
        ],
    },
    include_package_data=True,
    data_files=[("man/man1", ["manpage/modseccfg.1"])],
    entry_points={
        "console_scripts": [
            "modseccfg=modseccfg.mainwindow:main",
        ]
    }
)