Check-in [950ea0eb0c]
Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Performance fix for pyrewrite in range() check. |
---|---|
Downloads: | Tarball | ZIP archive | SQL archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA3-256: |
950ea0eb0c23653ecfc52761143fef4c |
User & Date: | mario 2021-03-29 21:23:05 |
Context
2021-03-29
| ||
21:24 | Shorten titles (rm modseccfg:) check-in: c7b7b39b4f user: mario tags: trunk | |
21:23 | Performance fix for pyrewrite in range() check. check-in: 950ea0eb0c user: mario tags: trunk | |
2021-03-26
| ||
12:13 | html2mallard update: support direct .md conversion, and http:// url params, doc updates. check-in: ada19bd287 user: mario tags: trunk | |
Changes
Added modseccfg/data/help/pyrewrite.md.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | # MSC_PyRewrite There's a plugin for rewriting core rules files (*.conf) per [msc_pyparser](https://pypi.org/project/msc-pyparser/). It can be enabled in [Settings], and then appears directly in the File menu. This plugin bundles a handful of examples, as listed in the "Transform" dropdown. But you can also edit the code block before running. ## Usage In the dialog, you can limit the operation to specific secrule *.conf files and optionally id ranges. Then there's a dropdown for default samples, and the actual code editor. * So first select the files you want to have rewritten. They must toggled on by one. * Optionally set a range to limit the processing to SecRules. The format is `0-9999999` to defined a range. Clear the field if you want to apply the changes to any and all rules instead. * Then pick a transform script to apply on each rule or file. The selection will update/overwrite the code block input. * The shown scripts can be adapted before executing them. What you see in the editor box is what will actually run. So it must be valid Python code with proper indentation. The "Run/Rewrite" button will execute that code block. Ideally you want to leave the `--dry-run` mode enabled first, to see if it runs into any errors. In this mode the files aren't rewritten, but potentially changes brought up in a preview window only. Note that any print() output will appear on the console/terminal where modseccfg was started from. ## Operation modes ### Rule-by-rule All default snippets only change properties on a rule by rule basis. They typically utilize the `rule.function()` convenience methods. But you can also use the raw `r` dictionary. ### Block-wise Alternatively, there's the block-wise mode, which might allow some msc_parser transform functions to be used literally. Simply prefix the code block with `# type: block` to mark up such scripts. Might be possible to utilize the examples/ from msc_pyparser directly, since the Transform function is often suitable for direct use as block-wise script. For example the [example04_reduce_escs.py](https://github.com/digitalwave/msc_pyparser/blob/master/examples/example04_reduce_escs.py) contains a "Transform" class. Just copy that into the input box. And append an instantiation and inocation call: # type: block # copy verbatim class Transform(object): def __init__(self, data): ... # execute Transform(self.data).stripescape() Where `self.data` is the raw parser structure, and `.stripescape()` the target method in this example/class. ### Available variables You can see the "help" Transform sample for some notes: * `r` is the plain msc_pyparser dict for a rule (alias: `d`) * r["type"] is often "SecRule" * r["actions"] a list of dicts (alias: `actions`) * `rule` is a wrapper object with some convenience methods * rule.id() * rule.has_tag("app") * rule.add_tag("my_new_tag") * for l in rule.find_actions("logdata"): * `parser` is the current MSCParser instance * `self.data` is parser.configlines |
Added modseccfg/data/help/pyrewrite.page.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | <page xmlns="http://projectmallard.org/1.0/" type="guide" id="pyrewrite"> <info> <link type="guide" xref="index#nav"/> <desc></desc> <?http header="X-Generator: html2mallard" ?> </info> <title>MSC_PyRewrite</title> <section> <title>MSC_PyRewrite</title> <p>There's a plugin for rewriting core rules files (*.conf) per <link type="seealso" href="https://pypi.org/project/msc-pyparser/">msc_pyparser</link>. It can be enabled in [Settings], and then appears directly in the File menu.</p> <p>This plugin bundles a handful of examples, as listed in the "Transform" dropdown. But you can also edit the code block before running.</p> </section> <section> <title>Usage</title> <p>In the dialog, you can limit the operation to specific secrule *.conf files and optionally id ranges. Then there's a dropdown for default samples, and the actual code editor.</p> <list> <item><p>So first select the files you want to have rewritten. They must toggled on by one.</p></item> <item><p>Optionally set a range to limit the processing to SecRules. The format is <code>0-9999999</code> to defined a range. Clear the field if you want to apply the changes to any and all rules instead.</p></item> <item><p>Then pick a transform script to apply on each rule or file. The selection will update/overwrite the code block input.</p></item> <item><p>The shown scripts can be adapted before executing them. What you see in the editor box is what will actually run. So it must be valid Python code with proper indentation.</p></item> </list> <p>The "Run/Rewrite" button will execute that code block. Ideally you want to leave the <code>--dry-run</code> mode enabled first, to see if it runs into any errors. In this mode the files aren't rewritten, but potentially changes brought up in a preview window only.</p> <p>Note that any print() output will appear on the console/terminal where modseccfg was started from.</p> </section> <section> <title>Operation modes</title> </section> <section> <subtitle>Rule-by-rule</subtitle> <p>All default snippets only change properties on a rule by rule basis. They typically utilize the <code>rule.function()</code> convenience methods.</p> <p>But you can also use the raw <code>r</code> dictionary.</p> </section> <section> <subtitle>Block-wise</subtitle> <p>Alternatively, there's the block-wise mode, which might allow some msc_parser transform functions to be used literally. Simply prefix the code block with <code># type: block</code> to mark up such scripts.</p> <p>Might be possible to utilize the examples/ from msc_pyparser directly, since the Transform function is often suitable for direct use as block-wise script.</p> <p>For example the <link type="seealso" href="https://github.com/digitalwave/msc_pyparser/blob/master/examples/example04_reduce_escs.py">example04_reduce_escs.py</link> contains a "Transform" class. Just copy that into the input box. And append an instantiation and inocation call:</p> <code># type: block # copy verbatim class Transform(object): def __init__(self, data): ... # execute Transform(self.data).stripescape() </code> <p>Where <code>self.data</code> is the raw parser structure, and <code>.stripescape()</code> the target method in this example/class.</p> </section> <section> <subtitle>Available variables</subtitle> <p>You can see the "help" Transform sample for some notes:</p> <list> <item><p><code>r</code> is the plain msc_pyparser dict for a rule (alias: <code>d</code>)</p> <list> <item><p>r["type"] is often "SecRule"</p></item> <item><p>r["actions"] a list of dicts (alias: <code>actions</code>)</p></item> </list> </item> <item><p><code>rule</code> is a wrapper object with some convenience methods</p> <list> <item><p>rule.id()</p></item> <item><p>rule.has_tag("app")</p></item> <item><p>rule.add_tag("my_new_tag")</p></item> <item><p>for l in rule.find_actions("logdata"):</p></item> </list> </item> <item><p><code>parser</code> is the current MSCParser instance</p></item> <item><p><code>self.data</code> is parser.configlines</p></item> </list> </section> </page> |
Changes to modseccfg/msc_pyrewrite.py.
︙ | ︙ | |||
141 142 143 144 145 146 147 148 149 150 151 152 153 154 | # empty line after the rule self.lineno += 1 else: d['lineno'] = self.lineno self.lineno += 1 """, } class rulewrap(): """ utility functions to simplify adapting rules and actions """ def __init__(self, r): self.r = r | > | 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 | # empty line after the rule self.lineno += 1 else: d['lineno'] = self.lineno self.lineno += 1 """, } class rulewrap(): """ utility functions to simplify adapting rules and actions """ def __init__(self, r): self.r = r |
︙ | ︙ | |||
183 184 185 186 187 188 189 190 191 192 193 194 195 196 | yield a["act_arg"] def id(self): """ find_action("id"), return as integer, or None """ for a in self.r["actions"]: if a['act_name'] == "id": return int(a['act_arg']) class show(): """ hook to file menu in mainwindow """ def __init__(self, mainwindow, data, *a, **kw): | > > > > > > > | 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 | yield a["act_arg"] def id(self): """ find_action("id"), return as integer, or None """ for a in self.r["actions"]: if a['act_name'] == "id": return int(a['act_arg']) def within(self, limit): id = self.id() if not id: return False if limit.start <= id <= limit.stop: return True class show(): """ hook to file menu in mainwindow """ def __init__(self, mainwindow, data, *a, **kw): |
︙ | ︙ | |||
249 250 251 252 253 254 255 | p(500) else: for r in parser.configlines: p(2) rule = rulewrap(r) if r["type"] != "SecRule": continue | | | 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 | p(500) else: for r in parser.configlines: p(2) rule = rulewrap(r) if r["type"] != "SecRule": continue if limit and not rule.within(limit): continue # aliases (msc_pyparser examples) d = r actions = r["actions"] # eval script block exec(script) p(10) |
︙ | ︙ |