Overview

Comment:	more pylint changes
Downloads:	Tarball | ZIP archive | SQL archive
Timelines:	family | ancestors | descendants | both | trunk
Files:	files | file ages | folders
SHA3-256:	7e6847fd5e6053098991cdbd0b45902aef757cfbd92b3180d1bd6982e03623af
User & Date:	mario on 2022-10-26 13:59:09
Other Links:	manifest | tags

Context

2022-10-26
13:59		setup and type map test, fix chdir() - which was redundant and causing more lookup errors in other tests check-in: 44cb252c3b user: mario tags: trunk
13:59		more pylint changes check-in: 7e6847fd5e user: mario tags: trunk
13:57		introduce flit wrapper building on .setup (individualized property extraction) check-in: 655c75da9b user: mario tags: trunk

Changes

Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Modified README.md from [c783f5eaf0] to [afd1caefa1].

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68	1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68	- + - + - + - +	Provides meta data extraction and plugin basename lookup. And it’s meant for in-application feature and option management. The [descriptor format](https://fossil.include-once.org/pluginspec/index) (*self-contained* atop each script) is basically: # encoding: utf-8 # api: python # type: handler # category: io # title: Plugin configuration # description: Read meta data, pyz/package contents, module locating # version: 0.5 # priority: core # docs: http://fossil.include-once.org/streamtuner2/wiki/plugin+meta+data # docs: https://fossil.include-once.org/pluginspec/ # config: # { name: xyz, value: 1, type: bool, description: "Sets..." } # { name: var, value: "init", description: "Another..." } # license: MITL # # Documentation goes here... The `key: value` format is language-agnostic. It’s basically YAML in the topmost script comment. For Python only # hash comments are used. Defaults to rather common field names, encourages a documentation block, and an obvious [config: { .. } scheme](https://fossil.include-once.org/pluginspec/wiki/config) for options and defaults. How it's used: import pluginconf meta = pluginconf.plugin_meta(fn="./plugin/func.py") meta = pluginconf.plugin_meta(filename="./plugin/func.py") print(meta) What it’s not: > * This is not another config reader/parser/storage class. > * Doesn’t impose a specific plugin API. > * Neither concerns itself with module/package loading. (See [pluginbase](https://pypi.org/project/pluginbase/) or just `__import__`.) What for then? > * Separates code from meta data. Avoids keeping seldomly used descriptors in variables. > * Does away with externalized ini/json files for modules, yet simplifies use of external tooling. > * Minimizes premature module loading just to inspect meta information. pluginconf is less about a concrete implementation/code, but pushing a universal meta data format. pluginconf is foremost about the universal meta comment format. # API Lookup configuration is currently just done through injection: plugin_base = [__package__, "myapp.plugins", "/usr/share/app/extensions"] module_base = "pluginconf" # or any top-level app module Which declares module and plugin basenames, which get used for lookups by just module= names in e.g. `module_list()`. (Works for literal setups and within PYZ bundles). This is unnecessary for plain `plugin_meta(fn=)` extraction. #### plugin_meta( module= | fn= | src= | frame= ) #### plugin_meta( module= | filename= | src= | frame= ) Returns a meta data dictionary for the given module name, file, source code, or caller frame: { "title": "Compound★", "description": "...", "version": "0.1",
︙
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111	93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111	- + - +	pluginconf.plugin_base = [__package__] conf = { "defaults": "123", "plugins": {} # ← stores the activation states } for module,meta in pluginconf.all_plugin_meta().items(): for module, meta in pluginconf.all_plugin_meta().items(): pluginconf.add_plugin_defaults(conf, conf["plugins"], meta, module) # share the same dict ↑ ↑ #### get_data( fn= ) #### get_data( filename= ) Is mostly an alias for pkgutil.get_data(). Abstracts usage within PYZ packages a little. #### argparse_map() Provides a simpler way to specify ugly argparse definitions. And allows to amass options from plugins.
︙
154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189	154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196	- + + + + + + + +	# setup.py wrapper Another obvious use case for PMD is simplifying packaging. A `setup()` script can become as short as: from pluginconf.setup import setup setup( fn="main/pkg.py" filename="main/pkg.py" ) Which will reuse version: and descriptors from the meta comment. For simple one-module packages, you might get away with just `setup()` and an all automatic lookup. The non-standard PMD field `# classifiers: x11, python` can be used to lookup trove categories (crude search on select topics). All other `setup(fields=…)` are passed on to distutils/setuptools as is. -- Btw, [setupmeta](https://pypi.org/project/setupmeta/) is an even more versatile wrapper with sensible defaults and source scanning. # other modules * DependencyValidation is now in `pluginconf.depends` * `pluginconf.flit` is meant as alternative to setup. #### Caveats * It’s mostly just an excerpt from streamtuner2. * Might need customization prior use. * The GUI implmentation is fairly simplistic. * Doesn't bundle any plugin repo loader logic. * So doesn't make use of the dependency class. * The description fields can double as packaging source (setup.py). There's also a [# pack: specifier](https://fossil.include-once.org/pluginspec/wiki/References) for fpm (deb/rpm/arch/exe/pyzw/pip generation), unused in the `setup.py` wrapper here however. * In Python `# type:` might neede doubled commenting (## pylint), or alternating to other descriptors like`class:` or `slot:`. (The whole scheme is agnostic to designators. Common keys are just recommended for potential interoparability.) * The format incidentally mixes well with established comment markers like `# format: off` or `# pylint: disable=…` for instance.

Modified pluginconf/__init__.py from [7d506ba315] to [4d5e4ffe7f].

︙
8 9 10 11 12 13 14 15 16 17 18 19 20 21	8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23	+ +	# state: stable # classifiers: documentation # license: PD # priority: core # docs: https://fossil.include-once.org/pluginspec/ # url: http://fossil.include-once.org/streamtuner2/wiki/plugin+meta+data # config: - # format: off # pylint: disable=invalid-name # # Provides plugin lookup and meta data extraction utility functions. # It's used to abstract module+option management in applications. # For consolidating internal use and external/tool accessibility. # # The key:value format is language-agnostic. It's basically YAML in # a topmost script comment. For Python only # hash comments though.
︙
70 71 72 73 74 75 76 77 78 79 80 81 82 83	72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87	+ +	# by attaching new locations itself via module.__path__ + ["./local"] # for example. # # Plugin loading thus becomes as simple as __import__("ext.local"). # The attached plugin_state config dictionary in most cases can just # list module basenames, if there's only one set to manage. """ Plugin meta extraction and module lookup""" import sys import os import re import functools import pkgutil import inspect
︙
112 113 114 115 116 117 118 119 120 121 122 123 124 125	116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143	+ + + + + + + + + + + + + +	""" Package/module names for module_list() and plugin_meta() lookups. All associated paths will be scanned for module/plugin basenames. (Equivalent to `searchpath` in PluginBase) """ plugin_base = ["channels"] """ normalize config type: names to `str`, `text`, `bool`, `int`, `select`, `dict` """ config_opt_type_map = { "longstr": "text", "string": "str", "boolean": "bool", "checkbox": "bool", "integer": "int", "number": "int", "choice": "select", "options": "select", "table": "dict", "array": "dict" } # Compatiblity # ‾‾‾‾‾‾‾‾‾‾‾‾ def renamed_arguments(renamed): """ map old argument names """ def wrapped(func):
︙
168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355	186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369	- - - - - + + + + + - - + + - - + + - + - + - + - - + + - + - + - - - - - - - - + + + + + + + + - + + - + - - - + + + - - + + - + - + - + - + - + - - - - - + + + + + + - - - - - -	names → paths) and get module basenames. :arg list extra_paths: in addition to plugin_base list """ # Convert plugin_base package names into paths for iter_modules paths = [] for mp in plugin_base: if sys.modules.get(mp): paths += sys.modules[mp].__path__ elif os.path.exists(mp): paths.append(mp) for module_or_path in plugin_base: if sys.modules.get(module_or_path): paths += sys.modules[module_or_path].__path__ elif os.path.exists(module_or_path): paths.append(module_or_path) # Should list plugins within zips as well as local paths ls = pkgutil.iter_modules(paths + (extra_paths or [])) return [name for loader, name, ispkg in ls] dirs = pkgutil.iter_modules(paths + (extra_paths or [])) return [name for loader, name, ispkg in dirs] # Plugin => meta dict # ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ def all_plugin_meta(): """ This is a trivial wrapper to assemble a complete dictionary of available/installed plugins. It associates each plugin name with a its meta{} fields. """ return { name: plugin_meta(module=name) for name in module_list() } # Plugin meta data extraction # ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ @renamed_arguments({"filename": "fn"}) def plugin_meta(fn=None, src=None, module=None, frame=1, **kwargs): @renamed_arguments({"fn": "filename"}) def plugin_meta(filename=None, src=None, module=None, frame=1, **kwargs): """ Extract plugin meta data block from different sources: :arg str fn: read literal files, or .pyz contents :arg str filename: read literal files, or .pyz contents :arg str src: from already uncovered script code :arg str module: lookup per pkgutil, from plugin_base or top-level modules :arg int frame: extract comment header of caller (default) :arg list extra_base: additional search directories :arg ist max_length: maximum size to read from files """ # Try via pkgutil first, # find any plugins.* modules, or main packages if module: fn = module filename = module for base in plugin_base + kwargs.get("extra_base", []): try: src = get_data(fn=fn+".py", decode=True, file_base=base) src = get_data(filename=filename+".py", decode=True, file_base=base) if src: break except: continue # plugin_meta_extract() will print a notice later # Real filename/path elif fn and os.path.exists(fn): src = open(fn).read(kwargs.get("max_length", 6144)) elif filename and os.path.exists(filename): src = open(filename).read(kwargs.get("max_length", 6144)) # Else get source directly from caller elif not src and not fn: elif not src and not filename: module = inspect.getmodule(sys._getframe(frame+1)) # decorator+1 fn = inspect.getsourcefile(module) filename = inspect.getsourcefile(module) src = inspect.getcomments(module) # Assume it's a filename within a zip elif fn: intfn = "" while fn and len(fn) and not os.path.exists(fn): fn, add = os.path.split(fn) intfn = add + "/" + intfn if len(fn) >= 3 and intfn and zipfile.is_zipfile(fn): src = zipfile.ZipFile(fn, "r").read(intfn.strip("/")) # Assume it's a filename matches …/base.zip/…/int.py elif filename: int_fn = "" while len(filename) and not os.path.exists(filename): # pylint: disable=len-as-condition filename, add = os.path.split(filename) int_fn = add + "/" + int_fn if len(filename) >= 3 and int_fn and zipfile.is_zipfile(filename): src = zipfile.ZipFile(filename, "r").read(int_fn.strip("/")) # Extract source comment into meta dict if not src: src = "" if not isinstance(src, str): src = src.decode("utf-8", errors='replace') return plugin_meta_extract(src, fn) return plugin_meta_extract(src, filename) # Comment and field extraction logic # ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ @renamed_arguments({"fn": "filename"}) def plugin_meta_extract(src="", fn=None, literal=False): def plugin_meta_extract(src="", filename=None, literal=False): """ Finds the first comment block. Splits key:value header fields from comment. Turns everything into an dict, with some stub fields if absent. :arg str src: from existing source code :arg int fn: set filename attribute :arg bool literla: just split comment from doc :arg str src: from existing source code :arg int filename: set filename attribute :arg bool literla: just split comment from doc """ # Defaults meta = { "id": os.path.splitext(os.path.basename(fn or ""))[0], "fn": fn, "id": os.path.splitext(os.path.basename(filename or ""))[0], "fn": filename, "api": "python", "type": "module", "category": None, "priority": None, "version": "0", "title": fn, "title": filename, "description": "no description", "config": [], "doc": "" } # Extract coherent comment block src = src.replace("\r", "") if not literal: src = rx.comment.search(src) if not src: log_ERR("Couldn't read source meta information:", fn) log_ERR("Couldn't read source meta information:", filename) return meta src = src.group(0) src = rx.hash.sub("", src).strip() # Split comment block if src.find("\n\n") > 0: src, meta["doc"] = src.split("\n\n", 1) # Turn key:value lines into dictionary for field in rx.keyval.findall(src): meta[field[0].replace("-", "_")] = field[1].strip() meta["config"] = plugin_meta_config(meta.get("config") or "") return meta # Unpack config: structures # ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ def plugin_meta_config(str): def plugin_meta_config(src): """ Further breaks up the meta['config'] descriptor. Creates an array from JSON/YAML option lists. :arg str str: unprocessed config: field :arg str src: unprocessed config: field Stubs out name, value, type, description if absent. # config: { name: 'var1', type: text, value: "default, ..." } { name=option2, type=boolean, $value=1, etc. } """ config = [] for entry in rx.config.findall(str): for entry in rx.config.findall(src): entry = entry[0] or entry[1] opt = { "type": None, "name": None, "description": "", "value": None } for field in rx.options.findall(entry): opt[field[0]] = (field[1] or field[2] or field[3] or "").strip() # normalize type opt["type"] = config_opt_type_map.get(opt["type"], opt["type"] or "str") # preparse select: if opt.get("select"): opt["select"] = config_opt_parse_select(opt.get("select", "")) config.append(opt) return config # split up `select: 1=on|2=more|3=title` or `select: foo|bar|lists` def config_opt_parse_select(s): if re.search("([=:])", s): return dict(rx.select_dict.findall(s)) else: return dict([(v, v) for v in rx.select_list.findall(s)]) def config_opt_parse_select(select): """ unpack 1|2|3 or title=lists """ if re.search("([=:])", select): return dict(rx.select_dict.findall(select)) #else: return {val: val for val in rx.select_list.findall(select)} # normalize type:names to `str`, `text`, `bool`, `int`, `select`, `dict` config_opt_type_map = dict( longstr="text", string="str", boolean="bool", checkbox="bool", integer="int", number="int", choice="select", options="select", table="dict", array="dict" ) # Comment extraction regexps # ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ class rx: """ Pretty crude comment splitting approach. But works
︙