Check-in [7ec987b9ba]
Overview
Comment: | Work atop Python3 by using io.BytesIO rather than compat2and3 module. |
---|---|
Downloads: | Tarball | ZIP archive | SQL archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA1: |
7ec987b9ba4cc3c90efd04d59b417562 |
User & Date: | mario on 2015-05-10 19:45:16 |
Other Links: | manifest | tags |
Context
2015-05-10
| ||
22:08 | Add ahttp.get( quieter= ) option for less log.HTTP notices. check-in: 529222eb9b user: mario tags: trunk | |
19:45 | Work atop Python3 by using io.BytesIO rather than compat2and3 module. check-in: 7ec987b9ba user: mario tags: trunk | |
19:20 | Move `favicon` module into extension/feature plugin. Simplify row["favicon"] cache filename pregeneration; separate from favicon module (but basically duplicated code there). Refactor most internal favicon+banner processing, rename methods for clarity. Plugin registers itself as .hooks["play"] callback. Uses main.thread() now instead of custom variant. Create icon cache dir on initialiation rather. Use combined row_to_fn() for cache filename generation instead of domain(), url(), file(), etc. Previous banner downloads are ignored, because the filename normalization is more in line with domain favicons now. Only update pixstore on successful downloads. Pre-check the content type per binary regex now, before saving image files. Combine resizing into store_image() function as well. Even PNG files will be piped through PIL (for sanitization). Completely got rid of urllib usage. Homepage/HTML extraction got rewritten, simpler, still inexact; but works now for most webpages. Favicon homepage downloading checks both returned MIME type and actual file content prior saving. Shorten timeouts to 2-3 seconds for Google and custom favicon retrieval. check-in: bd1a9cba05 user: mario tags: trunk | |
Changes
Modified channels/bookmarks.py from [74d1845f3d] to [9515f23d26].
︙ | ︙ | |||
17 18 19 20 21 22 23 | # Some feature extensions inject custom subcategories here. For example the # "search" feature adds its own result list here, as does the "timer" plugin. from config import * from uikit import * from channels import * | < | 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # Some feature extensions inject custom subcategories here. For example the # "search" feature adds its own result list here, as does the "timer" plugin. from config import * from uikit import * from channels import * # The bookmarks tab is a core feature and built into the GtkBuilder # layout. Which is why it derives from GenericChannel, and requires # less setup. # |
︙ | ︙ | |||
98 99 100 101 102 103 104 | # called from main window / menu / context menu, # when bookmark is to be added for a selected stream entry def add(self, row): # normalize data (this row originated in a gtk+ widget) row["favourite"] = 1 | | | | 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | # called from main window / menu / context menu, # when bookmark is to be added for a selected stream entry def add(self, row): # normalize data (this row originated in a gtk+ widget) row["favourite"] = 1 #if row.get("favicon"): # row["favicon"] = favicon.file(row.get("homepage")) if not row.get("listformat"): row["listformat"] = self.parent.channel().listformat # append to storage self.streams["favourite"].append(row) self.save() self.load(self.default) |
︙ | ︙ |
Modified channels/favicon.py from [444cf2c952] to [608e7fc3f8].
︙ | ︙ | |||
13 14 15 16 17 18 19 | # depends: streamtuner2 >= 2.1.9, python:pil # priority: standard # # This module fetches a favicon for each station, or a small banner # or logo for some channel modules. It converts .ico image files and # sanitizes .png or .jpeg images even prior display. # | | | | | | | | | 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | # depends: streamtuner2 >= 2.1.9, python:pil # priority: standard # # This module fetches a favicon for each station, or a small banner # or logo for some channel modules. It converts .ico image files and # sanitizes .png or .jpeg images even prior display. # # It prepares cache files in ~/.config/streamtuner2/icons/ in silent # agreement with the station list display logic. Either uses station # row["homepage"] or row["img"] URLs from any entry. # # While it can often discover favicons directly from station homepages, # it's often speedier to use the Google PNG conversion service. Both # depend on a recent Pillow2 python module (superseding the PIL module). # Else may display images with fragments if converted from ICO files. # # Has recently been rewritten, is somewhat less entangled with other # modules now: # · GenericChannel presets row["favicon"] with cache image filename # in any case, uses row["homepage"] or row["img"] as template # · The filename shortening functionality must be shared between # favicon and genericchannel.prepare() code # · uikit.columns() merely checks row["favicon"] for file existence # on redraws # · main.play() only calls .update_playing() or .update_all() # · urllib is no longer required, uses the main ahttp/requests API # · Still might need unhtml() from channels/__init__ later # · Reduce config options → move main favicon options here? import os, os.path from io import BytesIO import re from config import * import ahttp from PIL import Image from uikit import gtk |
︙ | ︙ | |||
142 143 144 145 146 147 148 | def update_pixstore(self, row, pixstore=None, row_i=None): log.FAVICON_UPDATE_PIXSTORE(pixstore, row_i) if not pixstore: return # Unpack ListStore, pixbuf column no, preset rowno ls, pix_entry, i = pixstore | | | 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | def update_pixstore(self, row, pixstore=None, row_i=None): log.FAVICON_UPDATE_PIXSTORE(pixstore, row_i) if not pixstore: return # Unpack ListStore, pixbuf column no, preset rowno ls, pix_entry, i = pixstore # Else use row index from update_all-iteration if i is None: i = row_i # Existing "favicon" cache filename if row.get("favicon"): fn = row["favicon"] else: |
︙ | ︙ | |||
166 167 168 169 170 171 172 | #--- somewhat unrelated --- # # Should become a distinct feature plugin. - It just depends on correct | | | | 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 | #--- somewhat unrelated --- # # Should become a distinct feature plugin. - It just depends on correct # invocation order for both plugins to interact. # Googling is often blocked anyway, because this is clearly a bot request. # And requests are tagged with ?client=streamtuner2 still purposefully. # def google_find_homepage(self, row): """ Searches for missing homepage URL via Google. """ if row.get("url") not in tried_urls: tried_urls.append(row.get("url")) if row.get("title"): |
︙ | ︙ | |||
229 230 231 232 233 234 235 | # Check valid image, possibly convert, and save to cache filename def store_image(imgdata, fn, resize=None): # Convert accepted formats -- even PNG for filtering now if re.match(br'^(.PNG|GIF\d+|.{0,15}JFIF|\x00\x00\x01\x00|.{0,255}<svg[^>]+svg)', imgdata): try: # Read from byte/str | | | | | | 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 | # Check valid image, possibly convert, and save to cache filename def store_image(imgdata, fn, resize=None): # Convert accepted formats -- even PNG for filtering now if re.match(br'^(.PNG|GIF\d+|.{0,15}JFIF|\x00\x00\x01\x00|.{0,255}<svg[^>]+svg)', imgdata): try: # Read from byte/str image = Image.open(BytesIO(imgdata)) log.FAVICON_IMAGE_TO_PNG(image, resize) # Resize if resize and image.size[0] > resize: image.resize((resize, resize), Image.ANTIALIAS) # Convert to PNG via string buffer out = BytesIO() image.save(out, "PNG", quality=98) imgdata = out.getvalue() except Exception as e: return log.ERR("favicon/logo conversion error:", e) and False else: log.WARN("Couldn't detect valig image type from its raw content") # PNG already? if re.match(b"^.(PNG)", imgdata): try: with open(fn, "wb") as f: f.write(imgdata) return True except Exception as e: log.ERR("favicon.store_image() failure:", e) # PNG via Google ico2png def fav_google_ico2png(url, fn): log.FAVICON("fav_google_ico2png()") # Download from service domain = re.sub("^\w+://|/.*$", "", url).lower() geturl = "http://www.google.com/s2/favicons?domain={}".format(domain) imgdata = ahttp.get(geturl, binary=1, timeout=2.5) # Check for stub sizes |
︙ | ︙ | |||
284 285 286 287 288 289 290 | def fav_from_homepage(url, fn): # Check for <link rel=icon> img = html_link_icon(url) if not img: return False | | | 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 | def fav_from_homepage(url, fn): # Check for <link rel=icon> img = html_link_icon(url) if not img: return False # Fetch image, verify MIME type r = ahttp.get(img, binary=1, content=0, timeout=2.75) if not re.match('image/(png|jpe?g|png|ico|x-ico|vnd.microsoft.ico)', r.headers["content-type"], re.I): log.WARN("content-type wrong", r.headers) return False # Convert, resize and save return store_image(r.content, fn, resize=16) |
︙ | ︙ | |||
312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 | pair = { name: val for name, val in pair } for name in ("shortcut icon", "favicon", "icon", "icon shortcut"): if name == pair.get("rel", "ignore") and pair.get("href"): href = pair["href"] # unhtml() break # Patch URL together (strip double proto/domain, or double slash) return re.sub("^(https?://\w[^/]+\w)?/?(https?://\w[^/]+\w)/?(/.+)$", "\g<2>\g<3>", url+href) #-- test if __name__ == "__main__": import sys favicon(None).download(sys.argv[1]) | > | 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 | pair = { name: val for name, val in pair } for name in ("shortcut icon", "favicon", "icon", "icon shortcut"): if name == pair.get("rel", "ignore") and pair.get("href"): href = pair["href"] # unhtml() break # Patch URL together (strip double proto/domain, or double slash) return re.sub("^(https?://\w[^/]+\w)?/?(https?://\w[^/]+\w)/?(/.+)$", "\g<2>\g<3>", url+href) # (should rather split this up again, a few more special cases w/ non-root homepages) #-- test if __name__ == "__main__": import sys favicon(None).download(sys.argv[1]) |