Internet radio browser GUI for music/video streams from various directory services.

⌈⌋ ⎇ branch:  streamtuner2


Check-in [7ec987b9ba]

Overview
Comment:Work atop Python3 by using io.BytesIO rather than compat2and3 module.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 7ec987b9ba4cc3c90efd04d59b41756247d7f8a4
User & Date: mario on 2015-05-10 19:45:16
Other Links: manifest | tags
Context
2015-05-10
22:08
Add ahttp.get( quieter= ) option for less log.HTTP notices. check-in: 529222eb9b user: mario tags: trunk
19:45
Work atop Python3 by using io.BytesIO rather than compat2and3 module. check-in: 7ec987b9ba user: mario tags: trunk
19:20
Move `favicon` module into extension/feature plugin. Simplify row["favicon"] cache filename pregeneration; separate from favicon module (but basically duplicated code there). Refactor most internal favicon+banner processing, rename methods for clarity. Plugin registers itself as .hooks["play"] callback. Uses main.thread() now instead of custom variant. Create icon cache dir on initialiation rather. Use combined row_to_fn() for cache filename generation instead of domain(), url(), file(), etc. Previous banner downloads are ignored, because the filename normalization is more in line with domain favicons now. Only update pixstore on successful downloads. Pre-check the content type per binary regex now, before saving image files. Combine resizing into store_image() function as well. Even PNG files will be piped through PIL (for sanitization). Completely got rid of urllib usage. Homepage/HTML extraction got rewritten, simpler, still inexact; but works now for most webpages. Favicon homepage downloading checks both returned MIME type and actual file content prior saving. Shorten timeouts to 2-3 seconds for Google and custom favicon retrieval. check-in: bd1a9cba05 user: mario tags: trunk
Changes

Modified channels/bookmarks.py from [74d1845f3d] to [9515f23d26].

17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Some feature extensions inject custom subcategories here. For example the
# "search" feature adds its own result list here, as does the "timer" plugin.


from config import *
from uikit import *
from channels import *
import favicon



# The bookmarks tab is a core feature and built into the GtkBuilder
# layout. Which is why it derives from GenericChannel, and requires
# less setup.
#







<







17
18
19
20
21
22
23

24
25
26
27
28
29
30
# Some feature extensions inject custom subcategories here. For example the
# "search" feature adds its own result list here, as does the "timer" plugin.


from config import *
from uikit import *
from channels import *




# The bookmarks tab is a core feature and built into the GtkBuilder
# layout. Which is why it derives from GenericChannel, and requires
# less setup.
#
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

    # called from main window / menu / context menu,
    # when bookmark is to be added for a selected stream entry
    def add(self, row):

        # normalize data (this row originated in a gtk+ widget)
        row["favourite"] = 1
        if row.get("favicon"):
           row["favicon"] = favicon.file(row.get("homepage"))
        if not row.get("listformat"):
            row["listformat"] = self.parent.channel().listformat
           
        # append to storage
        self.streams["favourite"].append(row)
        self.save()
        self.load(self.default)







|
|







97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

    # called from main window / menu / context menu,
    # when bookmark is to be added for a selected stream entry
    def add(self, row):

        # normalize data (this row originated in a gtk+ widget)
        row["favourite"] = 1
        #if row.get("favicon"):
        #   row["favicon"] = favicon.file(row.get("homepage"))
        if not row.get("listformat"):
            row["listformat"] = self.parent.channel().listformat
           
        # append to storage
        self.streams["favourite"].append(row)
        self.save()
        self.load(self.default)

Modified channels/favicon.py from [444cf2c952] to [608e7fc3f8].

13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# depends: streamtuner2 >= 2.1.9, python:pil
# priority: standard
#
# This module fetches a favicon for each station, or a small banner
# or logo for some channel modules. It converts .ico image files and
# sanitizes .png or .jpeg images even prior display.
# 
# It prepares cache files in ~/.config/streamtuner2/icons/ in agreement
# with the station list display logic. Either uses station "homepage"
# or "img" URLs from entry rows{}.
#
# While it can often discover favicons directly from station homepages,
# it's often speedier to user the Google PNG conversion service. Both
# depend on a recent Pillow2 python module (superseding the PIL module).
# Else may display images with fragments if converted from ICO files.
#
# Has recently been rewritten, is somewhat less entangled with other
# modules now:
#  · GenericChannel presets row["favicon"] with cache image filename
#    in any case, uses row["homepage"] or row["img"] as template
#  · the filename shortening functionality must be shared between
#    favicon and genericchannel.prepare() code
#  · uikit.columns() merely checks row["favicon"] for file existence
#    on redraws
#  · main.play() only calls .update_playing() or .update_all()
#  · urllib is no longer required, uses the main ahttp/requests API
#  · still might need unhtml() from channels/__init__ later
#  · Reduce config options → move main favicon options here?


import os, os.path
from compat2and3 import StringIO
import re
from config import *
import ahttp
from PIL import Image
from uikit import gtk









|
|
|


|







|





|




|







13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# depends: streamtuner2 >= 2.1.9, python:pil
# priority: standard
#
# This module fetches a favicon for each station, or a small banner
# or logo for some channel modules. It converts .ico image files and
# sanitizes .png or .jpeg images even prior display.
# 
# It prepares cache files in ~/.config/streamtuner2/icons/ in silent
# agreement with the station list display logic. Either uses station
# row["homepage"] or row["img"] URLs from any entry.
#
# While it can often discover favicons directly from station homepages,
# it's often speedier to use the Google PNG conversion service. Both
# depend on a recent Pillow2 python module (superseding the PIL module).
# Else may display images with fragments if converted from ICO files.
#
# Has recently been rewritten, is somewhat less entangled with other
# modules now:
#  · GenericChannel presets row["favicon"] with cache image filename
#    in any case, uses row["homepage"] or row["img"] as template
#  · The filename shortening functionality must be shared between
#    favicon and genericchannel.prepare() code
#  · uikit.columns() merely checks row["favicon"] for file existence
#    on redraws
#  · main.play() only calls .update_playing() or .update_all()
#  · urllib is no longer required, uses the main ahttp/requests API
#  · Still might need unhtml() from channels/__init__ later
#  · Reduce config options → move main favicon options here?


import os, os.path
from io import BytesIO
import re
from config import *
import ahttp
from PIL import Image
from uikit import gtk


142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
    def update_pixstore(self, row, pixstore=None, row_i=None):
        log.FAVICON_UPDATE_PIXSTORE(pixstore, row_i)
        if not pixstore:
            return

        # Unpack ListStore, pixbuf column no, preset rowno
        ls, pix_entry, i = pixstore
        # Else rows-iteration rowno
        if i is None:
            i = row_i

        # Existing "favicon" cache filename
        if row.get("favicon"):
            fn = row["favicon"]
        else:







|







142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
    def update_pixstore(self, row, pixstore=None, row_i=None):
        log.FAVICON_UPDATE_PIXSTORE(pixstore, row_i)
        if not pixstore:
            return

        # Unpack ListStore, pixbuf column no, preset rowno
        ls, pix_entry, i = pixstore
        # Else use row index from update_all-iteration
        if i is None:
            i = row_i

        # Existing "favicon" cache filename
        if row.get("favicon"):
            fn = row["favicon"]
        else:
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182




#--- somewhat unrelated ---
#
# Should become a distinct feature plugin. - It just depends on correct
# invocation order for plugins to work.
# Googling is often blocked anyway, because this is clearly a bot request.
# Tag requests with ?client=streamtuner2 purposefully still.
# 
def google_find_homepage(self, row):
    """ Searches for missing homepage URL via Google. """
    if row.get("url") not in tried_urls:
        tried_urls.append(row.get("url"))

    if row.get("title"):







|

|







166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182




#--- somewhat unrelated ---
#
# Should become a distinct feature plugin. - It just depends on correct
# invocation order for both plugins to interact.
# Googling is often blocked anyway, because this is clearly a bot request.
# And requests are tagged with ?client=streamtuner2 still purposefully.
# 
def google_find_homepage(self, row):
    """ Searches for missing homepage URL via Google. """
    if row.get("url") not in tried_urls:
        tried_urls.append(row.get("url"))

    if row.get("title"):
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
# Check valid image, possibly convert, and save to cache filename
def store_image(imgdata, fn, resize=None):

    # Convert accepted formats -- even PNG for filtering now
    if re.match(br'^(.PNG|GIF\d+|.{0,15}JFIF|\x00\x00\x01\x00|.{0,255}<svg[^>]+svg)', imgdata):
        try:
            # Read from byte/str
            image = Image.open(StringIO(imgdata))
            log.FAVICON_IMAGE_TO_PNG(image, resize)

            # Resize
            if resize and image.size[0] > resize:
                image.resize((resize, resize), Image.ANTIALIAS)

            # Convert to PNG via string buffer
            out = StringIO()
            image.save(out, "PNG", quality=98)
            imgdata = out.getvalue()

        except Exception as e:
            return log.ERR("favicon/logo conversion error:", e) and False
    else:
        log.WARN("couldn't detect mime type")

    # PNG already?
    if re.match(b"^.(PNG)", imgdata):
        try:
            with open(fn, "wb") as f:
                f.write(imgdata)
                return True
        except Exception as e:
            log.ERR("favicon.store_image() failure:", e)



# PNG via Google ico2png
def fav_google_ico2png(url, fn):
    log.FAVICON("google ico2png")

    # Download from service
    domain = re.sub("^\w+://|/.*$", "", url).lower()
    geturl = "http://www.google.com/s2/favicons?domain={}".format(domain)
    imgdata = ahttp.get(geturl, binary=1, timeout=2.5)
    
    # Check for stub sizes







|







|






|














|







229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
# Check valid image, possibly convert, and save to cache filename
def store_image(imgdata, fn, resize=None):

    # Convert accepted formats -- even PNG for filtering now
    if re.match(br'^(.PNG|GIF\d+|.{0,15}JFIF|\x00\x00\x01\x00|.{0,255}<svg[^>]+svg)', imgdata):
        try:
            # Read from byte/str
            image = Image.open(BytesIO(imgdata))
            log.FAVICON_IMAGE_TO_PNG(image, resize)

            # Resize
            if resize and image.size[0] > resize:
                image.resize((resize, resize), Image.ANTIALIAS)

            # Convert to PNG via string buffer
            out = BytesIO()
            image.save(out, "PNG", quality=98)
            imgdata = out.getvalue()

        except Exception as e:
            return log.ERR("favicon/logo conversion error:", e) and False
    else:
        log.WARN("Couldn't detect valig image type from its raw content")

    # PNG already?
    if re.match(b"^.(PNG)", imgdata):
        try:
            with open(fn, "wb") as f:
                f.write(imgdata)
                return True
        except Exception as e:
            log.ERR("favicon.store_image() failure:", e)



# PNG via Google ico2png
def fav_google_ico2png(url, fn):
    log.FAVICON("fav_google_ico2png()")

    # Download from service
    domain = re.sub("^\w+://|/.*$", "", url).lower()
    geturl = "http://www.google.com/s2/favicons?domain={}".format(domain)
    imgdata = ahttp.get(geturl, binary=1, timeout=2.5)
    
    # Check for stub sizes
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
def fav_from_homepage(url, fn):

    # Check for <link rel=icon>
    img = html_link_icon(url)
    if not img:
        return False
        
    # Fetc image, verify MIMEE type
    r = ahttp.get(img, binary=1, content=0, timeout=2.75)
    if not re.match('image/(png|jpe?g|png|ico|x-ico|vnd.microsoft.ico)', r.headers["content-type"], re.I):
        log.WARN("content-type wrong", r.headers)
        return False
        
    # Convert, resize and save
    return store_image(r.content, fn, resize=16)







|







284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
def fav_from_homepage(url, fn):

    # Check for <link rel=icon>
    img = html_link_icon(url)
    if not img:
        return False
        
    # Fetch image, verify MIME type
    r = ahttp.get(img, binary=1, content=0, timeout=2.75)
    if not re.match('image/(png|jpe?g|png|ico|x-ico|vnd.microsoft.ico)', r.headers["content-type"], re.I):
        log.WARN("content-type wrong", r.headers)
        return False
        
    # Convert, resize and save
    return store_image(r.content, fn, resize=16)
312
313
314
315
316
317
318

319
320
321
322
323
324
325
326
327
328
329
        pair = { name: val for name, val in pair }
        for name in ("shortcut icon", "favicon", "icon", "icon shortcut"):
            if name == pair.get("rel", "ignore") and pair.get("href"):
                href = pair["href"] # unhtml()
                break
    # Patch URL together (strip double proto/domain, or double slash)
    return re.sub("^(https?://\w[^/]+\w)?/?(https?://\w[^/]+\w)/?(/.+)$", "\g<2>\g<3>", url+href)

    




#-- test
if __name__ == "__main__":
    import sys
    favicon(None).download(sys.argv[1])









>











312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
        pair = { name: val for name, val in pair }
        for name in ("shortcut icon", "favicon", "icon", "icon shortcut"):
            if name == pair.get("rel", "ignore") and pair.get("href"):
                href = pair["href"] # unhtml()
                break
    # Patch URL together (strip double proto/domain, or double slash)
    return re.sub("^(https?://\w[^/]+\w)?/?(https?://\w[^/]+\w)/?(/.+)$", "\g<2>\g<3>", url+href)
    # (should rather split this up again, a few more special cases w/ non-root homepages)
    




#-- test
if __name__ == "__main__":
    import sys
    favicon(None).download(sys.argv[1])