Internet radio browser GUI for music/video streams from various directory services.

⌈⌋ ⎇ branch:  streamtuner2


Check-in [4e95040e40]

Overview
Comment:Move internal commentary out of plugin description block. Try/Catch processing failures in update_rows() loop. Use quieter=1 option for image downloads. Reimplement custom base_url+href favicon patching.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 4e95040e40f5f000c74a983e3218454920aa9d94
User & Date: mario on 2015-05-10 22:11:49
Other Links: manifest | tags
Context
2015-05-10
22:34
Fix URL patching (after actually testing it). And provide fallback for image resizing (.thumbnail in PIL 2.3.0 is broken). check-in: 12e7646cbf user: mario tags: trunk
22:11
Move internal commentary out of plugin description block. Try/Catch processing failures in update_rows() loop. Use quieter=1 option for image downloads. Reimplement custom base_url+href favicon patching. check-in: 4e95040e40 user: mario tags: trunk
22:08
Add ahttp.get( quieter= ) option for less log.HTTP notices. check-in: 529222eb9b user: mario tags: trunk
Changes

Modified channels/favicon.py from [608e7fc3f8] to [453360923b].

21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
























57
58
59
60
61
62
63
# agreement with the station list display logic. Either uses station
# row["homepage"] or row["img"] URLs from any entry.
#
# While it can often discover favicons directly from station homepages,
# it's often speedier to use the Google PNG conversion service. Both
# depend on a recent Pillow2 python module (superseding the PIL module).
# Else may display images with fragments if converted from ICO files.
#
# Has recently been rewritten, is somewhat less entangled with other
# modules now:
#  · GenericChannel presets row["favicon"] with cache image filename
#    in any case, uses row["homepage"] or row["img"] as template
#  · The filename shortening functionality must be shared between
#    favicon and genericchannel.prepare() code
#  · uikit.columns() merely checks row["favicon"] for file existence
#    on redraws
#  · main.play() only calls .update_playing() or .update_all()
#  · urllib is no longer required, uses the main ahttp/requests API
#  · Still might need unhtml() from channels/__init__ later
#  · Reduce config options → move main favicon options here?


import os, os.path
from io import BytesIO
import re
from config import *
import ahttp
from PIL import Image
from uikit import gtk


# Ensure that we don't try to download a single favicon twice per session.
# If it's not available the first time, we won't get it after switching
# stations back and forth either. So URLs are skipped simply.
tried_urls = []



























# Hook up as feature plugin
#
class favicon(object):

    # plugin attributes







<
<
<
<
<
<
<
<
<
<
<
<
<
<















>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







21
22
23
24
25
26
27














28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
# agreement with the station list display logic. Either uses station
# row["homepage"] or row["img"] URLs from any entry.
#
# While it can often discover favicons directly from station homepages,
# it's often speedier to use the Google PNG conversion service. Both
# depend on a recent Pillow2 python module (superseding the PIL module).
# Else may display images with fragments if converted from ICO files.















import os, os.path
from io import BytesIO
import re
from config import *
import ahttp
from PIL import Image
from uikit import gtk


# Ensure that we don't try to download a single favicon twice per session.
# If it's not available the first time, we won't get it after switching
# stations back and forth either. So URLs are skipped simply.
tried_urls = []



# Has recently been rewritten, is somewhat less entangled with other
# modules now:
#
#  · GenericChannel presets row["favicon"] with cache image filename
#    in any case. It uses row["homepage"] or row["img"] as template.
#
#  · The url-to-filename shortening functionality is therefore shared.
#    GenericChannel.prepare() duplicates all row_to_fn() logic.
#
#  · uikit.columns() merely checks row["favicon"] for file existence
#    when redrawing a station list.
#
#  · main.play() only calls .update_playing() or .update_all()
#
#  · urllib is no longer required. Using just ahttp/requests API now.
#
#  · Might need unhtml() utility from channels/__init__ later..
#
#  · Still need to consolidate config options → Move main favicon
#    options here?
#



# Hook up as feature plugin
#
class favicon(object):

    # plugin attributes
115
116
117
118
119
120
121

122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137




138
139
140
141
142
143
144
            # Cache image filename: have or can't have
            favicon_fn = row_to_fn(row)
            if not favicon_fn:
                continue
            if os.path.exists(favicon_fn):
                continue


            # Custom "img" banner/logo as favicon
            if row.get("img"):
                tried_urls.append(row["img"])
                ok = banner_localcopy(row["img"], favicon_fn)

            # Homepage to favicon
            elif row.get("homepage"):
                tried_urls.append(row["homepage"])
                if conf.favicon_google_first:
                    ok = fav_google_ico2png(row["homepage"], favicon_fn)
                else:
                    ok = fav_from_homepage(row["homepage"], favicon_fn)

            # Update TreeView
            if ok:
                self.update_pixstore(row, pixstore, i)




        pass


    # Update favicon in treeview/liststore
    def update_pixstore(self, row, pixstore=None, row_i=None):
        log.FAVICON_UPDATE_PIXSTORE(pixstore, row_i)
        if not pixstore:







>
|
|
|
|

|
|
|
|
|
|
|

|
|
|
>
>
>
>







125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
            # Cache image filename: have or can't have
            favicon_fn = row_to_fn(row)
            if not favicon_fn:
                continue
            if os.path.exists(favicon_fn):
                continue

            try:
                # Custom "img" banner/logo as favicon
                if row.get("img"):
                    tried_urls.append(row["img"])
                    ok = banner_localcopy(row["img"], favicon_fn)

                # Homepage to favicon
                elif row.get("homepage"):
                    tried_urls.append(row["homepage"])
                    if conf.favicon_google_first:
                        ok = fav_google_ico2png(row["homepage"], favicon_fn)
                    else:
                        ok = fav_from_homepage(row["homepage"], favicon_fn)

                # Update TreeView
                if ok:
                    self.update_pixstore(row, pixstore, i)

            # catch HTTP Timeouts etc., so update_all() row processing just continues..
            except Exception as e:
                log.WARN("favicon.update_rows():", e)
        pass


    # Update favicon in treeview/liststore
    def update_pixstore(self, row, pixstore=None, row_i=None):
        log.FAVICON_UPDATE_PIXSTORE(pixstore, row_i)
        if not pixstore:
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317








318
319
320
321
322
323
324
325
326
327
328
329
330
        except Exception as e:
            log.ERR("favicon.store_image() failure:", e)



# PNG via Google ico2png
def fav_google_ico2png(url, fn):
    log.FAVICON("fav_google_ico2png()")

    # Download from service
    domain = re.sub("^\w+://|/.*$", "", url).lower()
    geturl = "http://www.google.com/s2/favicons?domain={}".format(domain)
    imgdata = ahttp.get(geturl, binary=1, timeout=2.5)
    
    # Check for stub sizes
    if conf.favicon_delete_stub and len(imgdata) in (726,896): # google_placeholder_filesizes
        log.FAVICON("placeholder size, skipping")
        return False
    # Save
    else:
        return store_image(imgdata, fn)
    

  
# Peek at homepage URL, download favicon.ico <link rel>, convert to PNG file, resize to 16x16
def fav_from_homepage(url, fn):

    # Check for <link rel=icon>
    img = html_link_icon(url)
    if not img:
        return False
        
    # Fetch image, verify MIME type
    r = ahttp.get(img, binary=1, content=0, timeout=2.75)
    if not re.match('image/(png|jpe?g|png|ico|x-ico|vnd.microsoft.ico)', r.headers["content-type"], re.I):
        log.WARN("content-type wrong", r.headers)
        return False
        
    # Convert, resize and save
    return store_image(r.content, fn, resize=16)



# Download HTML, look for favicon name in <link rel=shortcut icon>.
#
# Very rough, doesn't respect any <base href=> and manually patches
# icon path to homepage url; nor does any entity decoding.
#
def html_link_icon(url, href="/favicon.png"):
    html = ahttp.get(url, encoding="iso-8859-1", timeout=3.5)
    # Extract
    for link in re.findall(r"""  <link ([^<>]+) >  """, html, re.X):
        pair = re.findall(r"""  \b(rel|href) \s*=\s* ["']? ([^<>"']+) ["']? """, link, re.X)
        pair = { name: val for name, val in pair }
        for name in ("shortcut icon", "favicon", "icon", "icon shortcut"):
            if name == pair.get("rel", "ignore") and pair.get("href"):
                href = pair["href"] # unhtml()
                break
    # Patch URL together (strip double proto/domain, or double slash)








    return re.sub("^(https?://\w[^/]+\w)?/?(https?://\w[^/]+\w)/?(/.+)$", "\g<2>\g<3>", url+href)
    # (should rather split this up again, a few more special cases w/ non-root homepages)
    




#-- test
if __name__ == "__main__":
    import sys
    favicon(None).download(sys.argv[1])









<




|




















|















|





|
|

|
>
>
>
>
>
>
>
>
|
<











274
275
276
277
278
279
280

281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340

341
342
343
344
345
346
347
348
349
350
351
        except Exception as e:
            log.ERR("favicon.store_image() failure:", e)



# PNG via Google ico2png
def fav_google_ico2png(url, fn):


    # Download from service
    domain = re.sub("^\w+://|/.*$", "", url).lower()
    geturl = "http://www.google.com/s2/favicons?domain={}".format(domain)
    imgdata = ahttp.get(geturl, binary=1, timeout=3.5, quieter=1)
    
    # Check for stub sizes
    if conf.favicon_delete_stub and len(imgdata) in (726,896): # google_placeholder_filesizes
        log.FAVICON("placeholder size, skipping")
        return False
    # Save
    else:
        return store_image(imgdata, fn)
    

  
# Peek at homepage URL, download favicon.ico <link rel>, convert to PNG file, resize to 16x16
def fav_from_homepage(url, fn):

    # Check for <link rel=icon>
    img = html_link_icon(url)
    if not img:
        return False
        
    # Fetch image, verify MIME type
    r = ahttp.get(img, binary=1, content=0, timeout=4.25, quieter=1)
    if not re.match('image/(png|jpe?g|png|ico|x-ico|vnd.microsoft.ico)', r.headers["content-type"], re.I):
        log.WARN("content-type wrong", r.headers)
        return False
        
    # Convert, resize and save
    return store_image(r.content, fn, resize=16)



# Download HTML, look for favicon name in <link rel=shortcut icon>.
#
# Very rough, doesn't respect any <base href=> and manually patches
# icon path to homepage url; nor does any entity decoding.
#
def html_link_icon(url, href="/favicon.png"):
    html = ahttp.get(url, encoding="iso-8859-1", timeout=4.5, quieter=1)
    # Extract
    for link in re.findall(r"""  <link ([^<>]+) >  """, html, re.X):
        pair = re.findall(r"""  \b(rel|href) \s*=\s* ["']? ([^<>"']+) ["']? """, link, re.X)
        pair = { name: val for name, val in pair }
        for name in ("shortcut icon", "favicon", "icon", "icon shortcut"):
            if name == pair.get("rel", "ignore").lower() and pair.get("href"):
                href = pair["href"].replace("&amp;", "&") # unhtml()
                break
    # Patch URL together
    if re.match("^https?://", href): # absolute URL
        return href
    elif re.startswith("//", href): # proto-absolute
        return "http:" + href
    elif re.startswith("/", href): # root path
        return re.sub("(https?://[^/]).*$", "\g<1>") + href
    else: # relative path references xyz/../
        href = re.sub("[^/]+$", "", url) + href
        return re.sub("[^/]+/../", "/", href)

    




#-- test
if __name__ == "__main__":
    import sys
    favicon(None).download(sys.argv[1])