LibreOffice plugin to pipe whole Writer documents through Google Translate, that ought to keep most of the page formatting.

⌈⌋ branch:  PageTranslate


Check-in [d0a93e3bf3]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Implement deepl_web instanceId/sessId/cookie retrieval, still yields 429 though. Add {text} and {lang} placeholders for cli backend.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1:d0a93e3bf3b5db6bcda392a1819803425c42abab
User & Date: mario 2020-06-01 19:07:11
Original Comment: (no comment)
Context
2020-06-02
16:59
Rewrite TextPortion handling (slow mode) according to reality. (.supportsService("XEnumeration") won't work, and getString/setString on TextPortion) However, formatting bleeds into adjoining text segments, so probably a useless operation mode. (Machine translations are expectedly worse.) check-in: 46943916e1 user: mario tags: trunk
2020-06-01
19:07
Implement deepl_web instanceId/sessId/cookie retrieval, still yields 429 though. Add {text} and {lang} placeholders for cli backend. check-in: d0a93e3bf3 user: mario tags: trunk
19:06
Add super slow mode: paragraph enumeration - breaks up sentences on style variations, but harms coherent translations and performance, obviously. check-in: ded540533c user: mario tags: trunk
Changes

Changes to description.xml.

1
2
3
4
5
6
7
8
9
10
11
<?xml version="1.0" encoding="UTF-8"?>
<description xmlns="http://openoffice.org/extensions/description/2006" xmlns:dep="http://openoffice.org/extensions/description/2006" xmlns:xlink="http://www.w3.org/1999/xlink">
  <identifier value="vnd.include-once.pagetranslate"/>
  <version value="1.5"/>
  <display-name>
    <name lang="en">PageTranslate</name>
  </display-name>
  <dependencies>
    <OpenOffice.org-minimal-version value="3.0" dep:name="OpenOffice.org 3.0"/>
  </dependencies>
  <registration>



|







1
2
3
4
5
6
7
8
9
10
11
<?xml version="1.0" encoding="UTF-8"?>
<description xmlns="http://openoffice.org/extensions/description/2006" xmlns:dep="http://openoffice.org/extensions/description/2006" xmlns:xlink="http://www.w3.org/1999/xlink">
  <identifier value="vnd.include-once.pagetranslate"/>
  <version value="1.5.5"/>
  <display-name>
    <name lang="en">PageTranslate</name>
  </display-name>
  <dependencies>
    <OpenOffice.org-minimal-version value="3.0" dep:name="OpenOffice.org 3.0"/>
  </dependencies>
  <registration>

Changes to pythonpath/translationbackends.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
..
89
90
91
92
93
94
95
96
97
98
99

100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117


118
119
120

121
122
123
124
125
126
127
128
129
130
131
132
133

134

135
136

137























138
139
140
141
142
143
144
145
146
147
148
149
150
...
161
162
163
164
165
166
167
168
169
170
171
172
173



174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
...
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
...
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
...
274
275
276
277
278
279
280
281


282
283
284
285
286
287
288
289
290
291
292

293
294
295
296
297
298
299
300
# encoding: utf-8
# api: pagetranslate
# type: classes
# category: language
# title: via_* translation backends
# description: Implements the alternative services (google, deepl, ...)
# version: 1.5
# state: beta
# depends: python:requests (>= 2.5)
# config: -
#
# Different online service backends and http interfaces are now coalesced here.
# Each class handles sentence/blockwise transfer to one of the online services,
# to get text snippets transformed.
................................................................................
            # use temporary placeholder `/#§/`
            text = self.translate(text.replace("\n\n", u"/#§/"))
            text = re.sub(rx_breakln, "\n\n", text)
        return text


# DeepL online translator
#
#  · uses some kind of json-rpc
#  · will easily yield HTTP 429 Too many requests,
#    so probably not useful for multi-paragraph translation anyway

#
# data origins:
#
#  · https://www.deepl.com/translator = nothing
#  · jsonrpcId = random integer
#  · sessionId = random client-side guid
#      (https://www.deepl.com/js/translator_glossary_late.min.js?v=… → generated in `function u()`)
#  · instanceId
#      (https://www.deepl.com/PHP/backend/clientState.php?request_type=jsonrpc&il=EN → "uid":"(.+=)")
#  · LMTBID cookie
#      (https://s.deepl.com/web/stats?request_type=jsonrpc ← jsonrpc+session+instId+clientinfos)
#
class deepl_web(google):
    # < https://www2.deepl.com/jsonrpc
    # cookies: LMTBID: GUID...
    # referer: https://www.deepl.com/translator
    # body:
    # > result.translations[0].beams[0].postprocessed_sentence


    
    def __init__(self, params):
        self.params = params

        self.id = rand(202002000, 959009000) # e.g. 702005000, arbitrary, part of jsonrpc req-resp association
        self.sess = uuid.uuid4()
        self.lang = params["lang"].upper()
        r = http.get("https://www.deepl.com/translator")  # should fetch us the cookie / No, it doesn't

        # "https://s.deepl.com/web/stats?request_type=jsonrpc" # this one does
        # → however it requires an  `instanceId`
        # {"jsonrpc":"2.0","method":"WebAppPushStatistics","params":{"value":{"instanceId":"9d363e65-2a95-4ca9-b0a3-e4b1c7605dd8","sessionId":"9e8fb310-4ff4-4176-b55a-238f1bc18bac","event":"web/pageview","url":"https://www.deepl.com/translator","userAgent":"Mozilla/5.0
        # (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","resolution":{"width":1920,"height":1080,"devicePixelRatio":1,"viewportWidth":1612,"viewportHeight":904},"data":{"referrer":""}}},"id":25220002}
        # → instanceID from clientStorage "clientVars{}"
        # → sessionID originates from sessionStorage "dap.sid"
        
        #-- instanceId from:

        # https://www.deepl.com/PHP/backend/clientState.php?request_type=jsonrpc&il=EN

        # {"jsonrpc":"2.0","method":"getClientState","params":{"v":"20180814"},"id":49050001}
        # → … "uid":"(.+=)" …

        























        
    def rpc(self, text):
        return json.dumps({
           "jsonrpc" : "2.0",
           "method" : "LMT_handle_jobs",
           "id" : self.id,
           "params" : {
              "lang" : {
                 "target_lang" : self.lang,
                 "user_preferred_langs" : [
                    self.lang,
                    "EN"
                 ],
................................................................................
                    "preferred_num_beams" : 4,
                    "raw_en_sentence" : text,
                    "quality" : "fast"
                 }
              ]
           }
        })
    
    def translate(self, text):
        # skip empty paragraph/table snippets
        if len(text) < 2 or rx_empty.match(text) or not rx_letters.search(text):
            return text
    



        # request
        r = http.post(
            "https://www2.deepl.com/jsonrpc",
            data=self.rpc(text),
            headers={"Referer": "https://www.deepl.com/translator", "Content-Type": "text/plain"}
        )
        if r.status_code != 200:
            log.error(repr(r.content))
            return text
            #return r, r.content
        
        # decode
        r = r.json()
        if r.get("id"):
            self.id = r["id"] + rand(1, 5)
        if r.get("result"):
            return r["result"]["translations"][0]["beams"][0]["postprocessed_sentence"]
        else:
            return text


# DeepL API costs money
................................................................................


# Translate-python
# requires `pip install translate`
#
#  · provides "microsoft" backend (requires OAuth secret in api_key)
#
#  · or "mymemory" (with email in `api_key` instead)
#
# https://translate-python.readthedocs.io/en/latest/
#
class translate_python(google):

    def __init__(self, params={}):
        self.params = params  # config+argparse
................................................................................
        except:
            log.error(format_exc())
            raise Exception("Run `pip install translate` to use this module.")
            
        # interestingly this backend function might just work as is.
        if params.get("mymemory"):
            self.translate = Translator(
                provider="mymemory", to_lang=params["lang"], email=params["api_key"]
            ).translate
        else:
            self.translate = Translator(
                provider="microsoft", to_lang=params["lang"], secret_access_key=params["api_key"]
            ).translate

        # though .linebreakwise has no equivalent, not sure if necessary,
................................................................................
    translate = None
    #linebreakwise = None


# Because, why not?
# Invokes a commandline tool for translating texts.
#
# → with e.g. `translate-cli -t {}` in "api_key"


#
class cli(google):

    def __init__(self, params):
        self.params = params
        self.cmd = params["api_key"]

    # pipe text through external program
    def translate(self, text):
        if rx_empty.match(text) or not rx_letters.search(text):
            return text

        cmd = [s.format(text) for s in shlex.split(self.cmd)]
        proc = subprocess.run(cmd, stdout=subprocess.PIPE)
        return proc.stdout.decode("utf-8")



# maps a pagetranslate.t.* object (in main module),
# according to config dict {"goog":1, "deepl":0}






|







 







<
<

|
>


<









|
|
|
|
|
|
>
>



>
|
<
|
|
|
|
|
|
|
<
<

|
>
|
>
|
|
>

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>





|







 







|




|
>
>
>













|
<







 







|







 







|







 







|
>
>





|





>
|







1
2
3
4
5
6
7
8
9
10
11
12
13
14
..
89
90
91
92
93
94
95


96
97
98
99
100

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

123
124
125
126
127
128
129


130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
...
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214

215
216
217
218
219
220
221
...
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
...
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
...
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
# encoding: utf-8
# api: pagetranslate
# type: classes
# category: language
# title: via_* translation backends
# description: Implements the alternative services (google, deepl, ...)
# version: 1.6
# state: beta
# depends: python:requests (>= 2.5)
# config: -
#
# Different online service backends and http interfaces are now coalesced here.
# Each class handles sentence/blockwise transfer to one of the online services,
# to get text snippets transformed.
................................................................................
            # use temporary placeholder `/#§/`
            text = self.translate(text.replace("\n\n", u"/#§/"))
            text = re.sub(rx_breakln, "\n\n", text)
        return text


# DeepL online translator


#  · will easily yield HTTP 429 Too many requests,
#    so probably not useful for multi-paragraph translation anyway (just text selections)
#  · uses some kind of json-rpc
#
# data origins:

#  · https://www.deepl.com/translator = nothing
#  · jsonrpcId = random integer
#  · sessionId = random client-side guid
#      (https://www.deepl.com/js/translator_glossary_late.min.js?v=… → generated in `function u()`)
#  · instanceId
#      (https://www.deepl.com/PHP/backend/clientState.php?request_type=jsonrpc&il=EN → "uid":"(.+=)")
#  · LMTBID cookie
#      (https://s.deepl.com/web/stats?request_type=jsonrpc ← jsonrpc+session+instId+clientinfos)
#
# translation requests:
#  < https://www2.deepl.com/jsonrpc
#    cookies: LMTBID: GUID...
#    referer: https://www.deepl.com/translator
# repsonse  body:
#  > result.translations[0].beams[0].postprocessed_sentence
#
class deepl_web(google):
    
    def __init__(self, params):
        self.params = params
        self.lang = params["lang"].upper()
        self.id_ = rand(202002000, 959009000) # e.g. 702005000, arbitrary, part of jsonrpc req-resp association

        self.sess = str(uuid.uuid4())    # e.g. 233beb7c-96bc-459c-ae20-157c0bebb2e4
        self.inst = ""   # e.g. ef629644-3d1b-41a4-a2de-0626d23c99ee
        
        # fetch homepage (redundant)
        html = http.get("https://www.deepl.com/translator").text  # should fetch us the cookie / No, it doesn't
        self.versions = dict(re.findall("([\w.]+)\?v=(\d+)", html))
        #print(html)


        
        # instanceId from clientState…
        j = http.post(
            "https://www.deepl.com/PHP/backend/clientState.php?request_type=jsonrpc&il=EN",
            data = json.dumps({"jsonrpc":"2.0", "method":"getClientState", "params":{"v":"20180814"}, "id":self.id()})
        ).json()
        self.inst = j.get("clientVars", {}).get("uid", "")
        #print(j)
        
        # aquire LMTBID cookie (not sure if needed)
        h = http.post(
            "https://s.deepl.com/web/stats?request_type=jsonrpc",
            data = json.dumps({
                "jsonrpc":"2.0", "method":"WebAppPushStatistics", "id":self.id(),
                "params":{
                    "value":{
                        "instanceId":self.inst,
                        "sessionId":self.sess,
                        "event":"web/pageview",
                        "url":"https://www.deepl.com/translator",
                        "userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0",
                        "resolution":{"width":1920,"height":1080,"devicePixelRatio":1,"viewportWidth":1900,"viewportHeight":916},
                        "data":{"referrer":""}
                    }
                }
            })
        )
        #print(h.headers)

    def id(self):
        self.id_ += 1
        return self.id_
        
    def rpc(self, text):
        return json.dumps({
           "jsonrpc" : "2.0",
           "method" : "LMT_handle_jobs",
           "id" : self.id(),
           "params" : {
              "lang" : {
                 "target_lang" : self.lang,
                 "user_preferred_langs" : [
                    self.lang,
                    "EN"
                 ],
................................................................................
                    "preferred_num_beams" : 4,
                    "raw_en_sentence" : text,
                    "quality" : "fast"
                 }
              ]
           }
        })

    def translate(self, text):
        # skip empty paragraph/table snippets
        if len(text) < 2 or rx_empty.match(text) or not rx_letters.search(text):
            return text
        
        # delay?
        time.sleep(rand(1, 15) / 10.0)
        
        # request
        r = http.post(
            "https://www2.deepl.com/jsonrpc",
            data=self.rpc(text),
            headers={"Referer": "https://www.deepl.com/translator", "Content-Type": "text/plain"}
        )
        if r.status_code != 200:
            log.error(repr(r.content))
            return text
            #return r, r.content
        
        # decode
        r = r.json()
        #print(r)

        if r.get("result"):
            return r["result"]["translations"][0]["beams"][0]["postprocessed_sentence"]
        else:
            return text


# DeepL API costs money
................................................................................


# Translate-python
# requires `pip install translate`
#
#  · provides "microsoft" backend (requires OAuth secret in api_key)
#
#  · or "mymemory" (with email in `email` instead)
#
# https://translate-python.readthedocs.io/en/latest/
#
class translate_python(google):

    def __init__(self, params={}):
        self.params = params  # config+argparse
................................................................................
        except:
            log.error(format_exc())
            raise Exception("Run `pip install translate` to use this module.")
            
        # interestingly this backend function might just work as is.
        if params.get("mymemory"):
            self.translate = Translator(
                provider="mymemory", to_lang=params["lang"], email=params.get("email", "")
            ).translate
        else:
            self.translate = Translator(
                provider="microsoft", to_lang=params["lang"], secret_access_key=params["api_key"]
            ).translate

        # though .linebreakwise has no equivalent, not sure if necessary,
................................................................................
    translate = None
    #linebreakwise = None


# Because, why not?
# Invokes a commandline tool for translating texts.
#
# → with e.g. `translate-cli -t {text}` in "cmd"
#
# Don't quote placeholders {}, {text} or {lang} in the command.
#
class cli(google):

    def __init__(self, params):
        self.params = params
        self.cmd = params.get("cmd", "translate-cli {}")

    # pipe text through external program
    def translate(self, text):
        if rx_empty.match(text) or not rx_letters.search(text):
            return text
        repl = {"{text}": text, "{lang}": self.params["lang"]}
        cmd = [arg.format(text, **repl) for arg in shlex.split(self.cmd)]
        proc = subprocess.run(cmd, stdout=subprocess.PIPE)
        return proc.stdout.decode("utf-8")



# maps a pagetranslate.t.* object (in main module),
# according to config dict {"goog":1, "deepl":0}