Enabling the pelican-minify plugin in Pythonista (iOS)


Enabling the pelican-minify plugin in Pythonista (iOS)

When I started this blog, I had a few technical requirements in mind. First of all, I wanted the whole static website build process to be based on Python and to be reproducible both on desktop and on iOS. So, I was very glad to find out I was able to pip install pelican and a few of its dependencies in Pythonista, using StaSh, and get my iPhone to process a bunch of Markdown formatted text files. But then I went on to add features and optimizations that required Pelican plugins. And soon I would get into trouble…

I decided not to use plugins that depended on executable binaries, since I wanted to keep my pipeline compatible with Pythonista. So, I had to read through the documentation and requirements files for each one and eventually decided to skip some that otherwise would be nice to use, like some image optimization automations.

When investigating website performance, I intended to include code ‘minification’, a process that compresses the HTML (and/or JavaScript + CSS) and shortens the time needed to download the pages. There are Pelican plugins available for that purpose, and I found pelican-minify, that didn’t use any external binaries. To keep Pythonista compatibility, I needed that pure-pythoness. However, things started to get a bit complicated.

I got some tracebacks like this in Pythonista:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Traceback (most recent call last):
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/pelican-stuff/make_html.py", line 15, in <module>
    pelican.run()
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/pelican/__init__.py", line 181, in run
    signals.finalized.send(self)
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/minify.py", line 32, in minify_html
    Parallel(n_jobs=-1)(delayed(create_minified_file)(filepath, options) for filepath in files_to_minify)
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/parallel.py", line 772, in __call__
    n_jobs = self._initialize_pool()
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/parallel.py", line 542, in _initialize_pool
    self._pool = MemmapingPool(n_jobs, **poolargs)
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/pool.py", line 580, in __init__
    super(MemmapingPool, self).__init__(**poolargs)
  File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/pool.py", line 419, in __init__
    super(PicklingPool, self).__init__(**poolargs)
  File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/pool.py", line 160, in __init__
    self._repopulate_pool()
  File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/pool.py", line 224, in _repopulate_pool
    w.start()
  File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/process.py", line 131, in start
    self._popen = Popen(self)
  File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/forking.py", line 122, in __init__
    self.pid = os.fork()
OSError: [Errno 1] Operation not permitted

I really wanted minification to work, so I decided to read through the minify module and it’s dependencies to find out a solution. That plugin uses joblib to parallelize the page minification jobs, which is nice because it allows it to run a lot faster on multi core and multiprocessor computers. But it turns out that multiprocessing on Pythonista is limited, so I should find out a way to turn it off. I soon discovered that changing a single character in site-packages/minify.py would solve issue for me.

Line 32 has this code:

1
    Parallel(n_jobs=-1)(delayed(create_minified_file)(filepath, options) for filepath in files_to_minify)

And in site-packages/joblib/parallel.py there was this explanation for the Parallel class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
class Parallel(Logger):
    ''' Helper class for readable parallel mapping.

        Parameters
        -----------
        n_jobs: int, default: 1
            The maximum number of concurrently running jobs, such as the number
            of Python worker processes when backend="multiprocessing"
            or the size of the thread-pool when backend="threading".
            If -1 all CPUs are used. If 1 is given, no parallel computing code
            is used at all, which is useful for debugging. For n_jobs below -1,
            (n_cpus + 1 + n_jobs) are used. Thus for n_jobs = -2, all
            CPUs but one are used.

So I went on and changed the instantiation call by removing the minus (-) sign:

1
    Parallel(n_jobs=1)(delayed(create_minified_file)(filepath, options) for filepath in files_to_minify)

And guess what? It worked! Finally I had this plugin working both on desktop and iPhone and iPad.