When I started this blog, I had a few technical requirements in mind. First of all, I wanted the whole static website build process to be based on Python and to be reproducible both on desktop and on iOS. So, I was very glad to find out I was able to
pip install pelican and a few of its dependencies in Pythonista, using StaSh, and get my iPhone to process a bunch of Markdown formatted text files. But then I went on to add features and optimizations that required Pelican plugins. And soon I would get into trouble…
I decided not to use plugins that depended on executable binaries, since I wanted to keep my pipeline compatible with Pythonista. So, I had to read through the documentation and requirements files for each one and eventually decided to skip some that otherwise would be nice to use, like some image optimization automations.
pelican-minify, that didn’t use any external binaries. To keep Pythonista compatibility, I needed that pure-pythoness. However, things started to get a bit complicated.
I got some tracebacks like this in Pythonista:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Traceback (most recent call last): File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/pelican-stuff/make_html.py", line 15, in <module> pelican.run() File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/pelican/__init__.py", line 181, in run signals.finalized.send(self) File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/blinker/base.py", line 267, in send for receiver in self.receivers_for(sender)] File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/minify.py", line 32, in minify_html Parallel(n_jobs=-1)(delayed(create_minified_file)(filepath, options) for filepath in files_to_minify) File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/parallel.py", line 772, in __call__ n_jobs = self._initialize_pool() File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/parallel.py", line 542, in _initialize_pool self._pool = MemmapingPool(n_jobs, **poolargs) File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/pool.py", line 580, in __init__ super(MemmapingPool, self).__init__(**poolargs) File "/private/var/mobile/Containers/Shared/AppGroup/F3C0E711-6D38-4FDF-81F2-DC3B97E4E9F1/Pythonista3/Documents/site-packages/joblib/pool.py", line 419, in __init__ super(PicklingPool, self).__init__(**poolargs) File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/pool.py", line 160, in __init__ self._repopulate_pool() File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/pool.py", line 224, in _repopulate_pool w.start() File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/process.py", line 131, in start self._popen = Popen(self) File "/var/containers/Bundle/Application/46E67BC7-8A23-4C29-9CBA-6E4CC2B2BB96/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/multiprocessing/forking.py", line 122, in __init__ self.pid = os.fork() OSError: [Errno 1] Operation not permitted
I really wanted minification to work, so I decided to read through the minify module and it’s dependencies to find out a solution. That plugin uses
joblib to parallelize the page minification jobs, which is nice because it allows it to run a lot faster on multi core and multiprocessor computers. But it turns out that multiprocessing on Pythonista is limited, so I should find out a way to turn it off. I soon discovered that changing a single character in
site-packages/minify.py would solve issue for me.
Line 32 has this code:
Parallel(n_jobs=-1)(delayed(create_minified_file)(filepath, options) for filepath in files_to_minify)
site-packages/joblib/parallel.py there was this explanation for the
1 2 3 4 5 6 7 8 9 10 11 12 13
class Parallel(Logger): ''' Helper class for readable parallel mapping. Parameters ----------- n_jobs: int, default: 1 The maximum number of concurrently running jobs, such as the number of Python worker processes when backend="multiprocessing" or the size of the thread-pool when backend="threading". If -1 all CPUs are used. If 1 is given, no parallel computing code is used at all, which is useful for debugging. For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. Thus for n_jobs = -2, all CPUs but one are used.
So I went on and changed the instantiation call by removing the minus (-) sign:
Parallel(n_jobs=1)(delayed(create_minified_file)(filepath, options) for filepath in files_to_minify)
And guess what? It worked! Finally I had this plugin working both on desktop and iPhone and iPad.