WaffleConf and multiprocess

2015-08-25 07:39

Version 0.2.0 of Flask-WaffleConf has just been released and includes support for multiprocess deployments. Here I will explain the problems encountered and the "solution" reached.

The problem

So here is the scenario: imagine you are deploying with uWSGI or another application server. This probably means that you may end up with several workers/processes when serving your application. With version 0.1.0, this led to inconsistencies in the worker's configurations, as an update would never reach the other workers.

The "solution"

In version 0.2.0, I've introduced additional functionality for this scenario, in particular, the usage of Redis to deliver update notifications to other workers/processes. It may not be the best solution, so if you can think of anything better, you know where to find me :)

Anyway, the way this works is as follows:

  • You install Redis, for Debian:
# apt-get install redis-server

$ pip install redis
  • You set the WAFFLE_MULTIPROC setting in your configuration to True
  • You set the WAFFLE_REDIS_HOST, WAFFLE_REDIS_PORT and WAFFLE_REDIS_CHANNEL settings (which default to 'localhost', 6379 and 'waffleconf' respectively)
  • Enjoy

That was the easy part. By default, the extension will use the threading module in order to create a listener for the Redis channel. If you are using uWSGI, this means that you need to enable the --enable-threads and --lazy-apps settings in your configuration. The reason behind --lazy-apps is due to the prefork model used by uWSGI when starting an application. For more information, check uWSGI's documentation.

If you don't want to deal with the GIL, there is another option available: the gevent module. If this module is installed, the extension will use greenlet instead of conventional threading. You should check the uWSGI and gevent section of the uWSGI documentation in order to check how to enable gevent support in your application and the gevent documentation for more information on gevent.

So that's it, as always, I'm open to suggestions, issues, pull requests and pretty much everything!

Enjoy your waffles!