Our modules are dockerized and this makes deployment very easy. The containers to deploy are:
- Backend web servers
- Frontends (to serve it through a NGinx server)
This is scalable: we can add more backend servers and configure the load balancers to use them. And adding new workers is very straightforward. In practice, we use several queues and we adapt the capacity (i.e. nb of workers) to each queue depending on the type of tasks it processes. These containers require access to:
- A PostGreSQL database for models.
- One or several data-warehouses (AWS Redshift or a PostGreSQL).
- A Redis (for queuing and cache engine).
- A MongoDB (for queuing).
- A file storage system (we use AWS S3 but it's also compatible with a local storage).
Mostly, we deploy on our multi-tenant AWS architecture. We have developed a small Slack bot to trigger a deployment on AWS. The bot also handles some other features (rollback, get status...).
We also deploy on our local machines for development (in this case, we also have Docker containers providing all the required services such as the PostGreSQL db, Redis, MongoDB...).
Our containers are orchestrated with simple docker-compose files. We had a period running Kubernetes. It went well (it's very easy to setup). But it failed one day in prod and it took us several hours to understand the issue and fix it. This day, we realized being an expert in Kubernetes require time/resource that we don't have. So we went back to a simpler solution which is perfectly fine for our needs.