Skip to content

Background processing

Overview

Pictures processing

For one uploaded picture, Panoramax can blur the picture using an external service (cf blurring API documentation).

After the blurring is done, Panoramax generate several derivatives versions:

  • Thumbnail: a small version of the picture, with a fixed width of 500 px (and the same width/height ratio than the original picture). For 360°, the thumbnail is a 500x300 px crop of the center of the picture to have a flat picture.
  • SD picture: a small definition picture, with a fixed width of 2048 px (and the same width/height ratio than the original picture)
  • Tiled version: for 360° pictures, a tiled version of the picture, consisting of several small jpeg files so we only load the needed tiles when viewing the picture.

Other background tasks are:

Picture deletion

When deleting pictures, the pictures are immediately deleted from the database but the associated files are deleted asynchronously.

Upload set dispatch

When an upload set is completed (all the pictures have been uploaded), the upload set is dispatched to deduplicate the pictures and create the associated collections.

Sequence finalization

This background task compute associated metadata for the sequence, like its geometry and statistics about its pictures.

Picture metadata reading

The picture's metadata parsing can be done again with this background task. It's especially handy when the pictures metadata parsing library has been updated.

Picture transfers

This task handle the transfer of pictures from another instance.

Deployment

Panoramax API has 2 modes to process background tasks.

This can either be done directly by the API (as background threads), or delegated to independent picture processing workers.

Using the API for this is easier to deploy, but using independent picture processing workers is way better to have a retry mechanism and distribute the load on different servers, especially since picture processing can be quite resource consuming for a limited time.

The 2 modes are not exclusives, you can process pictures from the API and also have independent picture processing workers.

Each worker will only process pictures one by one, but several workers can be run in the same time.

API

The parameter PICTURE_PROCESS_THREADS_LIMIT can be used to limit the number of background threads used to process the pictures. If set to 0, no background thread will be run, so no pictures will be processed unless a separate picture processing worker is run.

Tip

For production instances, it's recommended to set PICTURE_PROCESS_THREADS_LIMIT to 0 so the API focus on its API tasks, and have separate workers to process the pictures.

Separate picture processing workers

Separate picture processing workers can be run with the flask endpoint picture-worker:

Note: the picture workers use the same environment variables as the API.

When run directly with flask it is done like:

panoramax_backend picture-worker

When run directly with flask it is done like:

flask picture-worker

There is a separate docker entrypoint that can be used when using docker:

docker run \
    -e DB_URL=<database connection string>  -e <other variables> \
    --name geovisio-worker \
    panoramax/api:develop \
    picture-worker

A docker compose example can be viewed in the service background-worker of the docker compose full file.

In the docker compose example, 5 workers will be run (defined as a replica)

To change this number, you can do:

docker compose up background-worker -d --scale background-worker=<VALUE>