Skip to main content

☁️ Backend

The backend uses the following technologies:

  • Python
  • Django
  • ONNX Runtime

✨ Code Standards​

In order to keep the code consistently structured, the backend has a pre-commit hook. Run pre-commit install from apps/backend/ and the linter and the formatter will check your work before you commit.

We use ruff for both linting and formatting, configured in apps/backend/pyproject.toml. You can also run it by hand with ruff check . and ruff format .. Ruff is pinned (required-version); install the version from apps/backend/requirements.dev.txt.

Lint baseline​

Every rule selected in pyproject.toml (isort, pyupgrade, pep8-naming, flake8-django, blind except, print(), f-strings in logging calls, a pylint subset and the banned APIs) applies to new code straight away. Findings that existed when a rule was turned on are listed per file in apps/backend/ruff-baseline.toml, which pyproject.toml pulls in with extend. The file is generated by scripts/ruff_baseline.py; do not edit it by hand. From apps/backend/:

python scripts/ruff_baseline.py            # check the baseline against the code
python scripts/ruff_baseline.py --update   # rewrite it after fixing findings

The baseline only shrinks. The ruff job of the lint-backend workflow fails when:

  • a file has a finding of a code its baseline entry does not list (ruff check reports it too);
  • an entry no longer occurs. Run --update and commit the smaller file, otherwise the rule could come back in that file unnoticed;
  • the baseline gained an entry compared with the target branch, which is what running --update to absorb a new finding does. The only exception is a pull request that changes the ruff config (a new rule or a ruff upgrade); the job then lists the new entries for the reviewer.

The baseline is per file and code, not per finding, so a file listed for BLE001 can gain another blind except without failing; please do not rely on that. Do not add # noqa comments for baselined codes either. Permanent exceptions, such as print() in console scripts, and the banned APIs (ruff TID251, each with a message naming the replacement) live in pyproject.toml; exceptions go under [tool.ruff.lint.extend-per-file-ignores], because a plain per-file-ignores table would replace the baseline instead of adding to it.

New in 1.5.0 The supported Python versions are requires-python in apps/backend/pyproject.toml: 3.14, the Python of the Docker images (Ubuntu 26.04), the Windows standalone build and the prebuilt wheels (cp314). Ruff targets its floor, so it reports syntax 3.14 cannot parse, and the test-backend workflow runs the suite on 3.14 for every pull request. Python 3.15 waits for an onnxruntime release with cp315 wheels. Up to 1.4.0 the floor was 3.11 (the Windows build) and the images ran 3.12.

Type checking​

The backend is type-checked with mypy and the django-stubs and djangorestframework-stubs plugins, which understand the ORM: a null=True field is Optional, custom managers and <fk>_id attributes are typed, and so are reverse relations. Run it from apps/backend/:

python scripts/typecheck.py                      # Linux, Docker (DEBUG=1 images)
.venv-win\Scripts\python scripts\typecheck.py      # Windows

The config lives in [tool.mypy] in apps/backend/pyproject.toml. It pins platform = "linux", so a Windows machine reports exactly what CI reports. Migrations and tests are not checked. The script compares the result with apps/backend/mypy-baseline.txt, a list of accepted errors that is empty today and only ever shrinks. The django job of the test-backend workflow runs the same check and fails when:

  • mypy reports an error that is not in the baseline. Errors are matched by file, code and message, not by line, so moving existing code around is fine;
  • a baseline entry no longer occurs. Run python scripts/typecheck.py --update and commit the smaller file;
  • the baseline grew compared with the target branch. The only exception is a pull request that changes requirements*.txt or the mypy config, because a new stub or library release can change the messages; the job then lists the new entries for the reviewer. This is also why Renovate never automerges the backend type checking group;
  • a strict module (see below) has a baseline entry, or a strict-list entry names no module mypy checks (a module renamed or moved without updating the list).

Rules for new code:

  • Every function is annotated, including -> None (disallow_untyped_defs).
  • No explicit Any in an annotation; use object, a Protocol or a TypedDict instead. Ruff's ANN401 enforces this outside tests. *args: Any, **kwargs: Any on a pass-through override is allowed.
  • No bare # type: ignore. An ignore needs an error code, and an ignore that no longer suppresses anything is itself an error. Fix the type instead of ignoring it.
  • typing.cast only where the value really is narrower than the stubs say, with a comment saying why.
  • Never run --update to accept new errors.

The second [[tool.mypy.overrides]] table lists the strict modules, which additionally forbid explicit Any, untyped calls and bare generics such as dict. The list only grows: when you add a new module or package, add it there in the same pull request, and when you move or rename a listed module, rename its entry.

Dead code​

The dead-code job of the lint-backend workflow runs vulture (pinned in apps/backend/requirements.dev.txt) and fails on a function, method, class or property that nothing references. Run the same check from apps/backend/:

python scripts/check_dead_code.py

Tests are not scanned, so code that only a test calls counts as dead. Framework entry points such as @action methods, signal receivers, serializer get_<field> and validate_<field> methods and management commands are recognised by the script. A name that only a string refers to, such as a dotted path in the settings or a django_q schedule, goes into apps/backend/vulture_whitelist.py with a comment naming the caller. That file only shrinks: when the check flags something, delete it rather than whitelisting it.

Pagination​

New in 1.4.0 Every list endpoint pages the same way: api.drf.pagination.DefaultPagination in apps/backend/api/drf/pagination.py, which is DRF's DEFAULT_PAGINATION_CLASS. It reads ?page= and ?page_size= (100 rows by default, at most 1000; both are clamped, a value that is not a number is a 400, and a page past the end is a 404) and always answers

{ "count": 2345, "next": "…?page=3", "previous": "…?page=1", "results": [], "page": 2, "page_size": 100, "num_pages": 24 }
  • A ListModelMixin view gets it without declaring anything. A view that needs another page size names one of the presets in the same module, which change the page size and nothing else. Only WholeListPagination is left: the people, user album, auto album and thing album lists answer 1000 a page, because clients that read only their first page still use them (the third-party UhuruPhotos app and the old React Native app); its docstring names the reader of each. New in 1.5.0 Up to 1.4.0, the place album, tag, user, album collection and photo edit lists also answered 1000 a page, and the photo list and the shared photo and album lists 2500. The duplicates and stacks lists also sent has_next and has_previous, and the folder listing repeated its page as subfolders and pagination.
  • A view that is not a GenericAPIView pages through PaginatedViewMixin (self.paginate_queryset(), self.get_paginated_response()); no view imports Django's Paginator or slices a queryset into pages.
  • A list that answers every row at once sets pagination_class = None and is listed with the reason in UNPAGINATED_LIST_VIEWS. The one other style is the keyset cursor of the mobile sync feeds (api/views/sync.py).
  • PaginationConventionTest in api/tests/infra/test_api_conventions.py walks every routed list view and fails on anything else.

A client that shows a whole list follows next until it is null: fetchAllPages(client, endpoint) in packages/api-client does that for the web and mobile apps. Reading only the first page silently drops every row past it.

🐛 Debugging​

Usually nothing goes as planned, and you need to debug your shiny new feature. In the next paragraph, I will explain to you the two tools we have for debugging:

Using pdb​

Add the following in the python code where you want a breakpoint:

import pdb; pdb.set_trace()

Attach to the backend service:

docker attach $(docker ps --filter name=backend -q)

Debug as normal in pdb!

When you're done debugging, continue execution (c) and press Ctrl-P followed by Ctrl-Q to detach from the container without stopping it.

Turning up the log for one module​

Every backend module logs through its own logging.getLogger(__name__), so the logger name is the module path: api.directory_watcher.scan_jobs, api.runtime.supervisor, nextcloud.views. Loggers form a tree along the dots, which means a package name covers every module below it.

To see the DEBUG lines of just the part you are working on, set LOG_LEVELS on the backend container instead of raising LOG_LEVEL for everything:

services:
  backend:
    environment:
      - LOG_LEVELS=api.directory_watcher=DEBUG

Several overrides are comma-separated (api.directory_watcher=DEBUG,api.runtime.supervisor=WARNING), and they work in both directions - use WARNING or ERROR to quiet a noisy module. The output still goes to librephotos.log under BASE_LOGS and to the console, in the usual format. When you run the backend outside Docker, export the variable in the shell that starts manage.py. The whole configuration is built in librephotos/logging_bootstrap.py.

Using silk​

In order to debug queries, start the backend container in dev mode. Then you can access silk under /api/silk. Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection.

🏙️ Structure​

There are a lot of folders in our backend. Here is a quick rundown on where you can find what. The Django project itself — settings (librephotos/settings/), URL routing (librephotos/urls.py) and wsgi.py — lives in the librephotos/ folder, with supporting pieces in scripts/, chunked_upload/ and nextcloud/.

Django​

Most of the application code is within the API folder using Django. In the following section, I will explain where you can find what

management​

This exposes all the commands that you can use via the command line. If you want a new command, that's the place to add it.

migrations​

Every time we change our models, we have to migrate the database. We use Django migration feature to create migration files to migrate without the headaches.

Migrations 0001 to 0100 are squashed into 0001_squashed_0100, so the folder starts there and new migrations keep counting up from the newest file. Keep its replaces list: existing databases have the original names recorded and rely on it. A new migration that depends on anything up to 0100 depends on ("api", "0001_squashed_0100"). Why older installs have to upgrade in two steps is explained in Upgrading.

models​

Here are the actual data types. If you want to figure out how a photo works or how faces are connected to persons, then this is your folder.

views​

Here you can find our API implemented. They are separated, similar to the models. Views that expose the photos will be here in photos too.

serializers​

You have your python model and want to somehow convert that to JSON. That's what the serializer does!

Services​

Not everything runs inside Django. The heavy machine-learning work lives in standalone Flask processes that Django talks to over plain HTTP on localhost. Six of them sit under service/ — thumbnail, face_recognition, image_captioning, exif, tags and ocr — and image_similarity/ is a separate top-level folder. Each builds its Flask app with service/_common.py (create_app, the shared /health and /unload-model, json_fields for request parsing) and is served by a gevent WSGIServer on a fixed port; the ports are defined in the SERVICES dict in api/runtime/sidecars.py (image_similarity 8002, thumbnail 8003, face_recognition 8005, image_captioning 8007, exif 8010, tags 8011, ocr 8012).

The backend reaches them with sidecar_url(name) and calls them through api.runtime.sidecars.post/get, which retry a refused connection or a 503 and raise requests.HTTPError for any other error status; a caller turns that into its own error (for example MetadataReadError).

You start them with python manage.py start_service, which also schedules api.runtime.supervisor.check_services in django-q2 to poll each service's /health endpoint every minute: a service that died is restarted, and one that has held a model for two minutes without a request is asked to unload it. Stopping a service sends SIGTERM and kills it only if it is still there five seconds later.

Because these are separate processes, the docker attach + pdb trick above does not reach them — to debug a service, check its log under /logs/ or add logging in the service's own main.py.

Machine Learning​

Every model runs on ONNX Runtime; there is no PyTorch in the image, which keeps the image small. The same image runs the models on a GPU through ONNX Runtime's WebGPU plugin; service/onnx_session.py picks the device for every session, so a new model gets the GPU by creating its session with inference_session(). A new model has to come as an ONNX export (the onnx-community and Xenova organisations on Hugging Face publish many) with its tokenizer as a tokenizer.json for the tokenizers package.

Image Captioning​

Captions are generated on demand by the image captioning service (service/image_captioning/, port 8007), which runs Liquid AI's LFM2.5-VL-450M as three ONNX graphs (SigLIP 2 NaFlex vision encoder, token embedding, merged LFM2 decoder with its key/value and convolution caches), driven by service/image_captioning/lfm2_vl.py with greedy decoding. The Captioning Model site setting is lfm2_vl_450m or none; the model files are always downloaded so turning captioning on never waits.

The request carries a prompt. PhotoCaption.generate_captions_im2txt builds it from the user's caption-context switches (User.llm_settings, a historical name): the recognised person's name and the geocoded place go into the prompt, and the model writes them into the caption itself. There is no separate LLM pass any more. Photos are always fed as one tile of at most 256 image tokens; see the module docstring for why.

The caption is stored under the "im2txt" key in PhotoCaption.captions_json. For the user-facing comparison of these models, see the image captioning guide.

Face Recognition​

We use InsightFace to detect and recognise faces, running on ONNX Runtime: an SCRFD detector finds the faces and an ArcFace model turns each one into a 512-dimension embedding. The models are InsightFace's; the code that runs them is a port of the part of the insightface package we use (service/face_recognition/face_analysis.py), so the backend does not install the package and its matplotlib, scipy and scikit-image dependencies. It has to give the same boxes and embeddings as insightface 0.7.3, or new faces stop matching the ones already stored. This replaced the previous dlib / face_recognition pipeline (which produced 128-dimension encodings). The model is selectable via the Face Recognition Model site setting (buffalo_sc by default). Unknown faces are grouped with automatic clustering, and a separate classification step matches faces against already-labelled people.

Tags, semantic search and similar photos: OpenCLIP​

One image-text model does all three: OpenCLIP ViT-B/32 trained by LAION on DataComp-XL (laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K, MIT licence, 72.7 % ImageNet zero-shot), as ONNX in the tags service (service/tags/openclip/). There is no model to choose.

  • The bundle (openclip_vitb32 in api/runtime/ml_models.py, mirrored on derneuere/librephotos_models) is built by scripts/build_openclip_onnx.py, which runs in a throwaway venv with torch and open_clip (neither is a backend requirement): it exports the image tower (visual.onnx) and the text tower (textual.onnx) as fp32 graphs returning the raw 512-d projections, checks them against PyTorch (cosine ≥ 0.9999 on ~50 photos and prompts, through the runtime's own preprocessing and tokenizer), and writes preprocess.json from open_clip's pretrained config: shortest-edge bicubic resize with torchvision's arithmetic, centre crop, CLIP mean/std, and the model's logit scale. The tokenizer is the OpenAI CLIP BPE (tokenizer.json), laid out as open_clip's tokenize. An int8 text tower was tried and rejected (recall@10 against fp32 0.70).
  • Tags: zero-shot against the vocabulary in service/tags/tags.txt (prompted as a photo of <tag>, embedded once and cached as tag_embeddings.npy next to the model), cut on the softmax probability over all tags at the model's logit scale, because raw CLIP cosines are not comparable between photos; at most 10. They are stored as {"tags": [...]} in PhotoCaption.captions_json["openclip_vitb32"] and as AlbumThing rows of type openclip_vitb32_tag, which the Things albums, the word cloud, text search and the document detector read.
  • Search embeddings: the tags job stores the raw image embedding of the very run that tagged the photo (/generate-tags with with_embedding), and the scan's embedding job waits for it (batch_calculate_clip_embedding(wait_for_tags=True), then once more when the tags job finishes). /clip-embeddings embeds a batch (16 images per image-tower run) and, with with_tags, returns the tags of the same run; /query-embeddings runs the text tower, a session of its own loaded with the first query. With FEATURE_SCENE_CLASSIFICATION off nothing is tagged, but the service still runs: search needs it.
  • Thresholds are inner products of raw vectors (image and text norms are both ~13.5): search 31.3, similar photos 138.9, chosen by scripts/calibrate_openclip.py so a query and a photo find as many photos as CLIP ViT-B/32 did at its 27 and 90.

Photo.clip_embeddings_model records the model of each stored embedding (NULL means CLIP ViT-B/32, the model of everything stored before the column). The similarity index and similar photos use only OpenCLIP's embeddings. Migration 0152 drops the retired models' settings, tags and tag albums and the similarity index files, and manage.py build_similarity_index (container start) queues batch_calculate_clip_embedding for every user with another model's embeddings, which re-embeds them in place, re-tags them from the same run and rebuilds the index as it goes.