Skip to main content

Missing Photos Implementation

Overview​

The missing photos feature in LibrePhotos handles cases where photo files become unavailable on the file system while their metadata remains in the database. This document explains the architecture, implementation details, and how the system handles missing photos.

Architecture​

Data Models​

File Model​

Located in librephotos/api/models/file.py, the File model represents individual files on disk:

class File(models.Model):
    hash = models.CharField(primary_key=True, max_length=64)
    path = models.TextField(blank=True, default="")
    type = models.PositiveIntegerField(choices=FILE_TYPES)
    missing = models.BooleanField(default=False)  # Tracks if file is missing
    embedded_media = models.ManyToManyField("File")

Key fields:

  • hash: MD5 hash of file content + user ID (primary key)
  • path: Full file system path
  • missing: Boolean flag indicating if the file cannot be found
  • type: File type (IMAGE, VIDEO, METADATA_FILE, RAW_FILE, UNKNOWN)

Photo Model​

Located in librephotos/api/models/photo.py, the Photo model has relationships to files:

class Photo(models.Model):
    image_hash = models.CharField(primary_key=True, max_length=64)
    files = models.ManyToManyField(File)  # All associated files
    main_file = models.ForeignKey(
        File,
        related_name="main_photo",
        on_delete=models.SET_NULL,
        null=True,
    )
    # ... other fields

A photo is considered missing when:

  • files=None (no associated files), OR
  • main_file=None (no primary file reference)

Query for missing photos:

missing_photos = Photo.objects.filter(
    Q(owner=user) & Q(files=None) | Q(main_file=None)
)

Detection Mechanism​

_check_files() Method​

The Photo._check_files() method (lines 508-514 in librephotos/api/models/photo.py) is the core detection mechanism:

def _check_files(self):
    for file in self.files.all():
        if not file.path or not os.path.exists(file.path):
            self.files.remove(file)  # Remove from photo's file list
            file.missing = True       # Mark file as missing
            file.save()
    self.save()

This method:

  1. Iterates through all files associated with the photo
  2. Checks if the file path exists on the file system
  3. If missing, removes the file from the photo and sets file.missing = True
  4. Saves changes to the database

When is this called?

  • During photo scans (scan_missing_photos job)
  • When adding new files to existing photos
  • After detecting duplicate photos during import

Jobs​

Scan Missing Photos Job​

Type: JOB_SCAN_MISSING_PHOTOS (job type 14)

Function: scan_missing_photos(user, job_id) in librephotos/api/directory_watcher.py:356-386

Purpose: Checks all photos owned by a user to detect missing files

Implementation:

def scan_missing_photos(user, job_id: UUID):
    # Create or update job entry
    lrj = LongRunningJob.objects.create(
        started_by=user,
        job_id=job_id,
        job_type=LongRunningJob.JOB_SCAN_MISSING_PHOTOS,
    )
    
    # Get all photos for user
    existing_photos = Photo.objects.filter(owner=user.id).order_by("image_hash")
    
    # Process in batches of 5000 for memory efficiency
    paginator = Paginator(existing_photos, 5000)
    lrj.progress_target = paginator.num_pages
    
    for page in range(1, paginator.num_pages + 1):
        for existing_photo in paginator.page(page).object_list:
            existing_photo._check_files()  # Check each photo's files
        update_scan_counter(job_id)

Key features:

  • Processes photos in batches of 5,000 to manage memory
  • Updates progress counter for UI feedback
  • Automatically triggered after full scans if not scanning specific files

Delete Missing Photos Job​

Type: JOB_DELETE_MISSING_PHOTOS (job type 5)

Function: delete_missing_photos(user, job_id) in librephotos/api/autoalbum.py:188-232

Purpose: Permanently removes missing photos and their associated data from the database

Implementation:

def delete_missing_photos(user, job_id):
    # Find all photos with no files or no main file
    missing_photos = Photo.objects.filter(
        Q(owner=user) & Q(files=None) | Q(main_file=None)
    )
    
    # Remove from all album types
    for missing_photo in missing_photos:
        album_dates = AlbumDate.objects.filter(photos=missing_photo)
        for album_date in album_dates:
            album_date.photos.remove(missing_photo)
        
        album_things = AlbumThing.objects.filter(photos=missing_photo)
        for album_thing in album_things:
            album_thing.photos.remove(missing_photo)
        
        album_places = AlbumPlace.objects.filter(photos=missing_photo)
        for album_place in album_places:
            album_place.photos.remove(missing_photo)
        
        album_users = AlbumUser.objects.filter(photos=missing_photo)
        for album_user in album_users:
            album_user.photos.remove(missing_photo)
        
        # Delete associated faces
        faces = Face.objects.filter(photo=missing_photo)
        faces.delete()
    
    # Delete the photos
    missing_photos.delete()
    
    # Delete missing file records
    missing_files = File.objects.filter(Q(hash__endswith=user) & Q(missing=True))
    missing_files.delete()

What gets deleted:

  • Photo records from database
  • File records marked as missing
  • Associations with date-based albums
  • Associations with thing-based albums
  • Associations with place-based albums
  • Associations with user-created albums
  • Face detections linked to the photos

What doesn't get deleted (TODO):

  • Thumbnail files (line 221 notes: "To-Do: Remove thumbnails")

API Endpoints​

Delete Missing Photos​

Endpoint: POST /api/deletemissingphotos

Implementation: DeleteMissingPhotosView in librephotos/api/views/views.py:433-451

class DeleteMissingPhotosView(APIView):
    def post(self, request, format=None):
        try:
            job_id = uuid.uuid4()
            delete_missing_photos(request.user, job_id)
            return Response({"status": True, "job_id": job_id})
        except BaseException:
            logger.exception("An Error occurred")
            return Response({"status": False})

Response:

{
  "status": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000"
}

Note: Also supports GET method (deprecated) for backward compatibility.

Photo Statistics​

Missing photo count is included in the user statistics API response.

Calculation: get_count_stats(user) in librephotos/api/stats.py:382-384

num_missing_photos = Photo.objects.filter(
    Q(owner=user) & Q(files=None) | Q(main_file=None)
).count()

Returned in stats response as:

{
  "num_photos": 1234,
  "num_missing_photos": 5,
  // ... other stats
}

Hash-Based Relinking​

How It Works​

When files reappear in the scanned directories (even with different names or paths), LibrePhotos can automatically relink them using hash-based matching.

Hash Calculation: calculate_hash(user, path) in librephotos/api/models/file.py:136-145

def calculate_hash(user, path):
    hash_md5 = hashlib.md5()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(BUFFER_SIZE), b""):
            hash_md5.update(chunk)
    return hash_md5.hexdigest() + str(user.id)

Key points:

  • Hash is MD5 of file content + user ID
  • User ID ensures photos are scoped to individual users in multi-user setups
  • Buffer size of 65536 bytes for optimal performance

Relinking Process​

Function: create_new_image(user, path) in librephotos/api/directory_watcher.py:62-136

def create_new_image(user, path) -> Photo | None:
    # Calculate hash for the file
    hash = calculate_hash(user, path)
    
    # Check if photo with this hash already exists
    photos: QuerySet[Photo] = Photo.objects.filter(image_hash=hash, owner=user)
    
    if not photos.exists():
        # Create new photo
        photo = Photo()
        photo.image_hash = hash
        # ... initialize photo
    else:
        # Photo exists - add this file to it (relinking)
        file = File.create(path, user)
        photo = photos.first()
        photo.files.add(file)
        
        # Restore if previously marked as removed
        if photo.removed:
            photo.removed = False
            photo.in_trashcan = False
        
        photo.save()
        photo._check_files()  # Verify all files still exist
        
    return photo

Relinking behavior:

  1. Calculate hash of discovered file
  2. Query for existing photos with same hash
  3. If found, add new file to existing photo
  4. Restore photo if it was marked as removed
  5. Run _check_files() to clean up any still-missing files

Frontend Integration​

Photo Serializer​

File: librephotos/api/serializers/photos.py:348

The serializer includes a "Missing" label for photos without files:

def get_image_path(self, obj) -> list[str]:
    if not obj.files or obj.files.count() == 0:
        return ["Missing"]
    return [file.path for file in obj.files.all()]

This ensures the frontend can display appropriate UI for missing photos.

Video Error Handling​

File: librephotos-frontend/src/components/lightbox/MediaDisplay.tsx:64-69

The frontend displays an error alert when video files are missing:

if (videoError) {
  return (
    <Alert color="red" title="Video Not Found">
      <Text>The video file could not be found or is no longer available.</Text>
    </Alert>
  );
}

Future Implementation​

Real-Time File System Monitoring​

Goal: Eliminate most missing photo cases through proactive file tracking

Planned features:

  1. File System Watchers

    • Implement inotify (Linux), FSEvents (macOS), or watchdog library
    • Monitor scanned directories for file changes in real-time
    • Trigger immediate updates instead of waiting for manual scans

  2. Move/Rename Detection

    • Detect when files are moved within scanned directories
    • Automatically update file paths in database
    • Preserve all metadata, ratings, and associations

  3. Immediate Relinking

    • Hash-based matching happens immediately when files appear
    • No manual scan required
    • Significantly reduced "missing photo" window

  4. Benefits

    • Near-instant UI updates when files change
    • Reduced database queries (no periodic scanning)
    • Better user experience with fewer missing photos
    • Lower system resource usage

Implementation Considerations​

  • Performance impact of continuous monitoring
  • Handling large directory trees efficiently
  • Network storage compatibility (NAS, SMB, NFS)
  • Docker container file system event propagation
  • Graceful degradation if monitoring unavailable

Known Issues and TODOs​

Current TODOs​

  1. Remove thumbnails (line 221 in autoalbum.py)

    • When deleting missing photos, thumbnail files are not removed
    • Thumbnails remain in data/thumbnails/ directory
    • Should be cleaned up to free disk space

  2. Move delete_missing_photos function (line 187 in autoalbum.py)

    • Currently in autoalbum.py but doesn't belong there
    • Should be moved to appropriate module (e.g., photo_operations.py or similar)
    • Comment says: "To-Do: This does not belong here"

Edge Cases​

  1. Symbolic links: May not be handled correctly in all cases
  2. Network storage timeouts: Slow network storage may cause false positives
  3. Permissions: Permission changes could make files appear missing
  4. Race conditions: Files changed during scan may cause inconsistencies

Code Organization​

Key Files​

  • Models:

    • librephotos/api/models/file.py - File model with missing flag
    • librephotos/api/models/photo.py - Photo model and _check_files() method
    • librephotos/api/models/long_running_job.py - Job type definitions

  • Business Logic:

    • librephotos/api/directory_watcher.py - Scanning and relinking logic
    • librephotos/api/autoalbum.py - Delete missing photos function

  • API Views:

    • librephotos/api/views/views.py - Delete missing photos endpoint
    • librephotos/api/views/photos.py - Photo operations

  • Statistics:

    • librephotos/api/stats.py - Count calculations including missing photos

  • Serializers:

    • librephotos/api/serializers/photos.py - Photo serialization with "Missing" label

Testing Considerations​

When testing missing photos functionality:

  1. Setup: Create photos with valid files
  2. Trigger: Remove files from file system (outside LibrePhotos)
  3. Scan: Run scan_missing_photos job
  4. Verify: Check that photos marked as missing
  5. Restore: Add files back and rescan
  6. Verify relinking: Ensure photos automatically relink
  7. Delete: Test permanent deletion with delete_missing_photos