Update README.md
This commit is contained in:
@@ -1,3 +1,597 @@
|
||||
# image-backup-and-web-ui
|
||||
O-PHONE
|
||||
=======
|
||||
|
||||
This is a set of 2 programs, the first is to sort images from single folder and organise them by date into another folder and create a web ui to look at them with
|
||||
A Python command-line tool for organising and deduplicating media
|
||||
imported from phones and other devices.
|
||||
|
||||
The program is designed primarily for creating a long-term media
|
||||
backup on an external drive.
|
||||
|
||||
It uses SHA-256 file hashes and a SQLite database to identify files
|
||||
that have already been backed up. This allows the same source folder
|
||||
to be imported repeatedly without creating duplicate copies.
|
||||
|
||||
|
||||
WHAT IT DOES
|
||||
============
|
||||
|
||||
Given a source folder and a destination folder, o-phone.py:
|
||||
|
||||
1. Searches the source folder recursively.
|
||||
|
||||
2. Identifies supported image, video and other files.
|
||||
|
||||
3. Determines the best available date for each file.
|
||||
|
||||
4. Organises media into a date-based directory structure.
|
||||
|
||||
5. Renames imported media using a consistent filename format.
|
||||
|
||||
6. Calculates a SHA-256 hash for every imported file.
|
||||
|
||||
7. Stores the hash and file information in a SQLite database
|
||||
located in the destination folder.
|
||||
|
||||
8. Checks the database before importing a file.
|
||||
|
||||
9. Skips files whose hash is already present in the database.
|
||||
|
||||
10. Verifies files after they have been copied.
|
||||
|
||||
11. Records the imported file in the database only after the
|
||||
copy and verification have succeeded.
|
||||
|
||||
This makes it safe to run the program again later when new photos
|
||||
and videos have been copied from the phone.
|
||||
|
||||
|
||||
DIRECTORY STRUCTURE
|
||||
===================
|
||||
|
||||
The normal destination structure looks like this:
|
||||
|
||||
phone-media-backup/
|
||||
|
|
||||
+-- .phone-backup.sqlite
|
||||
+-- 2024/
|
||||
| +-- January/
|
||||
| | +-- 03-01-2024/
|
||||
| | | +-- Phone-03-01-2024-482193.jpg
|
||||
| | | +-- Phone-03-01-2024-817204.mp4
|
||||
| | |
|
||||
| | +-- February/
|
||||
| |
|
||||
| +-- December/
|
||||
|
|
||||
+-- 2025/
|
||||
| +-- January/
|
||||
| |
|
||||
| +-- ...
|
||||
|
|
||||
+-- 2026/
|
||||
|
|
||||
+-- screenshots/
|
||||
|
|
||||
+-- u-mp3/
|
||||
|
|
||||
+-- u-xxx/
|
||||
|
||||
|
||||
MEDIA ORGANISATION
|
||||
==================
|
||||
|
||||
Images and videos are organised by:
|
||||
|
||||
YEAR
|
||||
MONTH
|
||||
DAY-MONTH-YEAR
|
||||
|
||||
For example:
|
||||
|
||||
2026/
|
||||
August/
|
||||
14-08-2026/
|
||||
|
||||
The month is written as its full English name rather than a
|
||||
numeric month.
|
||||
|
||||
|
||||
FILENAMES
|
||||
=========
|
||||
|
||||
Imported files are renamed to:
|
||||
|
||||
Phone-DD-MM-YYYY-RANDOM.ext
|
||||
|
||||
For example:
|
||||
|
||||
Phone-14-08-2026-483921.jpg
|
||||
Phone-14-08-2026-182735.mp4
|
||||
|
||||
The six-digit number is randomly generated.
|
||||
|
||||
The number is not used as the duplicate identifier.
|
||||
|
||||
The SHA-256 hash of the actual file contents is the authoritative
|
||||
duplicate identifier.
|
||||
|
||||
|
||||
SCREENSHOTS
|
||||
===========
|
||||
|
||||
PNG files are treated as screenshots.
|
||||
|
||||
They are placed into:
|
||||
|
||||
screenshots/
|
||||
|
||||
The original date information is still retained in the database.
|
||||
|
||||
Other supported image formats remain in the normal date-based
|
||||
directory structure.
|
||||
|
||||
|
||||
UNKNOWN FILE TYPES
|
||||
==================
|
||||
|
||||
Files with formats that are not recognised as normal photo or
|
||||
video media are placed into a directory based on their extension.
|
||||
|
||||
For example:
|
||||
|
||||
.mp3
|
||||
|
||||
becomes:
|
||||
|
||||
u-mp3/
|
||||
|
||||
A file with extension:
|
||||
|
||||
.xyz
|
||||
|
||||
would become:
|
||||
|
||||
u-xyz/
|
||||
|
||||
|
||||
FILES WITHOUT AN EXTENSION
|
||||
==========================
|
||||
|
||||
Files without a usable extension are handled separately rather
|
||||
than being silently discarded.
|
||||
|
||||
The exact handling depends on the current version of o-phone.py.
|
||||
|
||||
|
||||
DUPLICATE DETECTION
|
||||
===================
|
||||
|
||||
Duplicate detection is based on SHA-256.
|
||||
|
||||
For each file being imported, o-phone.py calculates:
|
||||
|
||||
SHA-256(file contents)
|
||||
|
||||
The resulting hash is compared against hashes stored in:
|
||||
|
||||
.phone-backup.sqlite
|
||||
|
||||
If the hash already exists, the file is considered already backed
|
||||
up and is skipped.
|
||||
|
||||
This means duplicate detection is based on the contents of the
|
||||
file rather than:
|
||||
|
||||
- filename
|
||||
- folder name
|
||||
- file creation date
|
||||
- file modification date
|
||||
- device name
|
||||
|
||||
This is important because phones can rename files or give different
|
||||
metadata to the same media.
|
||||
|
||||
|
||||
WHY USE A DATABASE?
|
||||
===================
|
||||
|
||||
The SQLite database acts as an index of the backup.
|
||||
|
||||
Instead of repeatedly scanning every existing file on the external
|
||||
drive, o-phone.py can check the database for the SHA-256 hash.
|
||||
|
||||
Conceptually:
|
||||
|
||||
Source file
|
||||
|
|
||||
v
|
||||
Calculate SHA-256
|
||||
|
|
||||
v
|
||||
Search SQLite database
|
||||
|
|
||||
+---- hash exists ----> SKIP
|
||||
|
|
||||
+---- hash missing ---> COPY
|
||||
|
|
||||
v
|
||||
Verify copy
|
||||
|
|
||||
v
|
||||
Add to database
|
||||
|
||||
SQLite is built into Python, so no external database server is
|
||||
required.
|
||||
|
||||
|
||||
IMPORTANT: DATABASE UPDATE ORDER
|
||||
================================
|
||||
|
||||
The database should only be updated after a successful copy.
|
||||
|
||||
The intended sequence is:
|
||||
|
||||
1. Calculate source hash
|
||||
2. Check database
|
||||
3. If already present, skip
|
||||
4. Copy file
|
||||
5. Verify destination file
|
||||
6. Calculate/verify destination hash
|
||||
7. Confirm hashes match
|
||||
8. Add file to database
|
||||
|
||||
This prevents a failed or incomplete copy from being recorded as
|
||||
successfully backed up.
|
||||
|
||||
|
||||
RUNNING THE PROGRAM
|
||||
===================
|
||||
|
||||
The basic command is:
|
||||
|
||||
python3 o-phone.py SOURCE DESTINATION
|
||||
|
||||
Example:
|
||||
|
||||
python3 o-phone.py \
|
||||
"/Users/yourname/Desktop/phone-backup-moto-g13" \
|
||||
"/Volumes/all/phone-media-backup"
|
||||
|
||||
SOURCE
|
||||
------
|
||||
|
||||
The first argument is the folder containing the files imported from
|
||||
your phone.
|
||||
|
||||
The source folder can contain subdirectories.
|
||||
|
||||
o-phone.py searches recursively.
|
||||
|
||||
Example:
|
||||
|
||||
/Users/yourname/Desktop/phone-backup-moto-g13/
|
||||
|
||||
|
||||
DESTINATION
|
||||
-----------
|
||||
|
||||
The second argument is the root folder of the backup.
|
||||
|
||||
For example:
|
||||
|
||||
/Volumes/all/phone-media-backup/
|
||||
|
||||
The destination can be located on an external hard drive or SSD.
|
||||
|
||||
|
||||
RECOMMENDED WORKFLOW
|
||||
====================
|
||||
|
||||
When new photos are available on the phone:
|
||||
|
||||
1. Copy the new phone files onto your Mac.
|
||||
|
||||
2. Put them into a source folder.
|
||||
|
||||
3. Run o-phone.py using the same destination folder.
|
||||
|
||||
For example:
|
||||
|
||||
python3 o-phone.py \
|
||||
"/Users/yourname/Desktop/phone-backup-moto-g13" \
|
||||
"/Volumes/all/phone-media-backup"
|
||||
|
||||
The program will calculate hashes for the source files.
|
||||
|
||||
Files already present in the database will be skipped.
|
||||
|
||||
New files will be copied into the appropriate destination
|
||||
directories.
|
||||
|
||||
Therefore it is safe to use the same destination repeatedly.
|
||||
|
||||
|
||||
DRY RUN
|
||||
=======
|
||||
|
||||
The program supports a dry-run mode when available in the current
|
||||
version.
|
||||
|
||||
Dry-run mode allows the proposed changes to be inspected without
|
||||
actually copying files.
|
||||
|
||||
Example:
|
||||
|
||||
python3 o-phone.py SOURCE DESTINATION --dry-run
|
||||
|
||||
Always check the output before performing a large import.
|
||||
|
||||
|
||||
DATABASE
|
||||
========
|
||||
|
||||
The SQLite database is stored inside the destination directory.
|
||||
|
||||
Default database:
|
||||
|
||||
.phone-backup.sqlite
|
||||
|
||||
It should be treated as part of the backup.
|
||||
|
||||
If you move the entire backup to another drive, move the database
|
||||
with it.
|
||||
|
||||
Do not delete or manually modify the database unless you understand
|
||||
the consequences.
|
||||
|
||||
|
||||
DATABASE INTEGRITY
|
||||
==================
|
||||
|
||||
The database is an index of the files in the backup.
|
||||
|
||||
Over time it is possible for the database and the actual files to
|
||||
become different.
|
||||
|
||||
For example, this could happen if somebody manually deletes a file
|
||||
from the backup drive.
|
||||
|
||||
A database verification/check function is therefore intended to
|
||||
compare:
|
||||
|
||||
SQLite records
|
||||
against
|
||||
actual files on disk
|
||||
|
||||
The check should be used periodically, especially after manually
|
||||
changing files on the backup drive.
|
||||
|
||||
|
||||
DO NOT MANUALLY ORGANISE THE BACKUP
|
||||
===================================
|
||||
|
||||
Once files have been imported, it is recommended not to manually
|
||||
rename or move them.
|
||||
|
||||
The database records their paths.
|
||||
|
||||
Moving files outside of o-phone.py can therefore cause the database
|
||||
and the actual backup to become inconsistent.
|
||||
|
||||
If files need to be reorganised, future versions of the program may
|
||||
provide dedicated maintenance commands.
|
||||
|
||||
|
||||
SOURCE FILES ARE NOT DELETED
|
||||
============================
|
||||
|
||||
o-phone.py is intended to be a backup/import tool.
|
||||
|
||||
It should not delete the original files from the source folder as
|
||||
part of a normal import.
|
||||
|
||||
The source files should remain available until you have independently
|
||||
confirmed that the backup is complete and healthy.
|
||||
|
||||
|
||||
FILE DATES
|
||||
==========
|
||||
|
||||
The program attempts to determine the most useful date associated
|
||||
with each media file.
|
||||
|
||||
Depending on the file type, this may come from embedded metadata
|
||||
or filesystem information.
|
||||
|
||||
The exact date source can vary between:
|
||||
|
||||
- JPEG photos
|
||||
- HEIC photos
|
||||
- PNG screenshots
|
||||
- videos
|
||||
- RAW camera files
|
||||
- files copied from different phones
|
||||
- files whose metadata has been removed
|
||||
|
||||
When metadata is unavailable, the program uses an appropriate
|
||||
fallback.
|
||||
|
||||
|
||||
RAW FILES
|
||||
=========
|
||||
|
||||
RAW camera files are preserved rather than converted.
|
||||
|
||||
They are treated as source files and copied to the backup.
|
||||
|
||||
The original RAW data is not modified.
|
||||
|
||||
|
||||
SUPPORTED MEDIA
|
||||
===============
|
||||
|
||||
The exact list of supported extensions is defined in o-phone.py.
|
||||
|
||||
Typical photo formats include:
|
||||
|
||||
.jpg
|
||||
.jpeg
|
||||
.heic
|
||||
.heif
|
||||
.png
|
||||
.webp
|
||||
.tif
|
||||
.tiff
|
||||
RAW formats
|
||||
|
||||
Typical video formats include:
|
||||
|
||||
.mp4
|
||||
.mov
|
||||
.m4v
|
||||
.avi
|
||||
.mkv
|
||||
.3gp
|
||||
|
||||
The program may recognise additional formats.
|
||||
|
||||
Unknown formats are not automatically discarded.
|
||||
|
||||
|
||||
REQUIREMENTS
|
||||
============
|
||||
|
||||
macOS or another operating system with Python 3.
|
||||
|
||||
Python 3.9 or newer is recommended.
|
||||
|
||||
The standard Python libraries used by the program do not require a
|
||||
separate database server.
|
||||
|
||||
Some optional functionality may require additional Python packages.
|
||||
|
||||
|
||||
INSTALLATION
|
||||
============
|
||||
|
||||
Check Python:
|
||||
|
||||
python3 --version
|
||||
|
||||
Run the program directly:
|
||||
|
||||
python3 o-phone.py SOURCE DESTINATION
|
||||
|
||||
|
||||
SAFETY
|
||||
======
|
||||
|
||||
Before using o-phone.py with an important archive:
|
||||
|
||||
1. Test it with a small source folder.
|
||||
|
||||
2. Use --dry-run where available.
|
||||
|
||||
3. Confirm the destination path carefully.
|
||||
|
||||
4. Check the generated directory structure.
|
||||
|
||||
5. Open several copied files.
|
||||
|
||||
6. Confirm the SQLite database exists.
|
||||
|
||||
7. Run the database/filesystem integrity check if available.
|
||||
|
||||
8. Keep another independent backup of important media.
|
||||
|
||||
|
||||
IMPORTANT PRINCIPLE
|
||||
===================
|
||||
|
||||
The SHA-256 hash is the identity of the file.
|
||||
|
||||
The filename and directory are organisational information.
|
||||
|
||||
For example, these two files:
|
||||
|
||||
IMG_1234.JPG
|
||||
vacation-photo.jpg
|
||||
|
||||
may represent exactly the same file contents.
|
||||
|
||||
If their SHA-256 hashes match, o-phone.py considers them duplicates.
|
||||
|
||||
Conversely, two files with the same filename are not considered
|
||||
duplicates if their contents differ.
|
||||
|
||||
|
||||
PROJECT PHILOSOPHY
|
||||
==================
|
||||
|
||||
The goal of o-phone.py is to provide a simple, dependable and
|
||||
repeatable way to build a personal media archive.
|
||||
|
||||
The important principles are:
|
||||
|
||||
- Do not delete source files.
|
||||
- Do not overwrite existing backup files unnecessarily.
|
||||
- Use file contents rather than filenames for duplicate detection.
|
||||
- Verify files before recording them as successfully backed up.
|
||||
- Keep a local SQLite index.
|
||||
- Keep the archive organised by date.
|
||||
- Make repeated imports safe.
|
||||
- Prefer simple, inspectable storage over proprietary formats.
|
||||
|
||||
|
||||
EXAMPLE
|
||||
=======
|
||||
|
||||
Suppose the phone contains:
|
||||
|
||||
DCIM/Camera/IMG_1234.jpg
|
||||
DCIM/Camera/IMG_1235.jpg
|
||||
DCIM/Camera/VID_001.mp4
|
||||
Screenshots/screenshot.png
|
||||
|
||||
Run:
|
||||
|
||||
python3 o-phone.py \
|
||||
"/Users/yourname/Desktop/phone-import" \
|
||||
"/Volumes/all/phone-media-backup"
|
||||
|
||||
The destination might become:
|
||||
|
||||
phone-media-backup/
|
||||
|
|
||||
+-- .phone-backup.sqlite
|
||||
|
|
||||
+-- 2026/
|
||||
| |
|
||||
| +-- August/
|
||||
| |
|
||||
| +-- 14-08-2026/
|
||||
| |
|
||||
| +-- Phone-14-08-2026-483921.jpg
|
||||
| +-- Phone-14-08-2026-193827.mp4
|
||||
|
|
||||
+-- screenshots/
|
||||
|
|
||||
+-- Phone-14-08-2026-928374.png
|
||||
|
||||
If the same source files are imported again, their hashes will
|
||||
already exist in the database and they will be skipped.
|
||||
|
||||
|
||||
BACKUP REMINDER
|
||||
===============
|
||||
|
||||
A single external drive is not a complete backup strategy.
|
||||
|
||||
If the media is important, maintain at least one additional,
|
||||
independent copy.
|
||||
|
||||
A RAID array, external drive, or database does not by itself protect
|
||||
against every form of data loss.
|
||||
|
||||
o-phone.py is an organisation and import tool, not a replacement for
|
||||
a complete backup strategy.
|
||||
|
||||
Reference in New Issue
Block a user