diff --git a/README.md b/README.md index 7cde252..4bf9e19 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,597 @@ -# image-backup-and-web-ui +O-PHONE +======= -This is a set of 2 programs, the first is to sort images from single folder and organise them by date into another folder and create a web ui to look at them with \ No newline at end of file +A Python command-line tool for organising and deduplicating media +imported from phones and other devices. + +The program is designed primarily for creating a long-term media +backup on an external drive. + +It uses SHA-256 file hashes and a SQLite database to identify files +that have already been backed up. This allows the same source folder +to be imported repeatedly without creating duplicate copies. + + +WHAT IT DOES +============ + +Given a source folder and a destination folder, o-phone.py: + +1. Searches the source folder recursively. + +2. Identifies supported image, video and other files. + +3. Determines the best available date for each file. + +4. Organises media into a date-based directory structure. + +5. Renames imported media using a consistent filename format. + +6. Calculates a SHA-256 hash for every imported file. + +7. Stores the hash and file information in a SQLite database + located in the destination folder. + +8. Checks the database before importing a file. + +9. Skips files whose hash is already present in the database. + +10. Verifies files after they have been copied. + +11. Records the imported file in the database only after the + copy and verification have succeeded. + +This makes it safe to run the program again later when new photos +and videos have been copied from the phone. + + +DIRECTORY STRUCTURE +=================== + +The normal destination structure looks like this: + +phone-media-backup/ +| ++-- .phone-backup.sqlite ++-- 2024/ +| +-- January/ +| | +-- 03-01-2024/ +| | | +-- Phone-03-01-2024-482193.jpg +| | | +-- Phone-03-01-2024-817204.mp4 +| | | +| | +-- February/ +| | +| +-- December/ +| ++-- 2025/ +| +-- January/ +| | +| +-- ... +| ++-- 2026/ +| ++-- screenshots/ +| ++-- u-mp3/ +| ++-- u-xxx/ + + +MEDIA ORGANISATION +================== + +Images and videos are organised by: + + YEAR + MONTH + DAY-MONTH-YEAR + +For example: + + 2026/ + August/ + 14-08-2026/ + +The month is written as its full English name rather than a +numeric month. + + +FILENAMES +========= + +Imported files are renamed to: + + Phone-DD-MM-YYYY-RANDOM.ext + +For example: + + Phone-14-08-2026-483921.jpg + Phone-14-08-2026-182735.mp4 + +The six-digit number is randomly generated. + +The number is not used as the duplicate identifier. + +The SHA-256 hash of the actual file contents is the authoritative +duplicate identifier. + + +SCREENSHOTS +=========== + +PNG files are treated as screenshots. + +They are placed into: + + screenshots/ + +The original date information is still retained in the database. + +Other supported image formats remain in the normal date-based +directory structure. + + +UNKNOWN FILE TYPES +================== + +Files with formats that are not recognised as normal photo or +video media are placed into a directory based on their extension. + +For example: + + .mp3 + +becomes: + + u-mp3/ + +A file with extension: + + .xyz + +would become: + + u-xyz/ + + +FILES WITHOUT AN EXTENSION +========================== + +Files without a usable extension are handled separately rather +than being silently discarded. + +The exact handling depends on the current version of o-phone.py. + + +DUPLICATE DETECTION +=================== + +Duplicate detection is based on SHA-256. + +For each file being imported, o-phone.py calculates: + + SHA-256(file contents) + +The resulting hash is compared against hashes stored in: + + .phone-backup.sqlite + +If the hash already exists, the file is considered already backed +up and is skipped. + +This means duplicate detection is based on the contents of the +file rather than: + +- filename +- folder name +- file creation date +- file modification date +- device name + +This is important because phones can rename files or give different +metadata to the same media. + + +WHY USE A DATABASE? +=================== + +The SQLite database acts as an index of the backup. + +Instead of repeatedly scanning every existing file on the external +drive, o-phone.py can check the database for the SHA-256 hash. + +Conceptually: + + Source file + | + v + Calculate SHA-256 + | + v + Search SQLite database + | + +---- hash exists ----> SKIP + | + +---- hash missing ---> COPY + | + v + Verify copy + | + v + Add to database + +SQLite is built into Python, so no external database server is +required. + + +IMPORTANT: DATABASE UPDATE ORDER +================================ + +The database should only be updated after a successful copy. + +The intended sequence is: + + 1. Calculate source hash + 2. Check database + 3. If already present, skip + 4. Copy file + 5. Verify destination file + 6. Calculate/verify destination hash + 7. Confirm hashes match + 8. Add file to database + +This prevents a failed or incomplete copy from being recorded as +successfully backed up. + + +RUNNING THE PROGRAM +=================== + +The basic command is: + + python3 o-phone.py SOURCE DESTINATION + +Example: + + python3 o-phone.py \ + "/Users/yourname/Desktop/phone-backup-moto-g13" \ + "/Volumes/all/phone-media-backup" + +SOURCE +------ + +The first argument is the folder containing the files imported from +your phone. + +The source folder can contain subdirectories. + +o-phone.py searches recursively. + +Example: + + /Users/yourname/Desktop/phone-backup-moto-g13/ + + +DESTINATION +----------- + +The second argument is the root folder of the backup. + +For example: + + /Volumes/all/phone-media-backup/ + +The destination can be located on an external hard drive or SSD. + + +RECOMMENDED WORKFLOW +==================== + +When new photos are available on the phone: + +1. Copy the new phone files onto your Mac. + +2. Put them into a source folder. + +3. Run o-phone.py using the same destination folder. + +For example: + + python3 o-phone.py \ + "/Users/yourname/Desktop/phone-backup-moto-g13" \ + "/Volumes/all/phone-media-backup" + +The program will calculate hashes for the source files. + +Files already present in the database will be skipped. + +New files will be copied into the appropriate destination +directories. + +Therefore it is safe to use the same destination repeatedly. + + +DRY RUN +======= + +The program supports a dry-run mode when available in the current +version. + +Dry-run mode allows the proposed changes to be inspected without +actually copying files. + +Example: + + python3 o-phone.py SOURCE DESTINATION --dry-run + +Always check the output before performing a large import. + + +DATABASE +======== + +The SQLite database is stored inside the destination directory. + +Default database: + + .phone-backup.sqlite + +It should be treated as part of the backup. + +If you move the entire backup to another drive, move the database +with it. + +Do not delete or manually modify the database unless you understand +the consequences. + + +DATABASE INTEGRITY +================== + +The database is an index of the files in the backup. + +Over time it is possible for the database and the actual files to +become different. + +For example, this could happen if somebody manually deletes a file +from the backup drive. + +A database verification/check function is therefore intended to +compare: + + SQLite records + against + actual files on disk + +The check should be used periodically, especially after manually +changing files on the backup drive. + + +DO NOT MANUALLY ORGANISE THE BACKUP +=================================== + +Once files have been imported, it is recommended not to manually +rename or move them. + +The database records their paths. + +Moving files outside of o-phone.py can therefore cause the database +and the actual backup to become inconsistent. + +If files need to be reorganised, future versions of the program may +provide dedicated maintenance commands. + + +SOURCE FILES ARE NOT DELETED +============================ + +o-phone.py is intended to be a backup/import tool. + +It should not delete the original files from the source folder as +part of a normal import. + +The source files should remain available until you have independently +confirmed that the backup is complete and healthy. + + +FILE DATES +========== + +The program attempts to determine the most useful date associated +with each media file. + +Depending on the file type, this may come from embedded metadata +or filesystem information. + +The exact date source can vary between: + +- JPEG photos +- HEIC photos +- PNG screenshots +- videos +- RAW camera files +- files copied from different phones +- files whose metadata has been removed + +When metadata is unavailable, the program uses an appropriate +fallback. + + +RAW FILES +========= + +RAW camera files are preserved rather than converted. + +They are treated as source files and copied to the backup. + +The original RAW data is not modified. + + +SUPPORTED MEDIA +=============== + +The exact list of supported extensions is defined in o-phone.py. + +Typical photo formats include: + + .jpg + .jpeg + .heic + .heif + .png + .webp + .tif + .tiff + RAW formats + +Typical video formats include: + + .mp4 + .mov + .m4v + .avi + .mkv + .3gp + +The program may recognise additional formats. + +Unknown formats are not automatically discarded. + + +REQUIREMENTS +============ + +macOS or another operating system with Python 3. + +Python 3.9 or newer is recommended. + +The standard Python libraries used by the program do not require a +separate database server. + +Some optional functionality may require additional Python packages. + + +INSTALLATION +============ + +Check Python: + + python3 --version + +Run the program directly: + + python3 o-phone.py SOURCE DESTINATION + + +SAFETY +====== + +Before using o-phone.py with an important archive: + +1. Test it with a small source folder. + +2. Use --dry-run where available. + +3. Confirm the destination path carefully. + +4. Check the generated directory structure. + +5. Open several copied files. + +6. Confirm the SQLite database exists. + +7. Run the database/filesystem integrity check if available. + +8. Keep another independent backup of important media. + + +IMPORTANT PRINCIPLE +=================== + +The SHA-256 hash is the identity of the file. + +The filename and directory are organisational information. + +For example, these two files: + + IMG_1234.JPG + vacation-photo.jpg + +may represent exactly the same file contents. + +If their SHA-256 hashes match, o-phone.py considers them duplicates. + +Conversely, two files with the same filename are not considered +duplicates if their contents differ. + + +PROJECT PHILOSOPHY +================== + +The goal of o-phone.py is to provide a simple, dependable and +repeatable way to build a personal media archive. + +The important principles are: + + - Do not delete source files. + - Do not overwrite existing backup files unnecessarily. + - Use file contents rather than filenames for duplicate detection. + - Verify files before recording them as successfully backed up. + - Keep a local SQLite index. + - Keep the archive organised by date. + - Make repeated imports safe. + - Prefer simple, inspectable storage over proprietary formats. + + +EXAMPLE +======= + +Suppose the phone contains: + + DCIM/Camera/IMG_1234.jpg + DCIM/Camera/IMG_1235.jpg + DCIM/Camera/VID_001.mp4 + Screenshots/screenshot.png + +Run: + + python3 o-phone.py \ + "/Users/yourname/Desktop/phone-import" \ + "/Volumes/all/phone-media-backup" + +The destination might become: + + phone-media-backup/ + | + +-- .phone-backup.sqlite + | + +-- 2026/ + | | + | +-- August/ + | | + | +-- 14-08-2026/ + | | + | +-- Phone-14-08-2026-483921.jpg + | +-- Phone-14-08-2026-193827.mp4 + | + +-- screenshots/ + | + +-- Phone-14-08-2026-928374.png + +If the same source files are imported again, their hashes will +already exist in the database and they will be skipped. + + +BACKUP REMINDER +=============== + +A single external drive is not a complete backup strategy. + +If the media is important, maintain at least one additional, +independent copy. + +A RAID array, external drive, or database does not by itself protect +against every form of data loss. + +o-phone.py is an organisation and import tool, not a replacement for +a complete backup strategy.