hydrus/db/help my media files are broke.txt

*** The purpose of this document ***

If you have missing files or missing file records, either from files being deleted or having to restore a database file out of sync with its file storage, this document will walk you through the specific steps on how to resync. If you have encountered a related problem, such as losing entire 'fxx' sub-folder(s), it may also help as background reading.


*** Missing Files ***

If you have lost lots of media files due to a mistake or a drive failing, or if you have rolled back to an older backup and now your file structure is out of sync with your database record, there are several things to do.

First off, make sure you are running on good hard drives now. Check the 'help my db is broke' help document for info here. If your .db files were also on a failing disk, you'll want to run through that whole document to make sure your database itself is ok--there's no point trying to fix the file record on a malformed database.

Next, boot into the client. If you lost whole media file directories, or they moved to a new location, your client will throw up a repair dialog during boot to help you find the right locations again. If you have truly lost some directories, it should be able to recreate empty ones for you.

FYI: In hydrus's file storage, there are two sets of 256 sub folders. One is 'fxx' and one is 'txx', where 'f' is for files, 't' is for thumbnails, and 'xx' counts from 00 to ff in hexadecimal. You need all 512 directories located to boot.

Once you are booted, your client may throw up lots of missing thumbnail/file errors. Do not worry so much about these for now, but it would be best to close your larger pages and pause the import pages for now. Don't pause network traffic, since we'll be using that in a minute.

If you needed to remap your file locations during pre-boot repair, then hit _database->migrate database_. It may still have your old damaged location set as the 'ideal', and the remapped correct locations set as just a temporary location it currently wants to move away from. Correct this by giving the new locations some weight and removing the old locations. Once you are done, you should only see good locations listed, no desire to move anything, with 'current' and 'ideal' usage at the same percentage.

Next, we have two classes of problem: files in the database but not in storage, and files in storage but not in the database.


*** Files that the database thinks are in storage, but they are not ***

These are the cause of the missing file errors. They could have been deleted or damaged, or they might have been imported after you made your file storage backup. The are not on disk any more, but the database doesn't know that, so any time it tries to access where they _were_, it is surprised and throws the error. We need to let it fix all those records.

Hit _database->file maintenance->manage scheduled jobs_, and then hit the 'add new jobs' tab. Click the 'all media files' button, or just run a search for 'system:everything', and then queue up the job that says 'if file is missing and has URL, try to download, else remove record'. It should be the default job. There are other jobs--feel free to read their descriptions--but you want to deal with the 'missing' suite for now.

Queue up that job for everything and then on the first tab you can hurry all that work along. It may take an hour or more to check all your files. Any of the files with URLs (i.e. anything you once downloaded) will be queued up in a new download page that may get pretty large, so you might like to pause file maintenance under the same _database->file maintenance_ menu for a bit if it gets too big.

With luck, it will have redownloaded many of your missing files. Those it could not will no longer be listed in 'my files' (as if they were deleted) and will not cause 'missing file' errors. All file hashes and any known URLs will be written to your log file and a new folder in your database directory. If you have a manual way to find what could not be redownloaded, you can try re-importing them in future, but otherwise this is the best answer for now. Even though it sucks to lose files, all you can do now is let the database know about it too.

Once the file maintenance is done, and the download page is done, we then need to run 'if file is missing, remove record' on everything once more to deal with the files that had a URL but could not be downloaded successfully.

You do not need to run the 'regenerate thumbnail' jobs--the client does that automatically as needed when you load a file, so it will fill in gaps no problem--but if you lost whole thumbnail folders and some or all are now empty, running this job will speed up future load by doing that work in the background now.


*** Files that are in storage, but the database does not know it ***

~If you only had drive errors or you restored a backup that included both file storage and database files made at the same time, you can ignore this step. You probably have a couple of extra orphan files, but everyone does, it isn't a big problem.~

If you restored an older file storage backup to a newer database, these would be files that were deleted after the backup was made. If you restored an older database backup to a newer file storage, then these would be files that were imported after the backup was made. In either case, they are files in your file structure that the database does not know about, and we want to collect them together to A) delete them or B) reimport them.

Run _database->db maintenance->clear orphan files_. Choose a location for the files to go to, and then wait for it to finish. Browse through them to verify what you are looking at, and then either delete them or reimport them.


*** Repository Update Files ***

If you had missing files, that may include some update files for the PTR or another repo, which are stored outside the 'system:everything' search, but are still in your file storage. Don't worry about it too much--the next time repo processing happens, if there is a missing or damaged file, it will notice and run an automatic maintenance routine to fix things. It all happens through the same review file maintenance panel if you want to watch it. Once it has cleared records for the missing files, you can resume repo processing and it will redownload the missing files automatically and resume.

Note these files do not have URLs, so if you are an advanced user and try to run this maintenance yourself with the 'and has URL' qualifier, they will not load up in a download page.


*** OK ***

You should now be done! If you get weird file counts on autocomplete results or more missing file errors, let me know.

I recommend you take a breath, pour yourself a drink, and make a job for tomorrow to think about your backup routine.