diff --git a/docs/changelog.md b/docs/changelog.md index b63e7902..768be81e 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -7,6 +7,53 @@ title: Changelog !!! note This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html). +## [Version 541](https://github.com/hydrusnetwork/hydrus/releases/tag/v541) + +* misc +* fixed the gallery downloader and thread watcher loading with the 'clear highlight' button enabled despite there being nothing currently highlighted +* to fix the darkmode tooltips on the new Qt 6.5.2 on Windows (the text is stuck on a dark grey, which is unreadable in darkmodes), all the default darkmode styles now have an 'alternate-tooltip-colour' variant, which swaps out the tooltip background colour for the much brighter normal widget text colour +* rewrote the apng parser to work much faster on large files. someone encountered a 200MB giga apng that locked up the client for minutes. now it takes a second or two (unfortunately it looks like that huge apng breaks mpv, but there we go) +* the 'media' options page has two new checkboxes--'hide uninteresting import/modified times'--which allow you to turn off the media viewer behaivour where import and modified times similar to the 'added to my files xxx days ago' are hidden +* reworked the layout of the 'media' options page. everything is in sections now and re-ordered a bit +* the 'other file is a pixel-for-pixel duplicate png!' statements will now only show if the complement is a jpeg, gif, or webp. this statement isn't so appropriate for formats like PSD +* a variety of tricky tags like `:>=` are now searchable in normal autocomplete lookup. a test that determined whether to use a slower but more capable search was misfiring +* the client api key editing window has a new 'check all permissions' button +* fixed the updates I made last week to the missing-master-file-id recovery system. I made a stupid typo and didn't test it properly, fixed now. sorry for the trouble! +* thanks to a user, the help has a bunch of updated screenshots and fixed references to old concepts +* did a little more reformatting and cleanup of 'getting started with downloading' help document and added a short section on note import options +* cleaned up some of the syntax in our various batch files. fingers crossed, the setup_venv.bat script will absolutely retain the trailing space after its questions now, no matter what whitespace my IDE and github want to trim + +### string joiner + +* the parsing system has a new String Processor object--the 'String Joiner'. this is a simple concatenator that takes the list of strings and joins them together. it has two variables: what joining text to use, e.g. ', ', or '-', or empty string '' for simple concatenation; and an optional 'group size', which lets you join every two or three or n strings in 1-2-3, 1-2-3, 1-2-3 style patterns + +### new file types + +* thanks to a user; we now have support for QOI (a png-like lossless image type) and procreate (Apple image project file) files. the former has full support; the latter has thumbnails +* QOI needs Pillow 9.5 at least, so if you are on a super old 'running from source' version, try rebuilding your venv; or cope with you QOI-lessness + +### client api + +* thanks to a user, we now have `/add_tags/get_siblings_and_parents`, which, given a set of tags, shows their sibling and parent display rules for each service +* I wrote some help and unit tests for this +* client api version is now 51 + +### file storage (mostly boring) + +* the file storage system is creaky and ugly to use. I have prepped some longer-term upgrades, mostly by writing new tools and cleaning and reworking existing code. I am nowhere near to done, but I'd like us to have four new features in the nearish future: +* - dynamic-length subfolders (where instead of a fixed set of 256 x00-xff folders, we can bump up to 4096 x000-xfff, and beyond, based on total number of files) +* - setting fixed space limits on particular database locations (e.g. 'no more than 200GB of files here') to complement the current weight system +* - permitting multiple valid locations for a particular subfolder prefix +* - slow per-file background migration between valid subfolders, rather than the giganto folder-atomic program-blocking 'move files now' button in database maintenance +* so, it is pretty boring so far, but I did the following: +* wrote a new class to handle a specific file storage subfolder and spammed it everywhere, replacing previous location and prefix juggling +* wrote some new tools to scan and check the coverage of multiple locations and dynamic-length subfolders +* rewrote the file location database initialisation, storage, testing, updating, and repair to support multiple valid locations +* updated the database to hold 'max num bytes' per file storage location +* the feature to migrate the SQLite database files and then restart is removed from the 'migrate database' dialog. it was always ultrajank in a place that really shouldn't be, and it was completely user-unfriendly. just move things manually, while the client is closed +* the old 'recover and merge surplus database locations into the correct position' side feature in 'move files now' is removed. it was always a little jank, was very rarely actually helpful, and had zero reporting. it will return in the new system as a better one-shot maintenance job +* touched up the migrated database help a little + ## [Version 540](https://github.com/hydrusnetwork/hydrus/releases/tag/v540) ### misc @@ -368,35 +415,3 @@ title: Changelog * the `/get_files/file` command now has a `download=true` parameter which converts the `Content-Disposition` from `inline` (show the file) to `attachment` (auto-download or open save-as dialog) (issue #1375) * added help and a unit test for the above * client api version is now 47 - -## [Version 531](https://github.com/hydrusnetwork/hydrus/releases/tag/v531) - -### misc - -* fixed editing favourite searches, which I accidentally broke last week with the collect-by updates -* when you right-click a tag and get the siblings/parents menus, the list of copyable siblings, parents, and children is now truncated to 10 items each per service. stuff like pokemon has hundreds of children and for a very long time has been spamming giganto 11-column menus that cover the entire screen -* same menu truncation for the open/copy URLs menu. if there's a file that has 600 URLs for interesting technical reasons, it won't nuke you any more (issue #1037) -* updated the default pixiv file page parser, which recently broke for users who were not logged in. they seem to hide original size behind the login now, so if you do a lot of pixiv work, get Hydrus Companion or figure out a cookies.txt solution and get yourself logged in -* the downloader progress panels have a couple of status text improvements: first, they will stop saying 'waiting for a work slot' when the actual error is something unusual such as the gallery search hitting the file limit. second, when there is an unusual status and the downloader is in the paused state, it can now properly differentiate between 'paused' and 'pausing' -* some invalid URL strings now raise the correct error in the downloader system, causing them to be properly filtered away instead of sticking around and being unhelpful -* if there is a connection error because of an SSL issue, the network job is now retried like any other connection error. I originally thought these were all non-retryable like cert validation errors, but it seems some of them are just write timeouts etc.. during the negotiation, so let's see how it goes -* I believe I have fixed an error when selecting a tag in a list when that list had been previously shift-selected and then cleared and repopulated -* manage siblings and parents should be better about focusing the correct text input after they boot and load -* in future, if a taglist tries to deselect something it no longer has, it'll do an emergency 'deselect all' to exorcise the ghosts fully -* reworded the text around 'reset potential duplicates' action in the duplicates page to be more clear on what it does -* I tinkered with some of the shutdown code hoping to catch an odd issue of the exit 'last session' not saving correctly, but I don't think I figured the issue out. if you have noticed you boot up and get a session that missed up to the last 15 minutes of changes before you last shut down, please let me know you your details -* added a link to `tagrank`, a new Client API project at https://github.com/matjojo/tagrank, to the Client API help. it shows you pairs of comparison images over and over and uses `trueskill` ranking algorithm to figure out which tags are your favourite -* added a link to 'Send to Hydrus', a Client API project at https://github.com/Wyrrrd/send-to-hydrus, to the Client API help. it sends URLs from an Android device to your client - -### client api - -* as part of a plan to migrate to service_key indexing everywhere and reduce file_metadata bloat, the client api has a new `services` structure, a service information Object where `service_key` is the key. this is now in the `/get_services` call and `/get_files/file_metadata`, under `services` under the root. the old type-based structure in `/get_services` and the in-file embedding of service info in `/get_files/file_metadata` are still in place, so nothing breaks today, but I am officially declaring them deprecated, to be deleted in 2024, and recommend all Client API devs move to the new system before the new year -* the new service object also includes info on the local rating services. I'd like to add ratings to file_metadata fairly soon -* if you don't want the services object in `/get_files/file_metadata`, there's a new `include_services_object` param you can set to false to hide it -* updated the unit tests and client api help to reflect all this. main new section: https://hydrusnetwork.github.io/hydrus/developer_api.html#services_object -* the client api version is now 46 - -### update woes - -* I somewhat successfully pounded my head against an issue where the first tab (usually 'my tags') was disappearing in the _manage tags/siblings/parents_ dialogs for some users. this bug, for real, seems to be the combination of (Python 3.11 + PyQt6 6.5.x + two tabs + total tab text characters > ~12 + tab selection is set to 1 during init event). Change any of those things and it doesn't happen. This is so weird a problem to otherwise normal code that I won't pivot all my 50-odd instances of tab selection to handle it and instead have hacked an answer for the three tag dialogs and filename tagging. Sorry for the trouble if you got this! Let me know if you see any more -* in a similar-but-different thing, PySide6 6.5.1 has a bug related to certain Signal connections. don't use it with hydrus, it messes up all my menus! their dev notes suggest they are going to have a fix/revert for 6.5.1.1 diff --git a/docs/database_migration.md b/docs/database_migration.md index 095f6dbc..cad30ea2 100644 --- a/docs/database_migration.md +++ b/docs/database_migration.md @@ -14,7 +14,7 @@ A hydrus client consists of three components: It doesn't really matter where you put this. An SSD will load it marginally quicker the first time, but you probably won't notice. If you run it without command-line parameters, it will try to write to its own directory (to create the initial database), so if you mean to run it like that, it should not be in a protected place like _Program Files_. -2. **the actual database** +2. **the actual SQLite database** The client stores all its preferences and current state and knowledge _about_ files--like file size and resolution, tags, ratings, inbox status, and so on and so on--in a handful of SQLite database files, defaulting to _install_dir/db_. Depending on the size of your client, these might total 1MB in size or be as much as 10GB. @@ -29,7 +29,7 @@ A hydrus client consists of three components: ## these components can be put on different drives { id="different_drives" } -Although an initial install will keep these parts together, it is possible to, say, run the database on a fast drive but keep your media in cheap slow storage. This is an excellent arrangement that works for many users. And if you have a very large collection, you can even spread your files across multiple drives. It is not very technically difficult, but I do not recommend it for new users. +Although an initial install will keep these parts together, it is possible to, say, run the SQLite database on a fast drive but keep your media in cheap slow storage. This is an excellent arrangement that works for many users. And if you have a very large collection, you can even spread your files across multiple drives. It is not very technically difficult, but I do not recommend it for new users. Backing such an arrangement up is obviously more complicated, and the internal client backup is not sophisticated enough to capture everything, so I recommend you figure out a broader solution with a third-party backup program like FreeFileSync. @@ -44,8 +44,6 @@ Go _database->migrate database_, giving you this dialog: ![](images/db_migration.png) -This is an image from my old laptop's client. At that time, I had moved the main database and its files out of the install directory but otherwise kept everything together. Your situation may be simpler or more complicated. - To move your files somewhere else, add the new location, empty/remove the old location, and then click 'move files now'. **Portable** means that the path is beneath the main db dir and so is stored as a relative path. Portable paths will still function if the database changes location between boots (for instance, if you run the client from a USB drive and it mounts under a different location). @@ -54,13 +52,11 @@ To move your files somewhere else, add the new location, empty/remove the old lo The operations on this dialog are simple and atomic--at no point is your db ever invalid. Once you have the locations and ideal usage set how you like, hit the 'move files now' button to actually shuffle your files around. It will take some time to finish, but you can pause and resume it later if the job is large or you want to undo or alter something. -If you decide to move your actual database, the program will have to shut down first. Before you boot up again, you will have to create a new program shortcut: +## informing the software that the SQLite database is not in the default location { id="launch_parameter" } -## informing the software that the database is not in the default location { id="launch_parameter" } +A straight call to the hydrus_client executable will look for a SQLite database in _install_dir/db_. If one is not found, it will create one. If you move your database and then try to run the client again, it will try to create a new empty database in that old location! -A straight call to the hydrus_client executable will look for a database in _install_dir/db_. If one is not found, it will create one. So, if you move your database and then try to run the client again, it will try to create a new empty database in the previous location! - -So, pass it a -d or --db_dir command line argument, like so: +To tell it about the new database location, pass it a `-d` or `--db_dir` command line argument, like so: * `hydrus_client -d="D:\media\my_hydrus_database"` * _--or--_ @@ -70,11 +66,11 @@ So, pass it a -d or --db_dir command line argument, like so: And it will instead use the given path. If no database is found, it will similarly create a new empty one at that location. You can use any path that is valid in your system, but I would not advise using network locations and so on, as the database works best with some clever device locking calls these interfaces may not provide. -Rather than typing the path out in a terminal every time you want to launch your external database, create a new shortcut with the argument in. Something like this, which is from my main development computer and tests that a fresh default install will run an existing database ok: +Rather than typing the path out in a terminal every time you want to launch your external database, create a new shortcut with the argument in. Something like this: ![](images/db_migration_shortcut.png) -Note that an install with an 'external' database no longer needs access to write to its own path, so you can store it anywhere you like, including protected read-only locations (e.g. in 'Program Files'). If you do move it, just double-check your shortcuts are still good and you are done. +Note that an install with an 'external' database no longer needs access to write to its own path, so you can store it anywhere you like, including protected read-only locations (e.g. in 'Program Files'). Just double-check your shortcuts are good. ## finally { id="finally" } @@ -90,22 +86,18 @@ As an example, let's say you started using the hydrus client on your HDD, and no Specifically: -* Update your backup if you maintain one. -* Create an empty folder on your HDD that is outside of your current install folder. Call it 'hydrus_files' or similar. -* Create two empty folders on your SSD with names like 'hydrus\_db' and 'hydrus\_thumbnails'. - -* Set the 'thumbnail location override' to 'hydrus_thumbnails'. You should get that new location in the list, currently empty but prepared to take all your thumbs. -* Hit 'move files now' to actually move the thumbnails. Since this involves moving a lot of individual files from a high-latency source, it will take a long time to finish. The hydrus client may hang periodically as it works, but you can just leave it to work on its own--it will get there in the end. You can also watch it do its disk work under Task Manager. - -* Now hit 'add location' and select your new 'hydrus\_files'. 'hydrus\_files' should be added and willing to take 50% of the files. -* Select the old location (probably 'install\_dir/db/client\_files') and hit 'decrease weight' until it has weight 0 and you are prompted to remove it completely. 'hydrus_files' should now be willing to take all the files from the old location. -* Hit 'move files now' again to make this happen. This should be fast since it is just moving a bunch of folders across the same partition. - -* With everything now 'non-portable' and hence decoupled from the db, you can now easily migrate the install and db to 'hydrus_db' simply by shutting the client down and moving the install folder in a file explorer. -* Update your shortcut to the new hydrus_client.exe location and try to boot. - -* Update your backup scheme to match your new locations. -* Enjoy a much faster client. +1. Update your backup if you maintain one. +* Create an empty folder on your HDD that is outside of your current install folder. Call it 'hydrus_files' or similar. +* Create two empty folders on your SSD with names like 'hydrus\_db' and 'hydrus\_thumbnails'. +* Set the 'thumbnail location override' to 'hydrus_thumbnails'. You should get that new location in the list, currently empty but prepared to take all your thumbs. +* Hit 'move files now' to actually move the thumbnails. Since this involves moving a lot of individual files from a high-latency source, it will take a long time to finish. The hydrus client may hang periodically as it works, but you can just leave it to work on its own--it will get there in the end. You can also watch it do its disk work under Task Manager. +* Now hit 'add location' and select your new 'hydrus\_files'. 'hydrus\_files' should be added and willing to take 50% of the files. +* Select the old location (probably 'install\_dir/db/client\_files') and hit 'decrease weight' until it has weight 0 and you are prompted to remove it completely. 'hydrus_files' should now be willing to take all the files from the old location. +* Hit 'move files now' again to make this happen. This should be fast since it is just moving a bunch of folders across the same partition. +* With everything now 'non-portable' and hence decoupled from the db, you can now easily migrate the install and db to 'hydrus_db' simply by shutting the client down and moving the install folder in a file explorer. +* Update your shortcut to the new hydrus_client.exe location and try to boot. +* Update your backup scheme to match your new locations. +* Enjoy a much faster client. You should now have _something_ like this: @@ -113,4 +105,4 @@ You should now have _something_ like this: ## p.s. running multiple clients { id="multiple_clients" } -Since you now know how to tell the software about an external database, you can, if you like, run multiple clients from the same install (and if you previously had multiple install folders, now you can now just use the one). Just make multiple shortcuts to the same hydrus_client executable but with different database directories. They can run at the same time. You'll save yourself a little memory and update-hassle. I do this on my laptop client to run a regular client for my media and a separate 'admin' client to do PTR petitions and so on. +Since you now know how to tell the software about an external database, you can, if you like, run multiple clients from the same install (and if you previously had multiple install folders, now you can now just use the one). Just make multiple shortcuts to the same hydrus_client executable but with different database directories. They can run at the same time. You'll save yourself a little memory and update-hassle. diff --git a/docs/developer_api.md b/docs/developer_api.md index 78057ee6..babb1ee7 100644 --- a/docs/developer_api.md +++ b/docs/developer_api.md @@ -852,9 +852,9 @@ _Ask the client about how it will see certain tags._ Restricted access: : YES. Add Tags permission needed. - + Required Headers: n/a - + Arguments (in percent-encoded JSON): : * `tags`: (a list of the tags you want cleaned) @@ -876,12 +876,108 @@ Response: Mostly, hydrus simply trims excess whitespace, but the other examples are rare issues you might run into. 'system' is an invalid namespace, tags cannot be prefixed with hyphens, and any tag starting with ':' is secretly dealt with internally as "\[no namespace\]:\[colon-prefixed-subtag\]". Again, you probably won't run into these, but if you see a mismatch somewhere and want to figure it out, or just want to sort some numbered tags, you might like to try this. +### **GET `/add_tags/get_siblings_and_parents`** { id="add_tags_get_siblings_and_parents" } + +_Ask the client about tags' sibling and parent relationships._ + +Restricted access: +: YES. Add Tags permission needed. + +Required Headers: n/a + +Arguments (in percent-encoded JSON): +: +* `tags`: (a list of the tags you want info on) + +Example request: +: Given tags `#!json [ "blue eyes", "samus aran" ]`: + ``` + /add_tags/get_siblings_and_parents?tags=%5B%22blue%20eyes%22%2C%20%22samus%20aran%22%5D + ``` + +Response: +: An Object showing all the display relationships for each tag on each service. Also [The Services Object](#services_object). +```json title="Example response" +{ + "services" : "The Services Object" + "tags" : { + "blue eyes" : { + "6c6f63616c2074616773" : { + "ideal_tag" : "blue eyes", + "siblings" : [ + "blue eyes", + "blue_eyes", + "blue eye", + "blue_eye" + ], + "descendants" : [], + "ancestors" : [] + }, + "877bfcf81f56e7e3e4bc3f8d8669f92290c140ba0acfd6c7771c5e1dc7be62d7": { + "ideal_tag" : "blue eyes", + "siblings" : [ + "blue eyes" + ], + "descendants" : [], + "ancestors" : [] + } + }, + "samus aran" : { + "6c6f63616c2074616773" : { + "ideal_tag" : "character:samus aran", + "siblings" : [ + "samus aran", + "samus_aran", + "character:samus aran" + ], + "descendants" : [ + "character:samus aran (zero suit)" + "cosplay:samus aran" + ], + "ancestors" : [ + "series:metroid", + "studio:nintendo" + ] + }, + "877bfcf81f56e7e3e4bc3f8d8669f92290c140ba0acfd6c7771c5e1dc7be62d7": { + "ideal_tag" : "samus aran", + "siblings" : [ + "samus aran" + ], + "descendants" : [ + "zero suit samus", + "samus_aran_(cosplay)" + ], + "ancestors" : [] + } + } + } +} +``` + + This data is essentially how mappings in the `storage` `tag_display_type` become `display`. + + The hex keys are the service keys, which you will have seen elsewhere, like [GET /get\_files/file\_metadata](#get_files_file_metadata). Note that there is no concept of 'all known tags' here. If a tag is in 'my tags', it follows the rules of 'my tags', and then all the services' display tags are merged into the 'all known tags' pool for user display. + + **Also, the siblings and parents here are not just what is in _tags->manage tag siblings/parents_, they are the final computed combination of rules as set in _tags->manage where tag siblings and parents apply_.** The data given here is not guaranteed to be useful for editing siblings and parents on a particular service. That data, which is currently pair-based, will appear in a different API request in future. + + - `ideal_tag` is how the tag appears in normal display to the user. + - `siblings` is every tag that will show as the `ideal_tag`, including the `ideal_tag` itself. + - `descendants` is every child (and recursive grandchild, great-grandchild...) that implies the `ideal_tag`. + - `ancestors` is every parent (and recursive grandparent, great-grandparent...) that our tag implies. + + Every descendant and ancestor is an `ideal_tag` itself that may have its own siblings. + + Most situations are simple, but remember that siblings and parents in hydrus can get complex. If you want to display this data, I recommend you plan to support simple service-specific workflows, and add hooks to recognise conflicts and other difficulty and, when that happens, abandon ship (send the user back to Hydrus proper). Also, if you show summaries of the data anywhere, make sure you add a 'and 22 more...' overflow mechanism to your menus, since if you hit up 'azur lane' or 'pokemon', you are going to get hundreds of children. + + I generally warn you off computing sibling and parent mappings or counts yourself. The data from this request is best used for sibling and parent decorators on individual tags in a 'manage tags' presentation. The code that actually computes what siblings and parents look like in the 'display' context can be a pain at times, and I've already done it. Just run /search_tags or /file_metadata again after any changes you make and you'll get updated values. + ### **GET `/add_tags/search_tags`** { id="add_tags_search_tags" } _Search the client for tags._ Restricted access: -: YES. Search for Files permission needed. +: YES. Search for Files and Add Tags permission needed. Required Headers: n/a diff --git a/docs/getting_started_downloading.md b/docs/getting_started_downloading.md index 10ac03c9..c3b69c9b 100644 --- a/docs/getting_started_downloading.md +++ b/docs/getting_started_downloading.md @@ -14,7 +14,9 @@ The downloader is highly parallelisable, and while the default [bandwidth rules] It also takes a decent whack of CPU to import a file. You'll usually never notice this with just one hard drive import going, but if you have twenty different download queues all competing for database access and individual 0.1-second hits of heavy CPU work, you will discover your client starts to judder and lag. Keep it in mind, and you'll figure out what your computer is happy with. I also recommend you try to keep your total loaded files/urls to be under 20,000 to keep things snappy. Remember that you can pause your import queues, if you need to calm things down a bit. ## Downloader types -There are a number of different downloader types, each with its own purpose. This is a short summary of them: + +There are a number of different downloader types, each with its own purpose: + **URL download** : Intended for single posts or images. (Works with the [API](client_api.md)) @@ -30,54 +32,15 @@ There are a number of different downloader types, each with its own purpose. Thi **Simple downloader** : Intended for simple one-off jobs like grabbing all linked images in a page. -# Import options -In previous versions these were split into completely different windows called `file import options` and `tag import options` so if you see those anywhere, this is what they're talking about and not some hidden menu anywhere. - -## File import settings -File import settings has a number of options that deal with the files being downloaded and what should happen to them. There's a few more tickboxes if you turn on advanced mode. - -![](images/file_import.png) - -**pre-import checks** -: Pretty self-explanatory for the most part. If you want to redownload previously deleted files turning off `exclude previously deleted files` will have Hydrus ignore deletion status. -A few of the options have more information if you hover over them. - -**import destinations** -: See [multiple file services](advanced_multiple_local_file_services.md), an advanced feature. - -**post import actions** -: See the [files section on filtering](getting_started_files.md#inbox-and-archive) for the first option, the other two have information if you hover over them. - -## Parsing -By default, hydrus now starts with a local tag service called 'downloader tags' and it will parse (get) all the tags from normal gallery sites and put them in this service. You don't have to do anything, you will get some decent tags. As you use the client, you will figure out which tags you like and where you want them. On the downloader page, click `import options`: - -![](images/tag_import_options_default.png) - -This is an important dialog, although you will not need to use it much. It governs which tags are parsed and where they go. To keep things easy to manage, a new downloader will refer to the 'default' tag import options for a website, but for now let's set some values just for this downloader: - -![](images/tag_import_options_specific.png) - -You can see that each tag service on your client has a separate section. If you add the PTR, that will get a new box too. A new client is set to _get all tags_ for 'downloader tags' service. Things can get much more complicated. Have a play around with the options here as you figure things out. Most of the controls have tooltips or longer explainers in sub-dialogs, so don't be afraid to try things. - -It is easy to get tens of thousands of tags by downloading this way. Different sites offer different kinds and qualities of tags, and the client's downloaders (which were designed by me, the dev, or a user) may parse all or only some of them. Many users like to just get everything on offer, but others only ever want, say, `creator`, `series`, and `character` tags. If you feel brave, click that 'all tags' button, which will take you into hydrus's advanced 'tag filter', which allows you to select which of the incoming list of tags will be added. - -The blacklist button will let you skip downloading files that have certain tags (perhaps you would like to auto-skip all images with `gore`, `scat`, or `diaper`?), again using the tag filter, while the whitelist enables you to only allow files that have at least one of a set of tags. The 'additional tags' adds some fixed personal tags to all files coming in--for instance, you might like to add 'process into favourites' to your 'my tags' for some query you really like so you can find those files again later and process them separately. That little 'cog' icon button can also do some advanced things. - -To edit the defaults, hit up _network->downloaders->manage default tag import options_. You should do this as you get a better idea of your preferences. You can set them for all file posts generally, all watchers, and for specific sites as well. - - -!!! warning - The file limit and file/tag import options on the upper panel, if changed, will only apply to **new** queries. If you want to change the options for an existing queue, either do so on its highlight panel below or use the 'set options to queries' button. - -## URL download +### URL download The **url downloader** works like the gallery downloader but does not do searches. You can paste downloadable URLs to it, and it will work through them as one list. Dragging and dropping recognisable URLs onto the client (e.g. from your web browser) will also spawn and use this downloader. The button next to the input field lets you paste multiple URLs at once such as if you've copied from a document or browser bookmarks. The URLs need to be newline separated. -### API +#### API If you use [API-connected](client_api.md) programs such as the Hydrus Companion, then any [non-watchable](downloader_url_classes.md#url_types) URLs sent to Hydrus through them will end up in an URL downloader page, the specifics depending on the program's settings. You can't use this to force Hydrus to download paged galleries since the URL downloader page doesn't support traversing to the next page, use the gallery downloader for this. -## Gallery download +### Gallery download ![](images/downloader_page.png) The gallery page can download from multiple sources at the same time. Each entry in the list represents a basic combination of two things: @@ -100,7 +63,7 @@ _Note that some sites only serve 25 or 50 pages of results, despite their indice **In general, particularly when starting out, artist searches are best.** They are usually fewer than a thousand files and have fairly uniform quality throughout. -## Subscriptions { id="subscriptions" } +### Subscriptions { id="subscriptions" } Let's say you found an artist you like. You downloaded everything of theirs from some site, but every week, one or two new pieces is posted. You'd like to keep up with the new stuff, but you don't want to manually make a new download job every week for every single artist you like. Subscriptions are a way to automatically recheck a good query in future, to keep up with new files. Many users come to use them. You set up a number of saved queries, and the client will 'sync' with the latest files in the gallery and download anything new, just as if you were running the download yourself. @@ -112,7 +75,7 @@ Subscriptions only work for booru-like galleries that put the newest files first It is important to note that while subscriptions can have multiple queries (even hundreds!), they _generally_ only work on one site. Expect to create one subscription for safebooru, one for artstation, one for paheal, and so on for every site you care about. Advanced users may be able to think of ways to get around this, but I recommend against it as it throws off some of the internal check timing calculations. -### Setting up subscriptions +#### Setting up subscriptions Here's the dialog, which is under _network->manage subscriptions_: @@ -136,7 +99,7 @@ Despite all the controls, the basic idea is simple: Up top, I have selected the You might want to put subscriptions off until you are more comfortable with galleries. There is more help [here](getting_started_subscriptions.md). -## Watchers +### Watchers If you are an imageboard user, try going to a thread you like and drag-and-drop its URL (straight from your web browser's address bar) onto the hydrus client. It should open up a new 'watcher' page and import the thread's files! ![](images/watcher_page.png) @@ -145,12 +108,60 @@ With only one URL to check, watchers are a little simpler than gallery searches, In general, you can leave the checker options alone, but you might like to revisit them if you are always visiting faster or slower boards and find you are missing files or getting DEAD too early. -### API +#### API If you use [API-connected](client_api.md) programs such as the Hydrus Companion, then any [watchable](downloader_url_classes.md#url_types) URLs sent to Hydrus through them will end up in a watcher page, the specifics depending on the program's settings. -## Simple downloader +### Simple downloader The **simple downloader** will do very simple parsing for unusual jobs. If you want to download all the images in a page, or all the image link destinations, this is the one to use. There are several default parsing rules to choose from, and if you learn the downloader system yourself, it will be easy to make more. +## Import options +Every importer in Hydrus has some 'import options' that change what is allowed, what is blacklisted, and whether tags or notes should be saved. + +In previous versions these were split into completely different windows called `file import options` and `tag import options` so if you see those anywhere, this is what they're talking about and not some hidden menu anywhere. + +Importers that download from websites rely on a flexible 'defaults' system, so you do not have to set them up every time you start a new downloader. While you should play around with your import options, once you know what works for you, you should set that as the default under _network->downloaders->manage default import options_. You can set them for all file posts generally, all watchers, and for specific sites as well. + +### File import options +This deals with the files being downloaded and what should happen to them. There's a few more tickboxes if you turn on advanced mode. + +![](images/file_import.png) + +**pre-import checks** +: Pretty self-explanatory for the most part. If you want to redownload previously deleted files turning off `exclude previously deleted files` will have Hydrus ignore deletion status. +A few of the options have more information if you hover over them. + +**import destinations** +: See [multiple file services](advanced_multiple_local_file_services.md), an advanced feature. + +**post import actions** +: See the [files section on filtering](getting_started_files.md#inbox-and-archive) for the first option, the other two have information if you hover over them. + +### Tag Parsing +By default, hydrus now starts with a local tag service called 'downloader tags' and it will parse (get) all the tags from normal gallery sites and put them in this service. You don't have to do anything, you will get some decent tags. As you use the client, you will figure out which tags you like and where you want them. On the downloader page, click `import options`: + +![](images/tag_import_options_default.png) + +This is an important dialog, although you will not need to use it much. It governs which tags are parsed and where they go. To keep things easy to manage, a new downloader will refer to the 'default' tag import options for a website, but for now let's set some values just for this downloader: + +![](images/tag_import_options_specific.png) + +You can see that each tag service on your client has a separate section. If you add the PTR, that will get a new box too. A new client is set to _get all tags_ for 'downloader tags' service. Things can get much more complicated. Have a play around with the options here as you figure things out. Most of the controls have tooltips or longer explainers in sub-dialogs, so don't be afraid to try things. + +It is easy to get tens of thousands of tags by downloading this way. Different sites offer different kinds and qualities of tags, and the client's downloaders (which were designed by me, the dev, or a user) may parse all or only some of them. Many users like to just get everything on offer, but others only ever want, say, `creator`, `series`, and `character` tags. If you feel brave, click that 'all tags' button, which will take you into hydrus's advanced 'tag filter', which allows you to select which of the incoming list of tags will be added. + +The blacklist button will let you skip downloading files that have certain tags (perhaps you would like to auto-skip all images with `gore`, `scat`, or `diaper`?), again using the tag filter, while the whitelist enables you to only allow files that have at least one of a set of tags. The 'additional tags' adds some fixed personal tags to all files coming in--for instance, you might like to add 'process into favourites' to your 'my tags' for some query you really like so you can find those files again later and process them separately. That little 'cog' icon button can also do some advanced things. + +!!! warning + The file limit and import options on the upper panel of a gallery or watcher page, if changed, will only apply to **new** queries. If you want to change the options for an existing queue, either do so on its highlight panel below or use the 'set options to queries' button. + +### Note Parsing + +Hydrus alsos parse 'notes' from some sites. This is a young feature, and a little advanced at times, but it generally means the comments that artists leave on certain gallery sites, or something like a tweet text. Notes are editable by you and appear in a hovering window on the right side of the media viewer. + +![](images/note_import_options_normal.png) + +Most of the controls here ensure that successive parses do not duplicate existing notes. The default settings are fine for all normal purposes, and you can leave them alone unless you know you want something special (e.g. turning note parsing off completely). + ## Bandwidth It will not be too long until you see a "bandwidth free in xxxxx..." message. As a long-term storage solution, hydrus is designed to be polite in its downloading--both to the source server and your computer. The client's default bandwidth rules have some caps to stop big mistakes, spread out larger jobs, and at a bare minimum, no domain will be hit more than once a second. diff --git a/docs/images/db_migration.png b/docs/images/db_migration.png index 55ddca52..eaa0c7aa 100644 Binary files a/docs/images/db_migration.png and b/docs/images/db_migration.png differ diff --git a/docs/images/db_migration_example.png b/docs/images/db_migration_example.png index 01349eeb..ba118046 100644 Binary files a/docs/images/db_migration_example.png and b/docs/images/db_migration_example.png differ diff --git a/docs/images/note_import_options_normal.png b/docs/images/note_import_options_normal.png new file mode 100644 index 00000000..ce9f333b Binary files /dev/null and b/docs/images/note_import_options_normal.png differ diff --git a/docs/old_changelog.html b/docs/old_changelog.html index c5e28526..097c63de 100644 --- a/docs/old_changelog.html +++ b/docs/old_changelog.html @@ -34,6 +34,47 @@

changelog