diff --git a/db/extract_options.sql b/db/extract_options.sql deleted file mode 100644 index b1348a18..00000000 --- a/db/extract_options.sql +++ /dev/null @@ -1,9 +0,0 @@ -.open client.db -.out my_options.sql -.print .open client.db\r\n -.print delete from options;\r\n -.print delete from json_dumps where dump_type = 22;\r\n -.mode insert options -select * from options; -.mode insert json_dumps -select * from json_dumps where dump_type = 22; \ No newline at end of file diff --git a/db/extract_subscriptions.sql b/db/extract_subscriptions.sql deleted file mode 100644 index 87ff6c45..00000000 --- a/db/extract_subscriptions.sql +++ /dev/null @@ -1,6 +0,0 @@ -.print The subscriptions will lose their tag import options, so make sure to check them once they are imported back in. -.open client.db -.out my_subscriptions.sql -.print .open client.db\r\n -.mode insert json_dumps_named -select * from json_dumps_named where dump_type = 3; \ No newline at end of file diff --git a/db/extract_version.bat b/db/extract_version.bat new file mode 100644 index 00000000..947fca16 --- /dev/null +++ b/db/extract_version.bat @@ -0,0 +1,3 @@ +@ECHO off +sqlite3 < extract_version.sql +SET /P gumpf=Hit Enter to exit! \ No newline at end of file diff --git a/db/extract_version.sql b/db/extract_version.sql new file mode 100644 index 00000000..5c3802df --- /dev/null +++ b/db/extract_version.sql @@ -0,0 +1,2 @@ +.open --readonly client.db +SELECT "This database is version " || version FROM version; \ No newline at end of file diff --git a/db/using the extract scripts.txt b/db/using the extract scripts.txt deleted file mode 100644 index 2cef5d53..00000000 --- a/db/using the extract scripts.txt +++ /dev/null @@ -1,9 +0,0 @@ -If your db is completely broken and you need to extract some important data, please check out the emergency extract scripts. To use them, put your old database, the sqlite3 executable, and the script in the same folder and feed the script into sqlite3, like so: - -sqlite3 < extract_subscriptions.sql - -This will connect to the database and copy your subscriptions to the new file my_subscriptions.sql, which you can then move and import to a new db folder in the same way: - -sqlite3 < my_subscriptions.sql - -Some things are difficult to copy over at this basic level. Your tag options and anything else service-specific will be lost or reset back to default. \ No newline at end of file diff --git a/docs/changelog.md b/docs/changelog.md index a8ab4c24..bb9a5a45 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -3,6 +3,49 @@ !!! note This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html). +## [Version 479](https://github.com/hydrusnetwork/hydrus/releases/tag/v479) + +### misc +* when shift-selecting some thumbnails, you can now reverse the direction of the select and what you just selected will be deselected, basically a full undo (issue #1105) +* when ctrl-selecting thumbnails, if you add to the selection, the file you click is now focused and always previewed (previously this only happened if there was no focused file already). this is related to the shift-select logic above, but it may be annoying when making a big ctrl-selection of videos etc.. so let me know and I can make this more clever if needed +* added file sort 'file->hash', which sorts pseudorandomly but repeatably. it sounds not super clever, but it will be useful for certain comparison operations across clients +* when you hit 'copy->hash' on a file right-click, it now shows the sha256 hash for quick review +* in the duplicate filter, the zoom locking tech now works better™ when one of the pair is portrait and the other landscape. it now tries to select either width or height to lock both when going AB and BA. it also chooses the 'better' of width or height by choosing the zoom that'll change the size less radically. previously, it could do width on AB and height on BA, which lead to a variety of odd situations. there are probably still some issues here, most likely when one of the files almost exactly fills the whole canvas, so let me know how you get on +* webps with transparency should now load correct! previously they were going crazy in the transparent area. all webps are scheduled a thumbnail regen this week +* when import folders run, the count on their progress bar now ignores previous failed and ignored entries. it should always start 0, like 0/100, rather than 20/120 etc... +* when import folders run, any imports where the status type is set to 'leave the file alone' is now still scanned at the end of a job. if the path does not exist any more, it is removed from the import list +* fixed a typo bug in the recent delete code cleanup that meant 'delete files after export' after a manual export was only working on the last file in the selection. sorry for the trouble! +* the delete files dialog now starts with keyboard focus on the action radiobox (it was defaulting to ok button since I added the recent panel disable tech) +* if a network job has a connection error or serverside bandwidth block and then waits before retrying, it now checks if all network jobs have just been paused and will not reattempt the connection if so (issue #1095) +* fixed a bug in thumbnail fallback rendering +* fixed another problem with cloudscraper's new method names. it should work for users still on an old version +* wrote a little 'extract version' sql and bat file for the db folder that simply pull the version from the client.db file in the same directory. I removed the extract options/subscriptions sql scripts since they are super old and out of date, but this general system may return in future + +### file history chart +* added 'archive' line to the file history chart. this isn't exactly (current_count - inbox_count), but it pretty much is +* added a 'show deleted' checkbox to the file history chart. it will recalculate the y axis range on click, so if you have loads of deleted files, you can now hide them to see current better +* improved the way data is aggregated in the file history chart. diagonal lines should be reduced during any periods of client import-inactivity, and spikes should show better +* also bumped the number of steps up to 8,000, so it should look nice maximised on a 4k +* the file history chart now remembers its last size and position--it has an entry under options->gui + +### client api +* thanks to a user, the Client API now accepts any file_id, file_ids, hash, or hashes as arguments in any place where you need to specify a file or files +* like 'return_hashes', the 'search_files' command in the Client API now takes an optional 'return_file_ids' parameter, default true, to turn off the file ids if you only want hashes +* added 'only_return_basic_information' parameter, default false, to 'get_metadata' call, which is fast for first-time requests (it is slim but not well cached) and just delivers the basics like resolution and file size +* added unit tests and updated the help to reflect the above +* client api version is now 29 + +### help +* split up the 'more files' help section into 'powerful searching' and 'exporting files', both still under the 'next steps' section +* moved the semi-advanced 'OR' section from 'tags' to 'searching' +* brushed up misc help +* a couple of users added some misc help updates too, thank you! + +### misc boring cleanup +* cleaned up an old wx label patch +* cleaned up an old wx system colour patch +* cleaned up some misc initialisation code + ## [Version 478](https://github.com/hydrusnetwork/hydrus/releases/tag/v478) ### misc diff --git a/docs/developer_api.md b/docs/developer_api.md index f509a0ae..986be2ea 100644 --- a/docs/developer_api.md +++ b/docs/developer_api.md @@ -1242,7 +1242,8 @@ Arguments (in percent-encoded JSON): * `tag_service_key`: (optional, selective, hexadecimal, the tag domain on which to search) * `file_sort_type`: (optional, integer, the results sort method) * `file_sort_asc`: true or false (optional, the results sort order) - * `return_hashes`: true or false (optional, default false, returns hex hashes in addition to file ids, hashes and file ids are in the same order) + * `return_file_ids`: true or false (optional, default true, returns file id results) + * `return_hashes`: true or false (optional, default false, returns hex hash results) * _`system_inbox`: true or false (obsolete, use tags)_ * _`system_archive`: true or false (obsolete, use tags)_ @@ -1394,7 +1395,9 @@ Response: } ``` - File ids are internal and specific to an individual client. For a client, a file with hash H always has the same file id N, but two clients will have different ideas about which N goes with which H. They are a bit faster than hashes to retrieve and search with _en masse_, which is why they are exposed here. + You can of course also specify `return_hashes=true&return_file_ids=false` just to get the hashes. The order of both lists is the same. + + File ids are internal and specific to an individual client. For a client, a file with hash H always has the same file id N, but two clients will have different ideas about which N goes with which H. IDs are a bit faster to retrieve than hashes and search with _en masse_, which is why they are exposed here. This search does **not** apply the implicit limit that most clients set to all searches (usually 10,000), so if you do system:everything on a client with millions of files, expect to get boshed. Even with a system:limit included, complicated queries with large result sets may take several seconds to respond. Just like the client itself. @@ -1414,6 +1417,7 @@ Arguments (in percent-encoded JSON): * `hash`: (selective, a hexadecimal SHA256 hash) * `hashes`: (selective, a list of hexadecimal SHA256 hashes) * `only_return_identifiers`: true or false (optional, defaulting to false) + * `only_return_basic_information`: true or false (optional, defaulting to false) * `detailed_url_information`: true or false (optional, defaulting to false) * `hide_service_names_tags`: true or false (optional, defaulting to false) * `include_notes`: true or false (optional, defaulting to false) @@ -1559,6 +1563,38 @@ Response: ] } ``` +```json title="And where only_return_basic_information is true" +{ + "metadata": [ + { + "file_id": 123, + "hash": "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2", + "size": 63405, + "mime": "image/jpg", + "ext": ".jpg", + "width": 640, + "height": 480, + "duration": null, + "has_audio": false, + "num_frames": null, + "num_words": null, + }, + { + "file_id": 4567, + "hash": "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82", + "size": 199713, + "mime": "video/webm", + "ext": ".webm", + "width": 1920, + "height": 1080, + "duration": 4040, + "has_audio": true, + "num_frames": 102, + "num_words": null, + } + ] +} +``` Size is in bytes. Duration is in milliseconds, and may be an int or a float. @@ -1578,6 +1614,8 @@ The tag structure is duplicated for both `name` and `key`. The use of `name` is While `service_XXX_to_statuses_to_tags` represent the actual tags stored on the database for a file, the service_XXX_to_statuses_to_display_tags structures reflect how tags appear in the UI, after siblings are collapsed and parents are added. If you want to edit a file's tags, start with `service_keys_to_statuses_to_tags`. If you want to render to the user, use `service_keys_to_statuses_to_displayed_tags`. +If you set `only_return_basic_information=true`, this will be much faster for first-time requests than the full metadata result, but it will be slower for repeat requests. The full metadata object is cached after first fetch, the limited file info object is not. + If you add `hide_service_names_tags=true`, the `service_names_to_statuses_to_tags` and `service_names_to_statuses_to_display_tags` Objects will not be included. Use this to save data/CPU on large queries. If you add `detailed_url_information=true`, a new entry, `detailed_known_urls`, will be added for each file, with a list of the same structure as /`add_urls/get_url_info`. This may be an expensive request if you are querying thousands of files at once. diff --git a/docs/getting_started_exporting.md b/docs/getting_started_exporting.md new file mode 100644 index 00000000..68b13278 --- /dev/null +++ b/docs/getting_started_exporting.md @@ -0,0 +1,49 @@ +--- +title: exporting files +--- + +# exporting files + +## exporting { id="exporting" } + +There are many ways to export files from the client: + +* **drag and drop** + + Just dragging from the thumbnail view will export (copy) all the selected files to wherever you drop them. You can also start a drag and drop for single files from the media viewer using this arrow button on the top hover window: + + ![](images/media_viewer_dnd.png) + + If you want to drag and drop to discord, check the special BUGFIX option under _options->gui_. + + By default, the files will be named by their ugly hexadecimal [hash](faq.md#hashes), which is how they are stored inside the database. Once you learn filename patterns (practise with manual exports, as below!), you will be able to change this in the options if you wish. + + If you use a drag and drop to open a file inside an image editing program, remember to hit 'save as' and give it a new filename in a new location! The client does not expect files inside its db directory to ever change. + +* **share->copy->files** + + This will copy the files themselves to your clipboard. You can then paste them wherever you like, just as with normal files. They will have their hashes for filenames. + + This is a very quick operation. It can also be triggered by hitting Ctrl+C. + +* **share->copy->image (bitmap)** + + This copies a file's rendered image data to your clipboard. This is useful for pasting into an image editor, but do not use it to upload images to the internet. + +* **share->copy->hashes** + + This will copy the files' unique identifiers to your clipboard, in hexadecimal. + + You will not have to do this often. It is best when you want to identify a number of files to someone else without having to send them the actual files. + +* **export dialog** + + Right clicking some files and selecting _share->export->files_ will open this dialog: + + ![](images/export.png) + + Which lets you export the selected files with custom filenames. It will initialise trying to export the files named by their hashes, but once you are comfortable with tags, you'll be able to generate much cleverer and prettier filenames. + +* **export folders** + + You can set up a regularly repeating export under _file->import and export folders_. This is an advanced operation, so best left until you know the client a bit better, but it is very useful if you want to regularly export some of your collection to a revolving wallpaper directory or similar. diff --git a/docs/getting_started_more_files.md b/docs/getting_started_searching.md similarity index 69% rename from docs/getting_started_more_files.md rename to docs/getting_started_searching.md index d1ac2fa6..a7c09678 100644 --- a/docs/getting_started_more_files.md +++ b/docs/getting_started_searching.md @@ -1,8 +1,39 @@ --- -title: more files +title: powerful searching --- -# more getting started with files +# powerful searching + +## the dropdown controls + +Let's look at the tag autocomplete dropdown again: + +![](images/ac_dropdown.png) + +* **favourite searches star** + + Once you get experience with the client, have a play with this. Rather than leaving common search pages open, save them in here and load them up as needed. You will keep your client lightweight and save time. + +* **include current/pending tags** + + Turn these on and off to control whether tag _search predicates_ apply to tags the exist, or limit just to those pending to be uploaded to a tag repository. Just searching 'pending' tags is useful if you want to scan what you have pending to go up to the PTR--just turn off 'current' tags and search `system:num tags > 0`. + +* **searching immediately** + + This controls whether a change to the search tags will instantly run the new search and get new results. Turning this off is helpful if you want to add, remove, or replace several heavy search terms in a row without getting UI lag. + +* **OR** + + You only see this if you have 'advanced mode' on. It lets you enter some pretty complicated tags! + +* **file/tag domains** + + By default, you will search in 'my files' and 'all known tags' domain. This is the intersection of your local media files (on your hard disk) and the union of all known tag searches. If you search for `character:samus aran`, then you will get file results from your 'my files' domain that have `character:samus aran` in any tag service. For most purposes, this search domain is fine, but as you use the client more, you may want to access different search domains. + + For instance, if you change the file domain to 'trash', then you will instead get files that are in your trash. Setting the tag domain to 'my tags' will ignore other tag services (e.g. the PTR) for all tag search predicates, so a `system:num_tags` or a `character:samus aran` will only look 'my tags'. + + Turning on 'advanced mode' gives access to more search domains. Some of them are subtly complicated and only useful for clever jobs--most of the time, you still want 'my files' and 'all known tags'. + ## searching with wildcards { id="wildcards" } @@ -24,36 +55,27 @@ This is particularly useful if you have a number of files with commonly structur In this case, selecting the `title:cool pic*` predicate will return all three images in the same search, where you can conveniently give them some more-easily searched tags like `series:cool pic` and `page:1`, `page:2`, `page:3`. -## more searching +## OR searching -Let's look at the tag autocomplete dropdown again: +Searches find files that match every search 'predicate' in the list (it is an **AND** search), which makes it difficult to search for files that include one **OR** another tag. More recently, simple OR search support was added. All you have to do is hold down Shift when you enter/double-click a tag in the autocomplete entry area. Instead of sending the tag up to the active search list up top, it will instead start an under-construction 'OR chain' in the tag results below: -![](images/ac_dropdown.png) +![](images/or_under_construction.png) -* **favourite searches star** - - Once you get experience with the client, have a play with this. Rather than leaving common search pages open, save them in here and load them up as needed. You will keep your client lightweight and save time. - -* **include current/pending tags** - - Turn these on and off to control whether tag _search predicates_ apply to tags the exist, or limit just to those pending to be uploaded to a tag repository. Just searching 'pending' tags is useful if you want to scan what you have pending to go up to the PTR--just turn off 'current' tags and search `system:num tags > 0`. - -* **searching immediately** - - This controls whether a change to the search tags will instantly run the new search and get new results. Turning this off is helpful if you want to add, remove, or replace several heavy search terms in a row without getting UI lag. - -* **OR** - - You only see this if you have 'advanced mode' on. It is an experimental module. Have a play with it--it lets you enter some pretty complicated tags! - -* **file/tag domains** - - By default, you will search in 'my files' and 'all known tags' domain. This is the intersection of your local media files (on your hard disk) and the union of all known tag searches. If you search for `character:samus aran`, then you will get file results from your 'my files' domain that have `character:samus aran` in any tag service. For most purposes, this search domain is fine, but as you use the client more, you may want to access different search domains. - - For instance, if you change the file domain to 'trash', then you will instead get files that are in your trash. Setting the tag domain to 'my tags' will ignore other tag services (e.g. the PTR) for all tag search predicates, so a `system:num_tags` or a `character:samus aran` will only look 'my tags'. - - Turning on 'advanced mode' gives access to more search domains. Some of them are subtly complicated and only useful for clever jobs--most of the time, you still want 'my files' and 'all known tags'. - +You can keep searching for and entering new tags. Holding down Shift on new tags will extend the OR chain, and entering them as normal will 'cap' the chain and send it to the complete and active search predicates above. + +![](images/or_done.png) + +Any file that has one or more of those OR sub-tags will match. + +If you enter an OR tag incorrectly, you can either cancel or 'rewind' the under-construction search predicate with these new buttons that will appear: + +![](images/or_buttons.png) + +You can also cancel an under-construction OR by hitting Esc on an empty input. You can add any sort of search term to an OR search predicate, including system predicates. Some unusual sub-predicates (typically a `-tag`, or a very broad system predicate) can run very slowly, but they will run much faster if you include non-OR search predicates in the search: + +![](images/or_mixed.png) + +This search will return all files that have the tag `fanfic` and one or more of `medium:text`, a positive value for the like/dislike rating 'read later', or PDF mime. ## sorting with system limit @@ -62,35 +84,3 @@ If you add system:limit to a search, the client will consider what that page's f If you change the sort, hydrus will not refresh the search, it'll just re-sort the n files you have. Hit F5 to refresh the search with a new sort. Not all sorts are supported. Anything complicated like tag sort will result in a random sample instead. - -## exporting and uploading { id="intro" } - -There are many ways to export files from the client: - -* **drag and drop** - - Just dragging from the thumbnail view will export (copy) all the selected files to wherever you drop them. - - The files will be named by their ugly hexadecimal [hash](faq.md#hashes), which is how they are stored inside the database. - - If you use this to open a file inside an image editing program, remember to go 'save as' and give it a new filename! The client does not expect files inside its db directory to change. - -* **export dialog** - - Right clicking some files and selecting _share->export->files_ will open this dialog: - - ![](images/export.png) - - Which lets you export the selected files with custom filenames. It will initialise trying to export the files named by their hashes, but once you are comfortable with tags, you'll be able to generate much cleverer and prettier filenames. - -* **share->copy->files** - - This will copy the files themselves to your clipboard. You can then paste them wherever you like, just as with normal files. They will have their hashes for filenames. - - This is a very quick operation. It can also be triggered by hitting Ctrl+C. - -* **share->copy->hashes** - - This will copy the files' unique identifiers to your clipboard, in hexadecimal. - - You will not have to do this often. It is best when you want to identify a number of files to someone else without having to send them the actual files. diff --git a/docs/getting_started_subscriptions.md b/docs/getting_started_subscriptions.md index cc665169..8c66c500 100644 --- a/docs/getting_started_subscriptions.md +++ b/docs/getting_started_subscriptions.md @@ -2,7 +2,7 @@ title: subscriptions --- -# getting started with subscriptions +# subscriptions Do not try to create a subscription until you are comfortable with a normal gallery download page! Go [here](getting_started_downloading.md). @@ -125,4 +125,4 @@ The second case is a safety stopgap for hydrus. If a site decides to have `/post ## I put character queries in my artist sub, and now things are all mixed up { id="merging_and_separating" } -On the main subscription dialog, there are 'merge' and 'separate' buttons. These are powerful, but they will walk you through the process of pulling queries out of a sub and merging them back into a different one. Only subs that use the same download source can be merged. Give them a go, and if it all goes wrong, just hit the cancel button on the dialog. \ No newline at end of file +On the main subscription dialog, there are 'merge' and 'separate' buttons. These are powerful, but they will walk you through the process of pulling queries out of a sub and merging them back into a different one. Only subs that use the same download source can be merged. Give them a go, and if it all goes wrong, just hit the cancel button on the dialog. diff --git a/docs/getting_started_tags.md b/docs/getting_started_tags.md index 7c20a56b..97f1d811 100644 --- a/docs/getting_started_tags.md +++ b/docs/getting_started_tags.md @@ -41,28 +41,6 @@ If you add more tags or system predicates to a search, you will limit the result You can also exclude a tag by prefixing it with a hyphen (e.g. `-heresy`). -## OR searching - -Searches find files that match every search 'predicate' in the list (it is an **AND** search), which makes it difficult to search for files that include one **OR** another tag. More recently, simple OR search support was added. All you have to do is hold down Shift when you enter/double-click a tag in the autocomplete entry area. Instead of sending the tag up to the active search list up top, it will instead start an under-construction 'OR chain' in the tag results below: - -![](images/or_under_construction.png) - -You can keep searching for and entering new tags. Holding down Shift on new tags will extend the OR chain, and entering them as normal will 'cap' the chain and send it to the complete and active search predicates above. - -![](images/or_done.png) - -Any file that has one or more of those OR sub-tags will match. - -If you enter an OR tag incorrectly, you can either cancel or 'rewind' the under-construction search predicate with these new buttons that will appear: - -![](images/or_buttons.png) - -You can also cancel an under-construction OR by hitting Esc on an empty input. You can add any sort of search term to an OR search predicate, including system predicates. Some unusual sub-predicates (typically a `-tag`, or a very broad system predicate) can run very slowly, but they will run much faster if you include non-OR search predicates in the search: - -![](images/or_mixed.png) - -This search will return all files that have the tag `fanfic` and one or more of `medium:text`, a positive value for the like/dislike rating 'read later', or PDF mime. - ## tag repositories It can take a long time to tag even small numbers of files well, so I created _tag repositories_ so people can share the work. @@ -93,4 +71,4 @@ I recommend you not spam tags to the public tag repo until you get a rough feel You can connect to more than one tag repository if you like. When you are in the _manage tags_ dialog, pressing the up or down arrow keys on an empty input switches between your services. -[FAQ: why can my friend not see what I just uploaded?](faq.md#delays) \ No newline at end of file +[FAQ: why can my friend not see what I just uploaded?](faq.md#delays) diff --git a/docs/images/media_viewer_dnd.png b/docs/images/media_viewer_dnd.png new file mode 100644 index 00000000..25453ee8 Binary files /dev/null and b/docs/images/media_viewer_dnd.png differ diff --git a/docs/old_changelog.html b/docs/old_changelog.html index d47b91cf..123672f9 100644 --- a/docs/old_changelog.html +++ b/docs/old_changelog.html @@ -33,6 +33,49 @@

changelog