diff --git a/docs/changelog.md b/docs/changelog.md
index da8d5125..5f8935c2 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -7,6 +7,49 @@ title: Changelog
!!! note
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
+## [Version 535](https://github.com/hydrusnetwork/hydrus/releases/tag/v535)
+
+### misc
+
+* thanks to a user, we now have Krita (.kra, .krz) support! it even pulls thumbnails!
+* thanks to another user, we now have SVG (.svg) support! it even generates thumbnails!
+* I think I fixed a comparison statement calculator divide-by-zero error in the duplicate filter when you compare a file with a resolution with a file without one
+
+### petitions overview
+
+* _this is a workflow/usability update only for server janitors_
+* tl;dr: the petitions page now fetches many petitions at once. update your servers and clients for it all to work right
+* so, the petitions page now fetches lots of petitions with each 'fetch' button click. you can set how many it will fetch with a new number control
+* the petitions are shown in a new multi-column list that shows action, account id, reason, and total weight. the actual data for the petitions will load in quickly, reflected in the list. as soon as the first is loaded, it is highlighted, but double-click any to highlight it in the old petition UI as normal
+* when you process petitions, the client moves instantly to the next, all fitting into the existing workflow, without having to wait for the server to fetch a new one after you commit
+* you can also mass approve/deny from here! if one account is doing great or terrible stuff, you can now blang it all in one go
+
+### petitions details
+
+* the 'fetch x petition' buttons now show `(*)` in their label if they are the active petition type being worked on
+* petition pages now remember: the last petition type they were looking at; the number of petitions to fetch; and the number of files to show
+* the petition page will pause any ongoing petition fetches if you close it, and resume if you unclose it
+* a system where multi-mapping petitions would be broken up and delivered in tags with weight-similar chunks (e.g. if would say 'aaa for 11 files' and 'bbb in 15 files' in the same fetch, but not 'ccc in 542,154 files') is abandoned. this was not well explained and was causing confusion and code complexity. these petitions now appear clientside in full
+* another system, where multi-mapping petitions would be delivered in same-namespace chunks, is also abandoned, for similar reasons. it was causing more confusion, especially when compared to the newer petition counting tech I've added. perhaps it will come back in as a clientside filter option
+* the list of petitions you are given _should_ also be neatly grouped by account id, so rather than randomly sampling from all petitions, you'll get batches by user x, y, or z, and in most cases you'll be looking at everything by user x, and y, and then z up to the limit of num petitions you chose to fetch
+* drawback: since petitions' content can overlap in complicated ways, and janitors can work on the same list at the same time, in edge cases the list you see can be slightly out of sync with what the server actually has. this isn't a big deal, and the worst case is wasted work as you approve the same thing twice. I tried to implement 'refresh list if count drops more than expected' tech, but the situation is complicated and it was spamming too much. I will let you refresh the list with a button click yourself for now, as you like, and please let me know where it works and fails
+* drawback: I added some new objects, so you have to update both server and client for this to work. older/newer combinations will give you some harmless errors
+* also, if your list starts running low, but there are plenty more petitions to work on, it will auto-refresh. again, it won't interrupt your current work, but it will fetch more. let me know how it works out
+* drawback: while the new petition summary list is intentionally lightweight, I do spend some extra CPU figuring it out. with a high 'num petitions to fetch', it may take several seconds for a very busy server like the PTR just to fetch the initial list, so please play around with different fetch sizes and let me know what works well and what is way too slow
+* there are still some things I want to do to this page, which I want to slip in the near future. I want to hide/show the sort and 'num files to show' widgets as appropriate, figure out a right-click menu for the new list to retry failures, and get some shortcut support going
+
+### boring code cleanup
+
+* wrote a new petition header object to hold content type, petition status, account id, and reason for petitions
+* serverside petition fetching is now split into 'get petition headers' and 'get petition data'. the 'headers' section supports filtering by account id and in future reason
+* the clientside petition management UI code pretty much got a full pass
+* cleaned a bunch of ancient server db code
+* cleaned a bunch of the clientside petition code. it was a real tangle
+* improved the resilience of the hydrus server when it is given unacceptable tags in a content update
+* all fetches of multiple rows of data from multi-column lists now happen sorted. this is just a little thing, but it'll probably dejank a few operations where you edit several things at once or get some errors and are trying to figure out which of five things caused it
+* the hydrus official mimetype for psd files is now 'image/vnd.adobe.photoshop' (instead of 'application/x-photoshop')
+* with krita file (which are actually just zip files) support, we now have the very barebones of archive tech started. I'll expand it a bit more and we should be able to improve support for other archive-like formats in the future
+
## [Version 534](https://github.com/hydrusnetwork/hydrus/releases/tag/v534)
### user submissions
@@ -332,28 +375,3 @@ title: Changelog
* the main build script no longer uses set-output commands (these are deprecated and being dropped later in the year I think, in favour of some ENV stuff)
* tidied some cruft from the main build script
* I moved the 'new' python-mpv in the requirements.txts from 1.0.1 to 1.0.3. source users might like to rebuild their venvs again, particularly Windows users who updated to the new mpv dll recently
-
-## [Version 525](https://github.com/hydrusnetwork/hydrus/releases/tag/v525)
-
-### library updates
-
-* after successful testing amongst source users, I am finally updating the official builds and the respective requirements.txts for Qt, from 6.3.1 to 6.4.1 (with 'test' now 6.5.0), opencv-python-headless from 4.5.3.56 to 4.5.5.64 (with a new 'test' of 4.7.0.72), and in the Windows build, the mpv dll from 2022-05-01 to 2023-02-12 (API 2.0 to 2.1). if you use my normal builds, you don't have to do anything special in the update, and with luck you'll get slightly faster images, video, and UI, and with fewer bugs. if you run from source, you might want to re-run your setup_venv script--it'll update you automatically--and if you are a modern Windows source user and haven't yet, grab the new dll here and rename it to mpv-2.dll https://sourceforge.net/projects/mpv-player-windows/files/libmpv/mpv-dev-x86_64-20230212-git-a40958c.7z . there is a chance that some older OSes will not be able to boot this new build, but I think these people were already migrated to being source users when Win 7-level was no longer supported. in any case, let me know how you get on, and if you are on an older OS, be prepared to rollback if this version doesn't boot
-* setup_venv.bat (Windows source) now adds PyWin32, just like the builds (the new version of pympler, a memory management module, moans on boot if it doesn't have it)
-
-### timestamps
-
-* a couple places where fixed calendar time-deltas are converted to absolute datestrings now work better over longer times. going back (5 years, 3 months) should now work out the actual calendar dates (previously they used a rough total_num_seconds estimation) and go back to the same day of the destination month, also accounting for if that has fewer days than the starting month and handling leap years. it also handles >'12 months' better now
-* in system:time predicates that use since/before a delta, it now allows much larger values in the UI, like '72 months', and it won't merge those into the larger values in the label. so if you set a gap of 100 days, it'll say that, not 3 months 10 days or whatever
-* the main copy button on 'manage file times' is now a menu button letting you choose to copy all timestamps or just those for the file services. as a hacky experiment, you can also copy the file service timestamps plus one second (in case you want to try finick-ily going through a handful of files to force a certain import sort order)
-* the system predicate time parsing is now more flexible. for archived, modified, last viewed, and imported time, you can now generally say all variants in the form 'import' or 'imported' and 'time' or 'date' and 'time imported' or 'imported time'.
-* fixed an issue that meant editing existing delta 'system:archived time' predicates was launching the 'date' edit panel
-
-### misc
-
-* in the 'exif and other embedded metadata' review window, which is launched from a button on the the media viewer's top hover, jpegs now state their subsampling and whether they are progressive
-* every simple place where the client eats clipboard data and tries to import something now has a unified error-reporting process. before, it would make a popup with something like 'I could not understandwhat was in the clipboard!'. Now it makes a popup with info on what was pasted, what was expected, and actual exception info. Longer info is printed to the log
-* many places across the program say the specific exception type when they report errors now, not just the string summary
-* the sankaku downloader is updated with a new url class for their new md5 links. also, the file parser is updated to associate the old id URL, and the gallery parser is updated to skip the 'get sank pro' thumbnail links if you are not logged in. if you have sank subscriptions, they are going to go crazy this week due to the URL format changing--sorry, there's no nice way around it!--just ignore their popups about hitting file limits and wait them out. unfortunately, due to an unusual 404-based redirect, the id-based URLs will not work in hydrus any more
-* the 'API URL' system for url classes now supports File URLs--this may help you figure out some CDN redirects and similar. in a special rule for these File URLs, both URLs will be associated with the imported file (normally, Post API URLs are not saved as Known URLs). relatedly, I have renamed this system broadly to 'api/redirect url', since we use it for a bunch of non-API stuff now
-* fixed a problem where deleting one of the new inc/dec rating services was not clearing the actual number ratings for that service from the database, causing service-id error hell on loading files with those orphaned rating records. sorry for the trouble, this slipped through testing! any users who were affected by this will also be fixed (orphan records cleared out) on update (issue #1357)
-* the client cleans up the temporary paths used by file imports more carefully now: it tries more times to delete 'sticky' temp files; it tries to clear them again immediately on shutdown; and it stores them all in the hydrus temp subdirectory where they are less loose and will be captured by the final directory clear on shutdown (issue #1356)
diff --git a/docs/getting_started_files.md b/docs/getting_started_files.md
index 919f898e..68dc7dba 100644
--- a/docs/getting_started_files.md
+++ b/docs/getting_started_files.md
@@ -72,6 +72,7 @@ Now:
* **image/png** (.png)
* **image/apng** (.apng)
* **image/jpeg** (.jpg)
+* **image/svg+xml** (.svg)
* **image/tiff** (.tiff)
* **image/webp** (.webp)
* **video/x-msvideo** (.avi)
@@ -90,6 +91,7 @@ Now:
* **application/pdf** (.pdf)
* **application/x-photoshop** (.psd)
* **application/clip** (.clip)
+* **application/x-krita** (.kra, .krz)
* **application/sai2** (.sai2)
* **application/vnd.rar** (.rar)
* **application/zip** (.zip)
diff --git a/docs/old_changelog.html b/docs/old_changelog.html
index c8bcb42e..7eb85c64 100644
--- a/docs/old_changelog.html
+++ b/docs/old_changelog.html
@@ -34,6 +34,44 @@
+
+
+
+ misc
+ thanks to a user, we now have Krita (.kra, .krz) support! it even pulls thumbnails!
+ thanks to another user, we now have SVG (.svg) support! it even generates thumbnails!
+ I think I fixed a comparison statement calculator divide-by-zero error in the duplicate filter when you compare a file with a resolution with a file without one
+ petitions overview
+ _this is a workflow/usability update only for server janitors_
+ tl;dr: the petitions page now fetches many petitions at once. update your servers and clients for it all to work right
+ so, the petitions page now fetches lots of petitions with each 'fetch' button click. you can set how many it will fetch with a new number control
+ the petitions are shown in a new multi-column list that shows action, account id, reason, and total weight. the actual data for the petitions will load in quickly, reflected in the list. as soon as the first is loaded, it is highlighted, but double-click any to highlight it in the old petition UI as normal
+ when you process petitions, the client moves instantly to the next, all fitting into the existing workflow, without having to wait for the server to fetch a new one after you commit
+ you can also mass approve/deny from here! if one account is doing great or terrible stuff, you can now blang it all in one go
+ petitions details
+ the 'fetch x petition' buttons now show `(*)` in their label if they are the active petition type being worked on
+ petition pages now remember: the last petition type they were looking at; the number of petitions to fetch; and the number of files to show
+ the petition page will pause any ongoing petition fetches if you close it, and resume if you unclose it
+ a system where multi-mapping petitions would be broken up and delivered in tags with weight-similar chunks (e.g. if would say 'aaa for 11 files' and 'bbb in 15 files' in the same fetch, but not 'ccc in 542,154 files') is abandoned. this was not well explained and was causing confusion and code complexity. these petitions now appear clientside in full
+ another system, where multi-mapping petitions would be delivered in same-namespace chunks, is also abandoned, for similar reasons. it was causing more confusion, especially when compared to the newer petition counting tech I've added. perhaps it will come back in as a clientside filter option
+ the list of petitions you are given _should_ also be neatly grouped by account id, so rather than randomly sampling from all petitions, you'll get batches by user x, y, or z, and in most cases you'll be looking at everything by user x, and y, and then z up to the limit of num petitions you chose to fetch
+ drawback: since petitions' content can overlap in complicated ways, and janitors can work on the same list at the same time, in edge cases the list you see can be slightly out of sync with what the server actually has. this isn't a big deal, and the worst case is wasted work as you approve the same thing twice. I tried to implement 'refresh list if count drops more than expected' tech, but the situation is complicated and it was spamming too much. I will let you refresh the list with a button click yourself for now, as you like, and please let me know where it works and fails
+ drawback: I added some new objects, so you have to update both server and client for this to work. older/newer combinations will give you some harmless errors
+ also, if your list starts running low, but there are plenty more petitions to work on, it will auto-refresh. again, it won't interrupt your current work, but it will fetch more. let me know how it works out
+ drawback: while the new petition summary list is intentionally lightweight, I do spend some extra CPU figuring it out. with a high 'num petitions to fetch', it may take several seconds for a very busy server like the PTR just to fetch the initial list, so please play around with different fetch sizes and let me know what works well and what is way too slow
+ there are still some things I want to do to this page, which I want to slip in the near future. I want to hide/show the sort and 'num files to show' widgets as appropriate, figure out a right-click menu for the new list to retry failures, and get some shortcut support going
+ boring code cleanup
+ wrote a new petition header object to hold content type, petition status, account id, and reason for petitions
+ serverside petition fetching is now split into 'get petition headers' and 'get petition data'. the 'headers' section supports filtering by account id and in future reason
+ the clientside petition management UI code pretty much got a full pass
+ cleaned a bunch of ancient server db code
+ cleaned a bunch of the clientside petition code. it was a real tangle
+ improved the resilience of the hydrus server when it is given unacceptable tags in a content update
+ all fetches of multiple rows of data from multi-column lists now happen sorted. this is just a little thing, but it'll probably dejank a few operations where you edit several things at once or get some errors and are trying to figure out which of five things caused it
+ the hydrus official mimetype for psd files is now 'image/vnd.adobe.photoshop' (instead of 'application/x-photoshop')
+ with krita file (which are actually just zip files) support, we now have the very barebones of archive tech started. I'll expand it a bit more and we should be able to improve support for other archive-like formats in the future
+
+
diff --git a/hydrus/client/ClientController.py b/hydrus/client/ClientController.py
index 9439d45c..1faf52a4 100644
--- a/hydrus/client/ClientController.py
+++ b/hydrus/client/ClientController.py
@@ -1444,6 +1444,14 @@ class Controller( HydrusController.HydrusController ):
+ def PageAliveAndNotClosed( self, page_key ):
+
+ with self._page_key_lock:
+
+ return page_key in self._alive_page_keys and page_key not in self._closed_page_keys
+
+
+
def PageClosedButNotDestroyed( self, page_key ):
with self._page_key_lock:
@@ -1452,6 +1460,14 @@ class Controller( HydrusController.HydrusController ):
+ def PageDestroyed( self, page_key ):
+
+ with self._page_key_lock:
+
+ return page_key not in self._alive_page_keys and page_key not in self._closed_page_keys
+
+
+
def PrepStringForDisplay( self, text ):
return text.lower()
diff --git a/hydrus/client/ClientDuplicates.py b/hydrus/client/ClientDuplicates.py
index 103ec6ae..7a12c477 100644
--- a/hydrus/client/ClientDuplicates.py
+++ b/hydrus/client/ClientDuplicates.py
@@ -120,61 +120,66 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
if s_size != c_size:
- absolute_size_ratio = max( s_size, c_size ) / min( s_size, c_size )
+ all_measurements_are_good = None not in ( s_size, c_size ) and True not in ( d <= 0 for d in ( s_size, c_size ) )
- if absolute_size_ratio > 2.0:
+ if all_measurements_are_good:
- if s_size > c_size:
+ absolute_size_ratio = max( s_size, c_size ) / min( s_size, c_size )
+
+ if absolute_size_ratio > 2.0:
- operator = '>>'
- score = duplicate_comparison_score_much_higher_filesize
+ if s_size > c_size:
+
+ operator = '>>'
+ score = duplicate_comparison_score_much_higher_filesize
+
+ else:
+
+ operator = '<<'
+ score = -duplicate_comparison_score_much_higher_filesize
+
+
+ elif absolute_size_ratio > 1.05:
+
+ if s_size > c_size:
+
+ operator = '>'
+ score = duplicate_comparison_score_higher_filesize
+
+ else:
+
+ operator = '<'
+ score = -duplicate_comparison_score_higher_filesize
+
else:
- operator = '<<'
- score = -duplicate_comparison_score_much_higher_filesize
+ operator = CC.UNICODE_ALMOST_EQUAL_TO
+ score = 0
- elif absolute_size_ratio > 1.05:
-
if s_size > c_size:
- operator = '>'
- score = duplicate_comparison_score_higher_filesize
+ sign = '+'
+ percentage_difference = ( s_size / c_size ) - 1.0
else:
- operator = '<'
- score = -duplicate_comparison_score_higher_filesize
+ sign = ''
+ percentage_difference = ( s_size / c_size ) - 1.0
- else:
+ percentage_different_string = ' ({}{})'.format( sign, HydrusData.ConvertFloatToPercentage( percentage_difference ) )
- operator = CC.UNICODE_ALMOST_EQUAL_TO
- score = 0
+ if is_a_pixel_dupe:
+
+ score = 0
+
-
- if s_size > c_size:
+ statement = '{} {} {}{}'.format( HydrusData.ToHumanBytes( s_size ), operator, HydrusData.ToHumanBytes( c_size ), percentage_different_string )
- sign = '+'
- percentage_difference = ( s_size / c_size ) - 1.0
+ statements_and_scores[ 'filesize' ] = ( statement, score )
- else:
-
- sign = ''
- percentage_difference = ( s_size / c_size ) - 1.0
-
-
- percentage_different_string = ' ({}{})'.format( sign, HydrusData.ConvertFloatToPercentage( percentage_difference ) )
-
- if is_a_pixel_dupe:
-
- score = 0
-
-
- statement = '{} {} {}{}'.format( HydrusData.ToHumanBytes( s_size ), operator, HydrusData.ToHumanBytes( c_size ), percentage_different_string )
-
- statements_and_scores[ 'filesize' ] = ( statement, score )
# higher/same res
@@ -433,7 +438,7 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
if s_label != c_label:
- if c_jpeg_quality is None or s_jpeg_quality is None:
+ if c_jpeg_quality is None or s_jpeg_quality is None or c_jpeg_quality <= 0 or s_jpeg_quality <= 0:
score = 0
diff --git a/hydrus/client/gui/ClientGUIRatings.py b/hydrus/client/gui/ClientGUIRatings.py
index c98e2e01..6203a7be 100644
--- a/hydrus/client/gui/ClientGUIRatings.py
+++ b/hydrus/client/gui/ClientGUIRatings.py
@@ -7,7 +7,6 @@ from qtpy import QtWidgets as QW
from hydrus.core import HydrusData
from hydrus.core import HydrusExceptions
from hydrus.core import HydrusGlobals as HG
-from hydrus.core import HydrusTime
from hydrus.client.gui import QtPorting as QP
from hydrus.client.gui.widgets import ClientGUICommon
diff --git a/hydrus/client/gui/lists/ClientGUIListConstants.py b/hydrus/client/gui/lists/ClientGUIListConstants.py
index 96f84cd4..dfdd7f90 100644
--- a/hydrus/client/gui/lists/ClientGUIListConstants.py
+++ b/hydrus/client/gui/lists/ClientGUIListConstants.py
@@ -1528,3 +1528,22 @@ register_column_type( COLUMN_LIST_DOMAIN_MODIFIED_TIMESTAMPS.ID, COLUMN_LIST_DOM
register_column_type( COLUMN_LIST_DOMAIN_MODIFIED_TIMESTAMPS.ID, COLUMN_LIST_DOMAIN_MODIFIED_TIMESTAMPS.TIMESTAMP, 'time', False, 23, True )
default_column_list_sort_lookup[ COLUMN_LIST_DOMAIN_MODIFIED_TIMESTAMPS.ID ] = ( COLUMN_LIST_DOMAIN_MODIFIED_TIMESTAMPS.DOMAIN, True )
+
+class COLUMN_LIST_PETITIONS_SUMMARY( COLUMN_LIST_DEFINITION ):
+
+ ID = 71
+
+ ACTION = 0
+ ACCOUNT_KEY = 1
+ REASON = 2
+ CONTENT = 3
+
+
+column_list_type_name_lookup[ COLUMN_LIST_PETITIONS_SUMMARY.ID ] = 'petitions summary'
+
+register_column_type( COLUMN_LIST_PETITIONS_SUMMARY.ID, COLUMN_LIST_PETITIONS_SUMMARY.ACTION, 'action', False, 9, True )
+register_column_type( COLUMN_LIST_PETITIONS_SUMMARY.ID, COLUMN_LIST_PETITIONS_SUMMARY.ACCOUNT_KEY, 'account', False, 10, True )
+register_column_type( COLUMN_LIST_PETITIONS_SUMMARY.ID, COLUMN_LIST_PETITIONS_SUMMARY.REASON, 'reason', False, 15, True )
+register_column_type( COLUMN_LIST_PETITIONS_SUMMARY.ID, COLUMN_LIST_PETITIONS_SUMMARY.CONTENT, 'content', False, 16, True )
+
+default_column_list_sort_lookup[ COLUMN_LIST_PETITIONS_SUMMARY.ID ] = ( COLUMN_LIST_PETITIONS_SUMMARY.ACCOUNT_KEY, True )
diff --git a/hydrus/client/gui/lists/ClientGUIListCtrl.py b/hydrus/client/gui/lists/ClientGUIListCtrl.py
index 516ca74b..e12bdc00 100644
--- a/hydrus/client/gui/lists/ClientGUIListCtrl.py
+++ b/hydrus/client/gui/lists/ClientGUIListCtrl.py
@@ -329,7 +329,7 @@ class BetterListCtrl( QW.QTreeWidget ):
return ( display_tuple, sort_tuple )
- def _GetSelected( self ):
+ def _GetSelectedIndices( self ) -> typing.List[ int ]:
indices = []
@@ -520,7 +520,12 @@ class BetterListCtrl( QW.QTreeWidget ):
def DeleteDatas( self, datas: typing.Iterable[ object ] ):
- deletees = [ ( self._data_to_indices[ data ], data ) for data in datas ]
+ deletees = [ ( self._data_to_indices[ data ], data ) for data in datas if data in self._data_to_indices ]
+
+ if len( deletees ) == 0:
+
+ return
+
deletees.sort( reverse = True )
@@ -552,7 +557,7 @@ class BetterListCtrl( QW.QTreeWidget ):
def DeleteSelected( self ):
- indices = self._GetSelected()
+ indices = self._GetSelectedIndices()
indices.sort( reverse = True )
@@ -655,13 +660,15 @@ class BetterListCtrl( QW.QTreeWidget ):
if only_selected:
- indices = self._GetSelected()
+ indices = self._GetSelectedIndices()
else:
indices = list( self._indices_to_data_info.keys() )
+ indices.sort()
+
result = []
for index in indices:
diff --git a/hydrus/client/gui/pages/ClientGUIManagementController.py b/hydrus/client/gui/pages/ClientGUIManagementController.py
index 6a0d5777..417d03fd 100644
--- a/hydrus/client/gui/pages/ClientGUIManagementController.py
+++ b/hydrus/client/gui/pages/ClientGUIManagementController.py
@@ -166,6 +166,11 @@ def CreateManagementControllerPetitions( petition_service_key ):
management_controller.SetVariable( 'petition_service_key', petition_service_key )
+ management_controller.SetVariable( 'petition_type_content_type', None )
+ management_controller.SetVariable( 'petition_type_status', None )
+ management_controller.SetVariable( 'num_petitions_to_fetch', 40 )
+ management_controller.SetVariable( 'num_files_to_show', 256 )
+
return management_controller
@@ -188,7 +193,7 @@ class ManagementController( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_MANAGEMENT_CONTROLLER
SERIALISABLE_NAME = 'Client Page Management Controller'
- SERIALISABLE_VERSION = 12
+ SERIALISABLE_VERSION = 13
def __init__( self, page_name = 'page' ):
@@ -536,6 +541,27 @@ class ManagementController( HydrusSerialisable.SerialisableBase ):
return ( 12, new_serialisable_info )
+ if version == 12:
+
+ ( page_name, management_type, serialisable_variables ) = old_serialisable_info
+
+ if management_type == MANAGEMENT_TYPE_PETITIONS:
+
+ variables = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_variables )
+
+ variables[ 'petition_type_content_type' ] = None
+ variables[ 'petition_type_status' ] = None
+ variables[ 'num_petitions_to_fetch' ] = 40
+ variables[ 'num_files_to_show' ] = 256
+
+ serialisable_variables = variables.GetSerialisableTuple()
+
+
+ new_serialisable_info = ( page_name, management_type, serialisable_variables )
+
+ return ( 13, new_serialisable_info )
+
+
def GetAPIInfoDict( self, simple ):
diff --git a/hydrus/client/gui/pages/ClientGUIManagementPanels.py b/hydrus/client/gui/pages/ClientGUIManagementPanels.py
index b93f122a..b39c1d7a 100644
--- a/hydrus/client/gui/pages/ClientGUIManagementPanels.py
+++ b/hydrus/client/gui/pages/ClientGUIManagementPanels.py
@@ -1,5 +1,7 @@
+import collections
import os
import random
+import threading
import time
import typing
@@ -21,6 +23,7 @@ from hydrus.client import ClientDuplicates
from hydrus.client import ClientLocation
from hydrus.client import ClientParsing
from hydrus.client import ClientPaths
+from hydrus.client import ClientServices
from hydrus.client import ClientThreading
from hydrus.client import ClientTime
from hydrus.client.gui import ClientGUIAsync
@@ -3835,6 +3838,41 @@ class ManagementPanelImporterURLs( ManagementPanelImporter ):
management_panel_types_to_classes[ ClientGUIManagementController.MANAGEMENT_TYPE_IMPORT_URLS ] = ManagementPanelImporterURLs
+def GetPetitionActionInfo( petition: HydrusNetwork.Petition ):
+
+ add_contents = petition.GetContents( HC.CONTENT_UPDATE_PEND )
+ delete_contents = petition.GetContents( HC.CONTENT_UPDATE_PETITION )
+
+ have_add = len( add_contents ) > 0
+ have_delete = len( delete_contents ) > 0
+
+ action_text = 'UNKNOWN'
+ hydrus_text = 'default'
+ object_name = 'normal'
+
+ if have_add or have_delete:
+
+ if have_add and have_delete:
+
+ action_text = 'REPLACE'
+
+ elif have_add:
+
+ action_text = 'ADD'
+ hydrus_text = 'valid'
+ object_name = 'HydrusValid'
+
+ else:
+
+ action_text = 'DELETE'
+ hydrus_text = 'invalid'
+ object_name = 'HydrusInvalid'
+
+
+
+ return ( action_text, hydrus_text, object_name )
+
+
class ManagementPanelPetitions( ManagementPanel ):
TAG_DISPLAY_TYPE = ClientTags.TAG_DISPLAY_STORAGE
@@ -3850,47 +3888,71 @@ class ManagementPanelPetitions( ManagementPanel ):
service_type = self._service.GetServiceType()
- self._num_petition_info = None
+ self._petition_types_to_count = collections.Counter()
self._current_petition = None
- self._last_petition_type_fetched = None
+ content_type = management_controller.GetVariable( 'petition_type_content_type' )
+ status = management_controller.GetVariable( 'petition_type_status' )
+
+ if content_type is None or status is None:
+
+ self._last_petition_type_fetched = None
+
+ else:
+
+ self._last_petition_type_fetched = ( content_type, status )
+
self._last_fetched_subject_account_key = None
+ self._petition_headers_to_fetched_petitions_cache = {}
+ self._petition_headers_we_failed_to_fetch = set()
+ self._petition_headers_we_are_fetching = []
+ self._outgoing_petition_headers_to_petitions = {}
+ self._failed_outgoing_petition_headers_to_petitions = {}
+
+ self._petition_fetcher_and_uploader_work_lock = threading.Lock()
+
#
- self._petitions_info_panel = ClientGUICommon.StaticBox( self, 'petitions info' )
+ self._petition_numbers_panel = ClientGUICommon.StaticBox( self, 'counts' )
- self._petition_account_key = QW.QLineEdit( self._petitions_info_panel )
+ self._petition_account_key = QW.QLineEdit( self._petition_numbers_panel )
self._petition_account_key.setPlaceholderText( 'account id filter' )
- self._refresh_num_petitions_button = ClientGUICommon.BetterButton( self._petitions_info_panel, 'refresh counts', self._FetchNumPetitions )
+ self._num_petitions_to_fetch = ClientGUICommon.BetterSpinBox( self._petition_numbers_panel, min = 1, max = 10000 )
+
+ self._num_petitions_to_fetch.setValue( management_controller.GetVariable( 'num_petitions_to_fetch' ) )
+
+ self._refresh_num_petitions_button = ClientGUICommon.BetterButton( self._petition_numbers_panel, 'refresh counts', self._StartFetchNumPetitions )
self._petition_types_to_controls = {}
content_type_hboxes = []
- petition_types = []
+ self._my_petition_types = []
if service_type == HC.FILE_REPOSITORY:
- petition_types.append( ( HC.CONTENT_TYPE_FILES, HC.CONTENT_STATUS_PETITIONED ) )
+ self._my_petition_types.append( ( HC.CONTENT_TYPE_FILES, HC.CONTENT_STATUS_PETITIONED ) )
elif service_type == HC.TAG_REPOSITORY:
- petition_types.append( ( HC.CONTENT_TYPE_MAPPINGS, HC.CONTENT_STATUS_PETITIONED ) )
- petition_types.append( ( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_STATUS_PENDING ) )
- petition_types.append( ( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_STATUS_PETITIONED ) )
- petition_types.append( ( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_STATUS_PENDING ) )
- petition_types.append( ( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_STATUS_PETITIONED ) )
+ self._my_petition_types.append( ( HC.CONTENT_TYPE_MAPPINGS, HC.CONTENT_STATUS_PETITIONED ) )
+ self._my_petition_types.append( ( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_STATUS_PENDING ) )
+ self._my_petition_types.append( ( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_STATUS_PETITIONED ) )
+ self._my_petition_types.append( ( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_STATUS_PENDING ) )
+ self._my_petition_types.append( ( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_STATUS_PETITIONED ) )
- for ( content_type, status ) in petition_types:
+ for petition_type in self._my_petition_types:
- func = HydrusData.Call( self._FetchPetition, content_type, status )
+ ( content_type, status ) = petition_type
- st = ClientGUICommon.BetterStaticText( self._petitions_info_panel )
- button = ClientGUICommon.BetterButton( self._petitions_info_panel, 'fetch ' + HC.content_status_string_lookup[ status ] + ' ' + HC.content_type_string_lookup[ content_type ] + ' petition', func )
+ func = HydrusData.Call( self._FetchPetitionsSummary, petition_type )
+
+ st = ClientGUICommon.BetterStaticText( self._petition_numbers_panel )
+ button = ClientGUICommon.BetterButton( self._petition_numbers_panel, 'fetch ' + HC.content_status_string_lookup[ status ] + ' ' + HC.content_type_string_lookup[ content_type ] + ' petitions', func )
button.setEnabled( False )
@@ -3906,11 +3968,24 @@ class ManagementPanelPetitions( ManagementPanel ):
#
- self._petition_panel = ClientGUICommon.StaticBox( self, 'petition' )
+ self._petitions_panel = ClientGUICommon.StaticBox( self, 'petitions' )
+
+ self._petitions_summary_list_panel = ClientGUIListCtrl.BetterListCtrlPanel( self._petitions_panel )
+
+ self._petitions_summary_list = ClientGUIListCtrl.BetterListCtrl( self._petitions_summary_list_panel, CGLC.COLUMN_LIST_PETITIONS_SUMMARY.ID, 12, self._ConvertDataToListCtrlTuples, activation_callback = self._ActivateToHighlightPetition )
+
+ self._petitions_summary_list_panel.SetListCtrl( self._petitions_summary_list )
+
+ self._petitions_summary_list_panel.AddButton( 'mass-approve', self._ApproveSelected, enabled_check_func = self._OnlySelectingLoadedPetitions, tooltip = 'Approve the selected petitions' )
+ self._petitions_summary_list_panel.AddButton( 'mass-deny', self._DenySelected, enabled_check_func = self._OnlySelectingLoadedPetitions, tooltip = 'Deny the selected petitions' )
+
+ #
+
+ self._petition_panel = ClientGUICommon.StaticBox( self, 'highlighted petition' )
self._num_files_to_show = ClientGUICommon.NoneableSpinCtrl( self._petition_panel, message = 'number of files to show', min = 1 )
- self._num_files_to_show.SetValue( 256 )
+ self._num_files_to_show.SetValue( management_controller.GetVariable( 'num_files_to_show' ) )
self._action_text = ClientGUICommon.BetterStaticText( self._petition_panel, label = '' )
@@ -3948,7 +4023,7 @@ class ManagementPanelPetitions( ManagementPanel ):
self._contents_delete.setFixedHeight( min_height )
self._process = QW.QPushButton( 'process', self._petition_panel )
- self._process.clicked.connect( self.EventProcess )
+ self._process.clicked.connect( self.ProcessCurrentPetition )
self._process.setObjectName( 'HydrusAccept' )
self._copy_account_key_button = ClientGUICommon.BetterButton( self._petition_panel, 'copy petitioner account id', self._CopyAccountKey )
@@ -3960,14 +4035,21 @@ class ManagementPanelPetitions( ManagementPanel ):
#
- self._petitions_info_panel.Add( self._petition_account_key, CC.FLAGS_EXPAND_PERPENDICULAR )
- self._petitions_info_panel.Add( self._refresh_num_petitions_button, CC.FLAGS_EXPAND_PERPENDICULAR )
+ self._petition_numbers_panel.Add( self._petition_account_key, CC.FLAGS_EXPAND_PERPENDICULAR )
+ self._petition_numbers_panel.Add( self._refresh_num_petitions_button, CC.FLAGS_EXPAND_PERPENDICULAR )
+ self._petition_numbers_panel.Add( ClientGUICommon.WrapInText( self._num_petitions_to_fetch, self, 'number of petitions to fetch' ), CC.FLAGS_EXPAND_PERPENDICULAR )
for hbox in content_type_hboxes:
- self._petitions_info_panel.Add( hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
+ self._petition_numbers_panel.Add( hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
+ #
+
+ self._petitions_panel.Add( self._petitions_summary_list_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
+
+ #
+
check_hbox = QP.HBoxLayout()
QP.AddToLayout( check_hbox, check_all, CC.FLAGS_CENTER_PERPENDICULAR_EXPAND_DEPTH )
@@ -3979,14 +4061,14 @@ class ManagementPanelPetitions( ManagementPanel ):
QP.AddToLayout( sort_hbox, self._sort_by_left, CC.FLAGS_CENTER_PERPENDICULAR_EXPAND_DEPTH )
QP.AddToLayout( sort_hbox, self._sort_by_right, CC.FLAGS_CENTER_PERPENDICULAR_EXPAND_DEPTH )
- self._petition_panel.Add( ClientGUICommon.BetterStaticText( self._petition_panel, label = 'Double-click a petition to see its files, if it has them.' ), CC.FLAGS_EXPAND_PERPENDICULAR )
+ self._petition_panel.Add( ClientGUICommon.BetterStaticText( self._petition_panel, label = 'Double-click a petition row to see its files, if it has them.' ), CC.FLAGS_EXPAND_PERPENDICULAR )
self._petition_panel.Add( self._num_files_to_show, CC.FLAGS_EXPAND_PERPENDICULAR )
self._petition_panel.Add( self._action_text, CC.FLAGS_EXPAND_PERPENDICULAR )
self._petition_panel.Add( self._reason_text, CC.FLAGS_EXPAND_PERPENDICULAR )
- self._petition_panel.Add( check_hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self._petition_panel.Add( sort_hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self._petition_panel.Add( self._contents_add, CC.FLAGS_EXPAND_PERPENDICULAR )
self._petition_panel.Add( self._contents_delete, CC.FLAGS_EXPAND_PERPENDICULAR )
+ self._petition_panel.Add( check_hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self._petition_panel.Add( self._process, CC.FLAGS_EXPAND_PERPENDICULAR )
self._petition_panel.Add( self._copy_account_key_button, CC.FLAGS_EXPAND_PERPENDICULAR )
self._petition_panel.Add( self._modify_petitioner, CC.FLAGS_EXPAND_PERPENDICULAR )
@@ -3996,7 +4078,8 @@ class ManagementPanelPetitions( ManagementPanel ):
QP.AddToLayout( vbox, self._media_sort_widget, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._media_collect_widget, CC.FLAGS_EXPAND_PERPENDICULAR )
- QP.AddToLayout( vbox, self._petitions_info_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
+ QP.AddToLayout( vbox, self._petition_numbers_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
+ QP.AddToLayout( vbox, self._petitions_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._petition_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
if service_type == HC.TAG_REPOSITORY:
@@ -4017,10 +4100,65 @@ class ManagementPanelPetitions( ManagementPanel ):
self._petition_account_key.textChanged.connect( self._UpdateAccountKey )
+ self._num_files_to_show.valueChanged.connect( self._NotifyNumsUpdated )
+ self._num_petitions_to_fetch.valueChanged.connect( self._NotifyNumsUpdated )
+
self._UpdateAccountKey()
self._DrawCurrentPetition()
+ def _ActivateToHighlightPetition( self ):
+
+ for eligible_petition_header in self._petitions_summary_list.GetData( only_selected = True ):
+
+ if self._CanHighlight( eligible_petition_header ):
+
+ self._HighlightPetition( eligible_petition_header )
+
+ break
+
+
+
+
+ def _ApproveSelected( self ):
+
+ selected_petition_headers = self._petitions_summary_list.GetData( only_selected = True )
+
+ viable_petitions = [ self._petition_headers_to_fetched_petitions_cache[ petition_header ] for petition_header in selected_petition_headers if self._CanHighlight( petition_header ) ]
+
+ if len( viable_petitions ) > 0:
+
+ text = 'Approve all the content in these {} petitions?'.format( HydrusData.ToHumanInt( len( viable_petitions ) ) )
+
+ result = ClientGUIDialogsQuick.GetYesNo( self, text )
+
+ if result == QW.QDialog.Accepted:
+
+ for petition in viable_petitions:
+
+ petition.ApproveAll()
+
+
+ self._StartUploadingCompletedPetitions( viable_petitions )
+
+
+
+
+ def _CanHighlight( self, petition_header: HydrusNetwork.PetitionHeader ):
+
+ if petition_header in self._outgoing_petition_headers_to_petitions:
+
+ return False
+
+
+ if petition_header in self._failed_outgoing_petition_headers_to_petitions:
+
+ return False
+
+
+ return petition_header in self._petition_headers_to_fetched_petitions_cache
+
+
def _CheckAll( self ):
for i in range( self._contents_add.count() ):
@@ -4047,6 +4185,101 @@ class ManagementPanelPetitions( ManagementPanel ):
+ def _ClearCurrentPetition( self ):
+
+ if self._current_petition is not None:
+
+ petition_header = self._current_petition.GetPetitionHeader()
+
+ self._current_petition = None
+
+ if self._petitions_summary_list.HasData( petition_header ):
+
+ self._petitions_summary_list.UpdateDatas( ( petition_header, ) )
+
+
+ self._DrawCurrentPetition()
+
+ self._ShowHashes( [] )
+
+
+
+ def _ClearPetitionsSummary( self ):
+
+ self._petitions_summary_list.DeleteDatas( self._petitions_summary_list.GetData() )
+
+ self._petition_headers_we_are_fetching = []
+ self._petition_headers_we_failed_to_fetch = set()
+ self._failed_outgoing_petition_headers_to_petitions = {}
+
+ self._ClearCurrentPetition()
+
+
+ def _ConvertDataToListCtrlTuples( self, petition_header: HydrusNetwork.PetitionHeader ):
+
+ pretty_action = ''
+ pretty_content = 'fetching\u2026'
+
+ sort_content = 1
+
+ petition = None
+ this_is_current_petition = False
+
+ if petition_header in self._outgoing_petition_headers_to_petitions:
+
+ petition = self._outgoing_petition_headers_to_petitions[ petition_header ]
+
+ pretty_content = 'uploading\u2026'
+
+ elif petition_header in self._failed_outgoing_petition_headers_to_petitions:
+
+ petition = self._failed_outgoing_petition_headers_to_petitions[ petition_header ]
+
+ pretty_content = 'failed to upload!'
+
+ elif petition_header in self._petition_headers_to_fetched_petitions_cache:
+
+ petition = self._petition_headers_to_fetched_petitions_cache[ petition_header ]
+
+ pretty_content = petition.GetContentSummary()
+
+ this_is_current_petition = False
+
+ if self._current_petition is not None and petition_header == self._current_petition.GetPetitionHeader():
+
+ this_is_current_petition = True
+
+
+ elif petition_header in self._petition_headers_we_failed_to_fetch:
+
+ pretty_content = 'failed to fetch!'
+
+
+ if petition is not None:
+
+ pretty_action = GetPetitionActionInfo( petition )[0]
+
+ sort_content = petition.GetActualContentWeight()
+
+
+ if this_is_current_petition:
+
+ pretty_action = f'* {pretty_action}'
+
+
+ pretty_account_key = petition_header.account_key.hex()
+ pretty_reason = petition_header.reason
+
+ sort_action = pretty_action
+ sort_account_key = pretty_account_key
+ sort_reason = pretty_reason
+
+ display_tuple = ( pretty_action, pretty_account_key, pretty_reason, pretty_content )
+ sort_tuple = ( sort_action, sort_account_key, sort_reason, sort_content )
+
+ return ( display_tuple, sort_tuple )
+
+
def _CopyAccountKey( self ):
if self._current_petition is None:
@@ -4059,6 +4292,30 @@ class ManagementPanelPetitions( ManagementPanel ):
HG.client_controller.pub( 'clipboard', 'text', account_key.hex() )
+ def _DenySelected( self ):
+
+ selected_petition_headers = self._petitions_summary_list.GetData( only_selected = True )
+
+ viable_petitions = [ self._petition_headers_to_fetched_petitions_cache[ petition_header ] for petition_header in selected_petition_headers if self._CanHighlight( petition_header ) ]
+
+ if len( viable_petitions ) > 0:
+
+ text = 'Deny all the content in these {} petitions?'.format( HydrusData.ToHumanInt( len( viable_petitions ) ) )
+
+ result = ClientGUIDialogsQuick.GetYesNo( self, text )
+
+ if result == QW.QDialog.Accepted:
+
+ for petition in viable_petitions:
+
+ petition.DenyAll()
+
+
+ self._StartUploadingCompletedPetitions( viable_petitions )
+
+
+
+
def _DrawCurrentPetition( self ):
if self._current_petition is None:
@@ -4094,29 +4351,7 @@ class ManagementPanelPetitions( ManagementPanel ):
have_add = len( add_contents ) > 0
have_delete = len( delete_contents ) > 0
- action_text = 'UNKNOWN'
- hydrus_text = 'default'
- object_name = 'normal'
-
- if have_add or have_delete:
-
- if have_add and have_delete:
-
- action_text = 'REPLACE'
-
- elif have_add:
-
- action_text = 'ADD'
- hydrus_text = 'valid'
- object_name = 'HydrusValid'
-
- else:
-
- action_text = 'DELETE'
- hydrus_text = 'invalid'
- object_name = 'HydrusInvalid'
-
-
+ ( action_text, hydrus_text, object_name ) = GetPetitionActionInfo( self._current_petition )
self._action_text.setText( action_text )
self._action_text.setObjectName( object_name )
@@ -4165,197 +4400,104 @@ class ManagementPanelPetitions( ManagementPanel ):
def _DrawNumPetitions( self ):
- if self._num_petition_info is None:
+ for ( petition_type, ( st, button ) ) in self._petition_types_to_controls.items():
- for ( petition_type, ( st, button ) ) in self._petition_types_to_controls.items():
-
- st.setText( '0 petitions' )
-
- button.setEnabled( False )
-
+ count = self._petition_types_to_count[ petition_type ]
- else:
+ ( st, button ) = self._petition_types_to_controls[ petition_type ]
- for ( content_type, status, count ) in self._num_petition_info:
-
- petition_type = ( content_type, status )
-
- if petition_type in self._petition_types_to_controls:
-
- ( st, button ) = self._petition_types_to_controls[ petition_type ]
-
- st.setText( HydrusData.ToHumanInt( count )+' petitions' )
-
- if count > 0:
-
- button.setEnabled( True )
-
- else:
-
- button.setEnabled( False )
-
-
-
+ st.setText( '{} petitions'.format( HydrusData.ToHumanInt( count ) ) )
+
+ button.setEnabled( count > 0 )
- def _FetchBestPetition( self ):
+ def _FetchBestPetitionsSummary( self ):
top_petition_type_with_count = None
- for ( content_type, status, count ) in self._num_petition_info:
+ for petition_type in self._my_petition_types:
+
+ count = self._petition_types_to_count[ petition_type ]
if count == 0:
continue
- petition_type = ( content_type, status )
+ if self._last_petition_type_fetched is not None and self._last_petition_type_fetched == petition_type:
+
+ self._FetchPetitionsSummary( petition_type )
+
+ return
+
if top_petition_type_with_count is None:
top_petition_type_with_count = petition_type
- if self._last_petition_type_fetched is not None and self._last_petition_type_fetched == petition_type:
-
- self._FetchPetition( content_type, status )
-
- return
-
-
if top_petition_type_with_count is not None:
- ( content_type, status ) = top_petition_type_with_count
-
- self._FetchPetition( content_type, status )
+ self._FetchPetitionsSummary( top_petition_type_with_count )
- def _FetchNumPetitions( self ):
+ def _FetchPetitionsSummary( self, petition_type ):
- def do_it( service, subject_account_key = None ):
-
- def qt_draw( n_p_i ):
-
- if not self or not QP.isValid( self ):
-
- return
-
-
- self._num_petition_info = n_p_i
-
- self._DrawNumPetitions()
-
- if self._current_petition is None:
-
- self._FetchBestPetition()
-
-
-
- def qt_reset():
-
- if not self or not QP.isValid( self ):
-
- return
-
-
- self._refresh_num_petitions_button.setText( 'refresh counts' )
-
-
- try:
-
- if subject_account_key is None:
-
- response = service.Request( HC.GET, 'num_petitions' )
-
- else:
-
- try:
-
- response = service.Request( HC.GET, 'num_petitions', { 'subject_account_key' : subject_account_key } )
-
- except HydrusExceptions.NotFoundException:
-
- HydrusData.ShowText( 'That account id was not found!' )
-
- QP.CallAfter( qt_draw, None )
-
- return
-
-
-
- num_petition_info = response[ 'num_petitions' ]
-
- QP.CallAfter( qt_draw, num_petition_info )
-
- finally:
-
- QP.CallAfter( qt_reset )
-
-
+ ( st, button ) = self._petition_types_to_controls[ petition_type ]
- self._refresh_num_petitions_button.setText( 'Fetching\u2026' )
+ ( content_type, status ) = petition_type
+
+ num_to_fetch = self._num_petitions_to_fetch.value()
subject_account_key = self._GetSubjectAccountKey()
- self._last_fetched_subject_account_key = subject_account_key
-
- self._controller.CallToThread( do_it, self._service, subject_account_key )
-
-
- def _FetchPetition( self, content_type, status ):
-
- self._last_petition_type_fetched = ( content_type, status )
-
- ( st, button ) = self._petition_types_to_controls[ ( content_type, status ) ]
-
- def qt_setpet( petition ):
+ def qt_set_petitions_summary( petitions_summary ):
- if not self or not QP.isValid( self ):
+ if self._last_petition_type_fetched != petition_type:
- return
+ last_petition_type = self._last_petition_type_fetched
+
+ self._last_petition_type_fetched = petition_type
+
+ self._management_controller.SetVariable( 'petition_type_content_type', content_type )
+ self._management_controller.SetVariable( 'petition_type_status', status )
+
+ self._UpdateFetchButtonText( last_petition_type )
- self._current_petition = petition
-
- self._DrawCurrentPetition()
-
- self._ShowHashes( [] )
+ self._SetPetitionsSummary( petitions_summary )
def qt_done():
- if not self or not QP.isValid( self ):
-
- return
-
-
button.setEnabled( True )
- button.setText( 'fetch {} {} petition'.format( HC.content_status_string_lookup[ status ], HC.content_type_string_lookup[ content_type ] ) )
+
+ self._UpdateFetchButtonText( self._last_petition_type_fetched )
- def do_it( service, subject_account_key = None ):
+ def do_it( service ):
try:
if subject_account_key is None:
- response = service.Request( HC.GET, 'petition', { 'content_type' : content_type, 'status' : status } )
+ response = service.Request( HC.GET, 'petitions_summary', { 'content_type' : content_type, 'status' : status, 'num' : num_to_fetch } )
else:
- response = service.Request( HC.GET, 'petition', { 'content_type' : content_type, 'status' : status, 'subject_account_key' : subject_account_key } )
+ response = service.Request( HC.GET, 'petitions_summary', { 'content_type' : content_type, 'status' : status, 'num' : num_to_fetch, 'subject_account_key' : subject_account_key } )
- QP.CallAfter( qt_setpet, response['petition'] )
+ HG.client_controller.CallBlockingToQt( self, qt_set_petitions_summary, response[ 'petitions_summary' ] )
except HydrusExceptions.NotFoundException:
job_key = ClientThreading.JobKey()
- job_key.SetStatusText( 'Hey, the server did not have a petition after all. Please hit refresh counts.' )
+ job_key.SetStatusText( 'Hey, the server did not have that type of petition after all. Please hit refresh counts.' )
job_key.Delete( 5 )
@@ -4363,25 +4505,19 @@ class ManagementPanelPetitions( ManagementPanel ):
finally:
- QP.CallAfter( qt_done )
+ HG.client_controller.CallBlockingToQt( self, qt_done )
- if self._current_petition is not None:
+ if petition_type != self._last_petition_type_fetched:
- self._current_petition = None
-
- self._DrawCurrentPetition()
-
- self._ShowHashes( [] )
+ self._ClearPetitionsSummary()
button.setEnabled( False )
button.setText( 'Fetching\u2026' )
- subject_account_key = self._GetSubjectAccountKey()
-
- self._controller.CallToThread( do_it, self._service, subject_account_key )
+ self._controller.CallToThread( do_it, self._service )
def _FlipSelected( self ):
@@ -4453,6 +4589,67 @@ class ManagementPanelPetitions( ManagementPanel ):
+ def _HighlightAPetitionIfNeeded( self ):
+
+ if self._current_petition is None:
+
+ for eligible_petition_header in self._petitions_summary_list.GetData():
+
+ if self._CanHighlight( eligible_petition_header ):
+
+ self._HighlightPetition( eligible_petition_header )
+
+ break
+
+
+
+
+
+ def _HighlightPetition( self, petition_header ):
+
+ if not self._CanHighlight( petition_header ):
+
+ return
+
+
+ if self._current_petition is not None and petition_header == self._current_petition.GetPetitionHeader():
+
+ self._ClearCurrentPetition()
+
+ elif petition_header in self._petition_headers_to_fetched_petitions_cache:
+
+ petition = self._petition_headers_to_fetched_petitions_cache[ petition_header ]
+
+ self._SetCurrentPetition( petition )
+
+
+
+ def _OnlySelectingLoadedPetitions( self ):
+
+ petition_headers = self._petitions_summary_list.GetData( only_selected = True )
+
+ if len( petition_headers ) == 0:
+
+ return False
+
+
+ for petition_header in petition_headers:
+
+ if petition_header not in self._petition_headers_to_fetched_petitions_cache:
+
+ return False
+
+
+
+ return True
+
+
+ def _NotifyNumsUpdated( self ):
+
+ self._management_controller.SetVariable( 'num_petitions_to_fetch', self._num_petitions_to_fetch.value() )
+ self._management_controller.SetVariable( 'num_files_to_show', self._num_files_to_show.GetValue() )
+
+
def _SetContentsAndChecks( self, action, contents_and_checks, sort_type ):
def key( c_and_s ):
@@ -4529,6 +4726,49 @@ class ManagementPanelPetitions( ManagementPanel ):
contents.setFixedHeight( ideal_height_in_pixels )
+ def _SetCurrentPetition( self, petition: HydrusNetwork.Petition ):
+
+ self._ClearCurrentPetition()
+
+ self._current_petition = petition
+
+ self._petitions_summary_list.UpdateDatas( ( self._current_petition.GetPetitionHeader(), ) )
+
+ self._DrawCurrentPetition()
+
+ self._ShowHashes( [] )
+
+
+ def _SetPetitionsSummary( self, petitions_summary: typing.List[ HydrusNetwork.PetitionHeader ] ):
+
+ # note we can't make this a nice 'append' so easily, since we still need to cull petitions that were processed without us looking
+ # we'll keep the current since the user is looking, but otherwise we'll be good for now
+ # maybe add a hard refresh button in future? we'll see how common these issues are
+
+ if self._current_petition is not None:
+
+ current_petition_header = self._current_petition.GetPetitionHeader()
+
+ if current_petition_header not in petitions_summary:
+
+ petitions_summary.append( current_petition_header )
+
+
+
+ self._petitions_summary_list.SetData( petitions_summary )
+
+ sorted_petition_headers = self._petitions_summary_list.GetData()
+
+ self._petition_headers_we_are_fetching = [ petition_header for petition_header in sorted_petition_headers if petition_header not in self._petition_headers_to_fetched_petitions_cache ]
+
+ if len( self._petition_headers_we_are_fetching ) > 0:
+
+ HG.client_controller.CallToThread( self.THREADPetitionFetcherAndUploader, self._petition_fetcher_and_uploader_work_lock, self._service )
+
+
+ self._HighlightAPetitionIfNeeded()
+
+
def _ShowHashes( self, hashes ):
with ClientGUICommon.BusyCursor():
@@ -4555,6 +4795,110 @@ class ManagementPanelPetitions( ManagementPanel ):
+ def _StartFetchNumPetitions( self ):
+
+ def do_it( service, subject_account_key = None ):
+
+ def qt_draw( petition_count_rows ):
+
+ if not self or not QP.isValid( self ):
+
+ return
+
+
+ num_petitions_currently_listed = len( self._petitions_summary_list.GetData() )
+
+ old_petition_types_to_count = self._petition_types_to_count
+
+ self._petition_types_to_count = collections.Counter()
+
+ # we had a whole thing here that did 'if count dropped by more than 1, refresh summary' and 'if we only have 20% left of our desired count, refresh summary'
+ # but the count from the server and the count of what we see differs for mappings, where petitions are bunched, and it was just a pain
+ # maybe try again later, with better counting tech and more experience of what is actually wanted here
+
+ for ( content_type, status, count ) in petition_count_rows:
+
+ petition_type = ( content_type, status )
+
+ self._petition_types_to_count[ petition_type ] = count
+
+
+ self._DrawNumPetitions()
+
+ if num_petitions_currently_listed == 0:
+
+ self._FetchBestPetitionsSummary()
+
+
+
+ def qt_reset():
+
+ if not self or not QP.isValid( self ):
+
+ return
+
+
+ self._refresh_num_petitions_button.setText( 'refresh counts' )
+
+
+ try:
+
+ if subject_account_key is None:
+
+ response = service.Request( HC.GET, 'num_petitions' )
+
+ else:
+
+ try:
+
+ response = service.Request( HC.GET, 'num_petitions', { 'subject_account_key' : subject_account_key } )
+
+ except HydrusExceptions.NotFoundException:
+
+ HydrusData.ShowText( 'That account id was not found!' )
+
+ QP.CallAfter( qt_draw, [] )
+
+ return
+
+
+
+ num_petition_info = response[ 'num_petitions' ]
+
+ QP.CallAfter( qt_draw, num_petition_info )
+
+ finally:
+
+ QP.CallAfter( qt_reset )
+
+
+
+ self._refresh_num_petitions_button.setText( 'Fetching\u2026' )
+
+ subject_account_key = self._GetSubjectAccountKey()
+
+ self._last_fetched_subject_account_key = subject_account_key
+
+ self._controller.CallToThread( do_it, self._service, subject_account_key )
+
+
+ def _StartUploadingCompletedPetitions( self, petitions: typing.Collection[ HydrusNetwork.Petition ] ):
+
+ for petition in petitions:
+
+ self._outgoing_petition_headers_to_petitions[ petition.GetPetitionHeader() ] = petition
+
+ if petition == self._current_petition:
+
+ self._ClearCurrentPetition()
+
+
+
+ self._HighlightAPetitionIfNeeded()
+
+ HG.client_controller.CallToThread( self.THREADPetitionFetcherAndUploader, self._petition_fetcher_and_uploader_work_lock, self._service )
+
+
def _UpdateAccountKey( self ):
account_key_hex = self._petition_account_key.text()
@@ -4588,7 +4932,7 @@ class ManagementPanelPetitions( ManagementPanel ):
if self._GetSubjectAccountKey() != self._last_fetched_subject_account_key:
- self._FetchNumPetitions()
+ self._StartFetchNumPetitions()
else:
@@ -4599,6 +4943,25 @@ class ManagementPanelPetitions( ManagementPanel ):
self._petition_account_key.style().polish( self._petition_account_key )
+ def _UpdateFetchButtonText( self, petition_type ):
+
+ if petition_type is not None:
+
+ ( st, button ) = self._petition_types_to_controls[ petition_type ]
+
+ ( content_type, status ) = petition_type
+
+ label = 'fetch {} {} petitions'.format( HC.content_status_string_lookup[ status ], HC.content_type_string_lookup[ content_type ] )
+
+ if petition_type == self._last_petition_type_fetched:
+
+ label = f'{label} (*)'
+
+
+ button.setText( label )
+
+
+
def ContentsAddDoubleClick( self, item ):
selected_indices = self._contents_add.GetSelectedIndices()
@@ -4644,181 +5007,6 @@ class ManagementPanelPetitions( ManagementPanel ):
- def EventProcess( self ):
-
- def break_contents_into_chunks( some_contents ):
-
- chunks_of_some_contents = []
- chunk_of_some_contents = []
-
- weight = 0
-
- for content in some_contents:
-
- for content_chunk in content.IterateUploadableChunks(): # break 20K-strong mappings petitions into smaller bits to POST back
-
- chunk_of_some_contents.append( content_chunk )
-
- weight += content.GetVirtualWeight()
-
- if weight > 50:
-
- chunks_of_some_contents.append( chunk_of_some_contents )
-
- chunk_of_some_contents = []
-
- weight = 0
-
-
-
-
- if len( chunk_of_some_contents ) > 0:
-
- chunks_of_some_contents.append( chunk_of_some_contents )
-
-
- return chunks_of_some_contents
-
-
- def do_it( controller, service, petition_service_key, add_approved_contents, add_denied_contents, delete_approved_contents, delete_denied_contents, petition ):
-
- jobs = [
- ( HC.CONTENT_UPDATE_PEND, True, add_approved_contents ),
- ( HC.CONTENT_UPDATE_PEND, False, add_denied_contents ),
- ( HC.CONTENT_UPDATE_PETITION, True, delete_approved_contents ),
- ( HC.CONTENT_UPDATE_PETITION, False, delete_denied_contents ),
- ]
-
- num_done = 0
- num_to_do = 0
-
- for ( action, approved, contents ) in jobs:
-
- num_to_do += len( contents )
-
-
- if num_to_do > 1:
-
- job_key = ClientThreading.JobKey( cancellable = True )
-
- job_key.SetStatusTitle( 'committing petitions' )
-
- HG.client_controller.pub( 'message', job_key )
-
- else:
-
- job_key = None
-
-
- reason = petition.GetReason()
-
- try:
-
- for ( action, approved, contents ) in jobs:
-
- if len( contents ) == 0:
-
- continue
-
-
- chunks_of_contents = break_contents_into_chunks( contents )
-
- num_to_do += len( chunks_of_contents ) - 1
-
- for chunk_of_contents in chunks_of_contents:
-
- if job_key is not None:
-
- ( i_paused, should_quit ) = job_key.WaitIfNeeded()
-
- if should_quit:
-
- return
-
-
- job_key.SetVariable( 'popup_gauge_1', ( num_done, num_to_do ) )
-
-
- content_updates = []
-
- if approved:
-
- ( update, content_updates ) = petition.GetApproval( action, chunk_of_contents, reason )
-
- else:
-
- update = petition.GetDenial( action, chunk_of_contents, reason )
-
-
- service.Request( HC.POST, 'update', { 'client_to_server_update' : update } )
-
- if len( content_updates ) > 0:
-
- controller.WriteSynchronous( 'content_updates', { petition_service_key : content_updates } )
-
-
- num_done += 1
-
-
-
- finally:
-
- if job_key is not None:
-
- job_key.Delete()
-
-
- def qt_fetch():
-
- if not self or not QP.isValid( self ):
-
- return
-
-
- self._FetchNumPetitions()
-
-
- QP.CallAfter( qt_fetch )
-
-
-
- add_approved_contents = []
- add_denied_contents = []
-
- delete_approved_contents = []
- delete_denied_contents = []
-
- jobs = [
- ( self._contents_add, add_approved_contents, add_denied_contents ),
- ( self._contents_delete, delete_approved_contents, delete_denied_contents )
- ]
-
- for ( contents, approved_contents, denied_contents ) in jobs:
-
- for index in range( contents.count() ):
-
- content = contents.GetData( index )
-
- if contents.IsChecked( index ):
-
- approved_contents.append( content )
-
- else:
-
- denied_contents.append( content )
-
-
-
-
- HG.client_controller.CallToThread( do_it, self._controller, self._service, self._petition_service_key, add_approved_contents, add_denied_contents, delete_approved_contents, delete_denied_contents, self._current_petition )
-
- self._current_petition = None
-
- self._DrawCurrentPetition()
-
- self._ShowHashes( [] )
-
-
def EventModifyPetitioner( self ):
subject_account_key = self._current_petition.GetPetitionerAccount().GetAccountKey()
@@ -4918,6 +5106,45 @@ class ManagementPanelPetitions( ManagementPanel ):
+ def PageShown( self ):
+
+ ManagementPanel.PageShown( self )
+
+ HG.client_controller.CallToThread( self.THREADPetitionFetcherAndUploader, self._petition_fetcher_and_uploader_work_lock, self._service )
+
+
+ def ProcessCurrentPetition( self ):
+
+ if self._current_petition is None:
+
+ return
+
+
+ jobs = [
+ ( self._contents_add, HC.CONTENT_UPDATE_PEND ),
+ ( self._contents_delete, HC.CONTENT_UPDATE_PETITION )
+ ]
+
+ for ( contents_list, action ) in jobs:
+
+ for index in range( contents_list.count() ):
+
+ content = contents_list.GetData( index )
+
+ if contents_list.IsChecked( index ):
+
+ self._current_petition.Approve( action, content )
+
+ else:
+
+ self._current_petition.Deny( action, content )
+
+
+
+
+ self._StartUploadingCompletedPetitions( ( self._current_petition, ) )
+
+
def RefreshQuery( self ):
self._DrawCurrentPetition()
@@ -4925,9 +5152,239 @@ class ManagementPanelPetitions( ManagementPanel ):
def Start( self ):
- QP.CallAfter( self._FetchNumPetitions )
+ QP.CallAfter( self._StartFetchNumPetitions )
+ def THREADPetitionFetcherAndUploader( self, work_lock: threading.Lock, service: ClientServices.ServiceRepository ):
+
+ def qt_get_work():
+
+ fetch_petition_header = None
+ outgoing_petition = None
+
+ if len( self._petition_headers_we_are_fetching ) > 0:
+
+ if HG.client_controller.PageAliveAndNotClosed( self._page_key ):
+
+ fetch_petition_header = self._petition_headers_we_are_fetching[0]
+
+ elif HG.client_controller.PageDestroyed( self._page_key ):
+
+ self._petition_headers_we_are_fetching = []
+
+
+
+ if len( self._outgoing_petition_headers_to_petitions ) > 0:
+
+ item = list( self._outgoing_petition_headers_to_petitions.keys() )[0]
+
+ outgoing_petition = self._outgoing_petition_headers_to_petitions[ item ]
+
+
+ return ( fetch_petition_header, outgoing_petition )
+
+
+ def qt_petition_cleared( petition: HydrusNetwork.Petition ):
+
+ petition_header = petition.GetPetitionHeader()
+
+ if petition_header in self._outgoing_petition_headers_to_petitions:
+
+ del self._outgoing_petition_headers_to_petitions[ petition_header ]
+
+
+ if petition_header in self._failed_outgoing_petition_headers_to_petitions:
+
+ del self._failed_outgoing_petition_headers_to_petitions[ petition_header ]
+
+
+ if petition_header in self._petition_headers_to_fetched_petitions_cache:
+
+ del self._petition_headers_to_fetched_petitions_cache[ petition_header ]
+
+
+ if self._petitions_summary_list.HasData( petition_header ):
+
+ self._petitions_summary_list.DeleteDatas( ( petition_header, ) )
+
+
+ self._StartFetchNumPetitions()
+
+
+ def qt_petition_clear_failed( petition: HydrusNetwork.Petition ):
+
+ petition_header = petition.GetPetitionHeader()
+
+ if petition_header in self._outgoing_petition_headers_to_petitions:
+
+ del self._outgoing_petition_headers_to_petitions[ petition_header ]
+
+
+ self._failed_outgoing_petition_headers_to_petitions[ petition_header ] = petition
+
+ if self._petitions_summary_list.HasData( petition_header ):
+
+ self._petitions_summary_list.UpdateDatas( ( petition_header, ) )
+
+
+
+ def qt_petition_fetch_404( petition_header: HydrusNetwork.PetitionHeader ):
+
+ if petition_header in self._petition_headers_we_are_fetching:
+
+ self._petition_headers_we_are_fetching.remove( petition_header )
+
+
+ if self._petitions_summary_list.HasData( petition_header ):
+
+ self._petitions_summary_list.DeleteDatas( ( petition_header, ) )
+
+
+
+ def qt_petition_fetch_failed( petition_header: HydrusNetwork.PetitionHeader ):
+
+ if petition_header in self._petition_headers_we_are_fetching:
+
+ self._petition_headers_we_are_fetching.remove( petition_header )
+
+
+ self._petition_headers_we_failed_to_fetch.add( petition_header )
+
+ if self._petitions_summary_list.HasData( petition_header ):
+
+ self._petitions_summary_list.UpdateDatas( ( petition_header, ) )
+
+
+
+ def qt_petition_fetched( petition: HydrusNetwork.Petition ):
+
+ petition_header = petition.GetPetitionHeader()
+
+ if petition_header in self._petition_headers_we_are_fetching:
+
+ self._petition_headers_we_are_fetching.remove( petition_header )
+
+
+ self._petition_headers_we_failed_to_fetch.discard( petition_header )
+
+ if self._petitions_summary_list.HasData( petition_header ):
+
+ self._petition_headers_to_fetched_petitions_cache[ petition_header ] = petition
+
+ self._petitions_summary_list.UpdateDatas( ( petition_header, ) )
+
+
+ if self._current_petition is None:
+
+ self._HighlightAPetitionIfNeeded()
+
+
+
+ with work_lock:
+
+ while True:
+
+ fetch_petition_header = None
+ outgoing_petition = None
+
+ ( fetch_petition_header, outgoing_petition ) = HG.client_controller.CallBlockingToQt( self, qt_get_work )
+
+ if fetch_petition_header is None and outgoing_petition is None:
+
+ break
+
+
+ if fetch_petition_header is not None:
+
+ try:
+
+ request_dict = {
+ 'content_type' : fetch_petition_header.content_type,
+ 'status' : fetch_petition_header.status,
+ 'subject_account_key' : fetch_petition_header.account_key,
+ 'reason' : fetch_petition_header.reason
+ }
+
+ response = service.Request( HC.GET, 'petition', request_dict )
+
+ petition = response[ 'petition' ]
+
+ HG.client_controller.CallBlockingToQt( self, qt_petition_fetched, petition )
+
+ except HydrusExceptions.NotFoundException:
+
+ HG.client_controller.CallBlockingToQt( self, qt_petition_fetch_404, fetch_petition_header )
+
+ except Exception as e:
+
+ HydrusData.ShowText( 'Failed to fetch a petition!' )
+ HydrusData.ShowException( e )
+
+ HG.client_controller.CallBlockingToQt( self, qt_petition_fetch_failed, fetch_petition_header )
+
+
+
+ if outgoing_petition is not None:
+
+ try:
+
+ job_key = ClientThreading.JobKey( cancellable = True )
+
+ job_key.SetStatusTitle( 'committing petition' )
+
+ time_started = HydrusTime.GetNowFloat()
+
+ try:
+
+ updates_and_content_updates = outgoing_petition.GetAllCompletedUpdates()
+
+ num_to_do = len( updates_and_content_updates )
+
+ for ( num_done, ( update, content_updates ) ) in enumerate( updates_and_content_updates ):
+
+ if HydrusTime.TimeHasPassed( time_started + 3 ):
+
+ HG.client_controller.pub( 'message', job_key )
+
+
+ ( i_paused, should_quit ) = job_key.WaitIfNeeded()
+
+ if should_quit:
+
+ return
+
+
+ service.Request( HC.POST, 'update', { 'client_to_server_update' : update } )
+
+ if len( content_updates ) > 0:
+
+ HG.client_controller.WriteSynchronous( 'content_updates', { service.GetServiceKey() : content_updates } )
+
+
+ job_key.SetStatusText( HydrusData.ConvertValueRangeToPrettyString( num_done, num_to_do ) )
+ job_key.SetVariable( 'popup_gauge_1', ( num_done, num_to_do ) )
+
+
+ finally:
+
+ job_key.Delete()
+
+
+ HG.client_controller.CallBlockingToQt( self, qt_petition_cleared, outgoing_petition )
+
+ except Exception as e:
+
+ HydrusData.ShowText( 'Failed to upload a petition!' )
+ HydrusData.ShowException( e )
+
+ HG.client_controller.CallBlockingToQt( self, qt_petition_clear_failed, outgoing_petition )
+
+
+
+
+
+
+
management_panel_types_to_classes[ ClientGUIManagementController.MANAGEMENT_TYPE_PETITIONS ] = ManagementPanelPetitions
class ManagementPanelQuery( ManagementPanel ):
diff --git a/hydrus/client/gui/widgets/ClientGUICommon.py b/hydrus/client/gui/widgets/ClientGUICommon.py
index 816f829a..1e1ea047 100644
--- a/hydrus/client/gui/widgets/ClientGUICommon.py
+++ b/hydrus/client/gui/widgets/ClientGUICommon.py
@@ -1446,7 +1446,7 @@ class ListBook( QW.QWidget ):
class NoneableSpinCtrl( QW.QWidget ):
-
+
valueChanged = QC.Signal()
def __init__( self, parent, message = '', none_phrase = 'no limit', min = 0, max = 1000000, unit = None, multiplier = 1, num_dimensions = 1 ):
diff --git a/hydrus/client/networking/ClientLocalServerResources.py b/hydrus/client/networking/ClientLocalServerResources.py
index b627c1d8..fea3f921 100644
--- a/hydrus/client/networking/ClientLocalServerResources.py
+++ b/hydrus/client/networking/ClientLocalServerResources.py
@@ -702,7 +702,7 @@ def ParseHashes( request: HydrusServerRequest.HydrusRequest ):
if len( hash_ids_to_hashes ) > 0:
- hashes.extend( hash_ids_to_hashes[ hash_id ] )
+ hashes.append(hash_ids_to_hashes[ hash_id ])
diff --git a/hydrus/core/HydrusArchiveHandling.py b/hydrus/core/HydrusArchiveHandling.py
new file mode 100644
index 00000000..fae3eb7b
--- /dev/null
+++ b/hydrus/core/HydrusArchiveHandling.py
@@ -0,0 +1,26 @@
+import zipfile
+
+def ExtractSingleFileFromZip( path_to_zip, filename_to_extract, extract_into_file_path ):
+
+ with zipfile.ZipFile( path_to_zip ) as zip_handle:
+
+ with zip_handle.open( filename_to_extract ) as reader:
+
+ with open( extract_into_file_path, "wb" ) as writer:
+
+ writer.write( reader.read() )
+
+
+
+
+
+def ReadSingleFileFromZip( path_to_zip, filename_to_extract ):
+
+ with zipfile.ZipFile( path_to_zip ) as zip_handle:
+
+ with zip_handle.open( filename_to_extract ) as reader:
+
+ return reader.read()
+
+
+
diff --git a/hydrus/core/HydrusConstants.py b/hydrus/core/HydrusConstants.py
index 79c7286e..78cfca77 100644
--- a/hydrus/core/HydrusConstants.py
+++ b/hydrus/core/HydrusConstants.py
@@ -100,7 +100,7 @@ options = {}
# Misc
NETWORK_VERSION = 20
-SOFTWARE_VERSION = 534
+SOFTWARE_VERSION = 535
CLIENT_API_VERSION = 48
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@@ -795,6 +795,7 @@ mime_enum_lookup = {
'application/x-shockwave-flash' : APPLICATION_FLASH,
'application/x-photoshop' : APPLICATION_PSD,
'image/vnd.adobe.photoshop' : APPLICATION_PSD,
+ 'application/vnd.adobe.photoshop' : APPLICATION_PSD,
'application/clip' : APPLICATION_CLIP,
'application/sai2': APPLICATION_SAI2,
'application/x-krita': APPLICATION_KRITA,
@@ -857,10 +858,10 @@ mime_string_lookup = {
APPLICATION_JSON : 'json',
APPLICATION_CBOR : 'cbor',
APPLICATION_PDF : 'pdf',
- APPLICATION_PSD : 'photoshop psd',
+ APPLICATION_PSD : 'psd',
APPLICATION_CLIP : 'clip',
APPLICATION_SAI2 : 'sai2',
- APPLICATION_KRITA : 'kra',
+ APPLICATION_KRITA : 'krita',
APPLICATION_XCF : 'xcf',
APPLICATION_ZIP : 'zip',
APPLICATION_RAR : 'rar',
@@ -920,7 +921,7 @@ mime_mimetype_string_lookup = {
APPLICATION_JSON : 'application/json',
APPLICATION_CBOR : 'application/cbor',
APPLICATION_PDF : 'application/pdf',
- APPLICATION_PSD : 'application/x-photoshop',
+ APPLICATION_PSD : 'image/vnd.adobe.photoshop',
APPLICATION_CLIP : 'application/clip',
APPLICATION_SAI2: 'application/sai2',
APPLICATION_KRITA: 'application/x-krita',
diff --git a/hydrus/core/HydrusFileHandling.py b/hydrus/core/HydrusFileHandling.py
index 3c130cf2..60b6fa25 100644
--- a/hydrus/core/HydrusFileHandling.py
+++ b/hydrus/core/HydrusFileHandling.py
@@ -5,7 +5,18 @@ import struct
from hydrus.core import HydrusAudioHandling
from hydrus.core import HydrusClipHandling
from hydrus.core import HydrusKritaHandling
-from hydrus.core import HydrusSVGHandling
+
+try:
+
+ from hydrus.core import HydrusSVGHandling
+
+ SVG_OK = True
+
+except:
+
+ SVG_OK = False
+
+
from hydrus.core import HydrusConstants as HC
from hydrus.core import HydrusData
from hydrus.core import HydrusDocumentHandling
@@ -185,6 +196,11 @@ def GenerateThumbnailBytes( path, target_resolution, mime, duration, num_frames,
try:
+ if not SVG_OK:
+
+ raise Exception( 'No SVG thumbs' )
+
+
thumbnail_bytes = HydrusSVGHandling.GenerateThumbnailBytesFromSVGPath( path, target_resolution, clip_rect = clip_rect )
except Exception as e:
@@ -361,11 +377,14 @@ def GetFileInfo( path, mime = None, ok_to_look_for_hydrus_updates = False ):
elif mime == HC.APPLICATION_KRITA:
( width, height ) = HydrusKritaHandling.GetKraProperties( path )
-
+
elif mime == HC.IMAGE_SVG:
-
- ( width, height ) = HydrusSVGHandling.GetSVGResolution( path )
-
+
+ if SVG_OK:
+
+ ( width, height ) = HydrusSVGHandling.GetSVGResolution( path )
+
+
elif mime == HC.APPLICATION_FLASH:
( ( width, height ), duration, num_frames ) = HydrusFlashHandling.GetFlashProperties( path )
@@ -486,6 +505,18 @@ def GetMime( path, ok_to_look_for_hydrus_updates = False ):
if it_passes:
+ if mime == HC.APPLICATION_ZIP:
+
+ # TODO: since we'll be expanding this to other zip-likes, we should make the zipfile object up here and pass that to various checkers downstream
+ if HydrusKritaHandling.ZipLooksLikeAKrita( path ):
+
+ return HC.APPLICATION_KRITA
+
+ else:
+
+ return HC.APPLICATION_ZIP
+
+
if mime in ( HC.UNDETERMINED_WM, HC.UNDETERMINED_MP4 ):
return HydrusVideoHandling.GetMime( path )
@@ -512,9 +543,11 @@ def GetMime( path, ok_to_look_for_hydrus_updates = False ):
return HC.TEXT_HTML
+
if HydrusText.LooksLikeSVG( bit_to_check ):
return HC.IMAGE_SVG
+
# it is important this goes at the end, because ffmpeg has a billion false positives!
# for instance, it once thought some hydrus update files were mpegs
diff --git a/hydrus/core/HydrusKritaHandling.py b/hydrus/core/HydrusKritaHandling.py
index 94def9e0..8737fab2 100644
--- a/hydrus/core/HydrusKritaHandling.py
+++ b/hydrus/core/HydrusKritaHandling.py
@@ -1,93 +1,99 @@
-
+from hydrus.core import HydrusArchiveHandling
from hydrus.core import HydrusExceptions
from hydrus.core import HydrusTemp
-import zipfile
import re
KRITA_FILE_THUMB = "preview.png"
KRITA_FILE_MERGED = "mergedimage.png"
-
-
-def ExtractSingleFileFromZip( path_to_zip, filename_to_extract, extract_into_file_path ):
-
- with zipfile.ZipFile( path_to_zip ) as zip_handle:
-
- with zip_handle.open( filename_to_extract ) as reader:
-
- with open( extract_into_file_path, "wb" ) as writer:
-
- writer.write( reader.read() )
-
+KRITA_MIMETYPE = 'mimetype'
def ExtractZippedImageToPath( path_to_zip, temp_path_file ):
-
+
try:
- ExtractSingleFileFromZip( path_to_zip, KRITA_FILE_MERGED, temp_path_file )
-
+ HydrusArchiveHandling.ExtractSingleFileFromZip( path_to_zip, KRITA_FILE_MERGED, temp_path_file )
+
return
except KeyError:
-
- pass
-
+ pass
+
+
try:
- ExtractSingleFileFromZip( path_to_zip, KRITA_FILE_THUMB, temp_path_file )
+ HydrusArchiveHandling.ExtractSingleFileFromZip( path_to_zip, KRITA_FILE_THUMB, temp_path_file )
except KeyError:
-
+
raise HydrusExceptions.DamagedOrUnusualFileException( f'This krita file had no {KRITA_FILE_MERGED} or {KRITA_FILE_THUMB}, so no PNG thumb could be extracted!' )
-
+
+
# TODO: animation and frame stuff which is also in the maindoc.xml
def GetKraProperties( path ):
-
+
( os_file_handle, maindoc_xml ) = HydrusTemp.GetTempPath()
-
+
DOCUMENT_INFO_FILE = "maindoc.xml"
-
+
# TODO: probably actually parse the xml instead of using regex
- FIND_KEY_VALUE = re.compile(r"([a-z\-\_]+)\s*=\s*['\"]([^'\"]+)", re.IGNORECASE)
-
+ FIND_KEY_VALUE = re.compile(r"([a-z\-_]+)\s*=\s*['\"]([^'\"]+)", re.IGNORECASE)
+
width = None
height = None
-
+
try:
-
- ExtractSingleFileFromZip( path, DOCUMENT_INFO_FILE, maindoc_xml )
-
+
+ HydrusArchiveHandling.ExtractSingleFileFromZip( path, DOCUMENT_INFO_FILE, maindoc_xml )
+
with open(maindoc_xml, "r") as reader:
-
+
for line in reader:
-
+
for match in FIND_KEY_VALUE.findall( line ):
-
+
key, value = match
-
+
if key == "width" and value.isdigit():
-
+
width = int(value)
-
+
if key == "height" and value.isdigit():
-
+
height = int(value)
-
+
if width is not None and height is not None:
-
+
break
-
+
+
+
+
+
except KeyError:
-
+
raise HydrusExceptions.DamagedOrUnusualFileException( f'This krita file had no {DOCUMENT_INFO_FILE}, so no information could be extracted!' )
-
+
finally:
-
+
HydrusTemp.CleanUpTempPath( os_file_handle, maindoc_xml )
-
+
+
return width, height
+
-
-
+def ZipLooksLikeAKrita( path ):
+
+ try:
+
+ mimetype_data = HydrusArchiveHandling.ReadSingleFileFromZip( path, KRITA_MIMETYPE )
+
+ return b'application/x-krita' in mimetype_data
+
+ except KeyError:
+
+ return False
+
+
diff --git a/hydrus/core/HydrusSerialisable.py b/hydrus/core/HydrusSerialisable.py
index 11dc126f..332fbd2c 100644
--- a/hydrus/core/HydrusSerialisable.py
+++ b/hydrus/core/HydrusSerialisable.py
@@ -139,6 +139,7 @@ SERIALISABLE_TYPE_METADATA_SINGLE_FILE_IMPORTER_MEDIA_NOTES = 120
SERIALISABLE_TYPE_TIMESTAMP_DATA = 121
SERIALISABLE_TYPE_METADATA_SINGLE_FILE_EXPORTER_MEDIA_TIMESTAMPS = 122
SERIALISABLE_TYPE_METADATA_SINGLE_FILE_IMPORTER_MEDIA_TIMESTAMPS = 123
+SERIALISABLE_TYPE_PETITION_HEADER = 124
SERIALISABLE_TYPES_TO_OBJECT_TYPES = {}
diff --git a/hydrus/core/HydrusText.py b/hydrus/core/HydrusText.py
index 77ec56c4..4ffc0f47 100644
--- a/hydrus/core/HydrusText.py
+++ b/hydrus/core/HydrusText.py
@@ -114,8 +114,7 @@ def LooksLikeHTML( file_data ):
return False
def LooksLikeSVG( file_data ):
-
-
+
if isinstance( file_data, bytes ):
search_elements = ( b' 50:
+
+ chunks_of_some_contents.append( chunk_of_some_contents )
+
+ chunk_of_some_contents = []
+
+ weight_of_current_chunk = 0
+
+
+
+
+ if len( chunk_of_some_contents ) > 0:
+
+ chunks_of_some_contents.append( chunk_of_some_contents )
+
+
+ return chunks_of_some_contents
+
+
+ updates_and_content_updates = []
+
+ # make sure you delete before you add
+ for action in ( HC.CONTENT_UPDATE_DENY_PETITION, HC.CONTENT_UPDATE_DENY_PEND, HC.CONTENT_UPDATE_PETITION, HC.CONTENT_UPDATE_PEND ):
+
+ contents = self._completed_actions_to_contents[ action ]
+
+ if len( contents ) == 0:
+
+ continue
+
+
+ if action == HC.CONTENT_UPDATE_PEND:
+
+ content_update_action = HC.CONTENT_UPDATE_ADD
+
+ elif action == HC.CONTENT_UPDATE_PETITION:
+
+ content_update_action = HC.CONTENT_UPDATE_DELETE
+
+ else:
+
+ content_update_action = None
+
+
+ chunks_of_contents = break_contents_into_chunks( contents )
+
+ for chunk_of_contents in chunks_of_contents:
+
+ update = ClientToServerUpdate()
+ content_updates = []
+
+ for content in chunk_of_contents:
+
+ update.AddContent( action, content, self._petition_header.reason )
+
+ if content_update_action is not None:
+
+ content_type = content.GetContentType()
+
+ row = content.GetContentData()
+
+ content_update = HydrusData.ContentUpdate( content_type, content_update_action, row )
+
+ content_updates.append( content_update )
+
+
+
+ updates_and_content_updates.append( ( update, content_updates ) )
+
+
+
+ return updates_and_content_updates
+
+
def GetContents( self, action ):
actions_to_contents = dict( self._actions_and_contents )
@@ -2390,74 +2567,116 @@ class Petition( HydrusSerialisable.SerialisableBase ):
return self._petitioner_account
+ def GetPetitionHeader( self ) -> "PetitionHeader":
+
+ return self._petition_header
+
+
def GetReason( self ):
- return self._reason
+ return self._petition_header.reason
- @staticmethod
- def GetApproval( action, contents, reason ):
+ def GetActualContentWeight( self ) -> int:
- update = ClientToServerUpdate()
- content_updates = []
+ total_weight = 0
- if action == HC.CONTENT_UPDATE_PEND:
+ for ( action, contents ) in self._actions_and_contents:
- content_update_action = HC.CONTENT_UPDATE_ADD
+ for content in contents:
+
+ total_weight += content.GetActualWeight()
+
- elif action == HC.CONTENT_UPDATE_PETITION:
+
+ return total_weight
+
+
+ def GetContentSummary( self ) -> str:
+
+ num_sub_petitions = sum( ( len( contents ) for ( action, contents ) in self._actions_and_contents ) )
+
+ if self._petition_header.content_type == HC.CONTENT_TYPE_MAPPINGS and num_sub_petitions > 1:
- content_update_action = HC.CONTENT_UPDATE_DELETE
+ return '{} mappings in {} petitions'.format( HydrusData.ToHumanInt( self.GetActualContentWeight() ), HydrusData.ToHumanInt( num_sub_petitions ) )
else:
- raise Exception( 'Petition came with unexpected action: {}'.format( action ) )
+ return '{} {}'.format( HydrusData.ToHumanInt( self.GetActualContentWeight() ), HC.content_type_string_lookup[ self._petition_header.content_type ] )
- for content in contents:
-
- update.AddContent( action, content, reason )
-
- content_type = content.GetContentType()
-
- row = content.GetContentData()
-
- content_update = HydrusData.ContentUpdate( content_type, content_update_action, row )
-
- content_updates.append( content_update )
-
-
- return ( update, content_updates )
-
-
- @staticmethod
- def GetDenial( action, contents, reason ):
-
- update = ClientToServerUpdate()
-
- if action == HC.CONTENT_UPDATE_PEND:
-
- denial_action = HC.CONTENT_UPDATE_DENY_PEND
-
- elif action == HC.CONTENT_UPDATE_PETITION:
-
- denial_action = HC.CONTENT_UPDATE_DENY_PETITION
-
- else:
-
- raise Exception( 'Petition came with unexpected action: {}'.format( action ) )
-
-
- for content in contents:
-
- update.AddContent( denial_action, content, reason )
-
-
- return update
-
+
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_PETITION ] = Petition
+class PetitionHeader( HydrusSerialisable.SerialisableBase ):
+
+ SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_PETITION_HEADER
+ SERIALISABLE_NAME = 'Petitions Header'
+ SERIALISABLE_VERSION = 1
+
+ def __init__( self, content_type = None, status = None, account_key = None, reason = None ):
+
+ if content_type is None:
+
+ content_type = HC.CONTENT_TYPE_MAPPINGS
+
+
+ if status is None:
+
+ status = HC.CONTENT_STATUS_PETITIONED
+
+
+ if account_key is None:
+
+ account_key = b''
+
+
+ if reason is None:
+
+ reason = ''
+
+
+ HydrusSerialisable.SerialisableBase.__init__( self )
+
+ self.content_type = content_type
+ self.status = status
+ self.account_key = account_key
+ self.reason = reason
+
+
+ def __eq__( self, other ):
+
+ if isinstance( other, PetitionHeader ):
+
+ return self.__hash__() == other.__hash__()
+
+
+ return NotImplemented
+
+
+ def __hash__( self ):
+
+ return ( self.content_type, self.status, self.account_key, self.reason ).__hash__()
+
+
+ def _GetSerialisableInfo( self ):
+
+ serialisable_account_key = self.account_key.hex()
+
+ return ( self.content_type, self.status, serialisable_account_key, self.reason )
+
+
+ def _InitialiseFromSerialisableInfo( self, serialisable_info ):
+
+ ( self.content_type, self.status, serialisable_account_key, self.reason ) = serialisable_info
+
+ self.account_key = bytes.fromhex( serialisable_account_key )
+
+
+
+HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_PETITION_HEADER ] = PetitionHeader
+
class ServerService( object ):
def __init__( self, service_key, service_type, name, port, dictionary ):
diff --git a/hydrus/server/ServerDB.py b/hydrus/server/ServerDB.py
index e96ec058..2004c3de 100644
--- a/hydrus/server/ServerDB.py
+++ b/hydrus/server/ServerDB.py
@@ -109,6 +109,7 @@ class DB( HydrusDB.HydrusDB ):
'is_an_orphan' : self._IsAnOrphan,
'num_petitions' : self._RepositoryGetNumPetitions,
'petition' : self._RepositoryGetPetition,
+ 'petitions_summary' : self._RepositoryGetPetitionsSummary,
'registration_keys' : self._GenerateRegistrationKeysFromAccount,
'service_has_file' : self._RepositoryHasFile,
'service_info' : self._GetServiceInfo,
@@ -1929,6 +1930,58 @@ class DB( HydrusDB.HydrusDB ):
self._RepositoryUpdateServiceInfo( service_id, HC.SERVICE_INFO_NUM_TAG_SIBLINGS, self._GetRowCount() )
+ def _RepositoryConvertPetitionIdsToSummary( self, content_type: int, status: int, petition_ids: typing.Collection[ typing.Tuple[ int, int ] ], limit: int ) -> typing.List[ HydrusNetwork.PetitionHeader ]:
+
+ if len( petition_ids ) == 0:
+
+ raise HydrusExceptions.NotFoundException( 'No petitions!' )
+
+
+ if len( petition_ids ) > limit:
+
+ # we don't want a random sample, we want to sample grouped by account id, so let's be a bit more clever about it
+
+ petitioner_account_ids_to_reason_ids = HydrusData.BuildKeyToListDict( petition_ids )
+
+ petition_ids = []
+
+ num_rows = 0
+
+ petition_account_ids = list( petitioner_account_ids_to_reason_ids.keys() )
+
+ random.shuffle( petition_account_ids )
+
+ for petition_account_id in petition_account_ids:
+
+ reason_ids = petitioner_account_ids_to_reason_ids[ petition_account_id ]
+
+ if num_rows + len( reason_ids ) > limit:
+
+ num_to_add = limit - num_rows
+
+ reason_ids = reason_ids[ : num_to_add ]
+
+
+ petition_ids.extend( ( ( petition_account_id, reason_id ) for reason_id in reason_ids ) )
+
+ num_rows += len( reason_ids )
+
+ if num_rows >= limit:
+
+ break
+
+
+
+
+ petitioner_account_ids = { petitioner_account_id for ( petitioner_account_id, reason_id ) in petition_ids }
+ reason_ids = { reason_id for ( petitioner_account_id, reason_id ) in petition_ids }
+
+ petitioner_account_ids_to_account_keys = { petitioner_account_id : self._GetAccountKeyFromAccountId( petitioner_account_id ) for petitioner_account_id in petitioner_account_ids }
+ reason_ids_to_reasons = { reason_id : self._GetReason( reason_id ) for reason_id in reason_ids }
+
+ return HydrusSerialisable.SerialisableList( [ HydrusNetwork.PetitionHeader( content_type = content_type, status = status, account_key = petitioner_account_ids_to_account_keys[ petitioner_account_id ], reason = reason_ids_to_reasons[ reason_id ] ) for ( petitioner_account_id, reason_id ) in petition_ids ] )
+
+
def _RepositoryCreate( self, service_id ):
( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id )
@@ -2863,33 +2916,20 @@ class DB( HydrusDB.HydrusDB ):
- def _RepositoryGetFilePetition( self, service_id, account_id = None ):
+ def _RepositoryGetFilePetition( self, service_id, petitioner_account_id, reason_id ) -> HydrusNetwork.Petition:
( current_files_table_name, deleted_files_table_name, pending_files_table_name, petitioned_files_table_name, ip_addresses_table_name ) = GenerateRepositoryFilesTableNames( service_id )
- if account_id is None:
-
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} LIMIT 100;'.format( petitioned_files_table_name ) ).fetchall()
-
- else:
-
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} WHERE account_id = ? LIMIT 100;'.format( petitioned_files_table_name ), ( account_id, ) ).fetchall()
-
-
- if len( result ) == 0:
-
- raise HydrusExceptions.NotFoundException( 'No petitions!' )
-
-
- result = random.choice( result )
-
- ( petitioner_account_id, reason_id ) = result
-
petitioner_account = self._GetAccount( service_id, petitioner_account_id )
reason = self._GetReason( reason_id )
- service_hash_ids = [ service_hash_id for ( service_hash_id, ) in self._Execute( 'SELECT service_hash_id FROM ' + petitioned_files_table_name + ' WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ) ]
+ service_hash_ids = [ service_hash_id for ( service_hash_id, ) in self._Execute( f'SELECT service_hash_id FROM {petitioned_files_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ) ]
+
+ if len( service_hash_ids ) == 0:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a petition for that account and reason!' )
+
master_hash_ids = self._RepositoryGetMasterHashIds( service_id, service_hash_ids )
@@ -2903,7 +2943,44 @@ class DB( HydrusDB.HydrusDB ):
actions_and_contents = [ ( action, contents ) ]
- return HydrusNetwork.Petition( petitioner_account, reason, actions_and_contents )
+ petition_header = HydrusNetwork.PetitionHeader(
+ content_type = HC.CONTENT_TYPE_FILES,
+ status = HC.CONTENT_STATUS_PETITIONED,
+ account_key = petitioner_account.GetAccountKey(),
+ reason = reason
+ )
+
+ return HydrusNetwork.Petition( petitioner_account = petitioner_account, petition_header = petition_header, actions_and_contents = actions_and_contents )
+
+
+ def _RepositoryGetFilePetitionsSummary( self, service_id, limit = 100, account_id = None, reason_id = None ) -> typing.List:
+
+ ( current_files_table_name, deleted_files_table_name, pending_files_table_name, petitioned_files_table_name, ip_addresses_table_name ) = GenerateRepositoryFilesTableNames( service_id )
+
+ petition_search_limit = max( limit * 5, 100 )
+
+ preds = []
+
+ if account_id is not None:
+
+ preds.append( f'account_id = {account_id}' )
+
+
+ if reason_id is not None:
+
+ preds.append( f'reason_id = {reason_id}' )
+
+
+ pred_string = ''
+
+ if len( preds ) > 0:
+
+ pred_string = ' WHERE {}'.format( ' AND '.join( preds ) )
+
+
+ potential_petition_ids = self._Execute( f'SELECT DISTINCT account_id, reason_id FROM {petitioned_files_table_name}{pred_string} ORDER BY account_id LIMIT ?;', ( petition_search_limit, ) ).fetchall()
+
+ return self._RepositoryConvertPetitionIdsToSummary( HC.CONTENT_TYPE_FILES, HC.CONTENT_STATUS_PETITIONED, potential_petition_ids, limit )
def _RepositoryGetIPTimestamp( self, service_key, account, hash ):
@@ -2924,44 +3001,23 @@ class DB( HydrusDB.HydrusDB ):
return result
- def _RepositoryGetMappingPetition( self, service_id, account_id = None ):
+ def _RepositoryGetMappingPetition( self, service_id, petitioner_account_id, reason_id ) -> HydrusNetwork.Petition:
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
- if account_id is None:
-
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} LIMIT 100;'.format( petitioned_mappings_table_name ) ).fetchall()
-
- else:
-
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} WHERE account_id = ? LIMIT 100;'.format( petitioned_mappings_table_name ), ( account_id, ) ).fetchall()
-
-
- if len( result ) == 0:
-
- raise HydrusExceptions.NotFoundException( 'No petitions!' )
-
-
- result = random.choice( result )
-
- ( petitioner_account_id, reason_id ) = result
-
petitioner_account = self._GetAccount( service_id, petitioner_account_id )
reason = self._GetReason( reason_id )
- tag_ids_to_hash_ids = HydrusData.BuildKeyToListDict( self._Execute( 'SELECT service_tag_id, service_hash_id FROM ' + petitioned_mappings_table_name + ' WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ) )
+ tag_ids_to_hash_ids = HydrusData.BuildKeyToListDict( self._Execute( f'SELECT service_tag_id, service_hash_id FROM {petitioned_mappings_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ) )
+
+ if len( tag_ids_to_hash_ids ) == 0:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a petition for that account and reason!' )
+
contents = []
- total_num_petitions = 0
- total_weight = 0
-
- min_weight_permitted = None
- max_weight_permitted = None
-
- max_total_weight = None
-
petition_namespace = None
petition_pairs = list( tag_ids_to_hash_ids.items() )
@@ -2970,65 +3026,12 @@ class DB( HydrusDB.HydrusDB ):
for ( service_tag_id, service_hash_ids ) in petition_pairs:
- content_weight = len( service_hash_ids )
-
- if min_weight_permitted is None:
-
- # group petitions of similar weight together rather than mixing weight 5000 in with a hundred weight 1s
-
- if content_weight == 1:
-
- min_weight_permitted = 1
- max_weight_permitted = 1
-
- max_total_weight = 2000
-
- elif content_weight < 10:
-
- min_weight_permitted = 2
- max_weight_permitted = 9
-
- max_total_weight = 2000
-
- elif content_weight < 50:
-
- min_weight_permitted = 10
- max_weight_permitted = 49
-
- max_total_weight = 2000
-
- elif content_weight < 100:
-
- min_weight_permitted = 50
- max_weight_permitted = 99
-
- max_total_weight = 20000
-
- else:
-
- min_weight_permitted = 100
- max_weight_permitted = None
-
- max_total_weight = 100000
-
-
- else:
-
- if content_weight < min_weight_permitted:
-
- continue
-
-
- if max_weight_permitted is not None and content_weight > max_weight_permitted:
-
- continue
-
-
-
master_tag_id = self._RepositoryGetMasterTagId( service_id, service_tag_id )
tag = self._GetTag( master_tag_id )
+ # taking this out for now. it is confusing when you look at counts
+ '''
( namespace, subtag ) = HydrusTags.SplitTag( tag )
if petition_namespace is None:
@@ -3040,6 +3043,7 @@ class DB( HydrusDB.HydrusDB ):
continue
+ '''
master_hash_ids = self._RepositoryGetMasterHashIds( service_id, service_hash_ids )
@@ -3049,19 +3053,49 @@ class DB( HydrusDB.HydrusDB ):
contents.append( content )
- total_weight += content_weight
-
- if total_weight >= max_total_weight:
-
- break
-
-
action = HC.CONTENT_UPDATE_PETITION
actions_and_contents = [ ( action, contents ) ]
- return HydrusNetwork.Petition( petitioner_account, reason, actions_and_contents )
+ petition_header = HydrusNetwork.PetitionHeader(
+ content_type = HC.CONTENT_TYPE_MAPPINGS,
+ status = HC.CONTENT_STATUS_PETITIONED,
+ account_key = petitioner_account.GetAccountKey(),
+ reason = reason
+ )
+
+ return HydrusNetwork.Petition( petitioner_account = petitioner_account, petition_header = petition_header, actions_and_contents = actions_and_contents )
+
+
+ def _RepositoryGetMappingPetitionsSummary( self, service_id, limit = 100, account_id = None, reason_id = None ) -> typing.List:
+
+ ( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
+
+ petition_search_limit = max( limit * 5, 100 )
+
+ preds = []
+
+ if account_id is not None:
+
+ preds.append( f'account_id = {account_id}' )
+
+
+ if reason_id is not None:
+
+ preds.append( f'reason_id = {reason_id}' )
+
+
+ pred_string = ''
+
+ if len( preds ) > 0:
+
+ pred_string = ' WHERE {}'.format( ' AND '.join( preds ) )
+
+
+ potential_petition_ids = self._Execute( f'SELECT DISTINCT account_id, reason_id FROM {petitioned_mappings_table_name}{pred_string} ORDER BY account_id LIMIT ?;', ( petition_search_limit, ) ).fetchall()
+
+ return self._RepositoryConvertPetitionIdsToSummary( HC.CONTENT_TYPE_MAPPINGS, HC.CONTENT_STATUS_PETITIONED, potential_petition_ids, limit )
def _RepositoryGetMasterHashIds( self, service_id, service_hash_ids ):
@@ -3169,61 +3203,113 @@ class DB( HydrusDB.HydrusDB ):
return final_petition_count_info
- def _RepositoryGetPetition( self, service_key, account, content_type, status, subject_account_key = None ):
-
- # TODO: update this guy to take reason too, for (account key, reason) tuple lookups
+ def _RepositoryGetPetition( self, service_key, account, content_type, status, subject_account_key, reason ) -> HydrusNetwork.Petition:
service_id = self._GetServiceId( service_key )
- try:
+ subject_account_id = self._GetAccountId( subject_account_key )
+
+ reason_id = self._GetReasonId( reason )
+
+ if content_type == HC.CONTENT_TYPE_FILES:
- if subject_account_key is None:
+ petition = self._RepositoryGetFilePetition( service_id, subject_account_id, reason_id )
+
+ elif content_type == HC.CONTENT_TYPE_MAPPINGS:
+
+ petition = self._RepositoryGetMappingPetition( service_id, subject_account_id, reason_id )
+
+ elif content_type == HC.CONTENT_TYPE_TAG_PARENTS:
+
+ if status == HC.CONTENT_STATUS_PENDING:
- subject_account_id = None
+ petition = self._RepositoryGetTagParentPend( service_id, subject_account_id, reason_id )
else:
- subject_account_id = self._GetAccountId( subject_account_key )
+ petition = self._RepositoryGetTagParentPetition( service_id, subject_account_id, reason_id )
- if content_type == HC.CONTENT_TYPE_FILES:
+ elif content_type == HC.CONTENT_TYPE_TAG_SIBLINGS:
+
+ if status == HC.CONTENT_STATUS_PENDING:
- petition = self._RepositoryGetFilePetition( service_id, account_id = subject_account_id )
-
- elif content_type == HC.CONTENT_TYPE_MAPPINGS:
-
- petition = self._RepositoryGetMappingPetition( service_id, account_id = subject_account_id )
-
- elif content_type == HC.CONTENT_TYPE_TAG_PARENTS:
-
- if status == HC.CONTENT_STATUS_PENDING:
-
- petition = self._RepositoryGetTagParentPend( service_id, account_id = subject_account_id )
-
- else:
-
- petition = self._RepositoryGetTagParentPetition( service_id, account_id = subject_account_id )
-
-
- elif content_type == HC.CONTENT_TYPE_TAG_SIBLINGS:
-
- if status == HC.CONTENT_STATUS_PENDING:
-
- petition = self._RepositoryGetTagSiblingPend( service_id, account_id = subject_account_id )
-
- else:
-
- petition = self._RepositoryGetTagSiblingPetition( service_id, account_id = subject_account_id )
-
+ petition = self._RepositoryGetTagSiblingPend( service_id, subject_account_id, reason_id )
else:
- raise HydrusExceptions.BadRequestException( 'Unknown content type!' )
+ petition = self._RepositoryGetTagSiblingPetition( service_id, subject_account_id, reason_id )
- except HydrusExceptions.NotFoundException:
+ else:
- if subject_account_key is None:
+ raise HydrusExceptions.BadRequestException( 'Unknown content type!' )
+
+
+
+ return petition
+
+
+ def _RepositoryGetPetitionsSummary( self, service_key, account, content_type, status, limit = 100, subject_account_key = None, reason = None ) -> typing.List:
+
+ service_id = self._GetServiceId( service_key )
+
+ if subject_account_key is None:
+
+ subject_account_id = None
+
+ else:
+
+ subject_account_id = self._GetAccountId( subject_account_key )
+
+
+ if reason is None:
+
+ reason_id = None
+
+ else:
+
+ reason_id = self._GetReasonId( reason )
+
+
+ if content_type == HC.CONTENT_TYPE_FILES:
+
+ petitions_summary = self._RepositoryGetFilePetitionsSummary( service_id, limit = limit, account_id = subject_account_id, reason_id = reason_id )
+
+ elif content_type == HC.CONTENT_TYPE_MAPPINGS:
+
+ petitions_summary = self._RepositoryGetMappingPetitionsSummary( service_id, limit = limit, account_id = subject_account_id, reason_id = reason_id )
+
+ elif content_type == HC.CONTENT_TYPE_TAG_PARENTS:
+
+ if status == HC.CONTENT_STATUS_PENDING:
+
+ petitions_summary = self._RepositoryGetTagParentPendsSummary( service_id, limit = limit, account_id = subject_account_id, reason_id = reason_id )
+
+ else:
+
+ petitions_summary = self._RepositoryGetTagParentPetitionsSummary( service_id, limit = limit, account_id = subject_account_id, reason_id = reason_id )
+
+
+ elif content_type == HC.CONTENT_TYPE_TAG_SIBLINGS:
+
+ if status == HC.CONTENT_STATUS_PENDING:
+
+ petitions_summary = self._RepositoryGetTagSiblingPendsSummary( service_id, limit = limit, account_id = subject_account_id, reason_id = reason_id )
+
+ else:
+
+ petitions_summary = self._RepositoryGetTagSiblingPetitionsSummary( service_id, limit = limit, account_id = subject_account_id, reason_id = reason_id )
+
+
+ else:
+
+ raise HydrusExceptions.BadRequestException( 'Unknown content type!' )
+
+
+ if len( petitions_summary ) == 0:
+
+ if subject_account_key is None and reason is None:
info_type = None
@@ -3264,10 +3350,8 @@ class DB( HydrusDB.HydrusDB ):
- raise
-
- return petition
+ return petitions_summary
def _RepositoryGetServiceHashId( self, service_id, master_hash_id, timestamp ):
@@ -3336,9 +3420,6 @@ class DB( HydrusDB.HydrusDB ):
def _RepositoryGetServiceInfoSpecificForAccount( self, service_id: int, info_type: int, account_id: int ):
- service_name = self._GetServiceName( service_id )
-
- ( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id )
( current_files_table_name, deleted_files_table_name, pending_files_table_name, petitioned_files_table_name, ip_addresses_table_name ) = GenerateRepositoryFilesTableNames( service_id )
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
( current_tag_siblings_table_name, deleted_tag_siblings_table_name, pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ) = GenerateRepositoryTagSiblingsTableNames( service_id )
@@ -3472,33 +3553,20 @@ class DB( HydrusDB.HydrusDB ):
- def _RepositoryGetTagParentPend( self, service_id, account_id = None ):
+ def _RepositoryGetTagParentPend( self, service_id, petitioner_account_id, reason_id ) -> HydrusNetwork.Petition:
( current_tag_parents_table_name, deleted_tag_parents_table_name, pending_tag_parents_table_name, petitioned_tag_parents_table_name ) = GenerateRepositoryTagParentsTableNames( service_id )
- if account_id is None:
-
- result = self._Execute( 'SELECT DISTINCT account_id as a1, reason_id as r1 FROM {} WHERE 1 NOT IN ( SELECT 1 FROM {} WHERE account_id = a1 AND reason_id = r1 ) LIMIT 100;'.format( pending_tag_parents_table_name, petitioned_tag_parents_table_name ) ).fetchall()
-
- else:
-
- result = self._Execute( 'SELECT DISTINCT account_id as a1, reason_id as r1 FROM {} WHERE account_id = ? AND 1 NOT IN ( SELECT 1 FROM {} WHERE account_id = a1 AND reason_id = r1 ) LIMIT 100;'.format( pending_tag_parents_table_name, petitioned_tag_parents_table_name ), ( account_id, ) ).fetchall()
-
-
- if len( result ) == 0:
-
- raise HydrusExceptions.NotFoundException( 'No petitions!' )
-
-
- result = random.choice( result )
-
- ( petitioner_account_id, reason_id ) = result
-
petitioner_account = self._GetAccount( service_id, petitioner_account_id )
reason = self._GetReason( reason_id )
- pairs = self._Execute( 'SELECT child_master_tag_id, parent_master_tag_id FROM ' + pending_tag_parents_table_name + ' WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+ pairs = self._Execute( f'SELECT child_master_tag_id, parent_master_tag_id FROM {pending_tag_parents_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+
+ if len( pairs ) == 0:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a petition for that account and reason!' )
+
contents = []
@@ -3517,30 +3585,44 @@ class DB( HydrusDB.HydrusDB ):
actions_and_contents = [ ( action, contents ) ]
- return HydrusNetwork.Petition( petitioner_account, reason, actions_and_contents )
+ petition_header = HydrusNetwork.PetitionHeader(
+ content_type = HC.CONTENT_TYPE_TAG_PARENTS,
+ status = HC.CONTENT_STATUS_PENDING,
+ account_key = petitioner_account.GetAccountKey(),
+ reason = reason
+ )
+
+ return HydrusNetwork.Petition( petitioner_account = petitioner_account, petition_header = petition_header, actions_and_contents = actions_and_contents )
- def _RepositoryGetTagParentPetition( self, service_id, account_id = None ):
+ def _RepositoryGetTagParentPendsSummary( self, service_id, limit = 100, account_id = None, reason_id = None ) -> typing.List:
( current_tag_parents_table_name, deleted_tag_parents_table_name, pending_tag_parents_table_name, petitioned_tag_parents_table_name ) = GenerateRepositoryTagParentsTableNames( service_id )
- if account_id is None:
+ petition_search_limit = max( limit * 5, 100 )
+
+ preds = [ f'1 NOT IN ( SELECT 1 FROM {petitioned_tag_parents_table_name} WHERE account_id = a1 AND reason_id = r1 )' ]
+
+ if account_id is not None:
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} LIMIT 100;'.format( petitioned_tag_parents_table_name ) ).fetchall()
-
- else:
-
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} WHERE account_id = ? LIMIT 100;'.format( petitioned_tag_parents_table_name ), ( account_id, ) ).fetchall()
+ preds.append( f'account_id = {account_id}' )
- if len( result ) == 0:
+ if reason_id is not None:
- raise HydrusExceptions.NotFoundException( 'No petitions!' )
+ preds.append( f'reason_id = {reason_id}' )
- result = random.choice( result )
+ pred_string = ' WHERE {}'.format( ' AND '.join( preds ) )
- ( petitioner_account_id, reason_id ) = result
+ potential_petition_ids = self._Execute( f'SELECT DISTINCT account_id as a1, reason_id as r1 FROM {pending_tag_parents_table_name}{pred_string} ORDER BY account_id LIMIT ?;', ( petition_search_limit, ) ).fetchall()
+
+ return self._RepositoryConvertPetitionIdsToSummary( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_STATUS_PENDING, potential_petition_ids, limit )
+
+
+ def _RepositoryGetTagParentPetition( self, service_id, petitioner_account_id, reason_id ) -> HydrusNetwork.Petition:
+
+ ( current_tag_parents_table_name, deleted_tag_parents_table_name, pending_tag_parents_table_name, petitioned_tag_parents_table_name ) = GenerateRepositoryTagParentsTableNames( service_id )
petitioner_account = self._GetAccount( service_id, petitioner_account_id )
@@ -3550,7 +3632,12 @@ class DB( HydrusDB.HydrusDB ):
#
- pairs = self._Execute( 'SELECT child_service_tag_id, parent_service_tag_id FROM ' + petitioned_tag_parents_table_name + ' WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+ pairs = self._Execute( f'SELECT child_service_tag_id, parent_service_tag_id FROM {petitioned_tag_parents_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+
+ if len( pairs ) == 0:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a petition for that account and reason!' )
+
contents = []
@@ -3574,7 +3661,7 @@ class DB( HydrusDB.HydrusDB ):
#
- pairs = self._Execute( 'SELECT child_master_tag_id, parent_master_tag_id FROM ' + pending_tag_parents_table_name + ' WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+ pairs = self._Execute( f'SELECT child_master_tag_id, parent_master_tag_id FROM {pending_tag_parents_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
contents = []
@@ -3595,36 +3682,60 @@ class DB( HydrusDB.HydrusDB ):
#
- return HydrusNetwork.Petition( petitioner_account, reason, actions_and_contents )
+ petition_header = HydrusNetwork.PetitionHeader(
+ content_type = HC.CONTENT_TYPE_TAG_PARENTS,
+ status = HC.CONTENT_STATUS_PETITIONED,
+ account_key = petitioner_account.GetAccountKey(),
+ reason = reason
+ )
+
+ return HydrusNetwork.Petition( petitioner_account = petitioner_account, petition_header = petition_header, actions_and_contents = actions_and_contents )
- def _RepositoryGetTagSiblingPend( self, service_id, account_id = None ):
+ def _RepositoryGetTagParentPetitionsSummary( self, service_id, limit = 100, account_id = None, reason_id = None ) -> typing.List:
+
+ ( current_tag_parents_table_name, deleted_tag_parents_table_name, pending_tag_parents_table_name, petitioned_tag_parents_table_name ) = GenerateRepositoryTagParentsTableNames( service_id )
+
+ petition_search_limit = max( limit * 5, 100 )
+
+ preds = []
+
+ if account_id is not None:
+
+ preds.append( f'account_id = {account_id}' )
+
+
+ if reason_id is not None:
+
+ preds.append( f'reason_id = {reason_id}' )
+
+
+ pred_string = ''
+
+ if len( preds ) > 0:
+
+ pred_string = ' WHERE {}'.format( ' AND '.join( preds ) )
+
+
+ potential_petition_ids = self._Execute( f'SELECT DISTINCT account_id, reason_id FROM {petitioned_tag_parents_table_name}{pred_string} ORDER BY account_id LIMIT ?;', ( petition_search_limit, ) ).fetchall()
+
+ return self._RepositoryConvertPetitionIdsToSummary( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_STATUS_PETITIONED, potential_petition_ids, limit )
+
+
+ def _RepositoryGetTagSiblingPend( self, service_id, petitioner_account_id, reason_id ) -> HydrusNetwork.Petition:
( current_tag_siblings_table_name, deleted_tag_siblings_table_name, pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ) = GenerateRepositoryTagSiblingsTableNames( service_id )
- if account_id is None:
-
- result = self._Execute( 'SELECT DISTINCT account_id as a1, reason_id as r1 FROM {} WHERE 1 NOT IN ( SELECT 1 FROM {} WHERE account_id = a1 AND reason_id = r1 ) LIMIT 100;'.format( pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ) ).fetchall()
-
- else:
-
- result = self._Execute( 'SELECT DISTINCT account_id as a1, reason_id as r1 FROM {} WHERE account_id = ? AND 1 NOT IN ( SELECT 1 FROM {} WHERE account_id = a1 AND reason_id = r1 ) LIMIT 100;'.format( pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ), ( account_id, ) ).fetchall()
-
-
- if len( result ) == 0:
-
- raise HydrusExceptions.NotFoundException( 'No petitions!' )
-
-
- result = random.choice( result )
-
- ( petitioner_account_id, reason_id ) = result
-
petitioner_account = self._GetAccount( service_id, petitioner_account_id )
reason = self._GetReason( reason_id )
- pairs = self._Execute( 'SELECT bad_master_tag_id, good_master_tag_id FROM ' + pending_tag_siblings_table_name + ' WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+ pairs = self._Execute( f'SELECT bad_master_tag_id, good_master_tag_id FROM {pending_tag_siblings_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+
+ if len( pairs ) == 0:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a petition for that account and reason!' )
+
contents = []
@@ -3643,30 +3754,44 @@ class DB( HydrusDB.HydrusDB ):
actions_and_contents = [ ( action, contents ) ]
- return HydrusNetwork.Petition( petitioner_account, reason, actions_and_contents )
+ petition_header = HydrusNetwork.PetitionHeader(
+ content_type = HC.CONTENT_TYPE_TAG_SIBLINGS,
+ status = HC.CONTENT_STATUS_PENDING,
+ account_key = petitioner_account.GetAccountKey(),
+ reason = reason
+ )
+
+ return HydrusNetwork.Petition( petitioner_account = petitioner_account, petition_header = petition_header, actions_and_contents = actions_and_contents )
- def _RepositoryGetTagSiblingPetition( self, service_id, account_id = None ):
+ def _RepositoryGetTagSiblingPendsSummary( self, service_id, limit = 100, account_id = None, reason_id = None ) -> typing.List:
( current_tag_siblings_table_name, deleted_tag_siblings_table_name, pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ) = GenerateRepositoryTagSiblingsTableNames( service_id )
- if account_id is None:
+ petition_search_limit = max( limit * 5, 100 )
+
+ preds = [ f'1 NOT IN ( SELECT 1 FROM {petitioned_tag_siblings_table_name} WHERE account_id = a1 AND reason_id = r1 )' ]
+
+ if account_id is not None:
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} LIMIT 100;'.format( petitioned_tag_siblings_table_name ) ).fetchall()
-
- else:
-
- result = self._Execute( 'SELECT DISTINCT account_id, reason_id FROM {} WHERE account_id = ? LIMIT 100;'.format( petitioned_tag_siblings_table_name ), ( account_id, ) ).fetchall()
+ preds.append( f'account_id = {account_id}' )
- if len( result ) == 0:
+ if reason_id is not None:
- raise HydrusExceptions.NotFoundException( 'No petitions!' )
+ preds.append( f'reason_id = {reason_id}' )
- result = random.choice( result )
+ pred_string = ' WHERE {}'.format( ' AND '.join( preds ) )
- ( petitioner_account_id, reason_id ) = result
+ potential_petition_ids = self._Execute( f'SELECT DISTINCT account_id as a1, reason_id as r1 FROM {pending_tag_siblings_table_name}{pred_string} ORDER BY account_id LIMIT ?;', ( petition_search_limit, ) ).fetchall()
+
+ return self._RepositoryConvertPetitionIdsToSummary( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_STATUS_PENDING, potential_petition_ids, limit )
+
+
+ def _RepositoryGetTagSiblingPetition( self, service_id, petitioner_account_id, reason_id ) -> HydrusNetwork.Petition:
+
+ ( current_tag_siblings_table_name, deleted_tag_siblings_table_name, pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ) = GenerateRepositoryTagSiblingsTableNames( service_id )
petitioner_account = self._GetAccount( service_id, petitioner_account_id )
@@ -3676,7 +3801,12 @@ class DB( HydrusDB.HydrusDB ):
#
- pairs = self._Execute( 'SELECT bad_service_tag_id, good_service_tag_id FROM {} WHERE account_id = ? AND reason_id = ?;'.format( petitioned_tag_siblings_table_name ), ( petitioner_account_id, reason_id ) ).fetchall()
+ pairs = self._Execute( f'SELECT bad_service_tag_id, good_service_tag_id FROM {petitioned_tag_siblings_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+
+ if len( pairs ) == 0:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a petition for that account and reason!' )
+
contents = []
@@ -3700,7 +3830,7 @@ class DB( HydrusDB.HydrusDB ):
#
- pairs = self._Execute( 'SELECT bad_master_tag_id, good_master_tag_id FROM ' + pending_tag_siblings_table_name + ' WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
+ pairs = self._Execute( f'SELECT bad_master_tag_id, good_master_tag_id FROM {pending_tag_siblings_table_name} WHERE account_id = ? AND reason_id = ?;', ( petitioner_account_id, reason_id ) ).fetchall()
contents = []
@@ -3721,7 +3851,44 @@ class DB( HydrusDB.HydrusDB ):
#
- return HydrusNetwork.Petition( petitioner_account, reason, actions_and_contents )
+ petition_header = HydrusNetwork.PetitionHeader(
+ content_type = HC.CONTENT_TYPE_TAG_SIBLINGS,
+ status = HC.CONTENT_STATUS_PETITIONED,
+ account_key = petitioner_account.GetAccountKey(),
+ reason = reason
+ )
+
+ return HydrusNetwork.Petition( petitioner_account = petitioner_account, petition_header = petition_header, actions_and_contents = actions_and_contents )
+
+
+ def _RepositoryGetTagSiblingPetitionsSummary( self, service_id, limit = 100, account_id = None, reason_id = None ) -> typing.List:
+
+ ( current_tag_siblings_table_name, deleted_tag_siblings_table_name, pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ) = GenerateRepositoryTagSiblingsTableNames( service_id )
+
+ petition_search_limit = max( limit * 5, 100 )
+
+ preds = []
+
+ if account_id is not None:
+
+ preds.append( f'account_id = {account_id}' )
+
+
+ if reason_id is not None:
+
+ preds.append( f'reason_id = {reason_id}' )
+
+
+ pred_string = ''
+
+ if len( preds ) > 0:
+
+ pred_string = ' WHERE {}'.format( ' AND '.join( preds ) )
+
+
+ potential_petition_ids = self._Execute( f'SELECT DISTINCT account_id, reason_id FROM {petitioned_tag_siblings_table_name}{pred_string} ORDER BY account_id LIMIT ?;', ( petition_search_limit, ) ).fetchall()
+
+ return self._RepositoryConvertPetitionIdsToSummary( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_STATUS_PETITIONED, potential_petition_ids, limit )
def _RepositoryHasFile( self, service_key, hash ):
@@ -4113,7 +4280,14 @@ class DB( HydrusDB.HydrusDB ):
for ( ( tag, hashes ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_MAPPINGS, HC.CONTENT_UPDATE_PEND ):
- master_tag_id = self._GetMasterTagId( tag )
+ try:
+
+ master_tag_id = self._GetMasterTagId( tag )
+
+ except:
+
+ continue
+
master_hash_ids = self._GetMasterHashIds( hashes )
@@ -4127,7 +4301,14 @@ class DB( HydrusDB.HydrusDB ):
for ( ( tag, hashes ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_MAPPINGS, HC.CONTENT_UPDATE_PETITION ):
- master_tag_id = self._GetMasterTagId( tag )
+ try:
+
+ master_tag_id = self._GetMasterTagId( tag )
+
+ except:
+
+ continue
+
service_tag_id = self._RepositoryGetServiceTagId( service_id, master_tag_id, timestamp )
@@ -4152,7 +4333,14 @@ class DB( HydrusDB.HydrusDB ):
for ( ( tag, hashes ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_MAPPINGS, HC.CONTENT_UPDATE_DENY_PETITION ):
- master_tag_id = self._GetMasterTagId( tag )
+ try:
+
+ master_tag_id = self._GetMasterTagId( tag )
+
+ except:
+
+ continue
+
service_tag_id = self._RepositoryGetServiceTagId( service_id, master_tag_id, timestamp )
@@ -4170,8 +4358,15 @@ class DB( HydrusDB.HydrusDB ):
for ( ( child_tag, parent_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_UPDATE_PEND ):
- child_master_tag_id = self._GetMasterTagId( child_tag )
- parent_master_tag_id = self._GetMasterTagId( parent_tag )
+ try:
+
+ child_master_tag_id = self._GetMasterTagId( child_tag )
+ parent_master_tag_id = self._GetMasterTagId( parent_tag )
+
+ except:
+
+ continue
+
if can_create_tag_parents or can_moderate_tag_parents:
@@ -4189,8 +4384,15 @@ class DB( HydrusDB.HydrusDB ):
for ( ( child_tag, parent_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_UPDATE_PETITION ):
- child_master_tag_id = self._GetMasterTagId( child_tag )
- parent_master_tag_id = self._GetMasterTagId( parent_tag )
+ try:
+
+ child_master_tag_id = self._GetMasterTagId( child_tag )
+ parent_master_tag_id = self._GetMasterTagId( parent_tag )
+
+ except:
+
+ continue
+
child_service_tag_id = self._RepositoryGetServiceTagId( service_id, child_master_tag_id, timestamp )
parent_service_tag_id = self._RepositoryGetServiceTagId( service_id, parent_master_tag_id, timestamp )
@@ -4212,16 +4414,30 @@ class DB( HydrusDB.HydrusDB ):
for ( ( child_tag, parent_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_UPDATE_DENY_PEND ):
- child_master_tag_id = self._GetMasterTagId( child_tag )
- parent_master_tag_id = self._GetMasterTagId( parent_tag )
+ try:
+
+ child_master_tag_id = self._GetMasterTagId( child_tag )
+ parent_master_tag_id = self._GetMasterTagId( parent_tag )
+
+ except:
+
+ continue
+
self._RepositoryDenyTagParentPend( service_id, child_master_tag_id, parent_master_tag_id )
for ( ( child_tag, parent_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_PARENTS, HC.CONTENT_UPDATE_DENY_PETITION ):
- child_master_tag_id = self._GetMasterTagId( child_tag )
- parent_master_tag_id = self._GetMasterTagId( parent_tag )
+ try:
+
+ child_master_tag_id = self._GetMasterTagId( child_tag )
+ parent_master_tag_id = self._GetMasterTagId( parent_tag )
+
+ except:
+
+ continue
+
child_service_tag_id = self._RepositoryGetServiceTagId( service_id, child_master_tag_id, timestamp )
parent_service_tag_id = self._RepositoryGetServiceTagId( service_id, parent_master_tag_id, timestamp )
@@ -4238,8 +4454,15 @@ class DB( HydrusDB.HydrusDB ):
for ( ( bad_tag, good_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_UPDATE_PETITION ):
- bad_master_tag_id = self._GetMasterTagId( bad_tag )
- good_master_tag_id = self._GetMasterTagId( good_tag )
+ try:
+
+ bad_master_tag_id = self._GetMasterTagId( bad_tag )
+ good_master_tag_id = self._GetMasterTagId( good_tag )
+
+ except:
+
+ continue
+
bad_service_tag_id = self._RepositoryGetServiceTagId( service_id, bad_master_tag_id, timestamp )
good_service_tag_id = self._RepositoryGetServiceTagId( service_id, good_master_tag_id, timestamp )
@@ -4258,8 +4481,15 @@ class DB( HydrusDB.HydrusDB ):
for ( ( bad_tag, good_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_UPDATE_PEND ):
- bad_master_tag_id = self._GetMasterTagId( bad_tag )
- good_master_tag_id = self._GetMasterTagId( good_tag )
+ try:
+
+ bad_master_tag_id = self._GetMasterTagId( bad_tag )
+ good_master_tag_id = self._GetMasterTagId( good_tag )
+
+ except:
+
+ continue
+
if can_create_tag_siblings or can_moderate_tag_siblings:
@@ -4280,16 +4510,30 @@ class DB( HydrusDB.HydrusDB ):
for ( ( bad_tag, good_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_UPDATE_DENY_PEND ):
- bad_master_tag_id = self._GetMasterTagId( bad_tag )
- good_master_tag_id = self._GetMasterTagId( good_tag )
+ try:
+
+ bad_master_tag_id = self._GetMasterTagId( bad_tag )
+ good_master_tag_id = self._GetMasterTagId( good_tag )
+
+ except:
+
+ continue
+
self._RepositoryDenyTagSiblingPend( service_id, bad_master_tag_id, good_master_tag_id )
for ( ( bad_tag, good_tag ), reason ) in client_to_server_update.GetContentDataIterator( HC.CONTENT_TYPE_TAG_SIBLINGS, HC.CONTENT_UPDATE_DENY_PETITION ):
- bad_master_tag_id = self._GetMasterTagId( bad_tag )
- good_master_tag_id = self._GetMasterTagId( good_tag )
+ try:
+
+ bad_master_tag_id = self._GetMasterTagId( bad_tag )
+ good_master_tag_id = self._GetMasterTagId( good_tag )
+
+ except:
+
+ continue
+
bad_service_tag_id = self._RepositoryGetServiceTagId( service_id, bad_master_tag_id, timestamp )
good_service_tag_id = self._RepositoryGetServiceTagId( service_id, good_master_tag_id, timestamp )
diff --git a/hydrus/server/networking/ServerServer.py b/hydrus/server/networking/ServerServer.py
index 39fde5dd..cbbd3482 100644
--- a/hydrus/server/networking/ServerServer.py
+++ b/hydrus/server/networking/ServerServer.py
@@ -70,7 +70,7 @@ class HydrusServiceRepository( HydrusServiceRestricted ):
root.putChild( b'num_petitions', ServerServerResources.HydrusResourceRestrictedNumPetitions( self._service, HydrusServer.REMOTE_DOMAIN ) )
root.putChild( b'petition', ServerServerResources.HydrusResourceRestrictedPetition( self._service, HydrusServer.REMOTE_DOMAIN ) )
- root.putChild( b'petition_summary_list', ServerServerResources.HydrusResourceRestrictedPetitionSummaryList( self._service, HydrusServer.REMOTE_DOMAIN ) )
+ root.putChild( b'petitions_summary', ServerServerResources.HydrusResourceRestrictedPetitionsSummary( self._service, HydrusServer.REMOTE_DOMAIN ) )
root.putChild( b'update', ServerServerResources.HydrusResourceRestrictedUpdate( self._service, HydrusServer.REMOTE_DOMAIN ) )
#root.putChild( b'immediate_update', ServerServerResources.HydrusResourceRestrictedImmediateUpdate( self._service, HydrusServer.REMOTE_DOMAIN ) )
root.putChild( b'metadata', ServerServerResources.HydrusResourceRestrictedMetadataUpdate( self._service, HydrusServer.REMOTE_DOMAIN ) )
diff --git a/hydrus/server/networking/ServerServerResources.py b/hydrus/server/networking/ServerServerResources.py
index 0fa86597..03c7dba0 100644
--- a/hydrus/server/networking/ServerServerResources.py
+++ b/hydrus/server/networking/ServerServerResources.py
@@ -917,28 +917,7 @@ class HydrusResourceRestrictedNumPetitions( HydrusResourceRestricted ):
return response_context
-class HydrusResourceRestrictedPetitionSummaryList( HydrusResourceRestricted ):
-
- def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ):
-
- content_type = request.parsed_request_args[ 'content_type' ]
-
- request.hydrus_account.CheckPermission( content_type, HC.PERMISSION_ACTION_MODERATE )
-
-
- def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
-
- # fetch cached summary list
- # ( account_key, reason, size of petition )
- petition_summary_list = []
-
- body = HydrusNetworkVariableHandling.DumpHydrusArgsToNetworkBytes( { 'petition_summary_list' : petition_summary_list } )
-
- response_context = HydrusServerResources.ResponseContext( 200, body = body )
-
- return response_context
-
-
+
class HydrusResourceRestrictedPetition( HydrusResourceRestricted ):
def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ):
@@ -950,12 +929,38 @@ class HydrusResourceRestrictedPetition( HydrusResourceRestricted ):
def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
- subject_account_key = request.parsed_request_args.GetValueOrNone( 'subject_account_key', bytes )
- # add reason to here some time, for when we eventually select petitions from a summary list of ( account, reason, size ) stuff
content_type = request.parsed_request_args[ 'content_type' ]
status = request.parsed_request_args[ 'status' ]
+ subject_account_key = request.parsed_request_args.GetValueOrNone( 'subject_account_key', bytes )
+ reason = request.parsed_request_args.GetValueOrNone( 'reason', str )
- petition = HG.server_controller.Read( 'petition', self._service_key, request.hydrus_account, content_type, status, subject_account_key = subject_account_key )
+ if subject_account_key is None or reason is None:
+
+ petitions_summary = HG.server_controller.Read( 'petitions_summary', self._service_key, request.hydrus_account, content_type, status, limit = 1, subject_account_key = subject_account_key )
+
+ if len( petitions_summary ) == 0:
+
+ if subject_account_key is None and reason is None:
+
+ raise HydrusExceptions.NotFoundException( f'Sorry, no petitions were found!' )
+
+ elif subject_account_key is None:
+
+ raise HydrusExceptions.NotFoundException( f'Sorry, no petitions were found for the given reason {reason}!' )
+
+ else:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, no petitions were found for the given account_key {}!'.format( subject_account_key.hex() ) )
+
+
+
+ petition_header = petitions_summary[0]
+
+ subject_account_key = petition_header.account_key
+ reason = petition_header.reason
+
+
+ petition = HG.server_controller.Read( 'petition', self._service_key, request.hydrus_account, content_type, status, subject_account_key, reason )
body = HydrusNetworkVariableHandling.DumpHydrusArgsToNetworkBytes( { 'petition' : petition } )
@@ -964,6 +969,35 @@ class HydrusResourceRestrictedPetition( HydrusResourceRestricted ):
return response_context
+
+class HydrusResourceRestrictedPetitionsSummary( HydrusResourceRestricted ):
+
+ def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ):
+
+ content_type = request.parsed_request_args[ 'content_type' ]
+
+ request.hydrus_account.CheckPermission( content_type, HC.PERMISSION_ACTION_MODERATE )
+
+
+ def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
+
+ content_type = request.parsed_request_args.GetValue( 'content_type', int )
+ status = request.parsed_request_args.GetValue( 'status', int )
+ num = request.parsed_request_args.GetValue( 'num', int )
+
+ subject_account_key = request.parsed_request_args.GetValueOrNone( 'subject_account_key', bytes )
+ reason = request.parsed_request_args.GetValueOrNone( 'reason', str )
+
+ petitions_summary = HG.server_controller.Read( 'petitions_summary', self._service_key, request.hydrus_account, content_type, status, num, subject_account_key = subject_account_key, reason = reason )
+
+ body = HydrusNetworkVariableHandling.DumpHydrusArgsToNetworkBytes( { 'petitions_summary' : petitions_summary } )
+
+ response_context = HydrusServerResources.ResponseContext( 200, body = body )
+
+ return response_context
+
+
+
class HydrusResourceRestrictedRegistrationKeys( HydrusResourceRestricted ):
def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ):
diff --git a/hydrus/test/TestHydrusServer.py b/hydrus/test/TestHydrusServer.py
index 4aa8578f..389449d0 100644
--- a/hydrus/test/TestHydrusServer.py
+++ b/hydrus/test/TestHydrusServer.py
@@ -400,11 +400,18 @@ class TestServer( unittest.TestCase ):
# petition
- petitioner_account = HydrusNetwork.Account.GenerateUnknownAccount()
+ petitioner_account = HydrusNetwork.Account.GenerateUnknownAccount( HydrusData.GenerateKey() )
reason = 'it sucks'
actions_and_contents = ( HC.CONTENT_UPDATE_PETITION, [ HydrusNetwork.Content( HC.CONTENT_TYPE_FILES, [ HydrusData.GenerateKey() for i in range( 10 ) ] ) ] )
- petition = HydrusNetwork.Petition( petitioner_account, reason, actions_and_contents )
+ petition_header = HydrusNetwork.PetitionHeader(
+ content_type = HC.CONTENT_TYPE_FILES,
+ status = HC.CONTENT_STATUS_PETITIONED,
+ account_key = petitioner_account.GetAccountKey(),
+ reason = reason
+ )
+
+ petition = HydrusNetwork.Petition( petitioner_account = petitioner_account, petition_header = petition_header, actions_and_contents = actions_and_contents )
HG.test_controller.SetRead( 'petition', petition )
@@ -412,6 +419,12 @@ class TestServer( unittest.TestCase ):
self.assertEqual( response[ 'petition' ].GetSerialisableTuple(), petition.GetSerialisableTuple() )
+ HG.test_controller.SetRead( 'petition', petition )
+
+ response = service.Request( HC.GET, 'petition', { 'content_type' : HC.CONTENT_TYPE_FILES, 'status' : HC.CONTENT_UPDATE_PETITION, 'account_key' : petitioner_account.GetAccountKey(), reason : reason } )
+
+ self.assertEqual( response[ 'petition' ].GetSerialisableTuple(), petition.GetSerialisableTuple() )
+
# definitions
definitions_update = HydrusNetwork.DefinitionsUpdate()