Rooba
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Version 315

This commit is contained in:
Hydrus Network Developer 2018-07-18 16:07:15 -05:00
parent 804ffe8cec
commit 0646dd1f47
42 changed files with 3245 additions and 1502 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
client.py
View File

@ -135,6 +135,6 @@ except Exception as e:
f.write( traceback.format_exc() )
print( 'Critical error occured! Details written to crash.log!' )
print( 'Critical error occurred! Details written to crash.log!' )

2
client.pyw
View File

@ -158,6 +158,6 @@ except Exception as e:
f.write( traceback.format_exc() )
print( 'Critical error occured! Details written to crash.log!' )
print( 'Critical error occurred! Details written to crash.log!' )

37
help/changelog.html
View File

@ -8,6 +8,41 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 315</h3></li>
<ul>
<li>got started on the big gallery update, but decided not to pull the trigger just yet. I hope to do it next week, switching the whole thing over to a two-object multi-watcher kind of deal</li>
<li>updated to wxPython 4.0.3 for all platforms</li>
<li>cleaned up some menubar replacement code, and the update to the new wxPython should also fix a "event for a menu without associated window" bug some gtk2 users were seeing on quick menubar changes</li>
<li>manage default tag import options panel now has copy/paste buttons that work on the listctrl</li>
<li>added some 'paste tag import options' safety code to make sure no one accidentally pastes a subscription or something in there, wew</li>
<li>added default checker options for subscriptions to options->downloading</li>
<li>unified how checker options are edited from their button, much like how file and tag import options work. it also has a summary tooltip on the button</li>
<li>the checker options under options->downloading are now these slimmer buttons</li>
<li>in the manual import dialog (which pops up when you drop a folder/files on the client), the files will now be added in 'human friendly' number sorting, so files of the sort 'Favourites - 10.jpg' will sort [10, 11, ..., 99, 100] rather than the purely lexicographic [10, 100, 11, ..., 99]</li>
<li>gave the migrate database dialog a pass--a bunch of misc presentation changes and a general simplification of workflow, now based more on just increase/decrease location weight</li>
<li>a bunch of texts on page management (left-hand) panels that share horizontal space with buttons should now ellipsize ("downlo...") when they get too long for the width instead of drawing in an ugly way over the buttonspace</li>
<li>moved the manage import folders dialog to the new listctrl and added a 'paused' and better 'check period' column</li>
<li>if a user tries to run a 'paused' import folder specifically from the menu, the import folder will now unpause (I will probably remove this old paused variable in the future--it isn't of much use any more)</li>
<li>tightened up some repository reset code that wasn't deleting all service tables and hence recovering from some service id malformation errors correctly</li>
<li>wrote a 'clear orphan tables' db maintenance routine that kills some spare tables some users who have previously deleted/reset repositories may have floating around</li>
<li>fixed an issue with parsing folders after hitting cancel button on the import files pre-dialog</li>
<li>if watchers encounter non-404 network errors during check, they should now just delay checking for four hours (before, they were also pausing checking completely)</li>
<li>if watchers are in 'delay' mode, they'll also not work on files.</li>
<li>file and gallery downloads that hit a 403 (Forbidden) will now present a simpler error status, like they do for 404</li>
<li>the new post downloader will no longer fail if one of the parsed source urls is not a url. the borked string will also not be associated as a url</li>
<li>regular gallery downloads now override bandwidth for the file download step, which is almost always the second half of a pair of post_url/file downloads, just to keep things in sync in edge cases</li>
<li>cleaned up some timestamp generation and 'overriding in x seconds' strings to be more human friendly</li>
<li>improved some serverside file parse error handling to propagate the actual error description up to the client a bit better</li>
<li>fixed typo causing incorrect num_ignored count in file import status button right-click menu</li>
<li>parseexceptions will now present more data about which page and content parser caused the problem. I am not totally happy about how this solution works and may revisit it</li>
<li>the lz4 import error catching is now more broad to catch some odd problem I discovered in new Linux build environment</li>
<li>the moebooru parser now fetches the original png of an image, if available</li>
<li>added a new tumblr parser that also gets post tags--it _shouldn't_ be the default</li>
<li>the new login pipeline now kicks in for the legacy logins--pixiv and hentai foundry--on a per-url basis, so adding pixiv/hf urls to the url downloader will trigger a login even if needed (previously, this was tied to legacy gallery initialisation, which explains some pixiv 'missing' login stuff some users and I were having trouble with)</li>
<li>if the legacy login system fails in the new pipeline, it now sets a flag and won't try again that client boot</li>
<li>the old 'default tag import options' panel is now completely removed from options->importing. please check 'network->downloaders->manage default tag import options' for the new url-based settings</li>
<li>misc fixes</li>
</ul>
<li><h3>version 314</h3></li>
<ul>
<li>tag import options can now be set to 'default', meaning 'use whatever the default is at the time of import', which will be an easier way of managing TIOs for many subs that you'd prefer all share the same TIO settings anyway</li>
@ -37,7 +72,7 @@
<li>added a 'show the D on short file import summaries' checkbox to options->downloading--it defaults to off</li>
<li>the 'I' on short file import summaries is now 'Ig' to clear up 1/I confusion</li>
<li>added 'copy queries' to the edit subscription panel, which lets you copy all the selected queries' search texts to clipboard, newline separated</li>
<li>added a checbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on</li>
<li>added a checkbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on</li>
<li>fixed file import status button right-click, which I messed up somehow last week with the 'retry ignored' add</li>
<li>shook up and collapsed the network menu into neater categories</li>
<li>tightened-up the rarely used pre-parsing conversion panel on the edit page parser panel to just a button with a bit of explaining text</li>

2
help/tagging_schema.html
View File

@ -51,7 +51,7 @@
<h3>character:anna (frozen)</h3>
<p>I am not fond of putting a series name after a character because it looks unusual and is applied unreliably. It is done to separate same-named characters from each other (particularly when they have no canon surname), which is useful in places that search slowly or usually only deal in single-tag searches. I would prefer that namespaces say their namespace and nothing else. Some sites even say things like 'anna (disney)'. I don't really mind this stuff, but if you are adding a sibling to collapse these divergent tags into the 'proper' one, I'd prefer it all went to the simple and reliable 'character:anna'. Even better would be migrating towards a canon-ok unique name, like 'character:princess anna of arendelle'.</p>
<p>Including nicknames, like 'character:angela "mercy" ziegler' can be useful to establish uniqueness, but are not mandatory. 'character:harleen "harley quinn" frances quinzel' is probably overboard.</p>
<h3>protip: reign in your spergitude</h3>
<h3>protip: rein in your spergitude</h3>
<p>In developing hydrus, I have discovered two rules to happy tagging:</p>
<ol>
<li>Don't try to be perfect.</li>

12
include/ClientCaches.py
View File

@ -214,7 +214,7 @@ class ClientFilesManager( object ):
self._prefixes_to_locations = {}
self._bad_error_occured = False
self._bad_error_occurred = False
self._missing_locations = set()
self._Reinit()
@ -625,7 +625,7 @@ class ClientFilesManager( object ):
if len( self._missing_locations ) > 0:
self._bad_error_occured = True
self._bad_error_occurred = True
#
@ -1055,9 +1055,9 @@ class ClientFilesManager( object ):
self._GenerateFullSizeThumbnail( hash, mime )
if not self._bad_error_occured:
if not self._bad_error_occurred:
self._bad_error_occured = True
self._bad_error_occurred = True
HydrusData.ShowText( 'A thumbnail for a file, ' + hash.encode( 'hex' ) + ', was missing. It has been regenerated from the original file, but this event could indicate hard drive corruption. Please check everything is ok. This error may be occuring for many files, but this message will only display once per boot. If you are recovering from a fractured database, you may wish to run \'database->regenerate->all thumbnails\'.' )
@ -1093,9 +1093,9 @@ class ClientFilesManager( object ):
try:
if self._bad_error_occured:
if self._bad_error_occurred:
wx.MessageBox( 'A serious file error has previously occured during this session, so further file moving will not be reattempted. Please restart the client before trying again.' )
wx.MessageBox( 'A serious file error has previously occurred during this session, so further file moving will not be reattempted. Please restart the client before trying again.' )
return

2
include/ClientController.py
View File

@ -1254,7 +1254,7 @@ class Controller( HydrusController.HydrusController ):
except Exception as e:
text = 'A serious error occured while trying to start the program. The error will be shown next in a window. More information may have been written to client.log.'
text = 'A serious error occurred while trying to start the program. The error will be shown next in a window. More information may have been written to client.log.'
HydrusData.DebugPrint( 'If the db crashed, another error may be written just above ^.' )
HydrusData.DebugPrint( text )

94
include/ClientDB.py
View File

@ -2756,6 +2756,49 @@ class DB( HydrusDB.HydrusDB ):
def _ClearOrphanTables( self ):
service_ids = self._STL( self._c.execute( 'SELECT service_id FROM services;' ) )
table_prefixes = []
table_prefixes.append( 'repository_hash_id_map_' )
table_prefixes.append( 'repository_tag_id_map_' )
table_prefixes.append( 'repository_updates_' )
good_table_names = set()
for service_id in service_ids:
suffix = str( service_id )
for table_prefix in table_prefixes:
good_table_names.add( table_prefix + suffix )
existing_table_names = set()
existing_table_names.update( self._STS( self._c.execute( 'SELECT name FROM sqlite_master WHERE type = ?;', ( 'table', ) ) ) )
existing_table_names.update( self._STS( self._c.execute( 'SELECT name FROM external_master.sqlite_master WHERE type = ?;', ( 'table', ) ) ) )
existing_table_names = { name for name in existing_table_names if True in ( name.startswith( table_prefix ) for table_prefix in table_prefixes ) }
surplus_table_names = existing_table_names.difference( good_table_names )
surplus_table_names = list( surplus_table_names )
surplus_table_names.sort()
for table_name in surplus_table_names:
HydrusData.ShowText( 'Dropping ' + table_name )
self._c.execute( 'DROP table ' + table_name + ';' )
def _CreateDB( self ):
client_files_default = os.path.join( self._db_dir, 'client_files' )
@ -3192,6 +3235,18 @@ class DB( HydrusDB.HydrusDB ):
self._c.execute( 'DELETE FROM remote_thumbnails WHERE service_id = ?;', ( service_id, ) )
if service_type in HC.REPOSITORIES:
repository_updates_table_name = GenerateRepositoryRepositoryUpdatesTableName( service_id )
self._c.execute( 'DROP TABLE ' + repository_updates_table_name + ';' )
( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterCacheTableNames( service_id )
self._c.execute( 'DROP TABLE ' + hash_id_map_table_name + ';' )
self._c.execute( 'DROP TABLE ' + tag_id_map_table_name + ';' )
if service_type in HC.TAG_SERVICES:
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateMappingsTableNames( service_id )
@ -5309,7 +5364,7 @@ class DB( HydrusDB.HydrusDB ):
( timestamp, ) = result
note = 'Currently in trash. Sent there at ' + HydrusData.ConvertTimestampToPrettyTime( timestamp ) + ', which was ' + HydrusData.TimestampToPrettyTimeDelta( timestamp ) + ' (before this check).'
note = 'Currently in trash. Sent there at ' + HydrusData.ConvertTimestampToPrettyTime( timestamp ) + ', which was ' + HydrusData.TimestampToPrettyTimeDelta( timestamp, just_now_threshold = 0 ) + ' (before this check).'
return ( CC.STATUS_DELETED, hash, prefix + ': ' + note )
@ -5320,7 +5375,7 @@ class DB( HydrusDB.HydrusDB ):
( timestamp, ) = result
note = 'Imported at ' + HydrusData.ConvertTimestampToPrettyTime( timestamp ) + ', which was ' + HydrusData.TimestampToPrettyTimeDelta( timestamp ) + ' (before this check).'
note = 'Imported at ' + HydrusData.ConvertTimestampToPrettyTime( timestamp ) + ', which was ' + HydrusData.TimestampToPrettyTimeDelta( timestamp, just_now_threshold = 0 ) + ' (before this check).'
return ( CC.STATUS_SUCCESSFUL_BUT_REDUNDANT, hash, prefix + ': ' + note )
@ -6739,7 +6794,7 @@ class DB( HydrusDB.HydrusDB ):
if minimum_age is not None:
message += ' with minimum age ' + HydrusData.TimestampToPrettyTimeDelta( timestamp_cutoff ) + ','
message += ' with minimum age ' + HydrusData.TimestampToPrettyTimeDelta( timestamp_cutoff, just_now_threshold = 0 ) + ','
message += ' I found ' + HydrusData.ToHumanInt( len( hash_ids ) ) + '.'
@ -7125,7 +7180,7 @@ class DB( HydrusDB.HydrusDB ):
if HydrusData.TimeHasPassed( next_stop_time_presentation ):
HG.client_controller.pub( 'splash_set_status_subtext', 'cached ' + HydrusData.TimestampToPrettyTimeDelta( stop_time ) )
HG.client_controller.pub( 'splash_set_status_subtext', 'cached ' + HydrusData.TimestampToPrettyTimeDelta( stop_time, just_now_string = 'ok', just_now_threshold = 1 ) )
if HydrusData.TimeHasPassed( stop_time ):
@ -10585,6 +10640,36 @@ class DB( HydrusDB.HydrusDB ):
self.pub_initial_message( message )
if version == 314:
try:
domain_manager = self._GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )
domain_manager.Initialise()
#
domain_manager.OverwriteDefaultParsers( ( 'moebooru file page parser', 'tumblr api post page parser - with post tags' ) )
#
domain_manager.TryToLinkURLMatchesAndParsers()
#
self._SetJSONDump( domain_manager )
except Exception as e:
HydrusData.PrintException( e )
message = 'Trying to update some url classes and parsers failed! Please let hydrus dev know!'
self.pub_initial_message( message )
self._controller.pub( 'splash_set_title_text', 'updated db to v' + str( version + 1 ) )
self._c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
@ -11150,6 +11235,7 @@ class DB( HydrusDB.HydrusDB ):
elif action == 'associate_repository_update_hashes': result = self._AssociateRepositoryUpdateHashes( *args, **kwargs )
elif action == 'backup': result = self._Backup( *args, **kwargs )
elif action == 'clear_orphan_file_records': result = self._ClearOrphanFileRecords( *args, **kwargs )
elif action == 'clear_orphan_tables': result = self._ClearOrphanTables( *args, **kwargs )
elif action == 'content_updates': result = self._ProcessContentUpdates( *args, **kwargs )
elif action == 'db_integrity': result = self._CheckDBIntegrity( *args, **kwargs )
elif action == 'delete_hydrus_session_key': result = self._DeleteHydrusSessionKey( *args, **kwargs )

2
include/ClientData.py
View File

@ -410,7 +410,7 @@ def OrdIsNumber( o ):
def ReportShutdownException():
text = 'A serious error occured while trying to exit the program. Its traceback may be shown next. It should have also been written to client.log. You may need to quit the program from task manager.'
text = 'A serious error occurred while trying to exit the program. Its traceback may be shown next. It should have also been written to client.log. You may need to quit the program from task manager.'
HydrusData.DebugPrint( text )

5
include/ClientDownloading.py
View File

@ -262,6 +262,11 @@ class Gallery( object ):
network_job = self._network_job_factory( 'GET', url, referral_url = referral_url, temp_path = temp_path )
if temp_path is not None: # i.e. it is a file after a page fetch
network_job.OverrideBandwidth( 30 )
HG.client_controller.network_engine.AddJob( network_job )
try:

3
include/ClientFiles.py
View File

@ -1,4 +1,5 @@
import gc
import HydrusData
import HydrusExceptions
import HydrusGlobals as HG
import os
@ -35,5 +36,7 @@ def GetAllPaths( raw_paths ):
paths_to_process = next_paths_to_process
HydrusData.HumanTextSort( file_paths )
return file_paths

49
include/ClientGUI.py
View File

@ -704,6 +704,11 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
import_folder = self._controller.Read( 'serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_IMPORT_FOLDER, name )
if import_folder.Paused():
import_folder.PausePlay()
import_folders = [ import_folder ]
@ -771,6 +776,21 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _ClearOrphanTables( self ):
text = 'This will instruct the database to review its service tables and delete any orphans. This will typically do nothing, but hydrus dev may tell you to run this, just to check. Be sure you have a semi-recent backup before you run this.'
text += os.linesep * 2
text += 'It will create popups if it finds anything to delete.'
with ClientGUIDialogs.DialogYesNo( self, text, yes_label = 'do it', no_label = 'forget it' ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
self._controller.Write( 'clear_orphan_tables' )
def _DebugFetchAURL( self ):
def wx_code( network_job ):
@ -1079,7 +1099,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
if show:
menu_index = self._menubar.FindMenu( label )
menu_index = self._FindMenuBarIndex( menu )
self._menubar.EnableTop( menu_index, False )
@ -1119,6 +1139,19 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _FindMenuBarIndex( self, menu ):
for index in range( self._menubar.GetMenuCount() ):
if self._menubar.GetMenu( index ) == menu:
return index
raise HydrusExceptions.DataMissing( 'Menu not found!' )
def _ForceFitAllNonGUITLWs( self ):
tlws = wx.GetTopLevelWindows()
@ -1553,6 +1586,11 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
ClientGUIMenus.AppendMenuItem( self, submenu, 'clear orphan files', 'Clear out surplus files that have found their way into the file structure.', self._ClearOrphanFiles )
ClientGUIMenus.AppendMenuItem( self, submenu, 'clear orphan file records', 'Clear out surplus file records that have not been deleted correctly.', self._ClearOrphanFileRecords )
if self._controller.new_options.GetBoolean( 'advanced_mode' ):
ClientGUIMenus.AppendMenuItem( self, submenu, 'clear orphan tables', 'Clear out surplus db tables that have not been deleted correctly.', self._ClearOrphanTables )
ClientGUIMenus.AppendMenu( menu, submenu, 'maintain' )
submenu = wx.Menu()
@ -2832,7 +2870,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _RegenerateACCache( self ):
message = 'This will delete and then recreate the entire autocomplete cache. This is useful if miscounting has somehow occured.'
message = 'This will delete and then recreate the entire autocomplete cache. This is useful if miscounting has somehow occurred.'
message += os.linesep * 2
message += 'If you have a lot of tags and files, it can take a long time, during which the gui may hang.'
message += os.linesep * 2
@ -4291,7 +4329,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
if old_show:
old_menu_index = self._menubar.FindMenu( old_label )
old_menu_index = self._FindMenuBarIndex( old_menu )
if show:
@ -4310,7 +4348,10 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
for temp_name in MENU_ORDER:
if temp_name == name: break
if temp_name == name:
break
( temp_menu, temp_label, temp_show ) = self._menus[ temp_name ]

2
include/ClientGUICommon.py
View File

@ -3037,7 +3037,7 @@ class TagFilterButton( BetterButton ):
self.SetLabelText( tt[:32] )
self.SetToolTipString( tt )
self.SetToolTip( tt )
def GetValue( self ):

2
include/ClientGUIControls.py
View File

@ -540,7 +540,7 @@ class NetworkJobControl( wx.Panel ):
self._auto_override_bandwidth_rules = False
self._left_text = ClientGUICommon.BetterStaticText( self )
self._left_text = ClientGUICommon.BetterStaticText( self, style = wx.ST_ELLIPSIZE_END )
self._right_text = ClientGUICommon.BetterStaticText( self, style = wx.ALIGN_RIGHT )
# 512/768KB - 200KB/s

4
include/ClientGUIDialogs.py
View File

@ -1151,7 +1151,7 @@ class FrameInputLocalFiles( wx.Frame ):
unparsed_paths = []
unparsed_paths = collections.deque()
cancel_event.clear()
pause_event.clear()
@ -1789,7 +1789,7 @@ class DialogModifyAccounts( Dialog ):
def _DoModification( self ):
# change this to saveaccounts or whatever. the previous func changes the accounts, and then we push that change
# generate accounts, with the modification having occured
# generate accounts, with the modification having occurred
self._service.Request( HC.POST, 'account', { 'accounts' : self._accounts } )

111
include/ClientGUIDialogsManage.py
View File

@ -2220,16 +2220,15 @@ class DialogManageImportFolders( ClientGUIDialogs.Dialog ):
ClientGUIDialogs.Dialog.__init__( self, parent, 'manage import folders' )
self._import_folders = ClientGUIListCtrl.SaneListCtrlForSingleObject( self, 120, [ ( 'name', 120 ), ( 'path', -1 ), ( 'check period', 120 ) ], delete_key_callback = self.Delete, activation_callback = self.Edit )
import_folders_panel = ClientGUIListCtrl.BetterListCtrlPanel( self )
self._add_button = wx.Button( self, label = 'add' )
self._add_button.Bind( wx.EVT_BUTTON, self.EventAdd )
self._import_folders = ClientGUIListCtrl.BetterListCtrl( import_folders_panel, 'import_folders', 8, 36, [ ( 'name', 24 ), ( 'path', -1 ), ( 'paused', 8 ), ( 'check period', 24 ) ], self._ConvertImportFolderToListCtrlTuples, delete_key_callback = self._Delete, activation_callback = self._Edit )
self._edit_button = wx.Button( self, label = 'edit' )
self._edit_button.Bind( wx.EVT_BUTTON, self.EventEdit )
import_folders_panel.SetListCtrl( self._import_folders )
self._delete_button = wx.Button( self, label = 'delete' )
self._delete_button.Bind( wx.EVT_BUTTON, self.EventDelete )
import_folders_panel.AddButton( 'add', self._Add )
import_folders_panel.AddButton( 'edit', self._Edit, enabled_only_on_selection = True )
import_folders_panel.AddButton( 'delete', self._Delete, enabled_only_on_selection = True )
self._ok = wx.Button( self, id = wx.ID_OK, label = 'ok' )
self._ok.Bind( wx.EVT_BUTTON, self.EventOK )
@ -2242,21 +2241,12 @@ class DialogManageImportFolders( ClientGUIDialogs.Dialog ):
import_folders = HG.client_controller.Read( 'serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_IMPORT_FOLDER )
for import_folder in import_folders:
( display_tuple, sort_tuple ) = self._ConvertImportFolderToTuples( import_folder )
self._import_folders.Append( display_tuple, sort_tuple, import_folder )
self._import_folders.SetData( import_folders )
self._import_folders.Sort()
#
file_buttons = wx.BoxSizer( wx.HORIZONTAL )
file_buttons.Add( self._add_button, CC.FLAGS_VCENTER )
file_buttons.Add( self._edit_button, CC.FLAGS_VCENTER )
file_buttons.Add( self._delete_button, CC.FLAGS_VCENTER )
buttons = wx.BoxSizer( wx.HORIZONTAL )
buttons.Add( self._ok, CC.FLAGS_VCENTER )
@ -2275,8 +2265,7 @@ class DialogManageImportFolders( ClientGUIDialogs.Dialog ):
warning_st.SetForegroundColour( ( 128, 0, 0 ) )
vbox.Add( warning_st, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.Add( self._import_folders, CC.FLAGS_EXPAND_BOTH_WAYS )
vbox.Add( file_buttons, CC.FLAGS_BUTTON_SIZER )
vbox.Add( import_folders_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
vbox.Add( buttons, CC.FLAGS_BUTTON_SIZER )
self.SetSizer( vbox )
@ -2293,7 +2282,7 @@ class DialogManageImportFolders( ClientGUIDialogs.Dialog ):
wx.CallAfter( self._ok.SetFocus )
def _AddImportFolder( self ):
def _Add( self ):
import_folder = ClientImporting.ImportFolder( 'import folder' )
@ -2303,81 +2292,87 @@ class DialogManageImportFolders( ClientGUIDialogs.Dialog ):
import_folder = dlg.GetInfo()
self._import_folders.SetNonDupeName( import_folder )
import_folder.SetNonDupeName( self._GetExistingNames() )
( display_tuple, sort_tuple ) = self._ConvertImportFolderToTuples( import_folder )
self._import_folders.AddDatas( ( import_folder, ) )
self._import_folders.Append( display_tuple, sort_tuple, import_folder )
self._import_folders.Sort()
def _ConvertImportFolderToTuples( self, import_folder ):
def _ConvertImportFolderToListCtrlTuples( self, import_folder ):
sort_tuple = import_folder.ToListBoxTuple()
( name, path, paused, check_regularly, check_period ) = import_folder.ToListBoxTuple()
( name, path, check_period ) = sort_tuple
if paused:
pretty_paused = 'yes'
else:
pretty_paused = ''
pretty_check_period = HydrusData.TimeDeltaToPrettyTimeDelta( check_period )
if not check_regularly:
pretty_check_period = 'not checking regularly'
else:
pretty_check_period = HydrusData.TimeDeltaToPrettyTimeDelta( check_period )
display_tuple = ( name, path, pretty_check_period )
sort_tuple = ( name, path, paused, check_period )
display_tuple = ( name, path, pretty_paused, pretty_check_period )
return ( display_tuple, sort_tuple )
def Delete( self ):
def _Delete( self ):
with ClientGUIDialogs.DialogYesNo( self, 'Remove all selected?' ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
self._import_folders.RemoveAllSelected()
import_folders = self._import_folders.GetData( only_selected = True )
self._import_folders.DeleteDatas( import_folders )
def Edit( self ):
def _Edit( self ):
indices = self._import_folders.GetAllSelected()
import_folders = self._import_folders.GetData( only_selected = True )
for index in indices:
import_folder = self._import_folders.GetObject( index )
original_name = import_folder.GetName()
for import_folder in import_folders:
with DialogManageImportFoldersEdit( self, import_folder ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
import_folder = dlg.GetInfo()
self._import_folders.DeleteDatas( ( import_folder, ) )
if import_folder.GetName() != original_name:
self._import_folders.SetNonDupeName( import_folder )
edited_import_folder = dlg.GetInfo()
( display_tuple, sort_tuple ) = self._ConvertImportFolderToTuples( import_folder )
edited_import_folder.SetNonDupeName( self._GetExistingNames() )
self._import_folders.UpdateRow( index, display_tuple, sort_tuple, import_folder )
self._import_folders.AddDatas( ( edited_import_folder, ) )
self._import_folders.Sort()
def EventAdd( self, event ):
def _GetExistingNames( self ):
self._AddImportFolder()
import_folders = self._import_folders.GetData()
def EventDelete( self, event ):
names = { import_folder.GetName() for import_folder in import_folders }
self.Delete()
def EventEdit( self, event ):
self.Edit()
return names
def EventOK( self, event ):
@ -2386,7 +2381,7 @@ class DialogManageImportFolders( ClientGUIDialogs.Dialog ):
good_names = set()
import_folders = self._import_folders.GetObjects()
import_folders = self._import_folders.GetData()
for import_folder in import_folders:
@ -2395,7 +2390,7 @@ class DialogManageImportFolders( ClientGUIDialogs.Dialog ):
HG.client_controller.Write( 'serialisable', import_folder )
names_to_delete = existing_db_names - good_names
names_to_delete = existing_db_names.difference( good_names )
for name in names_to_delete:

2
include/ClientGUIFileSeedCache.py
View File

@ -567,7 +567,7 @@ class FileSeedCacheButton( ClientGUICommon.BetterBitmapButton ):
if num_vetoed > 0:
ClientGUIMenus.AppendMenuItem( self, menu, 'retry ' + HydrusData.ToHumanInt( num_errors ) + ' ignored', 'Tell this cache to reattempt all its ignored/vetoed results.', self._RetryIgnored )
ClientGUIMenus.AppendMenuItem( self, menu, 'retry ' + HydrusData.ToHumanInt( num_vetoed ) + ' ignored', 'Tell this cache to reattempt all its ignored/vetoed results.', self._RetryIgnored )
ClientGUIMenus.AppendSeparator( menu )

422
include/ClientGUIImport.py
View File

@ -1,11 +1,16 @@
import ClientConstants as CC
import ClientGUIACDropdown
import ClientGUICommon
import ClientGUIControls
import ClientGUIFileSeedCache
import ClientGUIGallerySeedLog
import ClientGUIListBoxes
import ClientGUIListCtrl
import ClientGUIMenus
import ClientGUIScrolledPanels
import ClientGUIScrolledPanelsEdit
import ClientGUIShortcuts
import ClientGUITime
import ClientGUITopLevelWindows
import ClientImporting
import ClientImportOptions
@ -21,6 +26,62 @@ import re
import wx
import wx.adv
class CheckerOptionsButton( ClientGUICommon.BetterButton ):
def __init__( self, parent, checker_options, update_callable = None ):
ClientGUICommon.BetterButton.__init__( self, parent, 'checker options', self._EditOptions )
self._checker_options = checker_options
self._update_callable = update_callable
self._SetToolTip()
def _EditOptions( self ):
with ClientGUITopLevelWindows.DialogEdit( self, 'edit checker options' ) as dlg:
panel = ClientGUITime.EditCheckerOptions( dlg, self._checker_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
checker_options = panel.GetValue()
self._SetValue( checker_options )
def _SetToolTip( self ):
self.SetToolTip( self._checker_options.GetSummary() )
def _SetValue( self, checker_options ):
self._checker_options = checker_options
self._SetToolTip()
if self._update_callable is not None:
self._update_callable( self._checker_options )
def GetValue( self ):
return self._checker_options
def SetValue( self, checker_options ):
self._SetValue( checker_options )
class FileImportOptionsButton( ClientGUICommon.BetterButton ):
def __init__( self, parent, file_import_options, update_callable = None ):
@ -1084,13 +1145,18 @@ class TagImportOptionsButton( ClientGUICommon.BetterButton ):
def _Paste( self ):
raw_text = HG.client_controller.GetClipboardText()
try:
tag_import_options = HydrusSerialisable.CreateFromString( raw_text )
if not isinstance( tag_import_options, ClientImportOptions.TagImportOptions ):
raise Exception( 'Not a Tag Import Options!' )
self._tag_import_options = tag_import_options
except Exception as e:
@ -1160,3 +1226,357 @@ class TagImportOptionsButton( ClientGUICommon.BetterButton ):
self._SetValue( tag_import_options )
class WatcherReviewPanel( ClientGUICommon.StaticBox ):
def __init__( self, parent, page_key ):
ClientGUICommon.StaticBox.__init__( self, parent, 'watcher' )
self._page_key = page_key
self._watcher = None
self._watcher_subject = ClientGUICommon.BetterStaticText( self, style = wx.ST_ELLIPSIZE_END )
self._url_input = wx.TextCtrl( self, style = wx.TE_PROCESS_ENTER )
self._url_input.Bind( wx.EVT_KEY_DOWN, self.EventKeyDown )
self._options_panel = wx.Panel( self )
#
imports_panel = ClientGUICommon.StaticBox( self._options_panel, 'file imports' )
self._files_pause_button = wx.BitmapButton( imports_panel, bitmap = CC.GlobalBMPs.pause )
self._files_pause_button.Bind( wx.EVT_BUTTON, self.EventPauseFiles )
self._current_action = ClientGUICommon.BetterStaticText( imports_panel, style = wx.ST_ELLIPSIZE_END )
self._file_seed_cache_control = ClientGUIFileSeedCache.FileSeedCacheStatusControl( imports_panel, self._controller, self._page_key )
self._file_download_control = ClientGUIControls.NetworkJobControl( imports_panel )
#
checker_panel = ClientGUICommon.StaticBox( self._options_panel, 'checker' )
self._file_velocity_status = ClientGUICommon.BetterStaticText( checker_panel, style = wx.ST_ELLIPSIZE_END )
self._checking_pause_button = wx.BitmapButton( checker_panel, bitmap = CC.GlobalBMPs.pause )
self._checking_pause_button.Bind( wx.EVT_BUTTON, self.EventPauseChecker )
self._watcher_status = ClientGUICommon.BetterStaticText( checker_panel, style = wx.ST_ELLIPSIZE_END )
self._check_now_button = wx.Button( checker_panel, label = 'check now' )
self._check_now_button.Bind( wx.EVT_BUTTON, self.EventCheckNow )
self._gallery_seed_log_control = ClientGUIGallerySeedLog.GallerySeedLogStatusControl( checker_panel, self._controller, True, page_key = self._page_key )
checker_options = ClientImportOptions.CheckerOptions()
self._checker_options_button = CheckerOptionsButton( checker_panel, checker_options, update_callable = self._SetCheckerOptions )
self._checker_download_control = ClientGUIControls.NetworkJobControl( checker_panel )
namespaces = []
file_import_options = ClientImportOptions.FileImportOptions()
tag_import_options = ClientImportOptions.TagImportOptions( is_default = True )
self._file_import_options = FileImportOptionsButton( self._watcher_panel, file_import_options, self._SetFileImportOptions )
self._tag_import_options = TagImportOptionsButton( self._watcher_panel, namespaces, tag_import_options, update_callable = self._SetTagImportOptions, allow_default_selection = True )
#
hbox = wx.BoxSizer( wx.HORIZONTAL )
hbox.Add( self._current_action, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
hbox.Add( self._files_pause_button, CC.FLAGS_VCENTER )
imports_panel.Add( hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
imports_panel.Add( self._file_seed_cache_control, CC.FLAGS_EXPAND_PERPENDICULAR )
imports_panel.Add( self._file_download_control, CC.FLAGS_EXPAND_PERPENDICULAR )
#
gridbox = wx.FlexGridSizer( 2 )
gridbox.AddGrowableCol( 0, 1 )
gridbox.Add( self._file_velocity_status, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
gridbox.Add( self._checking_pause_button, CC.FLAGS_LONE_BUTTON )
gridbox.Add( self._watcher_status, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
gridbox.Add( self._check_now_button, CC.FLAGS_VCENTER )
checker_panel.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
checker_panel.Add( self._gallery_seed_log_control, CC.FLAGS_EXPAND_PERPENDICULAR )
checker_panel.Add( self._checker_options_button, CC.FLAGS_EXPAND_PERPENDICULAR )
checker_panel.Add( self._checker_download_control, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.Add( imports_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.Add( checker_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
self._options_panel.SetSizer( vbox )
self._watcher_panel.Add( self._watcher_subject, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watcher_panel.Add( self._url_input, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watcher_panel.Add( self._options_panel, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
self._watcher_panel.Add( self._file_import_options, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watcher_panel.Add( self._tag_import_options, CC.FLAGS_EXPAND_PERPENDICULAR )
#
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.Add( self._sort_by, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self._collect_by.Hide()
vbox.Add( self._watcher_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
self._MakeCurrentSelectionTagsBox( vbox )
self.SetSizer( vbox )
#
self._UpdateStatus()
def _SetCheckerOptions( self, checker_options ):
if self._watcher is not None:
self._watcher_import.SetCheckerOptions( checker_options )
def _SetFileImportOptions( self, file_import_options ):
if self._watcher is not None:
self._watcher.SetFileImportOptions( file_import_options )
def _SetTagImportOptions( self, tag_import_options ):
if self._watcher is not None:
self._watcher.SetTagImportOptions( tag_import_options )
def _UpdateControlsForNewWatcher( self ):
if self._watcher is None:
pass # clear and disable everything
else:
# fetch url
url = 'blah'
if url is None or url == '':
pass # set up as a new url input, ready to init thread watcher
else:
pass # present everything else
# update CO button with value
# update FIO button with value
# update TIO button with value
file_seed_cache = self._watcher_import.GetFileSeedCache()
self._file_seed_cache_control.SetFileSeedCache( file_seed_cache )
gallery_seed_log = self._watcher_import.GetGallerySeedLog()
self._gallery_seed_log_control.SetGallerySeedLog( gallery_seed_log )
self._watcher_import.SetDownloadControlFile( self._file_download_control )
self._watcher_import.SetDownloadControlChecker( self._checker_download_control )
self._url_input.SetValue( url )
def _UpdateStatus( self ):
if self._watcher_import.HasURL():
self._url_input.SetEditable( False )
if not self._options_panel.IsShown():
self._watcher_subject.Show()
self._options_panel.Show()
self.Layout()
else:
if self._options_panel.IsShown():
self._watcher_subject.Hide()
self._options_panel.Hide()
self.Layout()
( current_action, files_paused, file_velocity_status, next_check_time, watcher_status, subject, checking_status, check_now, checking_paused ) = self._watcher_import.GetStatus()
if files_paused:
if current_action == '':
current_action = 'paused'
else:
current_action = 'pausing, ' + current_action
ClientGUICommon.SetBitmapButtonBitmap( self._files_pause_button, CC.GlobalBMPs.play )
else:
ClientGUICommon.SetBitmapButtonBitmap( self._files_pause_button, CC.GlobalBMPs.pause )
self._current_action.SetLabelText( current_action )
self._file_velocity_status.SetLabelText( file_velocity_status )
if checking_paused:
if watcher_status == '':
watcher_status = 'paused'
ClientGUICommon.SetBitmapButtonBitmap( self._checking_pause_button, CC.GlobalBMPs.play )
else:
if watcher_status == '' and next_check_time is not None:
if HydrusData.TimeHasPassed( next_check_time ):
watcher_status = 'checking imminently'
else:
watcher_status = 'next check ' + HydrusData.TimestampToPrettyTimeDelta( next_check_time, just_now_threshold = 0 )
ClientGUICommon.SetBitmapButtonBitmap( self._checking_pause_button, CC.GlobalBMPs.pause )
self._watcher_status.SetLabelText( watcher_status )
if checking_status == ClientImporting.CHECKER_STATUS_404:
self._checking_pause_button.Disable()
elif checking_status == ClientImporting.CHECKER_STATUS_DEAD:
self._checking_pause_button.Disable()
else:
self._checking_pause_button.Enable()
if subject in ( '', 'unknown subject' ):
subject = 'no subject'
self._watcher_subject.SetLabelText( subject )
if check_now:
self._check_now_button.Disable()
else:
self._check_now_button.Enable()
def EventCheckNow( self, event ):
self._watcher_import.CheckNow()
self._UpdateStatus()
def EventKeyDown( self, event ):
( modifier, key ) = ClientGUIShortcuts.ConvertKeyEventToSimpleTuple( event )
if key in ( wx.WXK_RETURN, wx.WXK_NUMPAD_ENTER ):
url = self._url_input.GetValue()
if url == '':
return
self._url_input.SetEditable( False )
self._watcher.SetURL( url )
publish_to_page = True
self._watcher.Start( self._page_key, publish_to_page )
self._UpdateControlsForNewWatcher()
else:
event.Skip()
def EventPauseFiles( self, event ):
self._watcher_import.PausePlayFiles()
self._UpdateStatus()
def EventPauseChecker( self, event ):
self._watcher_import.PausePlayChecker()
self._UpdateStatus()
def SetSearchFocus( self ):
wx.CallAfter( self._url_input.SetFocus )
def SetWatcher( self, watcher ):
self._watcher = watcher
self._UpdateControlsForNewWatcher()
def UpdateStatus( self ):
self._UpdateStatus()

15
include/ClientGUIListCtrl.py
View File

@ -496,9 +496,18 @@ class SaneListCtrlPanel( wx.Panel ):
class BetterListCtrl( wx.ListCtrl, ListCtrlAutoWidthMixin ):
def __init__( self, parent, name, height_num_chars, sizing_column_initial_width_num_chars, columns, data_to_tuples_func, delete_key_callback = None, activation_callback = None ):
def __init__( self, parent, name, height_num_chars, sizing_column_initial_width_num_chars, columns, data_to_tuples_func, delete_key_callback = None, activation_callback = None, style = None ):
wx.ListCtrl.__init__( self, parent, style = wx.LC_REPORT )
if style is None:
style = wx.LC_REPORT
else:
style = wx.LC_REPORT | style
wx.ListCtrl.__init__( self, parent, style = style )
ListCtrlAutoWidthMixin.__init__( self )
self._data_to_tuples_func = data_to_tuples_func
@ -512,7 +521,7 @@ class BetterListCtrl( wx.ListCtrl, ListCtrlAutoWidthMixin ):
self._indices_to_data_info = {}
self._data_to_indices = {}
( total_width, height ) = ClientGUICommon.ConvertTextToPixels( self, ( sizing_column_initial_width_num_chars, height_num_chars ) )
( total_width, height ) = ClientGUICommon.ConvertTextToPixels( self, ( sizing_column_initial_width_num_chars, height_num_chars + 2 ) ) # +2 for the header
resize_column = 1

141
include/ClientGUIManagement.py
View File

@ -28,6 +28,7 @@ import ClientGUIShortcuts
import ClientGUITime
import ClientGUITopLevelWindows
import ClientImporting
import ClientImportGallery
import ClientImportOptions
import ClientImportWatchers
import ClientMedia
@ -100,9 +101,9 @@ def CreateManagementControllerImportGallery( gallery_identifier ):
management_controller = CreateManagementController( page_name, MANAGEMENT_TYPE_IMPORT_GALLERY )
gallery_import = ClientImporting.GalleryImport( gallery_identifier = gallery_identifier )
multiple_gallery_import = ClientImportGallery.MultipleGalleryImport( gallery_identifier = gallery_identifier )
management_controller.SetVariable( 'gallery_import', gallery_import )
management_controller.SetVariable( 'multiple_gallery_import', multiple_gallery_import )
return management_controller
@ -561,7 +562,7 @@ class ManagementController( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_MANAGEMENT_CONTROLLER
SERIALISABLE_NAME = 'Client Page Management Controller'
SERIALISABLE_VERSION = 5
SERIALISABLE_VERSION = 6
def __init__( self, page_name = 'page' ):
@ -713,6 +714,22 @@ class ManagementController( HydrusSerialisable.SerialisableBase ):
return ( 5, new_serialisable_info )
if version == 5:
( page_name, management_type, serialisable_keys, serialisable_simples, serialisable_serialisables ) = old_serialisable_info
if 'gallery_import' in serialisable_serialisables:
serialisable_serialisables[ 'multiple_gallery_import' ] = serialisable_serialisables[ 'gallery_import' ]
del serialisable_serialisables[ 'gallery_import' ]
new_serialisable_info = ( page_name, management_type, serialisable_keys, serialisable_simples, serialisable_serialisables )
return ( 6, new_serialisable_info )
def GetKey( self, name ):
@ -735,9 +752,9 @@ class ManagementController( HydrusSerialisable.SerialisableBase ):
if self._management_type == MANAGEMENT_TYPE_IMPORT_GALLERY:
gallery_import = self._serialisables[ 'gallery_import' ]
multiple_gallery_import = self._serialisables[ 'multiple_gallery_import' ]
return gallery_import.GetValueRange()
return multiple_gallery_import.GetValueRange()
elif self._management_type == MANAGEMENT_TYPE_IMPORT_HDD:
@ -1430,13 +1447,13 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
ManagementPanelImporter.__init__( self, parent, page, controller, management_controller )
self._gallery_import = self._management_controller.GetVariable( 'gallery_import' )
self._multiple_gallery_import = self._management_controller.GetVariable( 'multiple_gallery_import' )
self._gallery_downloader_panel = ClientGUICommon.StaticBox( self, 'gallery downloader' )
self._import_queue_panel = ClientGUICommon.StaticBox( self._gallery_downloader_panel, 'imports' )
self._current_action = ClientGUICommon.BetterStaticText( self._import_queue_panel )
self._current_action = ClientGUICommon.BetterStaticText( self._import_queue_panel, style = wx.ST_ELLIPSIZE_END )
self._file_seed_cache_control = ClientGUIFileSeedCache.FileSeedCacheStatusControl( self._import_queue_panel, self._controller, self._page_key )
self._file_download_control = ClientGUIControls.NetworkJobControl( self._import_queue_panel )
@ -1445,7 +1462,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
self._gallery_panel = ClientGUICommon.StaticBox( self._gallery_downloader_panel, 'gallery parser' )
self._gallery_status = ClientGUICommon.BetterStaticText( self._gallery_panel )
self._gallery_status = ClientGUICommon.BetterStaticText( self._gallery_panel, style = wx.ST_ELLIPSIZE_END )
self._gallery_pause_button = wx.BitmapButton( self._gallery_panel, bitmap = CC.GlobalBMPs.pause )
self._gallery_pause_button.Bind( wx.EVT_BUTTON, self.EventGalleryPause )
@ -1476,16 +1493,16 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
self._file_limit.Bind( wx.EVT_SPINCTRL, self.EventFileLimit )
self._file_limit.SetToolTip( 'per query, stop searching the gallery once this many files has been reached' )
self._gallery_import.SetDownloadControls( self._file_download_control, self._gallery_download_control )
self._multiple_gallery_import.SetDownloadControls( self._file_download_control, self._gallery_download_control )
( file_import_options, tag_import_options, file_limit ) = self._gallery_import.GetOptions()
( file_import_options, tag_import_options, file_limit ) = self._multiple_gallery_import.GetOptions()
gallery_identifier = self._gallery_import.GetGalleryIdentifier()
gallery_identifier = self._multiple_gallery_import.GetGalleryIdentifier()
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( gallery_identifier )
self._file_import_options = ClientGUIImport.FileImportOptionsButton( self._gallery_downloader_panel, file_import_options, self._gallery_import.SetFileImportOptions )
self._tag_import_options = ClientGUIImport.TagImportOptionsButton( self._gallery_downloader_panel, namespaces, tag_import_options, update_callable = self._gallery_import.SetTagImportOptions, allow_default_selection = True )
self._file_import_options = ClientGUIImport.FileImportOptionsButton( self._gallery_downloader_panel, file_import_options, self._multiple_gallery_import.SetFileImportOptions )
self._tag_import_options = ClientGUIImport.TagImportOptionsButton( self._gallery_downloader_panel, namespaces, tag_import_options, update_callable = self._multiple_gallery_import.SetTagImportOptions, allow_default_selection = True )
#
@ -1549,11 +1566,11 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
#
file_seed_cache = self._gallery_import.GetFileSeedCache()
file_seed_cache = self._multiple_gallery_import.GetFileSeedCache()
self._file_seed_cache_control.SetFileSeedCache( file_seed_cache )
gallery_seed_log = self._gallery_import.GetGallerySeedLog()
gallery_seed_log = self._multiple_gallery_import.GetGallerySeedLog()
self._gallery_seed_log_control.SetGallerySeedLog( gallery_seed_log )
@ -1568,7 +1585,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
for query in queries:
self._gallery_import.PendQuery( query )
self._multiple_gallery_import.PendQuery( query )
self._UpdateStatus()
@ -1576,7 +1593,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
def _UpdateStatus( self ):
( pending_queries, gallery_status, current_action, files_paused, gallery_paused, gallery_cancellable ) = self._gallery_import.GetStatus()
( pending_queries, gallery_status, current_action, files_paused, gallery_paused, gallery_cancellable ) = self._multiple_gallery_import.GetStatus()
if self._pending_queries_listbox.GetStrings() != pending_queries:
@ -1655,7 +1672,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
def CheckAbleToClose( self ):
if self._gallery_import.CurrentlyWorking():
if self._multiple_gallery_import.CurrentlyWorking():
raise HydrusExceptions.VetoException( 'This page is still importing.' )
@ -1669,7 +1686,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
if len( selected_strings ) > 0:
self._gallery_import.AdvanceQueries( selected_strings )
self._multiple_gallery_import.AdvanceQueries( selected_strings )
self._UpdateStatus()
@ -1683,7 +1700,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
if len( selected_strings ) > 0:
self._gallery_import.DelayQueries( selected_strings )
self._multiple_gallery_import.DelayQueries( selected_strings )
self._UpdateStatus()
@ -1697,7 +1714,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
if len( selected_strings ) > 0:
self._gallery_import.DeleteQueries( selected_strings )
self._multiple_gallery_import.DeleteQueries( selected_strings )
self._UpdateStatus()
@ -1705,28 +1722,28 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
def EventFileLimit( self, event ):
self._gallery_import.SetFileLimit( self._file_limit.GetValue() )
self._multiple_gallery_import.SetFileLimit( self._file_limit.GetValue() )
event.Skip()
def EventFilesPause( self, event ):
self._gallery_import.PausePlayFiles()
self._multiple_gallery_import.PausePlayFiles()
self._UpdateStatus()
def EventGalleryCancel( self, event ):
self._gallery_import.FinishCurrentQuery()
self._multiple_gallery_import.FinishCurrentQuery()
self._UpdateStatus()
def EventGalleryPause( self, event ):
self._gallery_import.PausePlayGallery()
self._multiple_gallery_import.PausePlayGallery()
self._UpdateStatus()
@ -1738,7 +1755,7 @@ class ManagementPanelImporterGallery( ManagementPanelImporter ):
def Start( self ):
self._gallery_import.Start( self._page_key )
self._multiple_gallery_import.Start( self._page_key )
management_panel_types_to_classes[ MANAGEMENT_TYPE_IMPORT_GALLERY ] = ManagementPanelImporterGallery
@ -1854,7 +1871,7 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
self._highlighted_watcher = self._multiple_watcher_import.GetHighlightedWatcher()
( self._checker_options, file_import_options, tag_import_options ) = self._multiple_watcher_import.GetOptions()
( checker_options, file_import_options, tag_import_options ) = self._multiple_watcher_import.GetOptions()
#
@ -1890,10 +1907,9 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
self._watcher_url_input = ClientGUIControls.TextAndPasteCtrl( self._watchers_panel, self._AddURLs )
self._checker_options_button = ClientGUICommon.BetterButton( self._watchers_panel, 'check timings', self._EditCheckerOptions )
namespaces = []
self._checker_options = ClientGUIImport.CheckerOptionsButton( self._watchers_panel, checker_options, self._OptionsUpdated )
self._file_import_options = ClientGUIImport.FileImportOptionsButton( self._watchers_panel, file_import_options, self._OptionsUpdated )
self._tag_import_options = ClientGUIImport.TagImportOptionsButton( self._watchers_panel, namespaces, tag_import_options, update_callable = self._OptionsUpdated, allow_default_selection = True )
@ -1906,7 +1922,7 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
self._watchers_panel.Add( self._watchers_status_st_bottom, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watchers_panel.Add( self._watchers_listctrl_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
self._watchers_panel.Add( self._watcher_url_input, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watchers_panel.Add( self._checker_options_button, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watchers_panel.Add( self._checker_options, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watchers_panel.Add( self._file_import_options, CC.FLAGS_EXPAND_PERPENDICULAR )
self._watchers_panel.Add( self._tag_import_options, CC.FLAGS_EXPAND_PERPENDICULAR )
@ -2043,23 +2059,6 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
return ( display_tuple, sort_tuple )
def _EditCheckerOptions( self ):
with ClientGUITopLevelWindows.DialogEdit( self._checker_options_button, 'edit check timings' ) as dlg:
panel = ClientGUITime.EditCheckerOptions( dlg, self._checker_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
self._checker_options = panel.GetValue()
self._OptionsUpdated()
def _HighlightWatcher( self ):
selected = self._watchers_listctrl.GetData( only_selected = True )
@ -2107,7 +2106,7 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
def _OptionsUpdated( self, *args, **kwargs ):
self._multiple_watcher_import.SetOptions( self._checker_options, self._file_import_options.GetValue(), self._tag_import_options.GetValue() )
self._multiple_watcher_import.SetOptions( self._checker_options.GetValue(), self._file_import_options.GetValue(), self._tag_import_options.GetValue() )
def _PausePlay( self ):
@ -2202,12 +2201,13 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
if dlg.ShowModal() == wx.ID_YES:
checker_options = self._checker_options.GetValue()
file_import_options = self._file_import_options.GetValue()
tag_import_options = self._tag_import_options.GetValue()
for watcher in watchers:
watcher.SetCheckerOptions( self._checker_options )
watcher.SetCheckerOptions( checker_options )
watcher.SetFileImportOptions( file_import_options )
watcher.SetTagImportOptions( tag_import_options )
@ -2345,7 +2345,7 @@ class ManagementPanelImporterSimpleDownloader( ManagementPanelImporter ):
self._pause_files_button = wx.BitmapButton( self._import_queue_panel, bitmap = CC.GlobalBMPs.pause )
self._pause_files_button.Bind( wx.EVT_BUTTON, self.EventPauseFiles )
self._current_action = ClientGUICommon.BetterStaticText( self._import_queue_panel )
self._current_action = ClientGUICommon.BetterStaticText( self._import_queue_panel, style = wx.ST_ELLIPSIZE_END )
self._file_seed_cache_control = ClientGUIFileSeedCache.FileSeedCacheStatusControl( self._import_queue_panel, self._controller, self._page_key )
self._file_download_control = ClientGUIControls.NetworkJobControl( self._import_queue_panel )
@ -2358,7 +2358,7 @@ class ManagementPanelImporterSimpleDownloader( ManagementPanelImporter ):
self._pause_queue_button = wx.BitmapButton( self._simple_parsing_jobs_panel, bitmap = CC.GlobalBMPs.pause )
self._pause_queue_button.Bind( wx.EVT_BUTTON, self.EventPauseQueue )
self._parser_status = ClientGUICommon.BetterStaticText( self._simple_parsing_jobs_panel )
self._parser_status = ClientGUICommon.BetterStaticText( self._simple_parsing_jobs_panel, style = wx.ST_ELLIPSIZE_END )
self._gallery_seed_log_control = ClientGUIGallerySeedLog.GallerySeedLogStatusControl( self._simple_parsing_jobs_panel, self._controller, True, self._page_key )
@ -2781,7 +2781,7 @@ class ManagementPanelImporterWatcher( ManagementPanelImporter ):
self._files_pause_button = wx.BitmapButton( imports_panel, bitmap = CC.GlobalBMPs.pause )
self._files_pause_button.Bind( wx.EVT_BUTTON, self.EventPauseFiles )
self._current_action = ClientGUICommon.BetterStaticText( imports_panel )
self._current_action = ClientGUICommon.BetterStaticText( imports_panel, style = wx.ST_ELLIPSIZE_END )
self._file_seed_cache_control = ClientGUIFileSeedCache.FileSeedCacheStatusControl( imports_panel, self._controller, self._page_key )
self._file_download_control = ClientGUIControls.NetworkJobControl( imports_panel )
@ -2789,26 +2789,28 @@ class ManagementPanelImporterWatcher( ManagementPanelImporter ):
checker_panel = ClientGUICommon.StaticBox( self._options_panel, 'checker' )
self._file_velocity_status = ClientGUICommon.BetterStaticText( checker_panel )
self._file_velocity_status = ClientGUICommon.BetterStaticText( checker_panel, style = wx.ST_ELLIPSIZE_END )
self._checking_pause_button = wx.BitmapButton( checker_panel, bitmap = CC.GlobalBMPs.pause )
self._checking_pause_button.Bind( wx.EVT_BUTTON, self.EventPauseChecker )
self._watcher_status = ClientGUICommon.BetterStaticText( checker_panel )
self._watcher_status = ClientGUICommon.BetterStaticText( checker_panel, style = wx.ST_ELLIPSIZE_END )
self._check_now_button = wx.Button( checker_panel, label = 'check now' )
self._check_now_button.Bind( wx.EVT_BUTTON, self.EventCheckNow )
self._gallery_seed_log_control = ClientGUIGallerySeedLog.GallerySeedLogStatusControl( checker_panel, self._controller, True, page_key = self._page_key )
self._checker_options_button = ClientGUICommon.BetterButton( checker_panel, 'edit check timings', self._EditCheckerOptions )
self._watcher_import = self._management_controller.GetVariable( 'watcher_import' )
checker_options = self._watcher_import.GetCheckerOptions()
self._checker_options = ClientGUIImport.CheckerOptionsButton( checker_panel, checker_options, self._watcher_import.SetCheckerOptions )
self._checker_download_control = ClientGUIControls.NetworkJobControl( checker_panel )
#
self._watcher_import = self._management_controller.GetVariable( 'watcher_import' )
( url, file_import_options, tag_import_options ) = self._watcher_import.GetOptions()
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( ClientDownloading.GalleryIdentifier( HC.SITE_TYPE_WATCHER ) )
@ -2840,7 +2842,7 @@ class ManagementPanelImporterWatcher( ManagementPanelImporter ):
checker_panel.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
checker_panel.Add( self._gallery_seed_log_control, CC.FLAGS_EXPAND_PERPENDICULAR )
checker_panel.Add( self._checker_options_button, CC.FLAGS_EXPAND_PERPENDICULAR )
checker_panel.Add( self._checker_options, CC.FLAGS_EXPAND_PERPENDICULAR )
checker_panel.Add( self._checker_download_control, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox = wx.BoxSizer( wx.VERTICAL )
@ -2888,27 +2890,6 @@ class ManagementPanelImporterWatcher( ManagementPanelImporter ):
self._UpdateStatus()
def _EditCheckerOptions( self ):
checker_options = self._watcher_import.GetCheckerOptions()
with ClientGUITopLevelWindows.DialogEdit( self._checker_options_button, 'edit check timings' ) as dlg:
panel = ClientGUITime.EditCheckerOptions( dlg, checker_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
new_checker_options = panel.GetValue()
self._watcher_import.SetCheckerOptions( new_checker_options )
self._UpdateStatus()
def _UpdateStatus( self ):
if self._watcher_import.HasURL():
@ -2979,7 +2960,7 @@ class ManagementPanelImporterWatcher( ManagementPanelImporter ):
else:
watcher_status = 'next check ' + HydrusData.TimestampToPrettyTimeDelta( next_check_time )
watcher_status = 'next check ' + HydrusData.TimestampToPrettyTimeDelta( next_check_time, just_now_threshold = 0 )

128
include/ClientGUIScrolledPanelsEdit.py
View File

@ -340,6 +340,8 @@ class EditDefaultTagImportOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
self._list_ctrl_panel.SetListCtrl( self._list_ctrl )
self._list_ctrl_panel.AddButton( 'copy', self._Copy, enabled_check_func = self._OnlyOneTIOSelected )
self._list_ctrl_panel.AddButton( 'paste', self._Paste, enabled_only_on_selection = True )
self._list_ctrl_panel.AddButton( 'edit', self._Edit, enabled_only_on_selection = True )
self._list_ctrl_panel.AddButton( 'clear', self._Clear, enabled_only_on_selection = True )
@ -417,6 +419,27 @@ class EditDefaultTagImportOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
def _Copy( self ):
selected = self._list_ctrl.GetData( only_selected = True )
if len( selected ) == 1:
url_match = selected[0]
url_match_key = url_match.GetMatchKey()
if url_match_key in self._url_match_keys_to_tag_import_options:
tag_import_options = self._url_match_keys_to_tag_import_options[ url_match_key ]
json_string = tag_import_options.DumpToString()
HG.client_controller.pub( 'clipboard', 'text', json_string )
def _Edit( self ):
url_matches_to_edit = self._list_ctrl.GetData( only_selected = True )
@ -478,6 +501,55 @@ class EditDefaultTagImportOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
return ( namespaces, tag_import_options )
def _OnlyOneTIOSelected( self ):
selected = self._list_ctrl.GetData( only_selected = True )
if len( selected ) == 1:
url_match = selected[0]
url_match_key = url_match.GetMatchKey()
if url_match_key in self._url_match_keys_to_tag_import_options:
return True
return False
def _Paste( self ):
raw_text = HG.client_controller.GetClipboardText()
try:
tag_import_options = HydrusSerialisable.CreateFromString( raw_text )
if not isinstance( tag_import_options, ClientImportOptions.TagImportOptions ):
raise Exception( 'Not a Tag Import Options!' )
for url_match in self._list_ctrl.GetData( only_selected = True ):
url_match_key = url_match.GetMatchKey()
self._url_match_keys_to_tag_import_options[ url_match_key ] = tag_import_options.Duplicate()
self._list_ctrl.UpdateDatas()
except Exception as e:
wx.MessageBox( 'I could not understand what was in the clipboard' )
HydrusData.ShowException( e )
def GetValue( self ):
file_post_default_tag_import_options = self._file_post_default_tag_import_options_button.GetValue()
@ -2232,7 +2304,9 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
queries_panel.AddButton( 'check now', self._CheckNow, enabled_check_func = self._ListCtrlCanCheckNow )
queries_panel.AddButton( 'reset cache', self._ResetCache, enabled_check_func = self._ListCtrlCanResetCache )
self._checker_options_button = ClientGUICommon.BetterButton( self._query_panel, 'edit check timings', self._EditCheckerOptions )
( name, gallery_identifier, gallery_stream_identifiers, queries, checker_options, initial_file_limit, periodic_file_limit, paused, file_import_options, tag_import_options, self._no_work_until, self._no_work_until_reason ) = subscription.ToTuple()
self._checker_options = ClientGUIImport.CheckerOptionsButton( self._query_panel, checker_options, update_callable = self._CheckerOptionsUpdated )
#
@ -2260,8 +2334,6 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
#
( name, gallery_identifier, gallery_stream_identifiers, queries, self._checker_options, initial_file_limit, periodic_file_limit, paused, file_import_options, tag_import_options, self._no_work_until, self._no_work_until_reason ) = subscription.ToTuple()
self._file_import_options = ClientGUIImport.FileImportOptionsButton( self, file_import_options )
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( gallery_identifier )
@ -2312,7 +2384,7 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
self._query_panel.Add( self._site_type, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.Add( self._booru_selector, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.Add( queries_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
self._query_panel.Add( self._checker_options_button, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.Add( self._checker_options, CC.FLAGS_EXPAND_PERPENDICULAR )
#
@ -2388,6 +2460,16 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
def _CheckerOptionsUpdated( self, checker_options ):
for query in self._queries.GetData():
query.UpdateNextCheckTime( checker_options )
self._queries.UpdateDatas()
def _CheckNow( self ):
selected_queries = self._queries.GetData( only_selected = True )
@ -2447,7 +2529,7 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
if last_check_time == 0 or last_check_time is None:
pretty_last_check_time = '(initial check has not yet occured)'
pretty_last_check_time = '(initial check has not yet occurred)'
else:
@ -2456,8 +2538,10 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
pretty_next_check_time = query.GetNextCheckStatusString()
file_velocity = self._checker_options.GetRawCurrentVelocity( query.GetFileSeedCache(), last_check_time )
pretty_file_velocity = self._checker_options.GetPrettyCurrentVelocity( query.GetFileSeedCache(), last_check_time, no_prefix = True )
checker_options = self._checker_options.GetValue()
file_velocity = checker_options.GetRawCurrentVelocity( query.GetFileSeedCache(), last_check_time )
pretty_file_velocity = checker_options.GetPrettyCurrentVelocity( query.GetFileSeedCache(), last_check_time, no_prefix = True )
estimate = self._original_subscription.GetBandwidthWaitingEstimate( query )
@ -2521,28 +2605,6 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
def _EditCheckerOptions( self ):
with ClientGUITopLevelWindows.DialogEdit( self._checker_options_button, 'edit check timings' ) as dlg:
panel = ClientGUITime.EditCheckerOptions( dlg, self._checker_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
self._checker_options = panel.GetValue()
for query in self._queries.GetData():
query.UpdateNextCheckTime( self._checker_options )
self._queries.UpdateDatas()
def _EditQuery( self ):
selected_queries = self._queries.GetData( only_selected = True )
@ -2823,7 +2885,7 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
else:
status = 'delayed--retrying ' + HydrusData.TimestampToPrettyTimeDelta( self._no_work_until ) + ' because: ' + self._no_work_until_reason
status = 'delayed--retrying ' + HydrusData.TimestampToPrettyTimeDelta( self._no_work_until, just_now_threshold = 0 ) + ' because: ' + self._no_work_until_reason
self._delay_st.SetLabelText( status )
@ -2857,11 +2919,11 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
paused = self._paused.GetValue()
checker_options = self._checker_options.GetValue()
file_import_options = self._file_import_options.GetValue()
tag_import_options = self._tag_import_options.GetValue()
subscription.SetTuple( gallery_identifier, gallery_stream_identifiers, queries, self._checker_options, initial_file_limit, periodic_file_limit, paused, file_import_options, tag_import_options, self._no_work_until )
subscription.SetTuple( gallery_identifier, gallery_stream_identifiers, queries, checker_options, initial_file_limit, periodic_file_limit, paused, file_import_options, tag_import_options, self._no_work_until )
publish_files_to_popup_button = self._publish_files_to_popup_button.GetValue()
publish_files_to_page = self._publish_files_to_page.GetValue()
@ -3247,7 +3309,7 @@ class EditSubscriptionsPanel( ClientGUIScrolledPanels.EditPanel ):
else:
pretty_delay = 'delayed--retrying ' + HydrusData.TimestampToPrettyTimeDelta( no_work_until ) + ' - because: ' + no_work_until_reason
pretty_delay = 'delayed--retrying ' + HydrusData.TimestampToPrettyTimeDelta( no_work_until, just_now_threshold = 0 ) + ' - because: ' + no_work_until_reason
delay = HydrusData.GetTimeDeltaUntilTime( no_work_until )

96
include/ClientGUIScrolledPanelsManagement.py
View File

@ -7,6 +7,7 @@ import ClientGUIACDropdown
import ClientGUICommon
import ClientGUIControls
import ClientGUIDialogs
import ClientGUIImport
import ClientGUIListBoxes
import ClientGUIListCtrl
import ClientGUIPredicates
@ -1711,6 +1712,10 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._process_subs_in_random_order = wx.CheckBox( subscriptions )
self._process_subs_in_random_order.SetToolTip( 'Processing in random order is useful whenever bandwidth is tight, as it stops an \'aardvark\' subscription from always getting first whack at what is available. Otherwise, they will be processed in alphabetical order.' )
checker_options = self._new_options.GetDefaultSubscriptionCheckerOptions()
self._subscription_checker_options = ClientGUIImport.CheckerOptionsButton( subscriptions, checker_options )
#
watchers = ClientGUICommon.StaticBox( self, 'watchers' )
@ -1725,7 +1730,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
checker_options = self._new_options.GetDefaultWatcherCheckerOptions()
self._watcher_checker_options = ClientGUITime.EditCheckerOptions( watchers, checker_options )
self._watcher_checker_options = ClientGUIImport.CheckerOptionsButton( watchers, checker_options )
#
@ -1788,6 +1793,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
gridbox = ClientGUICommon.WrapInGrid( subscriptions, rows )
subscriptions.Add( gridbox, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
subscriptions.Add( self._subscription_checker_options, CC.FLAGS_EXPAND_PERPENDICULAR )
#
@ -1844,6 +1850,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._new_options.SetNoneableString( 'thread_watcher_paused_page_string', self._thread_watcher_paused_page_string.GetValue() )
self._new_options.SetDefaultWatcherCheckerOptions( self._watcher_checker_options.GetValue() )
self._new_options.SetDefaultSubscriptionCheckerOptions( self._subscription_checker_options.GetValue() )
self._new_options.SetBoolean( 'show_deleted_on_file_seed_short_summary', self._show_deleted_on_file_seed_short_summary.GetValue() )
@ -1940,28 +1947,6 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._loud_fios = ClientGUIImport.FileImportOptionsButton( default_fios, loud_file_import_options )
#
default_tios = ClientGUICommon.StaticBox( self, 'default tag import options' )
self._tag_import_options = wx.ListBox( default_tios )
self._tag_import_options.Bind( wx.EVT_LEFT_DCLICK, self.EventEdit )
self._edit = wx.Button( default_tios, label = 'edit' )
self._edit.Bind( wx.EVT_BUTTON, self.EventEdit )
self._delete = wx.Button( default_tios, label = 'delete' )
self._delete.Bind( wx.EVT_BUTTON, self.EventDelete )
#
for ( gallery_identifier, tag_import_options ) in self._new_options.GetDefaultTagImportOptions().items():
name = gallery_identifier.ToString()
self._tag_import_options.Append( name, ( gallery_identifier, tag_import_options ) )
#
rows = []
@ -1975,76 +1960,13 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
#
message = 'These options will be disappearing soon! Everything is moving over to the new downloader system! Default tag import options are now set under _network->downloaders->manage default tag import options_!'
st = ClientGUICommon.BetterStaticText( default_tios, message )
default_tios.Add( st, CC.FLAGS_EXPAND_PERPENDICULAR )
default_tios.Add( self._tag_import_options, CC.FLAGS_EXPAND_BOTH_WAYS )
hbox = wx.BoxSizer( wx.HORIZONTAL )
hbox.Add( self._edit, CC.FLAGS_VCENTER )
hbox.Add( self._delete, CC.FLAGS_VCENTER )
default_tios.Add( hbox, CC.FLAGS_BUTTON_SIZER )
#
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.Add( default_fios, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.Add( default_tios, CC.FLAGS_EXPAND_BOTH_WAYS )
self.SetSizer( vbox )
def EventDelete( self, event ):
selection = self._tag_import_options.GetSelection()
if selection != wx.NOT_FOUND:
name = self._tag_import_options.GetString( selection )
with ClientGUIDialogs.DialogYesNo( self, 'Delete \'' + name + '\' entry?' ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
self._tag_import_options.Delete( selection )
def EventEdit( self, event ):
selection = self._tag_import_options.GetSelection()
if selection != wx.NOT_FOUND:
name = self._tag_import_options.GetString( selection )
( gallery_identifier, tag_import_options ) = self._tag_import_options.GetClientData( selection )
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( gallery_identifier )
with ClientGUITopLevelWindows.DialogEdit( self, 'edit tag import options' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditTagImportOptionsPanel( dlg, namespaces, tag_import_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
tag_import_options = panel.GetValue()
self._tag_import_options.SetClientData( selection, ( gallery_identifier, tag_import_options ) )
def UpdateOptions( self ):
self._new_options.SetDefaultFileImportOptions( 'quiet', self._quiet_fios.GetValue() )
@ -2113,7 +2035,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows = []
rows.append( ( 'Run maintenance jobs when the client is idle and the system is not otherwise busy: ', self._idle_normal ) )
rows.append( ( 'Assume the client is idle if no general browsing activity has occured in the past: ', self._idle_period ) )
rows.append( ( 'Assume the client is idle if no general browsing activity has occurred in the past: ', self._idle_period ) )
rows.append( ( 'Assume the client is idle if the mouse has not been moved in the past: ', self._idle_mouse_period ) )
rows.append( ( 'Assume the system is busy if any CPU core has recent average usage above: ', self._idle_cpu_max ) )

226
include/ClientGUIScrolledPanelsReview.py
View File

@ -345,16 +345,15 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
current_media_locations_listctrl_panel = ClientGUIListCtrl.BetterListCtrlPanel( info_panel )
self._current_media_locations_listctrl = ClientGUIListCtrl.BetterListCtrl( current_media_locations_listctrl_panel, 'db_migration_locations', 6, 36, [ ( 'location', -1 ), ( 'portable?', 12 ), ( 'free space', 12 ), ( 'weight', 10 ), ( 'ideal usage', 24 ), ( 'current usage', 24 ) ], self._ConvertLocationToListCtrlTuples )
self._current_media_locations_listctrl = ClientGUIListCtrl.BetterListCtrl( current_media_locations_listctrl_panel, 'db_migration_locations', 8, 36, [ ( 'location', -1 ), ( 'portable?', 11 ), ( 'free space', 12 ), ( 'file weight', 10 ), ( 'current usage', 24 ), ( 'ideal usage', 24 ) ], self._ConvertLocationToListCtrlTuples, style = wx.LC_SINGLE_SEL )
self._current_media_locations_listctrl.Sort()
current_media_locations_listctrl_panel.SetListCtrl( self._current_media_locations_listctrl )
current_media_locations_listctrl_panel.AddButton( 'add location', self._AddPath )
current_media_locations_listctrl_panel.AddButton( 'empty/remove location', self._RemovePaths, enabled_check_func = self._FileLocationSelected )
current_media_locations_listctrl_panel.AddButton( 'increase weight', self._IncreaseWeight, enabled_check_func = self._FileLocationSelected )
current_media_locations_listctrl_panel.AddButton( 'decrease weight', self._DecreaseWeight, enabled_check_func = self._FileLocationSelected )
current_media_locations_listctrl_panel.AddButton( 'add new location for files', self._SelectPathToAdd )
current_media_locations_listctrl_panel.AddButton( 'increase file weight', self._IncreaseWeight, enabled_check_func = self._CanIncreaseWeight )
current_media_locations_listctrl_panel.AddButton( 'decrease file weight', self._DecreaseWeight, enabled_check_func = self._CanDecreaseWeight )
self._resized_thumbs_location = wx.TextCtrl( info_panel )
self._resized_thumbs_location.Disable()
@ -428,35 +427,27 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
self._Update()
def _AddPath( self ):
def _AddPath( self, path, starting_weight = 1 ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
with wx.DirDialog( self, 'Select the location' ) as dlg:
if path in locations_to_ideal_weights:
if dlg.ShowModal() == wx.ID_OK:
path = HydrusData.ToUnicode( dlg.GetPath() )
if path in locations_to_ideal_weights:
wx.MessageBox( 'You already have that location entered!' )
return
if path == resized_thumbnail_override or path == full_size_thumbnail_override:
wx.MessageBox( 'That path is already used as a special thumbnail location--please choose another.' )
return
self._new_options.SetClientFilesLocation( path, 1 )
self._Update()
wx.MessageBox( 'You already have that location entered!' )
return
if path == resized_thumbnail_override or path == full_size_thumbnail_override:
wx.MessageBox( 'That path is already used as a special thumbnail location--please choose another.' )
return
self._new_options.SetClientFilesLocation( path, 1 )
self._Update()
def _AdjustWeight( self, amount ):
@ -465,18 +456,14 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
adjustees = set()
for location in self._current_media_locations_listctrl.GetData( only_selected = True ):
locations = self._current_media_locations_listctrl.GetData( only_selected = True )
if len( locations ) > 0:
location = locations[0]
if location in locations_to_ideal_weights:
adjustees.add( location )
if len( adjustees ) > 0:
for location in adjustees:
current_weight = locations_to_ideal_weights[ location ]
new_amount = current_weight + amount
@ -485,12 +472,84 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
self._new_options.SetClientFilesLocation( location, new_amount )
elif new_amount <= 0:
self._RemovePath( location )
else:
if amount > 0:
if location not in ( resized_thumbnail_override, full_size_thumbnail_override ):
self._AddPath( location, starting_weight = amount )
self._Update()
def _CanDecreaseWeight( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
locations = self._current_media_locations_listctrl.GetData( only_selected = True )
if len( locations ) > 0:
location = locations[0]
if location in locations_to_ideal_weights:
selection_includes_ideal_locations = True
ideal_weight = locations_to_ideal_weights[ location ]
is_big = ideal_weight > 1
others_can_take_slack = len( locations_to_ideal_weights ) > 1
if is_big or others_can_take_slack:
return True
return False
def _CanIncreaseWeight( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights ) = self._GetLocationsToCurrentWeights()
locations = self._current_media_locations_listctrl.GetData( only_selected = True )
if len( locations ) > 0:
location = locations[0]
if location in locations_to_ideal_weights:
if len( locations_to_ideal_weights ) > 1:
return True
elif location in locations_to_file_weights:
return True
return False
def _ClearFullsizeThumbnailLocation( self ):
self._new_options.SetFullsizeThumbnailOverride( None )
@ -606,7 +665,14 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
ideal_weight = 0
pretty_ideal_weight = 'n/a'
if location in locations_to_file_weights:
pretty_ideal_weight = '0'
else:
pretty_ideal_weight = 'n/a'
if location in locations_to_ideal_weights:
@ -687,8 +753,8 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
pretty_ideal_usage = 'nothing'
display_tuple = ( pretty_location, pretty_portable, pretty_free_space, pretty_ideal_weight, pretty_ideal_usage, pretty_current_usage )
sort_tuple = ( location, portable, free_space, ideal_weight, ideal_usage, current_usage )
display_tuple = ( pretty_location, pretty_portable, pretty_free_space, pretty_ideal_weight, pretty_current_usage, pretty_ideal_usage )
sort_tuple = ( location, portable, free_space, ideal_weight, current_usage, ideal_usage )
return ( display_tuple, sort_tuple )
@ -698,25 +764,6 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
self._AdjustWeight( -1 )
def _FileLocationSelected( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights ) = self._GetLocationsToCurrentWeights()
locations = self._current_media_locations_listctrl.GetData( only_selected = True )
for location in locations:
if location in locations_to_file_weights or location in locations_to_ideal_weights:
return True
return False
def _GetLocationsToCurrentWeights( self ):
prefixes_to_locations = HG.client_controller.Read( 'client_files_locations' )
@ -892,43 +939,56 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
self._Update()
def _RemovePaths( self ):
def _RemovePath( self, location ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights ) = self._GetLocationsToCurrentWeights()
removees = set()
for location in self._current_media_locations_listctrl.GetData( only_selected = True ):
if location not in locations_to_ideal_weights:
if location in locations_to_ideal_weights:
removees.add( location )
wx.MessageBox( 'Please select a location with weight.' )
return
# eventually have a check and veto if not enough size on the destination partition
if len( locations_to_ideal_weights ) == 1:
wx.MessageBox( 'You cannot empty every single current file location--please add a new place for the files to be moved to and then try again.' )
if len( removees ) == 0:
if location in locations_to_file_weights:
wx.MessageBox( 'Please select some locations with weight.' )
elif len( removees ) == len( locations_to_ideal_weights ):
wx.MessageBox( 'You cannot empty every single location--please add a new place for the files to be moved to and then try again.' )
message = 'Are you sure you want to remove this location? This will schedule all of the files it is currently responsible for to be moved elsewhere.'
else:
with ClientGUIDialogs.DialogYesNo( self, 'Are you sure? This will schedule all the selected locations to have all their current files removed.' ) as dlg:
message = 'Are you sure you want to remove this location? The files it would be responsible for will be shared amongst the other file locations.'
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
if dlg.ShowModal() == wx.ID_YES:
for location in removees:
self._new_options.RemoveClientFilesLocation( location )
self._Update()
self._new_options.RemoveClientFilesLocation( location )
self._Update()
def _SelectPathToAdd( self ):
with wx.DirDialog( self, 'Select the location' ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
path = HydrusData.ToUnicode( dlg.GetPath() )
self._AddPath( path )

36
include/ClientImportFileSeeds.py
View File

@ -368,7 +368,21 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
def _NormaliseAndFilterAssociableURLs( self, urls ):
normalised_urls = { HG.client_controller.network_engine.domain_manager.NormaliseURL( url ) for url in urls }
normalised_urls = set()
for url in urls:
try:
url = HG.client_controller.network_engine.domain_manager.NormaliseURL( url )
except HydrusExceptions.URLMatchException:
continue # not a url--something like "file:///C:/Users/Tall%20Man/Downloads/maxresdefault.jpg" ha ha ha
normalised_urls.add( url )
associable_urls = { url for url in normalised_urls if HG.client_controller.network_engine.domain_manager.ShouldAssociateURLWithFiles( url ) }
@ -483,7 +497,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
self._CheckTagsBlacklist( self._tags, tag_import_options )
def DownloadAndImportRawFile( self, file_url, file_import_options, network_job_factory, network_job_presentation_context_factory ):
def DownloadAndImportRawFile( self, file_url, file_import_options, network_job_factory, network_job_presentation_context_factory, override_bandwidth = False ):
self.AddURL( file_url )
@ -502,6 +516,11 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
network_job = network_job_factory( 'GET', file_url, temp_path = temp_path, referral_url = referral_url )
if override_bandwidth:
network_job.OverrideBandwidth( 30 )
network_job.SetFileImportOptions( file_import_options )
HG.client_controller.network_engine.AddJob( network_job )
@ -996,7 +1015,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
status_hook( 'downloading file' )
self.DownloadAndImportRawFile( file_url, file_import_options, network_job_factory, network_job_presentation_context_factory )
self.DownloadAndImportRawFile( file_url, file_import_options, network_job_factory, network_job_presentation_context_factory, override_bandwidth = True )
elif url_type == HC.URL_TYPE_POST and can_parse:
@ -1090,6 +1109,17 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
time.sleep( 2 )
except HydrusExceptions.ForbiddenException:
status = CC.STATUS_VETOED
note = '403'
self.SetStatus( status, note = note )
status_hook( '403' )
time.sleep( 2 )
except HydrusExceptions.NotFoundException:
status = CC.STATUS_VETOED

1769
include/ClientImportGallery.py Normal file
View File

File diff suppressed because it is too large Load Diff

13
include/ClientImportGallerySeeds.py
View File

@ -312,6 +312,19 @@ class GallerySeed( HydrusSerialisable.SerialisableBase ):
time.sleep( 2 )
except HydrusExceptions.ForbiddenException:
status = CC.STATUS_VETOED
note = '403'
self.SetStatus( status, note = note )
status_hook( '403' )
time.sleep( 2 )
result_404 = True
except HydrusExceptions.NotFoundException:
status = CC.STATUS_VETOED

35
include/ClientImportOptions.py
View File

@ -169,11 +169,6 @@ class CheckerOptions( HydrusSerialisable.SerialisableBase ):
def GetRawCurrentVelocity( self, file_seed_cache, last_check_time ):
return self._GetCurrentFilesVelocity( file_seed_cache, last_check_time )
def GetPrettyCurrentVelocity( self, file_seed_cache, last_check_time, no_prefix = False ):
if len( file_seed_cache ) == 0:
@ -206,6 +201,36 @@ class CheckerOptions( HydrusSerialisable.SerialisableBase ):
return pretty_current_velocity
def GetRawCurrentVelocity( self, file_seed_cache, last_check_time ):
return self._GetCurrentFilesVelocity( file_seed_cache, last_check_time )
def GetSummary( self ):
if self._never_faster_than == self._never_slower_than:
timing_statement = 'Checking every ' + HydrusData.TimeDeltaToPrettyTimeDelta( self._never_faster_than ) + '.'
else:
timing_statement = 'Trying to get ' + HydrusData.ToHumanInt( self._intended_files_per_check ) + ' files per check, never faster than ' + HydrusData.TimeDeltaToPrettyTimeDelta( self._never_faster_than ) + ' and never slower than ' + HydrusData.TimeDeltaToPrettyTimeDelta( self._never_slower_than ) + '.'
( death_files_found, death_time_delta ) = self._death_file_velocity
if death_files_found == 0:
death_statement = 'Never stopping.'
else:
death_statement = 'Stopping if file velocity falls below ' + HydrusData.ToHumanInt( death_files_found ) + ' files per ' + HydrusData.TimeDeltaToPrettyTimeDelta( death_time_delta ) + '.'
return timing_statement + os.linesep * 2 + death_statement
def IsDead( self, file_seed_cache, last_check_time ):
if len( file_seed_cache ) == 0 and last_check_time == 0:

102
include/ClientImportWatchers.py
View File

@ -567,6 +567,20 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
if gallery_seed.status == CC.STATUS_ERROR:
# the [DEAD] stuff can override watcher status, so let's give a brief time for this to display the error
with self._lock:
self._checking_paused = True
self._watcher_status = gallery_seed.note
time.sleep( 5 )
except HydrusExceptions.NetworkException as e:
self._DelayWork( 4 * 3600, HydrusData.ToUnicode( e ) )
@ -575,10 +589,35 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
watcher_status = gallery_seed.note
error_occurred = gallery_seed.status == CC.STATUS_ERROR
watcher_status_should_stick = gallery_seed.status != CC.STATUS_SUCCESSFUL_AND_NEW
self._FinishCheck( watcher_status, error_occurred, watcher_status_should_stick )
with self._lock:
if self._check_now:
self._check_now = False
self._watcher_status = watcher_status
self._last_check_time = HydrusData.GetNow()
self._UpdateFileVelocityStatus()
self._UpdateNextCheckTime()
self._PublishPageName()
if not watcher_status_should_stick:
time.sleep( 5 )
with self._lock:
self._watcher_status = ''
def _DelayWork( self, time_delta, reason ):
@ -614,51 +653,6 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
return ClientImporting.NetworkJobPresentationContext( enter_call, exit_call )
def _FinishCheck( self, watcher_status, error_occurred, watcher_status_should_stick ):
if error_occurred:
# the [DEAD] stuff can override watcher status, so let's give a brief time for this to display the error
with self._lock:
self._checking_paused = True
self._watcher_status = watcher_status
time.sleep( 5 )
with self._lock:
if self._check_now:
self._check_now = False
self._watcher_status = watcher_status
self._last_check_time = HydrusData.GetNow()
self._UpdateFileVelocityStatus()
self._UpdateNextCheckTime()
self._PublishPageName()
if not watcher_status_should_stick:
time.sleep( 5 )
with self._lock:
self._watcher_status = ''
def _GetSerialisableInfo( self ):
serialisable_gallery_seed_log = self._gallery_seed_log.GetSerialisableTuple()
@ -1274,10 +1268,14 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
return
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._files_paused or HG.client_controller.PageClosedButNotDestroyed( self._page_key ) )
work_pending = self._file_seed_cache.WorkToDo() and not self._files_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
ok_to_work = work_pending and no_delays and page_shown
while work_to_do:
while ok_to_work:
try:
@ -1299,7 +1297,11 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
return
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._files_paused or HG.client_controller.PageClosedButNotDestroyed( self._page_key ) )
work_pending = self._file_seed_cache.WorkToDo() and not self._files_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
ok_to_work = work_pending and no_delays and page_shown

934
include/ClientImporting.py
View File

@ -357,917 +357,6 @@ def WakeRepeatingJob( job ):
job.Wake()
class GalleryImport( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_GALLERY_IMPORT
SERIALISABLE_NAME = 'Gallery Import'
SERIALISABLE_VERSION = 3
def __init__( self, gallery_identifier = None ):
if gallery_identifier is None:
gallery_identifier = ClientDownloading.GalleryIdentifier( HC.SITE_TYPE_DEVIANT_ART )
HydrusSerialisable.SerialisableBase.__init__( self )
self._gallery_identifier = gallery_identifier
self._gallery_stream_identifiers = ClientDownloading.GetGalleryStreamIdentifiers( self._gallery_identifier )
self._current_query = None
self._current_query_num_new_urls = 0
self._current_query_num_urls = 0
self._current_gallery_stream_identifier = None
self._current_gallery_stream_identifier_page_index = 0
self._current_gallery_stream_identifier_found_urls = set()
self._pending_gallery_stream_identifiers = []
self._pending_queries = []
new_options = HG.client_controller.new_options
self._file_limit = HC.options[ 'gallery_file_limit' ]
self._gallery_paused = False
self._files_paused = False
self._file_import_options = HG.client_controller.new_options.GetDefaultFileImportOptions( 'loud' )
self._tag_import_options = ClientImportOptions.TagImportOptions( is_default = True )
self._last_gallery_page_hit_timestamp = 0
self._gallery_seed_log = ClientImportGallerySeeds.GallerySeedLog()
self._file_seed_cache = ClientImportFileSeeds.FileSeedCache()
self._lock = threading.Lock()
self._gallery = None
self._gallery_status = ''
self._gallery_status_can_change_timestamp = 0
self._current_action = ''
self._download_control_file_set = None
self._download_control_file_clear = None
self._download_control_gallery_set = None
self._download_control_gallery_clear = None
self._files_repeating_job = None
self._gallery_repeating_job = None
HG.client_controller.sub( self, 'NotifyFileSeedsUpdated', 'file_seed_cache_file_seeds_updated' )
def _GetSerialisableInfo( self ):
serialisable_gallery_identifier = self._gallery_identifier.GetSerialisableTuple()
serialisable_gallery_stream_identifiers = [ gallery_stream_identifier.GetSerialisableTuple() for gallery_stream_identifier in self._gallery_stream_identifiers ]
if self._current_gallery_stream_identifier is None:
serialisable_current_gallery_stream_identifier = None
else:
serialisable_current_gallery_stream_identifier = self._current_gallery_stream_identifier.GetSerialisableTuple()
serialisable_current_gallery_stream_identifier_found_urls = list( self._current_gallery_stream_identifier_found_urls )
serialisable_pending_gallery_stream_identifiers = [ pending_gallery_stream_identifier.GetSerialisableTuple() for pending_gallery_stream_identifier in self._pending_gallery_stream_identifiers ]
serialisable_file_import_options = self._file_import_options.GetSerialisableTuple()
serialisable_tag_import_options = self._tag_import_options.GetSerialisableTuple()
serialisable_gallery_seed_log = self._gallery_seed_log.GetSerialisableTuple()
serialisable_file_seed_cache = self._file_seed_cache.GetSerialisableTuple()
serialisable_current_query_stuff = ( self._current_query, self._current_query_num_new_urls, serialisable_current_gallery_stream_identifier, self._current_gallery_stream_identifier_page_index, serialisable_current_gallery_stream_identifier_found_urls, serialisable_pending_gallery_stream_identifiers )
return ( serialisable_gallery_identifier, serialisable_gallery_stream_identifiers, serialisable_current_query_stuff, self._pending_queries, self._file_limit, self._gallery_paused, self._files_paused, serialisable_file_import_options, serialisable_tag_import_options, serialisable_gallery_seed_log, serialisable_file_seed_cache )
def _InitialiseFromSerialisableInfo( self, serialisable_info ):
( serialisable_gallery_identifier, serialisable_gallery_stream_identifiers, serialisable_current_query_stuff, self._pending_queries, self._file_limit, self._gallery_paused, self._files_paused, serialisable_file_import_options, serialisable_tag_import_options, serialisable_gallery_seed_log, serialisable_file_seed_cache ) = serialisable_info
( self._current_query, self._current_query_num_new_urls, serialisable_current_gallery_stream_identifier, self._current_gallery_stream_identifier_page_index, serialisable_current_gallery_stream_identifier_found_urls, serialisable_pending_gallery_stream_identifier ) = serialisable_current_query_stuff
self._gallery_identifier = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_gallery_identifier )
self._gallery_stream_identifiers = [ HydrusSerialisable.CreateFromSerialisableTuple( serialisable_gallery_stream_identifier ) for serialisable_gallery_stream_identifier in serialisable_gallery_stream_identifiers ]
if serialisable_current_gallery_stream_identifier is None:
self._current_gallery_stream_identifier = None
else:
self._current_gallery_stream_identifier = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_current_gallery_stream_identifier )
self._current_gallery_stream_identifier_found_urls = set( serialisable_current_gallery_stream_identifier_found_urls )
self._pending_gallery_stream_identifiers = [ HydrusSerialisable.CreateFromSerialisableTuple( serialisable_pending_gallery_stream_identifier ) for serialisable_pending_gallery_stream_identifier in serialisable_pending_gallery_stream_identifier ]
self._file_import_options = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_file_import_options )
self._tag_import_options = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_tag_import_options )
self._gallery_seed_log = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_gallery_seed_log )
self._file_seed_cache = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_file_seed_cache )
def _FileNetworkJobPresentationContextFactory( self, network_job ):
def enter_call():
with self._lock:
if self._download_control_file_set is not None:
wx.CallAfter( self._download_control_file_set, network_job )
def exit_call():
with self._lock:
if self._download_control_file_clear is not None:
wx.CallAfter( self._download_control_file_clear )
return NetworkJobPresentationContext( enter_call, exit_call )
def _SetGalleryStatus( self, status, timeout = None ):
if HydrusData.TimeHasPassed( self._gallery_status_can_change_timestamp ):
self._gallery_status = status
if timeout is not None:
self._gallery_status_can_change_timestamp = HydrusData.GetNow() + timeout
def _UpdateSerialisableInfo( self, version, old_serialisable_info ):
if version == 1:
( serialisable_gallery_identifier, serialisable_gallery_stream_identifiers, serialisable_current_query_stuff, pending_queries, get_tags_if_url_recognised_and_file_redundant, file_limit, gallery_paused, files_paused, serialisable_file_import_options, serialisable_tag_import_options, serialisable_file_seed_cache ) = old_serialisable_info
new_serialisable_info = ( serialisable_gallery_identifier, serialisable_gallery_stream_identifiers, serialisable_current_query_stuff, pending_queries, file_limit, gallery_paused, files_paused, serialisable_file_import_options, serialisable_tag_import_options, serialisable_file_seed_cache )
return ( 2, new_serialisable_info )
if version == 2:
( serialisable_gallery_identifier, serialisable_gallery_stream_identifiers, serialisable_current_query_stuff, pending_queries, file_limit, gallery_paused, files_paused, serialisable_file_import_options, serialisable_tag_import_options, serialisable_file_seed_cache ) = old_serialisable_info
gallery_seed_log = ClientImportGallerySeeds.GallerySeedLog()
serialisable_gallery_seed_log = gallery_seed_log.GetSerialisableTuple()
new_serialisable_info = ( serialisable_gallery_identifier, serialisable_gallery_stream_identifiers, serialisable_current_query_stuff, pending_queries, file_limit, gallery_paused, files_paused, serialisable_file_import_options, serialisable_tag_import_options, serialisable_gallery_seed_log, serialisable_file_seed_cache )
return ( 3, new_serialisable_info )
def _WorkOnFiles( self, page_key ):
file_seed = self._file_seed_cache.GetNextFileSeed( CC.STATUS_UNKNOWN )
if file_seed is None:
return
did_substantial_work = False
def network_job_factory( method, url, **kwargs ):
network_job = ClientNetworkingJobs.NetworkJobDownloader( page_key, method, url, **kwargs )
wx.CallAfter( self._download_control_file_set, network_job )
return network_job
try:
gallery = ClientDownloading.GetGallery( self._gallery_identifier )
except Exception as e:
HydrusData.PrintException( e )
with self._lock:
self._files_paused = True
self._gallery_paused = True
HydrusData.ShowText( 'A downloader could not load its gallery! It has been paused and the full error has been written to the log!' )
return
gallery.SetNetworkJobFactory( network_job_factory )
try:
if file_seed.WorksInNewSystem():
def status_hook( text ):
with self._lock:
self._current_action = text
did_substantial_work = file_seed.WorkOnURL( self._file_seed_cache, status_hook, GenerateDownloaderNetworkJobFactory( page_key ), self._FileNetworkJobPresentationContextFactory, self._file_import_options, self._tag_import_options )
if file_seed.ShouldPresent( self._file_import_options ):
file_seed.PresentToPage( page_key )
did_substantial_work = True
else:
with self._lock:
self._current_action = 'reviewing file'
file_seed.PredictPreImportStatus( self._file_import_options, self._tag_import_options )
status = file_seed.status
url = file_seed.file_seed_data
if status == CC.STATUS_SUCCESSFUL_BUT_REDUNDANT:
if self._tag_import_options.ShouldFetchTagsEvenIfURLKnownAndFileAlreadyInDB() and self._tag_import_options.WorthFetchingTags():
downloaded_tags = gallery.GetTags( url )
file_seed.AddTags( downloaded_tags )
elif status == CC.STATUS_UNKNOWN:
( os_file_handle, temp_path ) = ClientPaths.GetTempPath()
try:
with self._lock:
self._current_action = 'downloading file'
if self._tag_import_options.WorthFetchingTags():
downloaded_tags = gallery.GetFileAndTags( temp_path, url )
file_seed.AddTags( downloaded_tags )
else:
gallery.GetFile( temp_path, url )
file_seed.CheckPreFetchMetadata( self._tag_import_options )
with self._lock:
self._current_action = 'importing file'
file_seed.Import( temp_path, self._file_import_options )
did_substantial_work = True
finally:
HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
did_substantial_work = file_seed.WriteContentUpdates( self._tag_import_options )
if file_seed.ShouldPresent( self._file_import_options ):
file_seed.PresentToPage( page_key )
did_substantial_work = True
except HydrusExceptions.VetoException as e:
status = CC.STATUS_VETOED
note = HydrusData.ToUnicode( e )
file_seed.SetStatus( status, note = note )
if isinstance( e, HydrusExceptions.CancelledException ):
time.sleep( 2 )
except HydrusExceptions.NotFoundException:
status = CC.STATUS_VETOED
note = '404'
file_seed.SetStatus( status, note = note )
time.sleep( 2 )
except Exception as e:
status = CC.STATUS_ERROR
file_seed.SetStatus( status, exception = e )
time.sleep( 3 )
finally:
self._file_seed_cache.NotifyFileSeedsUpdated( ( file_seed, ) )
wx.CallAfter( self._download_control_file_clear )
with self._lock:
self._current_action = ''
if did_substantial_work:
time.sleep( DID_SUBSTANTIAL_FILE_WORK_MINIMUM_SLEEP_TIME )
def _WorkOnGallery( self, page_key ):
with self._lock:
if self._current_query is None:
if len( self._pending_queries ) == 0:
self._SetGalleryStatus( '' )
return False
else:
self._current_query = self._pending_queries.pop( 0 )
self._current_query_num_new_urls = 0
self._current_query_num_urls = 0
self._current_gallery_stream_identifier = None
self._pending_gallery_stream_identifiers = list( self._gallery_stream_identifiers )
if self._current_gallery_stream_identifier is None:
if len( self._pending_gallery_stream_identifiers ) == 0:
self._SetGalleryStatus( self._current_query + ': produced ' + HydrusData.ToHumanInt( self._current_query_num_new_urls ) + ' new urls', 5 )
self._current_query = None
return False
else:
self._current_gallery_stream_identifier = self._pending_gallery_stream_identifiers.pop( 0 )
self._current_gallery_stream_identifier_page_index = 0
self._current_gallery_stream_identifier_found_urls = set()
next_gallery_page_hit_timestamp = self._last_gallery_page_hit_timestamp + HG.client_controller.new_options.GetInteger( 'gallery_page_wait_period_pages' )
if not HydrusData.TimeHasPassed( next_gallery_page_hit_timestamp ):
if self._current_gallery_stream_identifier_page_index == 0:
page_check_status = 'checking first page ' + HydrusData.TimestampToPrettyTimeDelta( next_gallery_page_hit_timestamp )
else:
page_check_status = HydrusData.ToHumanInt( self._current_query_num_new_urls ) + ' new urls found, checking next page ' + HydrusData.TimestampToPrettyTimeDelta( next_gallery_page_hit_timestamp )
self._SetGalleryStatus( self._current_query + ': ' + page_check_status )
return True
def network_job_factory( method, url, **kwargs ):
network_job = ClientNetworkingJobs.NetworkJobDownloader( page_key, method, url, **kwargs )
network_job.OverrideBandwidth( 30 )
wx.CallAfter( self._download_control_gallery_set, network_job )
return network_job
try:
gallery = ClientDownloading.GetGallery( self._current_gallery_stream_identifier )
except Exception as e:
HydrusData.PrintException( e )
with self._lock:
self._files_paused = True
self._gallery_paused = True
HydrusData.ShowText( 'A downloader could not load its gallery! It has been paused and the full error has been written to the log!' )
return False
gallery.SetNetworkJobFactory( network_job_factory )
query = self._current_query
page_index = self._current_gallery_stream_identifier_page_index
self._SetGalleryStatus( self._current_query + ': ' + HydrusData.ToHumanInt( self._current_query_num_new_urls ) + ' new urls found, now checking page ' + HydrusData.ToHumanInt( self._current_gallery_stream_identifier_page_index + 1 ) )
error_occured = False
num_already_in_file_seed_cache = 0
new_file_seeds = []
try:
gallery_url = gallery.GetGalleryPageURL( query, page_index )
# can_generate_more_pages = if recognised in the new system, I guess
gallery_seed = ClientImportGallerySeeds.GallerySeed( gallery_url, can_generate_more_pages = False )
self._gallery_seed_log.AddGallerySeeds( ( gallery_seed, ) )
try:
( page_of_file_seeds, definitely_no_more_pages ) = gallery.GetPage( gallery_url )
finally:
self._last_gallery_page_hit_timestamp = HydrusData.GetNow()
with self._lock:
no_urls_found = len( page_of_file_seeds ) == 0
page_of_urls = [ file_seed.file_seed_data for file_seed in page_of_file_seeds ]
no_new_urls = len( self._current_gallery_stream_identifier_found_urls.intersection( page_of_urls ) ) == len( page_of_file_seeds )
if definitely_no_more_pages or no_urls_found or no_new_urls:
self._current_gallery_stream_identifier = None
else:
self._current_gallery_stream_identifier_page_index += 1
self._current_gallery_stream_identifier_found_urls.update( page_of_urls )
for file_seed in page_of_file_seeds:
if self._file_seed_cache.HasFileSeed( file_seed ):
num_already_in_file_seed_cache += 1
else:
with self._lock:
num_urls_estimate = max( self._current_query_num_new_urls, self._current_query_num_urls )
if self._file_limit is not None and num_urls_estimate + 1 > self._file_limit:
self._current_gallery_stream_identifier = None
self._pending_gallery_stream_identifiers = []
break
self._current_query_num_urls += 1
self._current_query_num_new_urls += 1
new_file_seeds.append( file_seed )
num_urls_added = self._file_seed_cache.AddFileSeeds( new_file_seeds )
status = query + ': ' + HydrusData.ToHumanInt( len( new_file_seeds ) ) + ' new urls found'
if num_already_in_file_seed_cache > 0:
status += ' (' + HydrusData.ToHumanInt( num_already_in_file_seed_cache ) + ' of last page already in queue)'
gallery_seed_status = CC.STATUS_SUCCESSFUL_AND_NEW
gallery_seed_note = status
if len( new_file_seeds ) > 0:
WakeRepeatingJob( self._files_repeating_job )
except Exception as e:
if isinstance( e, HydrusExceptions.NotFoundException ):
text = 'gallery 404'
gallery_seed_status = CC.STATUS_VETOED
gallery_seed_note = text
else:
text = HydrusData.ToUnicode( e )
gallery_seed_status = CC.STATUS_ERROR
gallery_seed_note = text
HydrusData.DebugPrint( traceback.format_exc() )
with self._lock:
self._current_gallery_stream_identifier = None
error_occured = True
finally:
wx.CallAfter( self._download_control_gallery_clear )
gallery_seed.SetStatus( gallery_seed_status, note = gallery_seed_note )
self._gallery_seed_log.NotifyGallerySeedsUpdated( ( gallery_seed, ) )
with self._lock:
status = gallery_seed_note
if error_occured:
self._SetGalleryStatus( status, 5 )
else:
self._SetGalleryStatus( status )
return True
def AdvanceQueries( self, queries ):
with self._lock:
queries_lookup = set( queries )
for query in queries:
if query in self._pending_queries:
index = self._pending_queries.index( query )
if index > 0 and self._pending_queries[ index - 1 ] not in queries_lookup:
self._pending_queries.remove( query )
self._pending_queries.insert( index - 1, query )
def CurrentlyWorking( self ):
with self._lock:
finished = not self._file_seed_cache.WorkToDo()
return not finished and not self._files_paused
def DelayQueries( self, queries ):
with self._lock:
queries = list( queries )
queries.reverse()
queries_lookup = set( queries )
for query in queries:
if query in self._pending_queries:
index = self._pending_queries.index( query )
if index + 1 < len( self._pending_queries ) and self._pending_queries[ index + 1 ] not in queries_lookup:
self._pending_queries.remove( query )
self._pending_queries.insert( index + 1, query )
def DeleteQueries( self, queries ):
with self._lock:
for query in queries:
if query in self._pending_queries:
self._pending_queries.remove( query )
def FinishCurrentQuery( self ):
with self._lock:
self._current_query = None
self._gallery_paused = False
WakeRepeatingJob( self._gallery_repeating_job )
def GetFileSeedCache( self ):
return self._file_seed_cache
def GetGalleryIdentifier( self ):
return self._gallery_identifier
def GetGallerySeedLog( self ):
return self._gallery_seed_log
def GetOptions( self ):
with self._lock:
return ( self._file_import_options, self._tag_import_options, self._file_limit )
def GetStatus( self ):
with self._lock:
cancellable = self._current_query is not None
return ( list( self._pending_queries ), self._gallery_status, self._current_action, self._files_paused, self._gallery_paused, cancellable )
def GetValueRange( self ):
with self._lock:
return self._file_seed_cache.GetValueRange()
def NotifyFileSeedsUpdated( self, file_seed_cache_key, file_seeds ):
if file_seed_cache_key == self._file_seed_cache.GetFileSeedCacheKey():
WakeRepeatingJob( self._files_repeating_job )
def PausePlayFiles( self ):
with self._lock:
self._files_paused = not self._files_paused
WakeRepeatingJob( self._files_repeating_job )
def PausePlayGallery( self ):
with self._lock:
self._gallery_paused = not self._gallery_paused
WakeRepeatingJob( self._gallery_repeating_job )
def PendQuery( self, query ):
with self._lock:
if query not in self._pending_queries:
self._pending_queries.append( query )
WakeRepeatingJob( self._gallery_repeating_job )
def SetDownloadControls( self, file_download_control, gallery_download_control ):
with self._lock:
self._download_control_file_set = file_download_control.SetNetworkJob
self._download_control_file_clear = file_download_control.ClearNetworkJob
self._download_control_gallery_set = gallery_download_control.SetNetworkJob
self._download_control_gallery_clear = gallery_download_control.ClearNetworkJob
def SetFileLimit( self, file_limit ):
with self._lock:
self._file_limit = file_limit
def SetFileImportOptions( self, file_import_options ):
with self._lock:
self._file_import_options = file_import_options
def SetTagImportOptions( self, tag_import_options ):
with self._lock:
self._tag_import_options = tag_import_options
def Start( self, page_key ):
self._files_repeating_job = HG.client_controller.CallRepeating( GetRepeatingJobInitialDelay(), REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnFiles, page_key )
self._gallery_repeating_job = HG.client_controller.CallRepeating( GetRepeatingJobInitialDelay(), REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnGallery, page_key )
def REPEATINGWorkOnFiles( self, page_key ):
with self._lock:
if PageImporterShouldStopWorking( page_key ):
self._files_repeating_job.Cancel()
return
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._files_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
while work_to_do:
try:
self._WorkOnFiles( page_key )
HG.client_controller.WaitUntilViewFree()
except Exception as e:
HydrusData.ShowException( e )
with self._lock:
if PageImporterShouldStopWorking( page_key ):
self._files_repeating_job.Cancel()
return
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._files_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
def REPEATINGWorkOnGallery( self, page_key ):
with self._lock:
if PageImporterShouldStopWorking( page_key ):
self._gallery_repeating_job.Cancel()
return
ok_to_work = not ( self._gallery_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
while ok_to_work:
try:
work_to_do = self._WorkOnGallery( page_key )
if work_to_do:
time.sleep( 1 )
else:
return
HG.client_controller.WaitUntilViewFree()
except Exception as e:
HydrusData.ShowException( e )
with self._lock:
if PageImporterShouldStopWorking( page_key ):
self._gallery_repeating_job.Cancel()
return
ok_to_work = not ( self._gallery_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_GALLERY_IMPORT ] = GalleryImport
class HDDImport( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_HDD_IMPORT
@ -1403,7 +492,7 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
HydrusData.ShowText( 'While attempting to delete ' + path + ', the following error occured:' )
HydrusData.ShowText( 'While attempting to delete ' + path + ', the following error occurred:' )
HydrusData.ShowException( e )
@ -1417,7 +506,7 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
HydrusData.ShowText( 'While attempting to delete ' + txt_path + ', the following error occured:' )
HydrusData.ShowText( 'While attempting to delete ' + txt_path + ', the following error occurred:' )
HydrusData.ShowException( e )
@ -2087,9 +1176,19 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
return self._file_seed_cache
def Paused( self ):
return self._paused
def PausePlay( self ):
self._paused = not self._paused
def ToListBoxTuple( self ):
return ( self._name, self._path, self._period )
return ( self._name, self._path, self._paused, self._check_regularly, self._period )
def ToTuple( self ):
@ -2786,7 +1885,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
new_options = HG.client_controller.new_options
self._checker_options = ClientDefaults.GetDefaultCheckerOptions( 'artist subscription' )
self._checker_options = HG.client_controller.new_options.GetDefaultSubscriptionCheckerOptions()
if HC.options[ 'gallery_file_limit' ] is None:
@ -2902,12 +2001,9 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
example_network_contexts = self._GetExampleNetworkContexts( query )
# just a little padding here
expected_requests = 3
expected_bytes = 1048576
threshold = 90
result = HG.client_controller.network_engine.bandwidth_manager.CanDoWork( example_network_contexts, expected_requests = expected_requests, expected_bytes = expected_bytes, threshold = threshold )
result = HG.client_controller.network_engine.bandwidth_manager.CanDoWork( example_network_contexts, threshold = threshold )
if HG.subscription_report_mode:

2
include/ClientNetworkingBandwidth.py
View File

@ -129,7 +129,7 @@ class NetworkBandwidthManager( HydrusSerialisable.SerialisableBase ):
def CanDoWork( self, network_contexts, expected_requests = 3, expected_bytes = 1048576, threshold = 30 ):
def CanDoWork( self, network_contexts, expected_requests = 1, expected_bytes = 1048576, threshold = 30 ):
with self._lock:

9
include/ClientNetworkingJobs.py
View File

@ -466,7 +466,7 @@ class NetworkJob( object ):
prefix = 'overriding bandwidth '
waiting_str = HydrusData.TimestampToPrettyTimeDelta( self._bandwidth_manual_override_delayed_timestamp )
waiting_str = HydrusData.TimestampToPrettyTimeDelta( self._bandwidth_manual_override_delayed_timestamp, just_now_string = 'imminently', just_now_threshold = 2 )
else:
@ -474,12 +474,7 @@ class NetworkJob( object ):
prefix = 'bandwidth free '
waiting_str = HydrusData.TimestampToPrettyTimeDelta( HydrusData.GetNow() + waiting_duration )
if waiting_duration < 2:
waiting_str = 'imminently'
waiting_str = HydrusData.TimestampToPrettyTimeDelta( HydrusData.GetNow() + waiting_duration, just_now_string = 'imminently', just_now_threshold = 2 )
self._status_text = prefix + waiting_str + u'\u2026'

222
include/ClientNetworkingLogin.py
View File

@ -36,6 +36,8 @@ class LoginCredentials( object ):
return self._credentials[ name ]
PIXIV_NETWORK_CONTEXT = ClientNetworkingContexts.NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, 'pixiv.net' )
HENTAI_FOUNDRY_NETWORK_CONTEXT = ClientNetworkingContexts.NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, 'hentai-foundry.com' )
class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_LOGIN_MANAGER
@ -107,6 +109,34 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
if network_context.context_type == CC.NETWORK_CONTEXT_DOMAIN:
domain = network_context.context_data
if 'pixiv.net' in domain:
if not LEGACY_LOGIN_OK:
raise Exception( 'Legacy login broke last time--please either restart the client or contact hydrus dev!' )
result = self.engine.controller.Read( 'serialisable_simple', 'pixiv_account' )
if result is None:
raise HydrusExceptions.DataMissing( 'You need to set up your pixiv credentials in services->manage pixiv account.' )
return
elif 'hentai-foundry.com' in domain:
if not LEGACY_LOGIN_OK:
raise Exception( 'Legacy login broke last time--please either restart the client or contact hydrus dev!' )
return
login_network_context = self._GetLoginNetworkContext( network_context )
if login_network_context is None:
@ -155,6 +185,17 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
if network_context.context_type == CC.NETWORK_CONTEXT_DOMAIN:
domain = network_context.context_data
if 'pixiv.net' in domain:
return LoginProcessLegacy( self.engine, PIXIV_NETWORK_CONTEXT, 'pixiv' )
elif 'hentai-foundry.com' in domain:
return LoginProcessLegacy( self.engine, HENTAI_FOUNDRY_NETWORK_CONTEXT, 'hentai foundry' )
login_network_context = self._GetLoginNetworkContext( network_context )
if login_network_context is None:
@ -181,6 +222,21 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
if network_context.context_type == CC.NETWORK_CONTEXT_DOMAIN:
domain = network_context.context_data
if 'pixiv.net' in domain:
required_cookies = [ 'PHPSESSID' ]
return not self._IsLoggedIn( PIXIV_NETWORK_CONTEXT, required_cookies )
elif 'hentai-foundry.com' in domain:
required_cookies = [ 'PHPSESSID', 'YII_CSRF_TOKEN' ]
return not self._IsLoggedIn( HENTAI_FOUNDRY_NETWORK_CONTEXT, required_cookies )
login_network_context = self._GetLoginNetworkContext( network_context )
if login_network_context is None:
@ -190,7 +246,7 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
( login_script, credentials ) = self._domains_to_login_scripts_and_credentials[ login_network_context.context_data ]
return login_script.IsLoggedIn( self.engine, login_network_context )
return not login_script.IsLoggedIn( self.engine, login_network_context )
elif network_context.context_type == CC.NETWORK_CONTEXT_HYDRUS:
@ -248,13 +304,13 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
if name == 'hentai foundry':
network_context = ClientNetworkingContexts.NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, 'hentai-foundry.com' )
network_context = HENTAI_FOUNDRY_NETWORK_CONTEXT
required_cookies = [ 'PHPSESSID', 'YII_CSRF_TOKEN' ]
elif name == 'pixiv':
network_context = ClientNetworkingContexts.NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, 'pixiv.net' )
network_context = PIXIV_NETWORK_CONTEXT
required_cookies = [ 'PHPSESSID' ]
@ -353,7 +409,7 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
# This updated login form is cobbled together from the example in PixivUtil2
# it is breddy shid because I'm not using mechanize or similar browser emulation (like requests's sessions) yet
# it is breddy shid but getting better
# Pixiv 400s if cookies and referrers aren't passed correctly
# I am leaving this as a mess with the hope the eventual login engine will replace it
def LoginPixiv( self, network_context, pixiv_id, password ):
@ -580,6 +636,162 @@ class LoginProcessHydrus( LoginProcess ):
self.login_script.Start( self.engine, self.network_context )
LEGACY_LOGIN_OK = True
class LoginProcessLegacy( LoginProcess ):
def _GetCookiesDict( self, network_context ):
session = self.engine.session_manager.GetSession( network_context )
cookies = session.cookies
cookies.clear_expired_cookies()
domains = cookies.list_domains()
for domain in domains:
if domain.endswith( network_context.context_data ):
return cookies.get_dict( domain )
return {}
def _Start( self ):
try:
name = self.login_script
if name == 'hentai foundry':
self.LoginHF( self.network_context )
elif name == 'pixiv':
result = self.engine.controller.Read( 'serialisable_simple', 'pixiv_account' )
if result is None:
raise HydrusExceptions.DataMissing( 'You need to set up your pixiv credentials in services->manage pixiv account.' )
( pixiv_id, password ) = result
self.LoginPixiv( self.network_context, pixiv_id, password )
except:
global LEGACY_LOGIN_OK
LEGACY_LOGIN_OK = False
def LoginHF( self, network_context ):
session = self.engine.session_manager.GetSession( network_context )
num_attempts = 0
while True:
try:
response = session.get( 'https://www.hentai-foundry.com/', timeout = 10 )
break
except ( requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout ):
if num_attempts < 3:
num_attempts += 1
time.sleep( 3 )
else:
raise HydrusExceptions.ConnectionException( 'Could not connect to HF to log in!' )
time.sleep( 1 )
response = session.get( 'https://www.hentai-foundry.com/?enterAgree=1' )
time.sleep( 1 )
cookie_dict = self._GetCookiesDict( network_context )
raw_csrf = cookie_dict[ 'YII_CSRF_TOKEN' ] # 19b05b536885ec60b8b37650a32f8deb11c08cd1s%3A40%3A%222917dcfbfbf2eda2c1fbe43f4d4c4ec4b6902b32%22%3B
processed_csrf = urllib.unquote( raw_csrf ) # 19b05b536885ec60b8b37650a32f8deb11c08cd1s:40:"2917dcfbfbf2eda2c1fbe43f4d4c4ec4b6902b32";
csrf_token = processed_csrf.split( '"' )[1] # the 2917... bit
hentai_foundry_form_info = ClientDefaults.GetDefaultHentaiFoundryInfo()
hentai_foundry_form_info[ 'YII_CSRF_TOKEN' ] = csrf_token
response = session.post( 'http://www.hentai-foundry.com/site/filters', data = hentai_foundry_form_info )
time.sleep( 1 )
# This updated login form is cobbled together from the example in PixivUtil2
# it is breddy shid but getting better
# Pixiv 400s if cookies and referrers aren't passed correctly
# I am leaving this as a mess with the hope the eventual login engine will replace it
def LoginPixiv( self, network_context, pixiv_id, password ):
session = self.engine.session_manager.GetSession( network_context )
response = session.get( 'https://accounts.pixiv.net/login' )
soup = ClientParsing.GetSoup( response.content )
# some whocking 20kb bit of json tucked inside a hidden form input wew lad
i = soup.find( 'input', id = 'init-config' )
raw_json = i['value']
j = json.loads( raw_json )
if 'pixivAccount.postKey' not in j:
raise HydrusExceptions.ForbiddenException( 'When trying to log into Pixiv, I could not find the POST key! This is a problem with hydrus\'s pixiv parsing, not your login! Please contact hydrus dev!' )
post_key = j[ 'pixivAccount.postKey' ]
form_fields = {}
form_fields[ 'pixiv_id' ] = pixiv_id
form_fields[ 'password' ] = password
form_fields[ 'captcha' ] = ''
form_fields[ 'g_recaptcha_response' ] = ''
form_fields[ 'return_to' ] = 'https://www.pixiv.net'
form_fields[ 'lang' ] = 'en'
form_fields[ 'post_key' ] = post_key
form_fields[ 'source' ] = 'pc'
headers = {}
headers[ 'referer' ] = "https://accounts.pixiv.net/login?lang=en^source=pc&view_type=page&ref=wwwtop_accounts_index"
headers[ 'origin' ] = "https://accounts.pixiv.net"
session.post( 'https://accounts.pixiv.net/api/login?lang=en', data = form_fields, headers = headers )
time.sleep( 1 )
class LoginScriptHydrus( object ):
def _IsLoggedIn( self, session ):
@ -653,7 +865,7 @@ class LoginScriptDomain( object ):
self._validity = VALIDITY_UNTESTED
self._error_reason = ''
self._expected_cookies_for_login = [] # [ name, stringmatch ]
self._expected_cookies_for_login = [] # [ name, stringmatch, minimum_expiry ]
def _IsLoggedIn( self, network_context, session ):

109
include/ClientOptions.py
View File

@ -362,10 +362,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
#
self._dictionary[ 'default_import_tag_options' ] = HydrusSerialisable.SerialisableDictionary()
#
self._dictionary[ 'frame_locations' ] = {}
# remember size, remember position, last_size, last_pos, default gravity, default position, maximised, fullscreen
@ -444,6 +440,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'misc' ] = HydrusSerialisable.SerialisableDictionary()
self._dictionary[ 'misc' ][ 'default_thread_watcher_options' ] = ClientDefaults.GetDefaultCheckerOptions( 'thread' )
self._dictionary[ 'misc' ][ 'default_subscription_checker_options' ] = ClientDefaults.GetDefaultCheckerOptions( 'artist subscription' )
#
@ -465,7 +462,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
for ( key, value ) in loaded_dictionary.items():
if isinstance( self._dictionary[ key ], dict ) and isinstance( value, dict ):
if key in self._dictionary and isinstance( self._dictionary[ key ], dict ) and isinstance( value, dict ):
self._dictionary[ key ].update( value )
@ -593,14 +590,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def ClearDefaultTagImportOptions( self ):
with self._lock:
self._dictionary[ 'default_import_tag_options' ] = HydrusSerialisable.SerialisableDictionary()
def FlipBoolean( self, name ):
with self._lock:
@ -672,76 +661,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def GetDefaultTagImportOptions( self, gallery_identifier = None ):
with self._lock:
default_tag_import_options = self._dictionary[ 'default_import_tag_options' ]
if gallery_identifier is None:
return default_tag_import_options
else:
if gallery_identifier in default_tag_import_options:
tag_import_options = default_tag_import_options[ gallery_identifier ]
else:
default_booru_gallery_identifier = ClientDownloading.GalleryIdentifier( HC.SITE_TYPE_BOORU )
default_hentai_foundry_gallery_identifier = ClientDownloading.GalleryIdentifier( HC.SITE_TYPE_HENTAI_FOUNDRY )
default_pixiv_gallery_identifier = ClientDownloading.GalleryIdentifier( HC.SITE_TYPE_PIXIV )
default_gallery_identifier = ClientDownloading.GalleryIdentifier( HC.SITE_TYPE_DEFAULT )
guidance_tag_import_options = None
site_type = gallery_identifier.GetSiteType()
if site_type == HC.SITE_TYPE_WATCHER:
import ClientImportOptions
return ClientImportOptions.TagImportOptions() # if nothing set, do nothing in this special case
if site_type == HC.SITE_TYPE_BOORU and default_booru_gallery_identifier in default_tag_import_options:
guidance_tag_import_options = default_tag_import_options[ default_booru_gallery_identifier ]
elif site_type in ( HC.SITE_TYPE_HENTAI_FOUNDRY_ARTIST, HC.SITE_TYPE_HENTAI_FOUNDRY_TAGS ) and default_hentai_foundry_gallery_identifier in default_tag_import_options:
guidance_tag_import_options = default_tag_import_options[ default_hentai_foundry_gallery_identifier ]
elif site_type in ( HC.SITE_TYPE_PIXIV_ARTIST_ID, HC.SITE_TYPE_PIXIV_TAG ) and default_pixiv_gallery_identifier in default_tag_import_options:
guidance_tag_import_options = default_tag_import_options[ default_pixiv_gallery_identifier ]
elif default_gallery_identifier in default_tag_import_options:
guidance_tag_import_options = default_tag_import_options[ default_gallery_identifier ]
if guidance_tag_import_options is None:
import ClientImportOptions
tag_import_options = ClientImportOptions.TagImportOptions()
else:
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( gallery_identifier )
tag_import_options = guidance_tag_import_options.DeriveTagImportOptionsFromSelf( namespaces )
return tag_import_options
def GetDefaultWatcherCheckerOptions( self ):
with self._lock:
@ -758,6 +677,14 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def GetDefaultSubscriptionCheckerOptions( self ):
with self._lock:
return self._dictionary[ 'misc' ][ 'default_subscription_checker_options' ]
def GetDuplicateActionOptions( self, duplicate_type ):
with self._lock:
@ -1021,14 +948,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def SetDefaultTagImportOptions( self, gallery_identifier, tag_import_options ):
with self._lock:
self._dictionary[ 'default_import_tag_options' ][ gallery_identifier ] = tag_import_options
def SetDefaultWatcherCheckerOptions( self, checker_options ):
with self._lock:
@ -1045,6 +964,14 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def SetDefaultSubscriptionCheckerOptions( self, checker_options ):
with self._lock:
self._dictionary[ 'misc' ][ 'default_subscription_checker_options' ] = checker_options
def SetDuplicateActionOptions( self, duplicate_type, duplicate_action_options ):
with self._lock:

96
include/ClientParsing.py
View File

@ -1688,7 +1688,18 @@ class ContentParser( HydrusSerialisable.SerialisableBase ):
def Parse( self, parsing_context, data ):
parsed_texts = self._formula.Parse( parsing_context, data )
try:
parsed_texts = self._formula.Parse( parsing_context, data )
except HydrusExceptions.ParseException as e:
prefix = 'Content Parser ' + self._name + ': '
e = HydrusExceptions.ParseException( prefix + HydrusData.ToUnicode( e ) )
raise e
if self._content_type == HC.CONTENT_TYPE_URLS:
@ -1739,6 +1750,14 @@ class ContentParser( HydrusSerialisable.SerialisableBase ):
results = [ 'veto: ' + HydrusData.ToUnicode( e ) ]
except HydrusExceptions.ParseException as e:
prefix = 'Content Parser ' + self._name + ': '
e = HydrusExceptions.ParseException( prefix + HydrusData.ToUnicode( e ) )
raise e
result_lines = [ '*** ' + HydrusData.ToHumanInt( len( results ) ) + ' RESULTS BEGIN ***' ]
@ -1921,14 +1940,34 @@ class PageParser( HydrusSerialisable.SerialisableBaseNamed ):
raise HydrusExceptions.ParseException( HydrusData.ToUnicode( e ) )
except HydrusExceptions.ParseException as e:
prefix = 'Page Parser ' + self._name + ': '
e = HydrusExceptions.ParseException( prefix + HydrusData.ToUnicode( e ) )
raise e
#
whole_page_parse_results = []
for content_parser in self._content_parsers:
try:
whole_page_parse_results.extend( content_parser.Parse( parsing_context, converted_page_data ) )
for content_parser in self._content_parsers:
whole_page_parse_results.extend( content_parser.Parse( parsing_context, converted_page_data ) )
except HydrusExceptions.ParseException as e:
prefix = 'Page Parser ' + self._name + ': '
e = HydrusExceptions.ParseException( prefix + HydrusData.ToUnicode( e ) )
raise e
#
@ -1955,28 +1994,39 @@ class PageParser( HydrusSerialisable.SerialisableBaseNamed ):
sub_page_parsers.sort( key = sort_key )
for ( formula, page_parser ) in self._sub_page_parsers:
try:
posts = formula.Parse( parsing_context, converted_page_data )
for ( formula, page_parser ) in self._sub_page_parsers:
posts = formula.Parse( parsing_context, converted_page_data )
for post in posts:
try:
page_parser_all_parse_results = page_parser.Parse( parsing_context, post )
except HydrusExceptions.VetoException:
continue
for page_parser_parse_results in page_parser_all_parse_results:
page_parser_parse_results.extend( whole_page_parse_results )
all_parse_results.append( page_parser_parse_results )
for post in posts:
try:
page_parser_all_parse_results = page_parser.Parse( parsing_context, post )
except HydrusExceptions.VetoException:
continue
for page_parser_parse_results in page_parser_all_parse_results:
page_parser_parse_results.extend( whole_page_parse_results )
all_parse_results.append( page_parser_parse_results )
except HydrusExceptions.ParseException as e:
prefix = 'Page Parser ' + self._name + ': '
e = HydrusExceptions.ParseException( prefix + HydrusData.ToUnicode( e ) )
raise e

2
include/ClientRendering.py
View File

@ -21,7 +21,7 @@ try:
LZ4_OK = True
except ImportError:
except: # ImportError wasn't enough here as Linux went up the shoot with a __version__ doesn't exist bs
pass

2
include/HydrusConstants.py
View File

@ -49,7 +49,7 @@ options = {}
# Misc
NETWORK_VERSION = 18
SOFTWARE_VERSION = 314
SOFTWARE_VERSION = 315
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )

17
include/HydrusData.py
View File

@ -12,6 +12,7 @@ import os
import pstats
import psutil
import random
import re
import shutil
import sqlite3
import struct
@ -376,7 +377,7 @@ def ConvertTimestampToPrettyTime( timestamp, in_gmt = False, include_24h_time =
return time.strftime( phrase, struct_time )
def TimestampToPrettyTimeDelta( timestamp ):
def TimestampToPrettyTimeDelta( timestamp, just_now_string = 'now', just_now_threshold = 3 ):
if HG.client_controller.new_options.GetBoolean( 'always_show_iso_time' ):
@ -385,9 +386,9 @@ def TimestampToPrettyTimeDelta( timestamp ):
time_delta = abs( timestamp - GetNow() )
if time_delta < 5:
if time_delta < just_now_threshold:
return 'now'
return just_now_string
time_delta_string = TimeDeltaToPrettyTimeDelta( time_delta )
@ -642,6 +643,16 @@ def GetTypeName( obj_type ):
return repr( obj_type )
def HumanTextSort( texts ):
"""Solves the 19, 20, 200, 21, 22 issue when sorting 'Page 21.jpg' type strings.
Breaks the string into groups of text and int."""
convert = lambda text: int( text ) if text.isdigit() else text
alphanum = lambda key: [ convert( c ) for c in re.split( '([0-9]+)', key ) ]
texts.sort( key = alphanum )
def IntelligentMassIntersect( sets_to_reduce ):
answer = None

24
include/HydrusExceptions.py
View File

@ -1,32 +1,34 @@
import os
import traceback
class CantRenderWithCVException( Exception ): pass
class DataMissing( Exception ): pass
class DBException( Exception ):
class HydrusException( Exception ):
def __str__( self ):
return os.linesep.join( self.args )
class CantRenderWithCVException( HydrusException ): pass
class DataMissing( HydrusException ): pass
class DBAccessException( Exception ): pass
class FileMissingException( Exception ): pass
class NameException( Exception ): pass
class ShutdownException( Exception ): pass
class DBException( HydrusException ): pass
class DBAccessException( HydrusException ): pass
class FileMissingException( HydrusException ): pass
class NameException( HydrusException ): pass
class ShutdownException( HydrusException ): pass
class VetoException( Exception ): pass
class VetoException( HydrusException ): pass
class CancelledException( VetoException ): pass
class MimeException( VetoException ): pass
class SizeException( VetoException ): pass
class DecompressionBombException( SizeException ): pass
class ParseException( Exception ): pass
class ParseException( HydrusException ): pass
class StringConvertException( ParseException ): pass
class StringMatchException( ParseException ): pass
class URLMatchException( ParseException ): pass
class NetworkException( Exception ): pass
class NetworkException( HydrusException ): pass
class NetworkInfrastructureException( NetworkException ): pass
class ConnectionException( NetworkInfrastructureException ): pass

6
include/HydrusSerialisable.py
View File

@ -9,11 +9,10 @@ try:
LZ4_OK = True
except ImportError:
except: # ImportError wasn't enough here as Linux went up the shoot with a __version__ doesn't exist bs
print( 'Could not import lz4.' )
SERIALISABLE_TYPE_BASE = 0
SERIALISABLE_TYPE_BASE_NAMED = 1
SERIALISABLE_TYPE_SHORTCUTS = 2
@ -34,7 +33,7 @@ SERIALISABLE_TYPE_EXPORT_FOLDER = 16
SERIALISABLE_TYPE_WATCHER_IMPORT = 17
SERIALISABLE_TYPE_SIMPLE_DOWNLOADER_IMPORT = 18
SERIALISABLE_TYPE_IMPORT_FOLDER = 19
SERIALISABLE_TYPE_GALLERY_IMPORT = 20
SERIALISABLE_TYPE_MULTIPLE_GALLERY_IMPORT = 20
SERIALISABLE_TYPE_DICTIONARY = 21
SERIALISABLE_TYPE_CLIENT_OPTIONS = 22
SERIALISABLE_TYPE_CONTENT = 23
@ -82,6 +81,7 @@ SERIALISABLE_TYPE_MULTIPLE_WATCHER_IMPORT = 64
SERIALISABLE_TYPE_SERVICE_TAG_IMPORT_OPTIONS = 65
SERIALISABLE_TYPE_GALLERY_SEED = 66
SERIALISABLE_TYPE_GALLERY_SEED_LOG = 67
SERIALISABLE_TYPE_GALLERY_IMPORT = 68
SERIALISABLE_TYPES_TO_OBJECT_TYPES = {}

8
include/HydrusServerResources.py
View File

@ -218,14 +218,6 @@ def ParseFileArguments( path, decompression_bombs_ok = False ):
( size, mime, width, height, duration, num_frames, num_words ) = HydrusFileHandling.GetFileInfo( path, mime )
except HydrusExceptions.SizeException:
raise HydrusExceptions.ForbiddenException( 'File is of zero length!' )
except HydrusExceptions.MimeException:
raise HydrusExceptions.ForbiddenException( 'Filetype is not permitted!' )
except Exception as e:
raise HydrusExceptions.ForbiddenException( HydrusData.ToUnicode( e ) )

2
server.py
View File

@ -143,6 +143,6 @@ except Exception as e:
f.write( traceback.format_exc() )
print( 'Critical error occured! Details written to crash.log!' )
print( 'Critical error occurred! Details written to crash.log!' )

BIN
static/default/parsers/moebooru file page parser.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.9 KiB

After

Width:  |  Height:  |  Size: 3.0 KiB

BIN
static/default/parsers/tumblr api post page parser - with post tags.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB