Rooba
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Version 132

This commit is contained in:
Hydrus 2014-10-01 17:58:32 -05:00
parent 30b4636cb8
commit 72a2605d5b
21 changed files with 701 additions and 488 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
help/changelog.html
View File

@ -8,6 +8,26 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 132</h3></li>
<ul>
<li>merged two complicated serverside account tables into two simpler tables</li>
<li>with this action, was able to clean out a lot of rubbish old server account code</li>
<li>made it so accounts can only be on one service. existing (admin) accounts that straddled sevices will have new access keys printed in a text file in the base installation directory on server update</li>
<li>rewrote the account object to be simpler and easier to maintain</li>
<li>swapped the old rubbish 'account_id' identifier in the account object for the much better 'account_key' identifier</li>
<li>harmonised some conflicting account-related variable names</li>
<li>refined the way the unknown account is stored and identified</li>
<li>split serverside account verification and identification into two separate paths, to reduce chance of security problems</li>
<li>reworked account identifiers (a general purpose account identifying object that is used in admin-server interactions) and their associated db functions to be more secure and reliable</li>
<li>simplified account data use checking</li>
<li>simplified and harmonised the way used bytes and used requests are stored and retrieved in the account and account type</li>
<li>with guarantee of account_key uniqueness across entire server, I have simplified session code in several places</li>
<li>updated help to reflect the new relationship between access keys and account keys</li>
<li>added 'copy account key' button to review services, which will now be the thing for users to use if they need an admin to modify their account</li>
<li>fixed serverside credential verification for non-instantiated (still have a registration key) access keys</li>
<li>added a bit of explaining text to the 'waiting' autocomplete state</li>
<li>fixed a typo when deleting files from a file repo</li>
</ul>
<li><h3>version 131</h3></li>
<ul>
<li>removed tag service precedence and its various expensive and overly complicated effects</li>

6
help/faq.html
View File

@ -35,12 +35,14 @@
<p>Hashes are a subject one usually has to be a software engineer to find interesting. If you don't care to digest the wiki page, the simple answer is that hashes are unique names for things. It can be proven that f099b5823f4e36a4bd6562812582f60e49e818cf445902b504b5533c6a5dad94 refers to one particular file and no other. Hashes make excellent—if ugly—identifiers. In the client's normal operation, you will never encounter a file's hash; if you want to see a thumbnail bigger, double-click it; the software handles the mathematics.</p>
<p><i>For those who </i>are<i> interested: hydrus uses SHA-256, which spits out 32-byte (256-bit) hashes. The software stores the hash densely, as 32 bytes, only encoding it to 64 hex characters when the user views it or copies to clipboard. SHA-256 is not perfect, but it is a great compromise candidate; it is secure for now, it is reasonably fast, it is available for most programming languages, and newer CPUs perform it more efficiently all the time.</i></p>
<a name="access_keys"><h3>hold up, what is an access key?</h3></a>
<p>The hydrus network's repositories do not use username/password, but instead a single combination identifier-password like this:</p>
<p>The hydrus network's repositories do not use username/password, but instead a single strong identifier-password like this:</p>
<p><i>7ce4dbf18f7af8b420ee942bae42030aab344e91dc0e839260fcd71a4c9879e3</i></p>
<p>These hex numbers give you access to a particular account on a particular repository, and are often combined like so:</p>
<p><i>7ce4dbf18f7af8b420ee942bae42030aab344e91dc0e839260fcd71a4c9879e3@hostname.com:45871</i></p>
<p>They are long enough to be impossible to guess, and also randomly generated, so they reveal nothing personally identifying about you. Many people can use the same access key (and hence the same account) on a repository without consequence, although they will have to share bandwidth limits, and if one person screws around and gets the account banned, everyone will lose access.</p>
<p>The access key is the account.</p>
<p>The access key is the account. Do not give it to anyone you do not want to have access to the account. An administrator will never need it; instead they will want your <i>account key</i>.</p>
<a name="account_keys"><h3>hold up, what is an account key?</h3></a>
<p>This is another long string of random hexadecimal that <i>identifies</i> your account without giving away any information about you. If you need to identify yourself to a repository administrator (say, to get your account's permissions modified), you will need to tell them your account key. You can copy it to your clipboard in <i>services->review services</i>.</p>
<h3>why aren't my swfs showing?</h3>
<p>If an Internet Explorer "Navigation Cancelled" page appears whenever you click on a swf thumbnail, try installing Flash Player for Internet Explorer. Just having it installed for Firefox/Opera is not enough; you need the ActiveX component that comes with the specific IE version. Just boot IE and download/run the installer from Adobe's site.</p>
<a name="delays"><h3>why can my friend not see what I just uploaded?</h3></a>

11
help/glossary.html
View File

@ -6,23 +6,24 @@
</head>
<body>
<div class="content">
<p><b>access key</b> A 32-byte identifier-password that gives you certain permissions with a repository. Usually represented as a 64-character hex string like so: 7ce4dbf18f7af8b420ee942bae42030aab344e91dc0e839260fcd71a4c9879e3</p>
<p><b>access key</b> A 32-byte identifier-password that gives you access to an account that has certain permissions with a repository. Usually represented as a 64-character hex string like so: 7ce4dbf18f7af8b420ee942bae42030aab344e91dc0e839260fcd71a4c9879e3</p>
<p><b>account key</b> A 32-byte identifier for a hydrus service account. Usually represented as a 64-character hex string like so: 0c3b554cb6fe7d55c945df88b2f6cf6ca0ae40824bca7534aa2fd483da7fb219</p>
<p><b>address</b> The pairing of both a server's host (be that an IP or a domain) with its port number, like so: 74.125.225.18:80, or google.com:80</p>
<p><b>archive</b> The store of files you have chosen to keep.</p>
<p><b>file repository</b> A service in the hydrus network that hosts files.</p>
<p><b>filtering</b> A method of quickly deleting and archiving files within the client.</p>
<p><b>hash</b> A file's unique identifier. The hydrus network uses SHA-256.</p>
<p><b>hash</b> A file's unique identifier. The hydrus network uses SHA-256, which generates 32-byte hashes.</p>
<p><b>hydrus client</b> An application that manages media and connects to services on the hydrus network.</p>
<p><b>hydrus network</b> A loose collection of clients and servers that attempt to make media management and distribution easier.</p>
<p><b>hydrus server</b> An executable that can run any number of hydrus services for any number of clients to plug into. It is controlled entirely by the client.</p>
<p><b>hydrus server</b> An executable that can run any number of hydrus services for any number of clients to plug into. It is managed entirely by the client.</p>
<p><b>inbox</b> A special tag the client gives to newly imported and downloaded files to make them easier to find and review.</p>
<p><b>mapping</b> The pairing of a particular file with a particular tag.</p>
<p><b>message depot</b> A service in the hydrus network that stores messages.</p>
<p><b>metadata</b> Information <i>about</i> a file, but not stored within the file. Filename, size, hash, modified dates, tags and location are all good examples.</p>
<p><b>petition</b> A request from an uploader for particular content to be removed from a repository.</p>
<p><b>registration key</b> A single-use 32-byte password that, when submitted to a repository, gives an access key. It allows moderators to distribute accounts through insecure channels.</a>
<p><b>registration key</b> A single-use 32-byte password that, when submitted to a repository, generates an account and an access key. They add a bit of privacy to the account creation process.</a>
<p><b>tag</b> A short string of text describing a file.</p>
<p><b>tag repository</b> The service in the hydrus network that hosts mappings.</p>
<p><b>tag repository</b> A service in the hydrus network that hosts mappings.</p>
</div>
</body>
</html>

2
help/privacy.html
View File

@ -50,7 +50,7 @@
<ul>
<li>Administrators for a particular repository can see which accounts uploaded what. If IP addresses are available, they can discover which IP uploaded a particular file, and when.</li>
<li>Repositories do not talk to each other.</li>
<li>All accounts are anonymous. Repositories do not <i>know</i> any of their accounts' access keys, and cannot produce them on demand; they can determine whether a particular access key refers to a particular account, but the access keys themselves are all irreversibly hashed inside the repository database.</li>
<li>All accounts are anonymous. Repositories do not <i>know</i> any of their accounts' access keys and cannot produce them on demand; they can determine whether a particular access key refers to a particular account, but the access keys themselves are all irreversibly hashed inside the repository database.</li>
</ul>
</p>
<p>There are of course some clever exceptions. If you tag a file three years before it surfaces on the internet, someone with enough knowledge will be able to infer it was most likely you who created it. If you set up a file repository for just a friend and yourself, it becomes trivial by elimination to guess who uploaded the NarutoXSonichu shota diaper fanon. If you sign up for a file repository that hosts only certain stuff and rack up a huge bandwidth record for the current month, anyone who knows that and also knows the account is yours alone will know basically what you were up to.</p>

8
help/server.html
View File

@ -25,7 +25,7 @@
</li>
<li>This is not for noobs. If you are wholly unfamiliar with NAT (opening ports on firewalls) and setting up servers in general, this may be a step too far. If you get stuck, drag a nerd away from Xel'Naga Caverns to give you a hand.</li>
<li>If you want your services to thrive in a community, you shall have to manage them. If you throw out a hundred uploader keys and go on a long vacation, do not expect your long-crafted thirteen-point 'Superman vs Goku ONLY!!!!!' ruleset to be upheld. Your repositories will clog with unrelated rubbish and your bandwidth will disappear. Banning worthless people and pruning worthless content is an important part of any online ecosystem.</li>
<li>There are legal issues with running a file-hosting service. If you plan to do this on any kind of large scale, <span class="warning">especially</span> if you plan to sell access keys for cash-money, familiarise yourself with your legal rights and responsibilities. You may—without even meaning to—make yourself liable for others' sins.</li>
<li>There are legal issues with running a file-hosting service. If you plan to do this on any kind of large scale, <span class="warning">especially</span> if you plan to sell access for cash-money, familiarise yourself with your legal rights and responsibilities. You may—without even meaning to—make yourself liable for others' sins.</li>
</ul>
<h3>still keen?</h3>
<p><b><i>If this stuff be-fuzzles you, and you just want a simple server set up on the same computer you run the client, you can now go </i>help->i don't know what I am doing->just set up the server on this computer, please<i> and you _should_ be all set up with an admin service and tag/file repos automatically.</i></b></p>
@ -40,7 +40,7 @@
<li>Start the server.</li>
<li>Set up your client with its address and initialise the administration interface</li>
<li>Set the server's options and services.</li>
<li>Make some access keys for your users.</li>
<li>Make some accounts for your users.</li>
<li>???</li>
<li>Profit</li>
</ul>
@ -49,7 +49,7 @@
<p>Since the server and client have so much common code, I have packaged them together. If you have the client, you have the server. To start it, you can hit the shortcut in your start menu or just go straight for server.exe/.pyw in the install directory. It will first try to take port 45870 or its administration interface, so make sure that is free. Open your firewall as appropriate.</p>
<h3>set up the client</h3>
<p>In the <i>services->manage services</i> dialog, go to the <i>servers admin</i> tab, give your server admin interface a nickname and set the credentials to whatever-hostname:45870. Don't enter any access key. Ok those changes, and go to <i>review services</i>.</p>
<p>On the tab+page for your new server, hit the initialise button. If you have everything set right, the server should generate its first administrator account and return the access key, which the client will automatically set to the account for you.</p>
<p>On the tab+page for your new server, hit the initialise button. If you have everything set right, the server should generate its first, administrator account and return the access key, which the client will automatically set to the account for you.</p>
<p class="warning">YOU'LL WANT TO SAVE THAT KEY IN A SAFE PLACE</p>
<p>If you lose your admin access key, there is no way to get it back, and if you are not sqlite-proficient, you'll have to restart from the beginning by deleting your server's database files.</p>
<p>If the client can't connect to the server, it is either not running or you have a firewall/port-mapping problem. If you want a quick way to test the server's visibility, just put its host:port into your browser; if working, it should return some simple html identifying itself.</p>
@ -66,7 +66,7 @@
<h3>???</h3>
<p>The most important part is to have fun! There are no losers on the INFORMATION SUPERHIGHWAY.</p>
<h3>profit</h3>
<p>I honestly hope you can get some benefit out of my code, whether just as a backup or as part of a far more complex system. Please mail me your comments as I am keen to make improvements.</p>
<p>I honestly hope you can get some benefit out of my code, whether just as a backup or as part of a far more complex system. Please mail me your comments as I am always keen to make improvements.</p>
<h3>btw, how to backup a repo's db</h3>
<p>All of a server's files and options are stored in its accompanying .db file and respective subdirectories, which are created on first startup (just like with the client). You can backup and restore these files just by copying them about, but you have to be careful how you do it with a server; when it is running, it has a live connection to its database, and all sorts of things could be written or read at any time. If you just try to copy the .db somewhere and someone uploads a file, something might break. Instead, you have two options:</p>
<ul>

10
include/ClientConstants.py
View File

@ -2535,7 +2535,15 @@ class Service( HC.HydrusYAMLBase ):
if isinstance( e, HydrusExceptions.PermissionException ):
HC.app.Write( 'service_updates', { self._service_key : [ HC.ServiceUpdate( HC.SERVICE_UPDATE_ACCOUNT, HC.GetUnknownAccount() ) ] } )
if 'account' in self._info:
account_key = self._info[ 'account' ].GetAccountKey()
unknown_account = HC.GetUnknownAccount( account_key )
else: unknown_account = HC.GetUnknownAccount()
HC.app.Write( 'service_updates', { self._service_key : [ HC.ServiceUpdate( HC.SERVICE_UPDATE_ACCOUNT, unknown_account ) ] } )
raise

29
include/ClientDB.py
View File

@ -1309,11 +1309,11 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
def _AddHydrusSession( self, c, service_key, session_key, expiry ):
def _AddHydrusSession( self, c, service_key, session_key, expires ):
service_id = self._GetServiceId( c, service_key )
c.execute( 'REPLACE INTO hydrus_sessions ( service_id, session_key, expiry ) VALUES ( ?, ?, ? );', ( service_id, sqlite3.Binary( session_key ), expiry ) )
c.execute( 'REPLACE INTO hydrus_sessions ( service_id, session_key, expiry ) VALUES ( ?, ?, ? );', ( service_id, sqlite3.Binary( session_key ), expires ) )
def _AddService( self, c, service_key, service_type, name, info ):
@ -1389,9 +1389,9 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
def _AddWebSession( self, c, name, cookies, expiry ):
def _AddWebSession( self, c, name, cookies, expires ):
c.execute( 'REPLACE INTO web_sessions ( name, cookies, expiry ) VALUES ( ?, ?, ? );', ( name, cookies, expiry ) )
c.execute( 'REPLACE INTO web_sessions ( name, cookies, expiry ) VALUES ( ?, ?, ? );', ( name, cookies, expires ) )
def _ArchiveFiles( self, c, hash_ids ):
@ -2240,13 +2240,13 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
results = c.execute( 'SELECT service_id, session_key, expiry FROM hydrus_sessions;' ).fetchall()
for ( service_id, session_key, expiry ) in results:
for ( service_id, session_key, expires ) in results:
service = self._GetService( c, service_id )
service_key = service.GetServiceKey()
sessions.append( ( service_key, session_key, expiry ) )
sessions.append( ( service_key, session_key, expires ) )
return sessions
@ -5294,6 +5294,21 @@ class DB( ServiceDB ):
self._RecalcCombinedMappings( c )
if version == 131:
service_info = c.execute( 'SELECT service_id, info FROM services;' ).fetchall()
for ( service_id, info ) in service_info:
if 'account' in info:
info[ 'account' ] = HC.GetUnknownAccount()
c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
HC.is_db_updated = True
@ -5988,7 +6003,7 @@ def DAEMONSynchroniseAccounts():
if info[ 'paused' ]: continue
if not account.IsBanned() and account.IsStale() and credentials.HasAccessKey() and not service.HasRecentError():
if account.IsStale() and credentials.HasAccessKey() and not service.HasRecentError():
try:

49
include/ClientGUI.py
View File

@ -315,15 +315,15 @@ class FrameGUI( ClientGUICommon.FrameThatResizes ):
def _AccountInfo( self, service_key ):
with ClientGUIDialogs.DialogTextEntry( self, 'Enter the account\'s access key.' ) as dlg:
with ClientGUIDialogs.DialogTextEntry( self, 'Enter the account\'s account key.' ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
subject_access_key = dlg.GetValue().decode( 'hex' )
subject_account_key = dlg.GetValue().decode( 'hex' )
service = HC.app.GetManager( 'services' ).GetService( service_key )
response = service.Request( HC.GET, 'account_info', { 'subject_access_key' : subject_access_key.encode( 'hex' ) } )
response = service.Request( HC.GET, 'account_info', { 'subject_account_key' : subject_account_key.encode( 'hex' ) } )
account_info = response[ 'account_info' ]
@ -1238,19 +1238,19 @@ class FrameGUI( ClientGUICommon.FrameThatResizes ):
service = HC.app.GetManager( 'services' ).GetService( service_key )
with ClientGUIDialogs.DialogTextEntry( self, 'Enter the access key for the account to be modified.' ) as dlg:
with ClientGUIDialogs.DialogTextEntry( self, 'Enter the account key for the account to be modified.' ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
try: access_key = dlg.GetValue().decode( 'hex' )
try: account_key = dlg.GetValue().decode( 'hex' )
except:
wx.MessageBox( 'Could not parse that access key' )
wx.MessageBox( 'Could not parse that account key' )
return
subject_identifiers = ( HC.AccountIdentifier( access_key = access_key ), )
subject_identifiers = ( HC.AccountIdentifier( account_key = account_key ), )
with ClientGUIDialogs.DialogModifyAccounts( self, service_key, subject_identifiers ) as dlg2: dlg2.ShowModal()
@ -2375,6 +2375,9 @@ class FrameReviewServices( ClientGUICommon.Frame ):
self._refresh = wx.Button( self, label = 'refresh account' )
self._refresh.Bind( wx.EVT_BUTTON, self.EventServiceRefreshAccount )
self._copy_account_key = wx.Button( self, label = 'copy account key' )
self._copy_account_key.Bind( wx.EVT_BUTTON, self.EventCopyAccountKey )
def PopulateControls():
@ -2478,6 +2481,7 @@ class FrameReviewServices( ClientGUICommon.Frame ):
if service_type in HC.RESTRICTED_SERVICES:
repo_buttons_hbox.AddF( self._refresh, FLAGS_MIXED )
repo_buttons_hbox.AddF( self._copy_account_key, FLAGS_MIXED )
vbox.AddF( repo_buttons_hbox, FLAGS_BUTTON_SIZERS )
@ -2566,21 +2570,24 @@ class FrameReviewServices( ClientGUICommon.Frame ):
created = account.GetCreated()
expiry = account.GetExpiry()
expires = account.GetExpires()
if expiry is None: self._age.Hide()
if expires is None: self._age.Hide()
else:
self._age.Show()
self._age.SetRange( expiry - created )
self._age.SetValue( min( now - created, expiry - created ) )
self._age.SetRange( expires - created )
self._age.SetValue( min( now - created, expires - created ) )
self._age_text.SetLabel( account.GetExpiryString() )
self._age_text.SetLabel( account.GetExpiresString() )
( max_num_bytes, max_num_requests ) = account_type.GetMaxMonthlyData()
( used_bytes, used_requests ) = account.GetUsedData()
max_num_bytes = account_type.GetMaxBytes()
max_num_requests = account_type.GetMaxRequests()
used_bytes = account.GetUsedBytes()
used_requests = account.GetUsedRequests()
if max_num_bytes is None: self._bytes.Hide()
else:
@ -2629,6 +2636,9 @@ class FrameReviewServices( ClientGUICommon.Frame ):
self._refresh.Enable()
if account.HasAccountKey(): self._copy_account_key.Enable()
else: self._copy_account_key.Disable()
def _DisplayNumThumbs( self ):
@ -2710,7 +2720,7 @@ class FrameReviewServices( ClientGUICommon.Frame ):
timeout = info[ 'timeout' ]
hashes = info[ 'hashes' ]
self._booru_shares.Append( ( name, text, HC.ConvertTimestampToPrettyExpiry( timeout ), len( hashes ) ), ( name, text, timeout, ( len( hashes ), hashes, share_key ) ) )
self._booru_shares.Append( ( name, text, HC.ConvertTimestampToPrettyExpires( timeout ), len( hashes ) ), ( name, text, timeout, ( len( hashes ), hashes, share_key ) ) )
@ -2784,6 +2794,15 @@ class FrameReviewServices( ClientGUICommon.Frame ):
def EventCopyAccountKey( self, event ):
account_key = self._service.GetInfo( 'account' ).GetAccountKey()
account_key_hex = account_key.encode( 'hex' )
HC.pubsub.pub( 'clipboard', 'text', account_key_hex )
def EventCopyExternalShareURL( self, event ):
shares = self._booru_shares.GetSelectedClientData()

2
include/ClientGUICommon.py
View File

@ -553,7 +553,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
self._include_pending_tags = OnOffButton( self._dropdown_window, self._page_key, 'notify_include_pending', on_label = 'include pending tags', off_label = 'exclude pending tags' )
self._include_pending_tags.SetToolTipString( 'select whether to include pending tags in the search' )
self._synchronised = OnOffButton( self._dropdown_window, self._page_key, 'notify_search_immediately', on_label = 'searching immediately', off_label = 'waiting' )
self._synchronised = OnOffButton( self._dropdown_window, self._page_key, 'notify_search_immediately', on_label = 'searching immediately', off_label = 'waiting -- tag counts may be inaccurate' )
self._synchronised.SetToolTipString( 'select whether to renew the search as soon as a new predicate is entered' )
button_hbox_1 = wx.BoxSizer( wx.HORIZONTAL )

59
include/ClientGUIDialogs.py
View File

@ -3203,7 +3203,8 @@ class DialogInputNewAccountType( Dialog ):
title = account_type.GetTitle()
permissions = account_type.GetPermissions()
( max_num_bytes, max_num_requests ) = account_type.GetMaxMonthlyData()
max_num_bytes = account_type.GetMaxBytes()
max_num_requests = account_type.GetMaxRequests()
Dialog.__init__( self, parent, 'edit account type' )
@ -3534,15 +3535,15 @@ class DialogModifyAccounts( Dialog ):
self._expiration_panel = ClientGUICommon.StaticBox( self, 'change expiration' )
self._add_to_expiry = wx.Choice( self._expiration_panel )
self._add_to_expires = wx.Choice( self._expiration_panel )
self._add_to_expiry_ok = wx.Button( self._expiration_panel, label = 'Ok' )
self._add_to_expiry_ok.Bind( wx.EVT_BUTTON, self.EventAddToExpiry )
self._add_to_expires_ok = wx.Button( self._expiration_panel, label = 'Ok' )
self._add_to_expires_ok.Bind( wx.EVT_BUTTON, self.EventAddToExpires )
self._set_expiry = wx.Choice( self._expiration_panel )
self._set_expires = wx.Choice( self._expiration_panel )
self._set_expiry_ok = wx.Button( self._expiration_panel, label = 'Ok' )
self._set_expiry_ok.Bind( wx.EVT_BUTTON, self.EventSetExpiry )
self._set_expires_ok = wx.Button( self._expiration_panel, label = 'Ok' )
self._set_expires_ok.Bind( wx.EVT_BUTTON, self.EventSetExpires )
#
@ -3589,21 +3590,21 @@ class DialogModifyAccounts( Dialog ):
for ( string, value ) in HC.lifetimes:
if value is not None: self._add_to_expiry.Append( string, value ) # don't want 'add no limit'
if value is not None: self._add_to_expires.Append( string, value ) # don't want 'add no limit'
self._add_to_expiry.SetSelection( 1 ) # three months
self._add_to_expires.SetSelection( 1 ) # three months
for ( string, value ) in HC.lifetimes: self._set_expiry.Append( string, value )
self._set_expiry.SetSelection( 1 ) # three months
for ( string, value ) in HC.lifetimes: self._set_expires.Append( string, value )
self._set_expires.SetSelection( 1 ) # three months
#
if not self._service.GetInfo( 'account' ).HasPermission( HC.GENERAL_ADMIN ):
self._account_types_ok.Disable()
self._add_to_expiry_ok.Disable()
self._set_expiry_ok.Disable()
self._add_to_expires_ok.Disable()
self._set_expires_ok.Disable()
@ -3618,20 +3619,20 @@ class DialogModifyAccounts( Dialog ):
self._account_types_panel.AddF( account_types_hbox, FLAGS_EXPAND_PERPENDICULAR )
add_to_expiry_box = wx.BoxSizer( wx.HORIZONTAL )
add_to_expires_box = wx.BoxSizer( wx.HORIZONTAL )
add_to_expiry_box.AddF( wx.StaticText( self._expiration_panel, label = 'add to expires: ' ), FLAGS_MIXED )
add_to_expiry_box.AddF( self._add_to_expiry, FLAGS_EXPAND_BOTH_WAYS )
add_to_expiry_box.AddF( self._add_to_expiry_ok, FLAGS_MIXED )
add_to_expires_box.AddF( wx.StaticText( self._expiration_panel, label = 'add to expires: ' ), FLAGS_MIXED )
add_to_expires_box.AddF( self._add_to_expires, FLAGS_EXPAND_BOTH_WAYS )
add_to_expires_box.AddF( self._add_to_expires_ok, FLAGS_MIXED )
set_expiry_box = wx.BoxSizer( wx.HORIZONTAL )
set_expires_box = wx.BoxSizer( wx.HORIZONTAL )
set_expiry_box.AddF( wx.StaticText( self._expiration_panel, label = 'set expires to: ' ), FLAGS_MIXED )
set_expiry_box.AddF( self._set_expiry, FLAGS_EXPAND_BOTH_WAYS )
set_expiry_box.AddF( self._set_expiry_ok, FLAGS_MIXED )
set_expires_box.AddF( wx.StaticText( self._expiration_panel, label = 'set expires to: ' ), FLAGS_MIXED )
set_expires_box.AddF( self._set_expires, FLAGS_EXPAND_BOTH_WAYS )
set_expires_box.AddF( self._set_expires_ok, FLAGS_MIXED )
self._expiration_panel.AddF( add_to_expiry_box, FLAGS_EXPAND_PERPENDICULAR )
self._expiration_panel.AddF( set_expiry_box, FLAGS_EXPAND_PERPENDICULAR )
self._expiration_panel.AddF( add_to_expires_box, FLAGS_EXPAND_PERPENDICULAR )
self._expiration_panel.AddF( set_expires_box, FLAGS_EXPAND_PERPENDICULAR )
self._ban_panel.AddF( self._ban, FLAGS_EXPAND_PERPENDICULAR )
self._ban_panel.AddF( self._superban, FLAGS_EXPAND_PERPENDICULAR )
@ -3687,7 +3688,7 @@ class DialogModifyAccounts( Dialog ):
if len( self._subject_identifiers ) > 1: wx.MessageBox( 'Done!' )
def EventAddToExpiry( self, event ): self._DoModification( HC.ADD_TO_EXPIRY, timespan = self._add_to_expiry.GetClientData( self._add_to_expiry.GetSelection() ) )
def EventAddToExpires( self, event ): self._DoModification( HC.ADD_TO_EXPIRES, timespan = self._add_to_expires.GetClientData( self._add_to_expires.GetSelection() ) )
def EventBan( self, event ):
@ -3699,13 +3700,13 @@ class DialogModifyAccounts( Dialog ):
def EventChangeAccountType( self, event ): self._DoModification( HC.CHANGE_ACCOUNT_TYPE, title = self._account_types.GetClientData( self._account_types.GetSelection() ).GetTitle() )
def EventSetExpiry( self, event ):
def EventSetExpires( self, event ):
expiry = self._set_expiry.GetClientData( self._set_expiry.GetSelection() )
expires = self._set_expires.GetClientData( self._set_expires.GetSelection() )
if expiry is not None: expiry += HC.GetNow()
if expires is not None: expires += HC.GetNow()
self._DoModification( HC.SET_EXPIRY, expiry = expiry )
self._DoModification( HC.SET_EXPIRES, expires = expires )
def EventSuperban( self, event ):
@ -4083,7 +4084,7 @@ class DialogPathsToTagsRegex( Dialog ):
account = service.GetInfo( 'account' )
if account.HasPermission( HC.POST_DATA ) or account.HasNoPermissions():
if account.HasPermission( HC.POST_DATA ) or account.IsUnknownAccount():
service_key = service.GetServiceKey()

28
include/ClientGUIDialogsManage.py
View File

@ -132,7 +132,7 @@ class DialogManage4chanPass( ClientGUIDialogs.Dialog ):
if self._timeout == 0: label = 'not authenticated'
elif self._timeout < HC.GetNow(): label = 'timed out'
else: label = 'authenticated - ' + HC.ConvertTimestampToPrettyExpiry( self._timeout )
else: label = 'authenticated - ' + HC.ConvertTimestampToPrettyExpires( self._timeout )
self._status.SetLabel( label )
@ -227,9 +227,11 @@ class DialogManageAccountTypes( ClientGUIDialogs.Dialog ):
permissions_string = ', '.join( [ HC.permissions_string_lookup[ permission ] for permission in permissions ] )
( max_num_bytes, max_num_requests ) = account_type.GetMaxMonthlyData()
max_num_bytes = account_type.GetMaxBytes()
max_num_requests = account_type.GetMaxRequests()
( max_num_bytes_string, max_num_requests_string ) = account_type.GetMaxMonthlyDataString()
max_num_bytes_string = account_type.GetMaxBytesString()
max_num_requests_string = account_type.GetMaxRequestsString()
self._ctrl_account_types.Append( ( title, permissions_string, max_num_bytes_string, max_num_requests_string ), ( title, len( permissions ), max_num_bytes, max_num_requests ) )
@ -290,9 +292,11 @@ class DialogManageAccountTypes( ClientGUIDialogs.Dialog ):
permissions_string = ', '.join( [ HC.permissions_string_lookup[ permission ] for permission in permissions ] )
( max_num_bytes, max_num_requests ) = account_type.GetMaxMonthlyData()
max_num_bytes = account_type.GetMaxBytes()
max_num_requests = account_type.GetMaxRequests()
( max_num_bytes_string, max_num_requests_string ) = account_type.GetMaxMonthlyDataString()
max_num_bytes_string = account_type.GetMaxBytesString()
max_num_requests_string = account_type.GetMaxRequestsString()
if title in self._titles_to_account_types: raise Exception( 'You already have an account type called ' + title + '; delete or edit that one first' )
@ -360,9 +364,11 @@ class DialogManageAccountTypes( ClientGUIDialogs.Dialog ):
permissions_string = ', '.join( [ HC.permissions_string_lookup[ permission ] for permission in permissions ] )
( max_num_bytes, max_num_requests ) = account_type.GetMaxMonthlyData()
max_num_bytes = account_type.GetMaxBytes()
max_num_requests = account_type.GetMaxRequests()
( max_num_bytes_string, max_num_requests_string ) = account_type.GetMaxMonthlyDataString()
max_num_bytes_string = account_type.GetMaxBytesString()
max_num_requests_string = account_type.GetMaxRequestsString()
if old_title != title:
@ -6282,7 +6288,7 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
account = service.GetInfo( 'account' )
if account.HasPermission( HC.POST_DATA ) or account.HasNoPermissions():
if account.HasPermission( HC.POST_DATA ) or account.IsUnknownAccount():
name = service.GetName()
service_key = service.GetServiceKey()
@ -6751,7 +6757,7 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
account = service.GetInfo( 'account' )
if account.HasPermission( HC.POST_DATA ) or account.HasNoPermissions():
if account.HasPermission( HC.POST_DATA ) or account.IsUnknownAccount():
name = service.GetName()
service_key = service.GetServiceKey()
@ -7422,7 +7428,7 @@ class DialogManageTags( ClientGUIDialogs.Dialog ):
if self._i_am_local_tag_service: self._modify_mappers.Hide()
else:
if not ( self._account.HasPermission( HC.POST_DATA ) or self._account.HasNoPermissions() ): self._add_tag_box.Hide()
if not ( self._account.HasPermission( HC.POST_DATA ) or self._account.IsUnknownAccount() ): self._add_tag_box.Hide()
if not self._account.HasPermission( HC.MANAGE_USERS ): self._modify_mappers.Hide()
@ -7626,7 +7632,7 @@ class DialogManageTags( ClientGUIDialogs.Dialog ):
def SetTagBoxFocus( self ):
if self._i_am_local_tag_service or self._account.HasPermission( HC.POST_DATA ) or self._account.HasNoPermissions(): self._add_tag_box.SetFocus()
if self._i_am_local_tag_service or self._account.HasPermission( HC.POST_DATA ) or self._account.IsUnknownAccount(): self._add_tag_box.SetFocus()

4
include/ClientGUIManagement.py
View File

@ -164,7 +164,7 @@ class CaptchaControl( wx.Panel ):
self._refresh_button.SetLabel( 'get new captcha' )
self._refresh_button.Enable()
self._captcha_time_left.SetLabel( HC.ConvertTimestampToPrettyExpiry( self._captcha_runs_out ) )
self._captcha_time_left.SetLabel( HC.ConvertTimestampToPrettyExpires( self._captcha_runs_out ) )
del dc
@ -2271,7 +2271,7 @@ class ManagementPanelMessages( wx.ScrolledWindow ):
self._current_predicates_box = ClientGUICommon.ListBoxMessagesPredicates( self._search_panel, self._page_key, [ 'system:inbox' ] )
self._synchronised = ClientGUICommon.OnOffButton( self._search_panel, self._page_key, 'notify_search_immediately', on_label = 'searching immediately', off_label = 'waiting' )
self._synchronised = ClientGUICommon.OnOffButton( self._search_panel, self._page_key, 'notify_search_immediately', on_label = 'searching immediately', off_label = 'waiting -- counts may be inaccurate' )
self._synchronised.SetToolTipString( 'select whether to renew the search as soon as a new predicate is entered' )
self._searchbox = ClientGUICommon.AutoCompleteDropdownMessageTerms( self._search_panel, self._page_key, self._identity )

6
include/ClientGUIMedia.py
View File

@ -228,7 +228,7 @@ class MediaPanel( ClientGUIMixins.ListeningMediaList, wx.ScrolledWindow ):
service_keys_to_content_updates = { file_service_key : ( content_update, ) }
HC.app.Write( 'content_updates', service_local_keys_to_content_updates )
HC.app.Write( 'content_updates', service_keys_to_content_updates )
@ -1705,8 +1705,8 @@ class MediaPanelThumbnails( MediaPanel ):
i_can_post_ratings = len( local_ratings_services ) > 0
downloadable_file_service_keys = { repository.GetServiceKey() for repository in file_repositories if repository.GetInfo( 'account' ).HasPermission( HC.GET_DATA ) or repository.GetInfo( 'account' ).HasNoPermissions() }
uploadable_file_service_keys = { repository.GetServiceKey() for repository in file_repositories if repository.GetInfo( 'account' ).HasPermission( HC.POST_DATA ) or repository.GetInfo( 'account' ).HasNoPermissions() }
downloadable_file_service_keys = { repository.GetServiceKey() for repository in file_repositories if repository.GetInfo( 'account' ).HasPermission( HC.GET_DATA ) or repository.GetInfo( 'account' ).IsUnknownAccount() }
uploadable_file_service_keys = { repository.GetServiceKey() for repository in file_repositories if repository.GetInfo( 'account' ).HasPermission( HC.POST_DATA ) or repository.GetInfo( 'account' ).IsUnknownAccount() }
petition_resolvable_file_service_keys = { repository.GetServiceKey() for repository in file_repositories if repository.GetInfo( 'account' ).HasPermission( HC.RESOLVE_PETITIONS ) }
petitionable_file_service_keys = { repository.GetServiceKey() for repository in file_repositories if repository.GetInfo( 'account' ).HasPermission( HC.POST_PETITIONS ) } - petition_resolvable_file_service_keys
user_manageable_file_service_keys = { repository.GetServiceKey() for repository in file_repositories if repository.GetInfo( 'account' ).HasPermission( HC.MANAGE_USERS ) }

224
include/HydrusConstants.py
View File

@ -63,8 +63,8 @@ options = {}
# Misc
NETWORK_VERSION = 14
SOFTWARE_VERSION = 131
NETWORK_VERSION = 15
SOFTWARE_VERSION = 132
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@ -141,6 +141,7 @@ RESOLVE_PETITIONS = 3
MANAGE_USERS = 4
GENERAL_ADMIN = 5
EDIT_SERVICES = 6
UNKNOWN_PERMISSION = 7
CREATABLE_PERMISSIONS = [ GET_DATA, POST_DATA, POST_PETITIONS, RESOLVE_PETITIONS, MANAGE_USERS, GENERAL_ADMIN ]
ADMIN_PERMISSIONS = [ RESOLVE_PETITIONS, MANAGE_USERS, GENERAL_ADMIN, EDIT_SERVICES ]
@ -154,6 +155,7 @@ permissions_string_lookup[ RESOLVE_PETITIONS ] = 'resolve petitions'
permissions_string_lookup[ MANAGE_USERS ] = 'manage users'
permissions_string_lookup[ GENERAL_ADMIN ] = 'general administration'
permissions_string_lookup[ EDIT_SERVICES ] = 'edit services'
permissions_string_lookup[ UNKNOWN_PERMISSION ] = 'unknown'
TAG_REPOSITORY = 0
FILE_REPOSITORY = 1
@ -202,8 +204,8 @@ DELETE_TAG_PETITION = 1
BAN = 0
SUPERBAN = 1
CHANGE_ACCOUNT_TYPE = 2
ADD_TO_EXPIRY = 3
SET_EXPIRY = 4
ADD_TO_EXPIRES = 3
SET_EXPIRES = 4
CURRENT = 0
PENDING = 1
@ -909,46 +911,46 @@ def ConvertTimestampToPrettyAgo( timestamp ):
elif hours > 0: return ' '.join( ( h, m ) ) + ' ago'
else: return ' '.join( ( m, s ) ) + ' ago'
def ConvertTimestampToPrettyExpiry( timestamp ):
def ConvertTimestampToPrettyExpires( timestamp ):
if timestamp is None: return 'does not expire'
if timestamp == 0: return 'unknown expiration'
expiry = GetNow() - timestamp
expires = GetNow() - timestamp
if expiry >= 0: already_happend = True
if expires >= 0: already_happend = True
else:
expiry *= -1
expires *= -1
already_happend = False
seconds = expiry % 60
seconds = expires % 60
if seconds == 1: s = '1 second'
else: s = u( seconds ) + ' seconds'
expiry = expiry / 60
minutes = expiry % 60
expires = expires / 60
minutes = expires % 60
if minutes == 1: m = '1 minute'
else: m = u( minutes ) + ' minutes'
expiry = expiry / 60
hours = expiry % 24
expires = expires / 60
hours = expires % 24
if hours == 1: h = '1 hour'
else: h = u( hours ) + ' hours'
expiry = expiry / 24
days = expiry % 30
expires = expires / 24
days = expires % 30
if days == 1: d = '1 day'
else: d = u( days ) + ' days'
expiry = expiry / 30
months = expiry % 12
expires = expires / 30
months = expires % 12
if months == 1: mo = '1 month'
else: mo = u( months ) + ' months'
years = expiry / 12
years = expires / 12
if years == 1: y = '1 year'
else: y = u( years ) + ' years'
@ -1285,18 +1287,21 @@ class Account( HydrusYAMLBase ):
yaml_tag = u'!Account'
def __init__( self, account_id, account_type, created, expiry, used_data, banned_info = None ):
def __init__( self, account_key, account_type, created, expires, used_bytes, used_requests, banned_info = None ):
HydrusYAMLBase.__init__( self )
self._account_id = account_id
self._account_type = account_type
self._created = created
self._expiry = expiry
self._used_data = used_data
self._banned_info = banned_info
self._info = {}
self._object_instantiation_timestamp = GetNow()
self._info[ 'account_key' ] = account_key
self._info[ 'account_type' ] = account_type
self._info[ 'created' ] = created
self._info[ 'expires' ] = expires
self._info[ 'used_bytes' ] = used_bytes
self._info[ 'used_requests' ] = used_requests
if banned_info is not None: self._info[ 'banned_info' ] = banned_info
self._info[ 'fresh_timestamp' ] = GetNow()
def __repr__( self ): return self.ConvertToString()
@ -1305,20 +1310,42 @@ class Account( HydrusYAMLBase ):
def _IsBanned( self ):
if self._banned_info is None: return False
if 'banned_info' not in self._info: return False
else:
( reason, created, expiry ) = self._banned_info
( reason, created, expires ) = self._info[ 'banned_info' ]
if expiry is None: return True
else: return GetNow() > expiry
if expires is None: return True
else: return GetNow() > expires
def _IsBytesExceeded( self ):
account_type = self._info[ 'account_type' ]
max_num_bytes = account_type.GetMaxBytes()
used_bytes = self._info[ 'used_bytes' ]
return max_num_bytes is not None and used_bytes > max_num_bytes
def _IsExpired( self ):
if self._expiry is None: return False
else: return GetNow() > self._expiry
if self._info[ 'expires' ] is None: return False
else: return GetNow() > self._info[ 'expires' ]
def _IsRequestsExceeded( self ):
account_type = self._info[ 'account_type' ]
max_num_requests = account_type.GetMaxRequests()
used_requests = self._info[ 'used_requests' ]
return max_num_requests is not None and used_requests > max_num_requests
def CheckPermission( self, permission ):
@ -1327,46 +1354,39 @@ class Account( HydrusYAMLBase ):
if self._IsExpired(): raise HydrusExceptions.PermissionException( 'This account is expired.' )
( max_num_bytes, max_num_requests ) = self._account_type.GetMaxMonthlyData()
if self._IsBytesExceeded(): raise HydrusExceptions.PermissionException( 'You have hit your data transfer limit, and cannot make any more requests for the month.' )
( used_bytes, used_requests ) = self._used_data
if self._IsRequestsExceeded(): raise HydrusExceptions.PermissionException( 'You have hit your requests limit, and cannot make any more requests for the month.' )
if max_num_bytes is not None and used_bytes > max_num_bytes: raise HydrusExceptions.PermissionException( 'You have hit your data transfer limit (' + ConvertIntToBytes( max_num_bytes ) + '), and cannot make any more requests for the month.' )
if max_num_requests is not None and used_requests > max_num_requests: raise HydrusExceptions.PermissionException( 'You have hit your requests limit (' + ConvertIntToPrettyString( max_num_requests ) + '), and cannot make any more requests for the month.' )
if not self._account_type.HasPermission( permission ): raise HydrusExceptions.PermissionException( 'You do not have permission to do that.' )
if not self._info[ 'account_type' ].HasPermission( permission ): raise HydrusExceptions.PermissionException( 'You do not have permission to do that.' )
def ConvertToString( self ): return ConvertTimestampToPrettyAge( self._created ) + os.linesep + self._account_type.ConvertToString( self._used_data ) + os.linesep + 'which '+ ConvertTimestampToPrettyExpiry( self._expiry )
def ConvertToString( self ): return ConvertTimestampToPrettyAge( self._info[ 'created' ] ) + os.linesep + self._info[ 'account_type' ].ConvertToString() + os.linesep + 'which '+ ConvertTimestampToPrettyExpires( self._info[ 'expires' ] )
def GetAccountIdentifier( self ): return AccountIdentifier( account_id = self._account_id )
def GetAccountKey( self ): return self._info[ 'account_key' ]
def GetAccountType( self ): return self._account_type
def GetAccountType( self ): return self._info[ 'account_type' ]
def GetBannedInfo( self ): return self._banned_info
def GetCreated( self ): return self._info[ 'created' ]
def GetCreated( self ): return self._created
def GetExpires( self ): return self._info[ 'expires' ]
def GetExpiry( self ): return self._expiry
def GetExpiryString( self ):
def GetExpiresString( self ):
if self._IsBanned():
( reason, created, expiry ) = self._banned_info
( reason, created, expires ) = self._info[ 'banned_info' ]
return 'banned ' + ConvertTimestampToPrettyAge( created ) + ', ' + ConvertTimestampToPrettyExpiry( expiry ) + ' because: ' + reason
return 'banned ' + ConvertTimestampToPrettyAge( created ) + ', ' + ConvertTimestampToPrettyExpires( expires ) + ' because: ' + reason
else: return ConvertTimestampToPrettyAge( self._created ) + ' and ' + ConvertTimestampToPrettyExpiry( self._expiry )
else: return ConvertTimestampToPrettyAge( self._info[ 'created' ] ) + ' and ' + ConvertTimestampToPrettyExpires( self._info[ 'expires' ] )
def GetAccountId( self ): return self._account_id
def GetUsedBytesString( self ):
( max_num_bytes, max_num_requests ) = self._account_type.GetMaxMonthlyData()
( used_bytes, used_requests ) = self._used_data
max_num_bytes = self._info[ 'account_type' ].GetMaxBytes()
used_bytes = self._info[ 'used_bytes' ]
if max_num_bytes is None: return ConvertIntToBytes( used_bytes ) + ' used this month'
else: return ConvertIntToBytes( used_bytes ) + '/' + ConvertIntToBytes( max_num_bytes ) + ' used this month'
@ -1374,16 +1394,24 @@ class Account( HydrusYAMLBase ):
def GetUsedRequestsString( self ):
( max_num_bytes, max_num_requests ) = self._account_type.GetMaxMonthlyData()
( used_bytes, used_requests ) = self._used_data
max_num_requests = self._info[ 'account_type' ].GetMaxRequests()
used_requests = self._info[ 'used_requests' ]
if max_num_requests is None: return ConvertIntToPrettyString( used_requests ) + ' requests used this month'
else: return ConvertIntToPrettyString( used_requests ) + '/' + ConvertIntToPrettyString( max_num_requests ) + ' requests used this month'
def GetUsedData( self ): return self._used_data
def GetUsedBytes( self ): return self._info[ 'used_bytes' ]
def HasNoPermissions( self ): return self._account_type.HasNoPermissions()
def GetUsedRequests( self ): return self._info[ 'used_bytes' ]
def HasAccountKey( self ):
if 'account_key' in self._info and self._info[ 'account_key' ] is not None: return True
return False
def HasPermission( self, permission ):
@ -1391,40 +1419,34 @@ class Account( HydrusYAMLBase ):
if self._IsExpired(): return False
( max_num_bytes, max_num_requests ) = self._account_type.GetMaxMonthlyData()
if self._IsBytesExceeded(): return False
( used_bytes, used_requests ) = self._used_data
if self._IsRequestsExceeded(): return False
if max_num_bytes is not None and used_bytes >= max_num_bytes: return False
if max_num_requests is not None and used_requests >= max_num_requests: return False
return self._account_type.HasPermission( permission )
return self._info[ 'account_type' ].HasPermission( permission )
def IsAdmin( self ): return True in [ self.HasPermissions( permission ) for permission in ADMIN_PERMISSIONS ]
def IsBanned( self ): return self._IsBanned()
def IsStale( self ): return self._object_instantiation_timestamp + UPDATE_DURATION * 5 < GetNow()
def IsStale( self ): return self._info[ 'fresh_timestamp' ] + UPDATE_DURATION * 5 < GetNow()
def MakeFresh( self ): self._object_instantiation_timestamp = GetNow()
def IsUnknownAccount( self ): return self._info[ 'account_type' ].IsUnknownAccountType()
def MakeStale( self ): self._object_instantiation_timestamp = 0
def MakeFresh( self ): self._info[ 'fresh_timestamp' ] = GetNow()
def MakeStale( self ): self._info[ 'fresh_timestamp' ] = 0
def RequestMade( self, num_bytes ):
( used_bytes, used_requests ) = self._used_data
used_bytes += num_bytes
used_requests += 1
self._used_data = ( used_bytes, used_requests )
self._info[ 'used_bytes' ] += num_bytes
self._info[ 'used_requests' ] += 1
class AccountIdentifier( HydrusYAMLBase ):
TYPE_ACCOUNT_ID = 0
TYPE_ACCESS_KEY = 1
TYPE_ACCOUNT_KEY = 1
TYPE_HASH = 2
TYPE_MAPPING = 3
TYPE_SIBLING = 4
@ -1432,19 +1454,14 @@ class AccountIdentifier( HydrusYAMLBase ):
yaml_tag = u'!AccountIdentifier'
def __init__( self, access_key = None, hash = None, tag = None, account_id = None ):
def __init__( self, account_key = None, hash = None, tag = None ):
HydrusYAMLBase.__init__( self )
if account_id is not None:
if account_key is not None:
self._type = self.TYPE_ACCOUNT_ID
self._data = account_id
elif access_key is not None:
self._type = self.TYPE_ACCESS_KEY
self._data = access_key
self._type = self.TYPE_ACCOUNT_KEY
self._data = account_key
elif hash is not None:
@ -1471,9 +1488,7 @@ class AccountIdentifier( HydrusYAMLBase ):
def GetData( self ): return self._data
def HasAccessKey( self ): return self._type == self.TYPE_ACCESS_KEY
def HasAccountId( self ): return self._type == self.TYPE_ACCOUNT_ID
def HasAccountKey( self ): return self._type == self.TYPE_ACCOUNT_KEY
def HasHash( self ): return self._type == self.TYPE_HASH
@ -1498,38 +1513,57 @@ class AccountType( HydrusYAMLBase ):
def GetTitle( self ): return self._title
def GetMaxMonthlyData( self ): return self._max_monthly_data
def GetMaxBytes( self ):
( max_num_bytes, max_num_requests ) = self._max_monthly_data
return max_num_bytes
def GetMaxMonthlyDataString( self ):
def GetMaxRequests( self ):
( max_num_bytes, max_num_requests ) = self._max_monthly_data
return max_num_requests
def GetMaxBytesString( self ):
( max_num_bytes, max_num_requests ) = self._max_monthly_data
if max_num_bytes is None: max_num_bytes_string = 'No limit'
else: max_num_bytes_string = ConvertIntToBytes( max_num_bytes )
return max_num_bytes_string
def GetMaxRequestsString( self ):
( max_num_bytes, max_num_requests ) = self._max_monthly_data
if max_num_requests is None: max_num_requests_string = 'No limit'
else: max_num_requests_string = ConvertIntToPrettyString( max_num_requests )
return ( max_num_bytes_string, max_num_requests_string )
return max_num_requests_string
def ConvertToString( self, data_usage = None ):
def ConvertToString( self ):
result_string = self._title + ' with '
if len( self._permissions ) == 0: result_string += 'no permissions'
if self._permissions == [ UNKNOWN_PERMISSION ]: result_string += 'no permissions'
else: result_string += ', '.join( [ permissions_string_lookup[ permission ] for permission in self._permissions ] ) + ' permissions'
return result_string
def HasNoPermissions( self ): return len( self._permissions ) == 0
def IsUnknownAccountType( self ): return self._permissions == [ UNKNOWN_PERMISSION ]
def HasPermission( self, permission ): return permission in self._permissions
UNKNOWN_ACCOUNT_TYPE = AccountType( 'unknown account', [], ( None, None ) )
UNKNOWN_ACCOUNT_TYPE = AccountType( 'unknown account', [ UNKNOWN_PERMISSION ], ( None, None ) )
def GetUnknownAccount(): return Account( 0, UNKNOWN_ACCOUNT_TYPE, 0, None, ( 0, 0 ) )
def GetUnknownAccount( account_key = None ): return Account( account_key, UNKNOWN_ACCOUNT_TYPE, 0, None, 0, 0 )
class ClientServiceIdentifier( HydrusYAMLBase ):

3
include/HydrusNetworking.py
View File

@ -64,8 +64,7 @@ def ConvertHydrusGETArgsToQuery( request_args ):
data = subject_identifier.GetData()
if subject_identifier.HasAccessKey(): request_args[ 'subject_access_key' ] = data.encode( 'hex' )
elif subject_identifier.HasAccountId(): request_args[ 'subject_account_id' ] = data
if subject_identifier.HasAccountKey(): request_args[ 'subject_account_key' ] = data.encode( 'hex' )
elif subject_identifier.HasHash(): request_args[ 'subject_hash' ] = data.encode( 'hex' )
if subject_identifier.HasMapping():

43
include/HydrusServerResources.py
View File

@ -152,20 +152,19 @@ class HydrusResourceCommand( Resource ):
value = values[0]
if name in ( 'begin', 'expiry', 'lifetime', 'num', 'subject_account_id', 'service_type', 'service_port', 'since', 'timespan' ):
if name in ( 'begin', 'expires', 'lifetime', 'num', 'service_type', 'service_port', 'since', 'timespan' ):
try: hydrus_args[ name ] = int( value )
except: raise HydrusExceptions.ForbiddenException( 'I was expecting to parse \'' + name + '\' as an integer, but it failed.' )
elif name in ( 'access_key', 'title', 'subject_access_key', 'contact_key', 'hash', 'subject_hash', 'subject_tag', 'message_key', 'share_key' ):
elif name in ( 'access_key', 'title', 'subject_account_key', 'contact_key', 'hash', 'subject_hash', 'subject_tag', 'message_key', 'share_key' ):
try: hydrus_args[ name ] = value.decode( 'hex' )
except: raise HydrusExceptions.ForbiddenException( 'I was expecting to parse \'' + name + '\' as a hex-encoded string, but it failed.' )
if 'subject_account_id' in hydrus_args: hydrus_args[ 'subject_identifier' ] = HC.AccountIdentifier( account_id = hydrus_args[ 'subject_account_id' ] )
elif 'subject_access_key' in hydrus_args: hydrus_args[ 'subject_identifier' ] = HC.AccountIdentifier( access_key = hydrus_args[ 'subject_access_key' ] )
if 'subject_account_key' in hydrus_args: hydrus_args[ 'subject_identifier' ] = HC.AccountIdentifier( account_key = hydrus_args[ 'subject_account_key' ] )
elif 'subject_hash' in hydrus_args:
if 'subject_tag' in hydrus_args: hydrus_args[ 'subject_identifier' ] = HC.AccountIdentifier( tag = hydrus_args[ 'subject_tag' ], hash = hydrus_args[ 'subject_hash' ] )
@ -596,7 +595,7 @@ class HydrusResourceCommandBooruGallery( HydrusResourceCommandBooru ):
<body>'''
body += '''
<div class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpiry( timeout ) + '''.</div>'''
<div class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpires( timeout ) + '''.</div>'''
if name != '': body += '''
<h3>''' + name + '''</h3>'''
@ -693,7 +692,7 @@ class HydrusResourceCommandBooruPage( HydrusResourceCommandBooru ):
<body>'''
body += '''
<div class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpiry( timeout ) + '''.</div>'''
<div class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpires( timeout ) + '''.</div>'''
if name != '': body += '''
<h3>''' + name + '''</h3>'''
@ -827,11 +826,11 @@ class HydrusResourceCommandSessionKey( HydrusResourceCommand ):
session_manager = HC.app.GetManager( 'restricted_services_sessions' )
( session_key, expiry ) = session_manager.AddSession( self._service_key, access_key )
( session_key, expires ) = session_manager.AddSession( self._service_key, access_key )
now = HC.GetNow()
max_age = now - expiry
max_age = now - expires
cookies = [ ( 'session_key', session_key.encode( 'hex' ), { 'max_age' : max_age, 'path' : '/' } ) ]
@ -915,7 +914,7 @@ class HydrusResourceCommandRestricted( HydrusResourceCommand ):
account.RequestMade( num_bytes )
HC.pubsub.pub( 'request_made', ( self._service_key, account, num_bytes ) )
HC.pubsub.pub( 'request_made', ( account.GetAccountKey(), num_bytes ) )
@ -940,17 +939,21 @@ class HydrusResourceCommandRestrictedAccount( HydrusResourceCommandRestricted ):
admin_account = request.hydrus_account
admin_account_key = admin_account.GetAccountKey()
action = request.hydrus_args[ 'action' ]
subject_identifiers = request.hydrus_args[ 'subject_identifiers' ]
kwargs = request.hydrus_args # for things like expiry, title, and so on
subject_account_keys = { HC.app.Read( 'account_key_from_identifier', self._service_key, subject_identifier ) for subject_identifier in subject_identifiers }
HC.app.Write( 'account', self._service_key, admin_account, action, subject_identifiers, kwargs )
kwargs = request.hydrus_args # for things like expires, title, and so on
HC.app.Write( 'account', self._service_key, admin_account_key, action, subject_account_keys, kwargs )
session_manager = HC.app.GetManager( 'restricted_services_sessions' )
session_manager.RefreshAccounts( self._service_key, subject_identifiers )
session_manager.RefreshAccounts( self._service_key, subject_account_keys )
response_context = HC.ResponseContext( 200 )
@ -965,7 +968,9 @@ class HydrusResourceCommandRestrictedAccountInfo( HydrusResourceCommandRestricte
subject_identifier = request.hydrus_args[ 'subject_identifier' ]
account_info = HC.app.Read( 'account_info', self._service_key, subject_identifier )
subject_account_key = HC.app.Read( 'account_key_from_identifier', self._service_key, subject_identifier )
account_info = HC.app.Read( 'account_info', self._service_key, subject_account_key )
body = yaml.safe_dump( { 'account_info' : account_info } )
@ -1123,11 +1128,13 @@ class HydrusResourceCommandRestrictedRepositoryFile( HydrusResourceCommandRestri
account = request.hydrus_account
account_key = account.GetAccountKey()
file_dict = request.hydrus_args
file_dict[ 'ip' ] = request.getClientIP()
HC.app.Write( 'file', self._service_key, account, file_dict )
HC.app.Write( 'file', self._service_key, account_key, file_dict )
response_context = HC.ResponseContext( 200 )
@ -1161,9 +1168,11 @@ class HydrusResourceCommandRestrictedServices( HydrusResourceCommandRestricted )
account = request.hydrus_account
account_key = account.GetAccountKey()
edit_log = request.hydrus_args[ 'edit_log' ]
service_keys_to_access_keys = HC.app.Write( 'services', account, edit_log )
service_keys_to_access_keys = HC.app.Write( 'services', account_key, edit_log )
body = yaml.safe_dump( { 'service_keys_to_access_keys' : service_keys_to_access_keys } )
@ -1225,9 +1234,11 @@ class HydrusResourceCommandRestrictedUpdate( HydrusResourceCommandRestricted ):
account = request.hydrus_account
account_key = account.GetAccountKey()
update = request.hydrus_args[ 'update' ]
HC.app.Write( 'update', self._service_key, account, update )
HC.app.Write( 'update', self._service_key, account_key, update )
response_context = HC.ResponseContext( 200 )

107
include/HydrusSessions.py
View File

@ -28,7 +28,7 @@ class HydrusMessagingSessionManagerServer( object ):
for ( service_key, session_tuples ) in existing_sessions:
self._service_keys_to_sessions[ service_key ] = { session_key : ( account, identifier, name, expiry ) for ( session_Key, account, identifier, name, expiry ) in session_tuples }
self._service_keys_to_sessions[ service_key ] = { session_key : ( account, name, expires ) for ( session_Key, account, name, expires ) in session_tuples }
self._lock = threading.Lock()
@ -41,18 +41,18 @@ class HydrusMessagingSessionManagerServer( object ):
if session_key not in self._service_keys_to_sessions[ service_key ]: raise HydrusExceptions.SessionException( 'Did not find that session!' )
else:
( account, identity, name, expiry ) = self._service_keys_to_sessions[ service_key ][ session_key ]
( account, name, expires ) = self._service_keys_to_sessions[ service_key ][ session_key ]
now = HC.GetNow()
if now > expiry:
if now > expires:
del self._service_keys_to_sessions[ service_key ][ session_key ]
raise HydrusExceptions.SessionException( 'Session expired!' )
return ( identity, name )
return ( account.GetAccountKey(), name )
@ -61,24 +61,17 @@ class HydrusMessagingSessionManagerServer( object ):
session_key = os.urandom( 32 )
account_identifier = HC.AccountIdentifier( access_key = access_key )
account_key = HC.app.Read( 'account_key', service_key, account_identifier )
account_key = HC.app.Read( 'account_key_from_access_key', service_key, access_key )
account = HC.app.Read( 'account', service_key, account_key )
identity = hashlib.sha256( access_key ).digest()
now = HC.GetNow()
expiry = now + HYDRUS_SESSION_LIFETIME
expires = now + HYDRUS_SESSION_LIFETIME
with self._lock:
self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, identity, name, expiry )
with self._lock: self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, name, expires )
HC.app.Write( 'messaging_session', service_key, session_key, account_key, identity, name, expiry )
HC.app.Write( 'messaging_session', service_key, session_key, account_key, name, expires )
return session_key
@ -89,7 +82,7 @@ class HydrusSessionManagerClient( object ):
existing_sessions = HC.app.Read( 'hydrus_sessions' )
self._service_keys_to_sessions = { service_key : ( session_key, expiry ) for ( service_key, session_key, expiry ) in existing_sessions }
self._service_keys_to_sessions = { service_key : ( session_key, expires ) for ( service_key, session_key, expires ) in existing_sessions }
self._lock = threading.Lock()
@ -112,9 +105,9 @@ class HydrusSessionManagerClient( object ):
if service_key in self._service_keys_to_sessions:
( session_key, expiry ) = self._service_keys_to_sessions[ service_key ]
( session_key, expires ) = self._service_keys_to_sessions[ service_key ]
if now + 600 > expiry: del self._service_keys_to_sessions[ service_key ]
if now + 600 > expires: del self._service_keys_to_sessions[ service_key ]
else: return session_key
@ -127,11 +120,11 @@ class HydrusSessionManagerClient( object ):
try: session_key = cookies[ 'session_key' ].decode( 'hex' )
except: raise Exception( 'Service did not return a session key!' )
expiry = now + HYDRUS_SESSION_LIFETIME
expires = now + HYDRUS_SESSION_LIFETIME
self._service_keys_to_sessions[ service_key ] = ( session_key, expiry )
self._service_keys_to_sessions[ service_key ] = ( session_key, expires )
HC.app.Write( 'hydrus_session', service_key, session_key, expiry )
HC.app.Write( 'hydrus_session', service_key, session_key, expires )
return session_key
@ -152,33 +145,31 @@ class HydrusSessionManagerServer( object ):
with self._lock:
account_identifier = HC.AccountIdentifier( access_key = access_key )
account_key = HC.app.Read( 'account_key_from_access_key', service_key, access_key )
account_key = HC.app.Read( 'account_key', service_key, account_identifier )
if account_key not in self._account_keys_to_accounts[ service_key ]:
if account_key not in self._account_keys_to_accounts:
account = HC.app.Read( 'account', service_key, account_key )
account = HC.app.Read( 'account', account_key )
self._account_keys_to_accounts[ service_key ][ account_key ] = account
self._account_keys_to_accounts[ account_key ] = account
account = self._account_keys_to_accounts[ service_key ][ account_key ]
account = self._account_keys_to_accounts[ account_key ]
session_key = os.urandom( 32 )
self._account_keys_to_session_keys[ service_key ][ account_key ].add( session_key )
self._account_keys_to_session_keys[ account_key ].add( session_key )
now = HC.GetNow()
expiry = now + HYDRUS_SESSION_LIFETIME
expires = now + HYDRUS_SESSION_LIFETIME
HC.app.Write( 'session', session_key, service_key, account_key, expiry )
HC.app.Write( 'session', session_key, service_key, account_key, expires )
self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, expiry )
self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, expires )
return ( session_key, expiry )
return ( session_key, expires )
def GetAccount( self, service_key, session_key ):
@ -189,11 +180,11 @@ class HydrusSessionManagerServer( object ):
if session_key in service_sessions:
( account, expiry ) = service_sessions[ session_key ]
( account, expires ) = service_sessions[ session_key ]
now = HC.GetNow()
if now > expiry: del service_sessions[ session_key ]
if now > expires: del service_sessions[ session_key ]
else: return account
@ -201,27 +192,25 @@ class HydrusSessionManagerServer( object ):
def RefreshAccounts( self, service_key, account_identifiers ):
def RefreshAccounts( self, service_key, account_keys ):
with self._lock:
for account_identifier in account_identifiers:
for account_key in account_keys:
account_key = HC.app.Read( 'account_key', service_key, account_identifier )
account = HC.app.Read( 'account', account_key )
account = HC.app.Read( 'account', service_key, account_key )
self._account_keys_to_accounts[ account_key ] = account
self._account_keys_to_accounts[ service_key ][ account_key ] = account
if account_key in self._account_keys_to_session_keys[ service_key ]:
if account_key in self._account_keys_to_session_keys:
session_keys = self._account_keys_to_session_keys[ service_key ][ account_key ]
session_keys = self._account_keys_to_session_keys[ account_key ]
for session_key in session_keys:
( old_account, expiry ) = self._service_keys_to_sessions[ service_key ][ session_key ]
( old_account, expires ) = self._service_keys_to_sessions[ service_key ][ session_key ]
self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, expiry )
self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, expires )
@ -234,21 +223,23 @@ class HydrusSessionManagerServer( object ):
self._service_keys_to_sessions = collections.defaultdict( dict )
self._account_keys_to_session_keys = collections.defaultdict( HC.default_dict_set )
self._account_keys_to_session_keys = HC.default_dict_set()
self._account_keys_to_accounts = collections.defaultdict( dict )
self._account_keys_to_accounts = {}
#
existing_sessions = HC.app.Read( 'sessions' )
for ( session_key, service_key, account_key, account, expiry ) in existing_sessions:
for ( session_key, service_key, account, expires ) in existing_sessions:
self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, expiry )
account_key = account.GetAccountKey()
self._account_keys_to_session_keys[ service_key ][ account_key ].add( session_key )
self._service_keys_to_sessions[ service_key ][ session_key ] = ( account, expires )
self._account_keys_to_accounts[ service_key ][ account_key ] = account
self._account_keys_to_session_keys[ account_key ].add( session_key )
self._account_keys_to_accounts[ account_key ] = account
@ -259,7 +250,7 @@ class WebSessionManagerClient( object ):
existing_sessions = HC.app.Read( 'web_sessions' )
self._names_to_sessions = { name : ( cookies, expiry ) for ( name, cookies, expiry ) in existing_sessions }
self._names_to_sessions = { name : ( cookies, expires ) for ( name, cookies, expires ) in existing_sessions }
self._lock = threading.Lock()
@ -272,9 +263,9 @@ class WebSessionManagerClient( object ):
if name in self._names_to_sessions:
( cookies, expiry ) = self._names_to_sessions[ name ]
( cookies, expires ) = self._names_to_sessions[ name ]
if now + 300 > expiry: del self._names_to_sessions[ name ]
if now + 300 > expires: del self._names_to_sessions[ name ]
else: return cookies
@ -284,7 +275,7 @@ class WebSessionManagerClient( object ):
( response_gumpf, cookies ) = HC.http.Request( HC.GET, 'http://www.hentai-foundry.com/?enterAgree=1', return_cookies = True )
expiry = now + 60 * 60
expires = now + 60 * 60
elif name == 'pixiv':
@ -312,12 +303,12 @@ class WebSessionManagerClient( object ):
# _ only given to logged in php sessions
if 'PHPSESSID' not in cookies or '_' not in cookies[ 'PHPSESSID' ]: raise Exception( 'Pixiv login credentials not accepted!' )
expiry = now + 30 * 86400
expires = now + 30 * 86400
self._names_to_sessions[ name ] = ( cookies, expiry )
self._names_to_sessions[ name ] = ( cookies, expires )
HC.app.Write( 'web_session', name, cookies, expiry )
HC.app.Write( 'web_session', name, cookies, expires )
return cookies

438
include/ServerDB.py
View File

@ -23,11 +23,11 @@ import wx
class FileDB( object ):
def _AddFile( self, c, service_key, account, file_dict ):
def _AddFile( self, c, service_key, account_key, file_dict ):
service_id = self._GetServiceId( c, service_key )
account_id = account.GetAccountId()
account_id = self._GetAccountId( c, account_key )
hash = file_dict[ 'hash' ]
@ -226,7 +226,9 @@ class FileDB( object ):
( account_id, reason_id ) = result
petitioner_account_identifier = HC.AccountIdentifier( account_id = account_id )
account_key = self._GetAccountKeyFromAccountId( c, account_id )
petitioner_account_identifier = HC.AccountIdentifier( account_key = account_key )
reason = self._GetReason( c, reason_id )
@ -778,7 +780,9 @@ class TagDB( object ):
if status == HC.PENDING: action = HC.CONTENT_UPDATE_PENDING
elif status == HC.PETITIONED: action = HC.CONTENT_UPDATE_PETITION
petitioner_account_identifier = HC.AccountIdentifier( account_id = account_id )
account_key = self._GetAccountKeyFromAccountId( c, account_id )
petitioner_account_identifier = HC.AccountIdentifier( account_key = account_key )
reason = self._GetReason( c, reason_id )
@ -892,7 +896,7 @@ class RatingDB( object ):
class ServiceDB( FileDB, MessageDB, TagDB ):
def _AccountTypeExists( self, c, service_id, title ): return c.execute( 'SELECT 1 FROM account_types, account_type_map USING ( account_type_id ) WHERE service_id = ? AND title = ?;', ( service_id, title ) ).fetchone() is not None
def _AccountTypeExists( self, c, service_id, title ): return c.execute( 'SELECT 1 FROM account_types WHERE service_id = ? AND title = ?;', ( service_id, title ) ).fetchone() is not None
def _AddNews( self, c, service_key, news ):
@ -903,37 +907,37 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
c.execute( 'INSERT INTO news ( service_id, news, timestamp ) VALUES ( ?, ?, ? );', ( service_id, news, now ) )
def _AddMessagingSession( self, c, service_key, session_key, account_key, identifier, name, expiry ):
def _AddMessagingSession( self, c, service_key, session_key, account_key, name, expires ):
service_id = self._GetServiceId( c, service_key )
account_id = self._GetAccountId( c, account_key )
c.execute( 'INSERT INTO sessions ( service_id, session_key, account_id, identifier, name, expiry ) VALUES ( ?, ?, ?, ?, ?, ? );', ( service_id, sqlite3.Binary( session_key ), account_id, sqlite3.Binary( identifier ), name, expiry ) )
c.execute( 'INSERT INTO sessions ( service_id, session_key, account_id, identifier, name, expiry ) VALUES ( ?, ?, ?, ?, ?, ? );', ( service_id, sqlite3.Binary( session_key ), account_id, sqlite3.Binary( account_key ), name, expires ) )
def _AddSession( self, c, session_key, service_key, account_key, expiry ):
def _AddSession( self, c, session_key, service_key, account_key, expires ):
service_id = self._GetServiceId( c, service_key )
account_id = self._GetAccountId( c, account_key )
c.execute( 'INSERT INTO sessions ( session_key, service_id, account_id, expiry ) VALUES ( ?, ?, ?, ? );', ( sqlite3.Binary( session_key ), service_id, account_id, expiry ) )
c.execute( 'INSERT INTO sessions ( session_key, service_id, account_id, expiry ) VALUES ( ?, ?, ?, ? );', ( sqlite3.Binary( session_key ), service_id, account_id, expires ) )
def _AddToExpiry( self, c, service_id, account_ids, timespan ): c.execute( 'UPDATE account_map SET expires = expires + ? WHERE service_id = ? AND account_id IN ' + HC.SplayListForDB( account_ids ) + ';', ( timespan, service_id ) )
def _AddToExpires( self, c, account_ids, timespan ): c.execute( 'UPDATE accounts SET expires = expires + ? WHERE account_id IN ' + HC.SplayListForDB( account_ids ) + ';', ( timespan, ) )
def _Ban( self, c, service_id, action, admin_account_id, subject_account_ids, reason_id, expiry = None, lifetime = None ):
def _Ban( self, c, service_id, action, admin_account_id, subject_account_ids, reason_id, expires = None, lifetime = None ):
splayed_subject_account_ids = HC.SplayListForDB( subject_account_ids )
now = HC.GetNow()
if expiry is not None: pass
elif lifetime is not None: expiry = now + lifetime
else: expiry = None
if expires is not None: pass
elif lifetime is not None: expires = now + lifetime
else: expires = None
c.executemany( 'INSERT OR IGNORE INTO bans ( service_id, account_id, admin_account_id, reason_id, created, expires ) VALUES ( ?, ?, ?, ?, ? );', [ ( service_id, subject_account_id, admin_account_id, reason_id, now, expiry ) for subject_account_id in subject_account_ids ] )
c.executemany( 'INSERT OR IGNORE INTO bans ( service_id, account_id, admin_account_id, reason_id, created, expires ) VALUES ( ?, ?, ?, ?, ? );', [ ( service_id, subject_account_id, admin_account_id, reason_id, now, expires ) for subject_account_id in subject_account_ids ] )
c.execute( 'DELETE FROM file_petitions WHERE service_id = ? AND account_id IN ' + splayed_subject_account_ids + ' AND status = ?;', ( service_id, HC.PETITIONED ) )
@ -958,12 +962,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
def _ChangeAccountType( self, c, service_id, account_ids, account_type_id ):
splayed_account_ids = HC.SplayListForDB( account_ids )
c.execute( 'UPDATE account_map SET account_type_id = ? WHERE service_id = ? AND account_id IN ' + splayed_account_ids + ';', ( account_type_id, service_id ) )
def _ChangeAccountType( self, c, account_ids, account_type_id ): c.execute( 'UPDATE accounts SET account_type_id = ? WHERE account_id IN ' + HC.SplayListForDB( account_ids ) + ';', ( account_type_id, ) )
def _CheckDataUsage( self, c ):
@ -977,7 +976,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
c.execute( 'UPDATE version SET year = ?, month = ?;', ( current_year, current_month ) )
c.execute( 'UPDATE account_map SET used_bytes = ?, used_requests = ?;', ( 0, 0 ) )
c.execute( 'UPDATE accounts SET used_bytes = ?, used_requests = ?;', ( 0, 0 ) )
self.pub_after_commit( 'update_all_session_accounts' )
@ -997,7 +996,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
if service_type != HC.SERVER_ADMIN:
( total_used_bytes, ) = c.execute( 'SELECT SUM( used_bytes ) FROM account_map WHERE service_id = ?;', ( service_id, ) ).fetchone()
( total_used_bytes, ) = c.execute( 'SELECT SUM( used_bytes ) FROM accounts WHERE service_id = ?;', ( service_id, ) ).fetchone()
if total_used_bytes is None: total_used_bytes = 0
@ -1095,18 +1094,11 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
for message_key in deletees: os.remove( SC.GetPath( 'message', message_key ) )
def _FlushRequestsMade( self, c, all_services_requests ):
def _FlushRequestsMade( self, c, all_requests ):
all_services_request_dict = HC.BuildKeyToListDict( [ ( service_key, ( account, num_bytes ) ) for ( service_key, account, num_bytes ) in all_services_requests ] )
requests_dict = HC.BuildKeyToListDict( all_requests )
for ( service_key, requests ) in all_services_request_dict.items():
service_id = self._GetServiceId( c, service_key )
requests_dict = HC.BuildKeyToListDict( requests )
c.executemany( 'UPDATE account_map SET used_bytes = used_bytes + ?, used_requests = used_requests + ? WHERE service_id = ? AND account_id = ?;', [ ( sum( num_bytes_list ), len( num_bytes_list ), service_id, account.GetAccountId() ) for ( account, num_bytes_list ) in requests_dict.items() ] )
c.executemany( 'UPDATE account SET used_bytes = used_bytes + ?, used_requests = used_requests + ? WHERE account_key = ?;', [ ( sum( num_bytes_list ), len( num_bytes_list ), sqlite3.Binary( account_key ) ) for ( account_key, num_bytes_list ) in requests_dict.items() ] )
def _GenerateRegistrationKeys( self, c, service_key, num, title, lifetime = None ):
@ -1117,12 +1109,12 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
now = HC.GetNow()
if lifetime is not None: expiry = now + lifetime
else: expiry = None
if lifetime is not None: expires = now + lifetime
else: expires = None
keys = [ ( os.urandom( HC.HYDRUS_KEY_LENGTH ), os.urandom( HC.HYDRUS_KEY_LENGTH ), os.urandom( HC.HYDRUS_KEY_LENGTH ) ) for i in range( num ) ]
c.executemany( 'INSERT INTO registration_keys ( registration_key, service_id, account_type_id, account_key, access_key, expiry ) VALUES ( ?, ?, ?, ?, ?, ? );', [ ( sqlite3.Binary( hashlib.sha256( registration_key ).digest() ), service_id, account_type_id, sqlite3.Binary( account_key ), sqlite3.Binary( access_key ), expiry ) for ( registration_key, account_key, access_key ) in keys ] )
c.executemany( 'INSERT INTO registration_keys ( registration_key, service_id, account_type_id, account_key, access_key, expiry ) VALUES ( ?, ?, ?, ?, ?, ? );', [ ( sqlite3.Binary( hashlib.sha256( registration_key ).digest() ), service_id, account_type_id, sqlite3.Binary( account_key ), sqlite3.Binary( access_key ), expires ) for ( registration_key, account_key, access_key ) in keys ] )
return [ registration_key for ( registration_key, account_key, access_key ) in keys ]
@ -1135,17 +1127,13 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
return access_key
def _GetAccount( self, c, service_key, account_key ):
def _GetAccount( self, c, account_key ):
service_id = self._GetServiceId( c, service_key )
( account_id, account_type, created, expiry, used_bytes, used_requests ) = c.execute( 'SELECT account_id, account_type, created, expires, used_bytes, used_requests FROM account_types, ( accounts, account_map USING ( account_id ) ) USING ( account_type_id ) WHERE service_id = ? AND account_key = ?;', ( service_id, sqlite3.Binary( account_key ) ) ).fetchone()
used_data = ( used_bytes, used_requests )
( account_id, service_id, account_key, account_type, created, expires, used_bytes, used_requests ) = c.execute( 'SELECT account_id, service_id, account_key, account_type, created, expires, used_bytes, used_requests FROM accounts, account_types USING ( service_id, account_type_id ) WHERE account_key = ?;', ( sqlite3.Binary( account_key ), ) ).fetchone()
banned_info = c.execute( 'SELECT reason, created, expires FROM bans, reasons USING ( reason_id ) WHERE service_id = ? AND account_id = ?;', ( service_id, account_id ) ).fetchone()
return HC.Account( account_id, account_type, created, expiry, used_data, banned_info = banned_info )
return HC.Account( account_key, account_type, created, expires, used_bytes, used_requests, banned_info = banned_info )
def _GetAccountFileInfo( self, c, service_id, account_id ):
@ -1174,43 +1162,55 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
return account_info
def _GetAccountKey( self, c, service_key, account_identifier ):
def _GetAccountKeyFromAccessKey( self, c, service_key, access_key ):
if account_identifier.HasAccessKey():
try: ( account_key, ) = c.execute( 'SELECT account_key FROM accounts WHERE hashed_access_key = ?;', ( sqlite3.Binary( hashlib.sha256( access_key ).digest() ), ) ).fetchone()
except:
access_key = account_identifier.GetData()
# we do not delete the registration_key (and hence the raw unhashed access_key)
# until the first attempt to create a session to make sure the user
# has the access_key saved
try: ( account_key, ) = c.execute( 'SELECT account_key FROM accounts WHERE access_key = ?;', ( sqlite3.Binary( hashlib.sha256( access_key ).digest() ), ) ).fetchone()
except:
# we do not delete the registration_key (and hence the raw unhashed access_key)
# until the first attempt to create a session to make sure the user
# has the access_key saved
try: ( account_type_id, account_key, expiry ) = c.execute( 'SELECT account_type_id, account_key, expiry FROM registration_keys WHERE access_key = ?;', ( sqlite3.Binary( access_key ), ) ).fetchone()
except: raise HydrusExceptions.ForbiddenException( 'The service could not find that account in its database.' )
c.execute( 'DELETE FROM registration_keys WHERE access_key = ?;', ( sqlite3.Binary( access_key ), ) )
#
c.execute( 'INSERT INTO accounts ( account_key, access_key ) VALUES ( ?, ? );', ( sqlite3.Binary( account_key ), sqlite3.Binary( hashlib.sha256( access_key ).digest() ), ) )
account_id = c.lastrowid
now = HC.GetNow()
service_id = self._GetServiceId( c, service_key )
c.execute( 'INSERT INTO account_map ( service_id, account_id, account_type_id, created, expires, used_bytes, used_requests ) VALUES ( ?, ?, ?, ?, ?, ?, ? );', ( service_id, account_id, account_type_id, now, expiry, 0, 0 ) )
try: ( account_type_id, account_key, expires ) = c.execute( 'SELECT account_type_id, account_key, expiry FROM registration_keys WHERE access_key = ?;', ( sqlite3.Binary( access_key ), ) ).fetchone()
except: raise HydrusExceptions.ForbiddenException( 'The service could not find that account in its database.' )
c.execute( 'DELETE FROM registration_keys WHERE access_key = ?;', ( sqlite3.Binary( access_key ), ) )
#
now = HC.GetNow()
service_id = self._GetServiceId( c, service_key )
c.execute( 'INSERT INTO accounts ( service_id, account_key, hashed_access_key, account_type_id, created, expires, used_bytes, used_requests ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ? );', ( service_id, sqlite3.Binary( account_key ), sqlite3.Binary( hashlib.sha256( access_key ).digest() ), account_type_id, now, expires, 0, 0 ) )
return account_key
def _GetAccountKeyFromAccountId( self, c, account_id ):
try: ( account_key, ) = c.execute( 'SELECT account_key FROM accounts WHERE account_id = ?;', ( account_id, ) ).fetchone()
except: raise HydrusExceptions.ForbiddenException( 'The service could not find that account_id in its database.' )
return account_key
def _GetAccountKeyFromIdentifier( self, c, service_key, account_identifier ):
service_id = self._GetServiceId( c, service_key )
if account_identifier.HasAccountKey():
account_key = account_identifier.GetData()
result = c.execute( 'SELECT 1 FROM accounts WHERE service_id = ? AND account_key = ?;', ( service_id, sqlite3.Binary( account_key ) ) ).fetchone()
if result is None: raise HydrusExceptions.ForbiddenException( 'The service could not find that hash in its database.')
else:
if account_identifier.HasAccountId(): account_id = account_identifier.GetData()
elif account_identifier.HasHash():
service_id = self._GetServiceId( c, service_key )
if account_identifier.HasHash():
try:
@ -1228,8 +1228,6 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
elif account_identifier.HasMapping():
service_id = self._GetServiceId( c, service_key )
try:
( hash, tag ) = account_identifier.GetData()
@ -1249,14 +1247,6 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
return account_key
def _GetAccountIdentifier( self, c, access_key ):
try: ( account_id, ) = c.execute( 'SELECT account_id FROM accounts WHERE access_key = ?;', ( sqlite3.Binary( hashlib.sha256( access_key ).digest() ), ) ).fetchone()
except: raise HydrusExceptions.ForbiddenException( 'The service could not find that account in its database.' )
return HC.AccountIdentifier( account_id = account_id )
def _GetAccountIdFromContactKey( self, c, service_id, contact_key ):
try:
@ -1272,38 +1262,20 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
result = c.execute( 'SELECT account_id FROM accounts WHERE account_key = ?;', ( sqlite3.Binary( account_key ), ) ).fetchone()
if result is None: raise HydrusExceptions.ForbiddenException( 'That account key was not found!' )
if result is None: raise HydrusExceptions.ForbiddenException( 'The service could not find that account key in its database.' )
( account_id, ) = result
return account_id
def _GetAccountIds( self, c, access_keys ):
account_ids = []
if type( access_keys ) == set: access_keys = list( access_keys )
for i in range( 0, len( access_keys ), 250 ): # there is a limit on the number of parameterised variables in sqlite, so only do a few at a time
access_keys_subset = access_keys[ i : i + 250 ]
account_ids.extend( [ account_id for ( account_id , ) in c.execute( 'SELECT account_id FROM accounts WHERE access_key IN (' + ','.join( '?' * len( access_keys_subset ) ) + ');', [ sqlite3.Binary( hashlib.sha256( access_key ).digest() ) for access_key in access_keys_subset ] ) ] )
return account_ids
def _GetAccountInfo( self, c, service_key, account_identifier ):
def _GetAccountInfo( self, c, service_key, account_key ):
service_id = self._GetServiceId( c, service_key )
account_key = self._GetAccountKey( c, service_key, account_identifier )
account = self._GetAccount( c, account_key )
account = self._GetAccount( c, service_key, account_key )
account_id = account.GetAccountId()
account_id = self._GetAccountId( c, account_key )
service_type = self._GetServiceType( c, service_id )
@ -1342,7 +1314,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
def _GetAccountTypeId( self, c, service_id, title ):
result = c.execute( 'SELECT account_type_id FROM account_types, account_type_map USING ( account_type_id ) WHERE service_id = ? AND title = ?;', ( service_id, title ) ).fetchone()
result = c.execute( 'SELECT account_type_id FROM account_types WHERE service_id = ? AND title = ?;', ( service_id, title ) ).fetchone()
if result is None: raise HydrusExceptions.NotFoundException( 'Could not find account title ' + HC.u( title ) + ' in db for this service.' )
@ -1355,7 +1327,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
service_id = self._GetServiceId( c, service_key )
return [ account_type for ( account_type, ) in c.execute( 'SELECT account_type FROM account_type_map, account_types USING ( account_type_id ) WHERE service_id = ?;', ( service_id, ) ) ]
return [ account_type for ( account_type, ) in c.execute( 'SELECT account_type FROM account_types WHERE service_id = ?;', ( service_id, ) ) ]
def _GetDirtyUpdates( self, c ):
@ -1380,7 +1352,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
c.execute( 'DELETE FROM messaging_sessions WHERE ? > expiry;', ( now, ) )
existing_session_ids = HC.BuildKeyToListDict( [ ( service_id, ( session_key, account_id, identifier, name, expiry ) ) for ( service_id, identifier, name, expiry ) in c.execute( 'SELECT service_id, session_key, account_id, identifier, name, expiry FROM messaging_sessions;' ) ] )
existing_session_ids = HC.BuildKeyToListDict( [ ( service_id, ( session_key, account_id, identifier, name, expires ) ) for ( service_id, identifier, name, expires ) in c.execute( 'SELECT service_id, session_key, account_id, identifier, name, expiry FROM messaging_sessions;' ) ] )
existing_sessions = {}
@ -1390,15 +1362,13 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
processed_tuples = []
for ( account_id, identifier, name, expiry ) in tuples:
for ( account_id, identifier, name, expires ) in tuples:
account_identifier = HC.AccountIdentifier( account_id = account_id )
account_key = self._GetAccountKeyFromAccountId( c, account_id )
account_key = self._GetAccountKey( c, service_key, account_identifier )
account = self._GetAccount( c, account_key )
account = self._GetAccount( c, service_key, account_key )
processed_tuples.append( ( account, identifier, name, expiry ) )
processed_tuples.append( ( account, name, expires ) )
existing_sessions[ service_key ] = processed_tuples
@ -1532,28 +1502,26 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
service_ids_to_service_keys = {}
service_id_and_account_ids_to_account_info = {}
account_ids_to_accounts = {}
for ( session_key, service_id, account_id, expiry ) in results:
for ( session_key, service_id, account_id, expires ) in results:
if service_id not in service_ids_to_service_keys: service_ids_to_service_keys[ service_id ] = self._GetServiceKey( c, service_id )
service_key = service_ids_to_service_keys[ service_id ]
if ( service_id, account_id ) not in service_id_and_account_ids_to_account_info:
if account_id not in account_ids_to_accounts:
account_identifier = HC.AccountIdentifier( account_id = account_id )
account_key = self._GetAccountKeyFromAccountId( c, account_id )
account_key = self._GetAccountKey( c, service_key, account_identifier )
account = self._GetAccount( c, account_key )
account = self._GetAccount( c, service_key, account_key )
service_id_and_account_ids_to_account_info[ ( service_id, account_id ) ] = ( account_key, account )
account_ids_to_accounts[ account_id ] = account
( account_key, account ) = service_id_and_account_ids_to_account_info[ ( service_id, account_id ) ]
account = account_ids_to_accounts[ account_id ]
sessions.append( ( session_key, service_key, account_key, account, expiry ) )
sessions.append( ( session_key, service_key, account, expires ) )
return sessions
@ -1565,7 +1533,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
stats = {}
( stats[ 'num_accounts' ], ) = c.execute( 'SELECT COUNT( * ) FROM account_map WHERE service_id = ?;', ( service_id, ) ).fetchone()
( stats[ 'num_accounts' ], ) = c.execute( 'SELECT COUNT( * ) FROM accounts WHERE service_id = ?;', ( service_id, ) ).fetchone()
( stats[ 'num_banned' ], ) = c.execute( 'SELECT COUNT( * ) FROM bans WHERE service_id = ?;', ( service_id, ) ).fetchone()
@ -1592,9 +1560,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
def _InitAdmin( self, c ):
if c.execute( 'SELECT 1 FROM account_map;' ).fetchone() is not None: raise HydrusExceptions.ForbiddenException( 'This server is already initialised!' )
# this is a little odd, but better to keep it all the same workflow
if c.execute( 'SELECT 1 FROM accounts;' ).fetchone() is not None: raise HydrusExceptions.ForbiddenException( 'This server is already initialised!' )
num = 1
@ -1607,17 +1573,13 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
return access_key
def _ModifyAccount( self, c, service_key, admin_account, action, subject_identifiers, kwargs ):
def _ModifyAccount( self, c, service_key, admin_account_key, action, subject_account_keys, kwargs ):
service_id = self._GetServiceId( c, service_key )
admin_account_id = admin_account.GetAccountId()
admin_account_id = self._GetAccountId( c, admin_account_key )
subject_account_keys = [ self._GetAccountKey( c, service_key, subject_identifier ) for subject_identifier in subject_identifiers ]
subjects = [ self._GetAccount( c, service_key, subject_account_key ) for subject_account_key in subject_account_keys ]
subject_account_ids = [ subject.GetAccountId() for subject in subjects ]
subject_account_ids = [ self._GetAccountId( c, subject_account_key ) for subject_account_key in subject_account_keys ]
if action in ( HC.BAN, HC.SUPERBAN ):
@ -1640,19 +1602,19 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
account_type_id = self._GetAccountTypeId( c, service_id, title )
self._ChangeAccountType( c, service_id, subject_account_ids, account_type_id )
self._ChangeAccountType( c, subject_account_ids, account_type_id )
elif action == HC.ADD_TO_EXPIRY:
elif action == HC.ADD_TO_EXPIRES:
timespan = kwargs[ 'timespan' ]
self._AddToExpiry( c, service_id, subject_account_ids, timespan )
self._AddToExpires( c, subject_account_ids, timespan )
elif action == HC.SET_EXPIRY:
elif action == HC.SET_EXPIRES:
expiry = kwargs[ 'expiry' ]
expires = kwargs[ 'expires' ]
self._SetExpiry( c, service_id, subject_account_ids, expiry )
self._SetExpires( c, subject_account_ids, expires )
@ -1671,11 +1633,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
if self._AccountTypeExists( c, service_id, title ): raise HydrusExceptions.ForbiddenException( 'Already found account type ' + HC.u( title ) + ' in the db for this service, so could not add!' )
c.execute( 'INSERT OR IGNORE INTO account_types ( title, account_type ) VALUES ( ?, ? );', ( title, account_type ) )
account_type_id = c.lastrowid
c.execute( 'INSERT OR IGNORE INTO account_type_map ( service_id, account_type_id ) VALUES ( ?, ? );', ( service_id, account_type_id ) )
c.execute( 'INSERT OR IGNORE INTO account_types ( service_id, title, account_type ) VALUES ( ?, ?, ? );', ( service_id, title, account_type ) )
elif action == HC.DELETE:
@ -1685,13 +1643,11 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
new_account_type_id = self._GetAccountTypeId( c, service_id, new_title )
c.execute( 'UPDATE account_map SET account_type_id = ? WHERE service_id = ? AND account_type_id = ?;', ( new_account_type_id, service_id, account_type_id ) )
c.execute( 'UPDATE registration_keys SET account_type_id = ? WHERE service_id = ? AND account_type_id = ?;', ( new_account_type_id, service_id, account_type_id ) )
c.execute( 'UPDATE accounts SET account_type_id = ? WHERE account_type_id = ?;', ( new_account_type_id, account_type_id ) )
c.execute( 'UPDATE registration_keys SET account_type_id = ? WHERE account_type_id = ?;', ( new_account_type_id, account_type_id ) )
c.execute( 'DELETE FROM account_types WHERE account_type_id = ?;', ( account_type_id, ) )
c.execute( 'DELETE FROM account_type_map WHERE service_id = ? AND account_type_id = ?;', ( service_id, account_type_id ) )
elif action == HC.EDIT:
( old_title, account_type ) = details
@ -1707,9 +1663,9 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
def _ModifyServices( self, c, account, edit_log ):
def _ModifyServices( self, c, account_key, edit_log ):
account_id = account.GetAccountId()
account_id = self._GetAccountId( c, account_key )
service_keys_to_access_keys = {}
@ -1727,11 +1683,7 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
service_admin_account_type = HC.AccountType( 'service admin', [ HC.GET_DATA, HC.POST_DATA, HC.POST_PETITIONS, HC.RESOLVE_PETITIONS, HC.MANAGE_USERS, HC.GENERAL_ADMIN ], ( None, None ) )
c.execute( 'INSERT INTO account_types ( title, account_type ) VALUES ( ?, ? );', ( 'service admin', service_admin_account_type ) )
service_admin_account_type_id = c.lastrowid
c.execute( 'INSERT INTO account_type_map ( service_id, account_type_id ) VALUES ( ?, ? );', ( service_id, service_admin_account_type_id ) )
c.execute( 'INSERT INTO account_types ( service_id, title, account_type ) VALUES ( ?, ?, ? );', ( service_id, 'service admin', service_admin_account_type ) )
[ registration_key ] = self._GenerateRegistrationKeys( c, service_key, 1, 'service admin', None )
@ -1774,11 +1726,13 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
return service_keys_to_access_keys
def _ProcessUpdate( self, c, service_key, account, update ):
def _ProcessUpdate( self, c, service_key, account_key, update ):
service_id = self._GetServiceId( c, service_key )
account_id = account.GetAccountId()
account_id = self._GetAccountId( c, account_key )
account = self._GetAccount( c, account_key )
service_type = self._GetServiceType( c, service_id )
@ -1976,27 +1930,33 @@ class ServiceDB( FileDB, MessageDB, TagDB ):
c.executemany( 'UPDATE account_scores SET score = score + ? WHERE service_id = ? AND account_id = ? and score_type = ?;', [ ( score, service_id, account_id, score_type ) for ( account_id, score ) in scores ] )
def _SetExpiry( self, c, service_id, account_ids, expiry ): c.execute( 'UPDATE account_map SET expires = ? WHERE service_id = ? AND account_id IN ' + HC.SplayListForDB( account_ids ) + ';', ( expiry, service_id ) )
def _SetExpires( self, c, account_ids, expires ): c.execute( 'UPDATE accounts SET expires = ? WHERE account_id IN ' + HC.SplayListForDB( account_ids ) + ';', ( expires, ) )
def _UnbanKey( self, c, service_id, account_id ): c.execute( 'DELETE FROM bans WHERE service_id = ? AND account_id = ?;', ( account_id, ) )
def _VerifyAccessKey( self, c, service_key, access_key ):
try:
service_id = self._GetServiceId( c, service_key )
result = c.execute( 'SELECT 1 FROM accounts WHERE service_id = ? AND hashed_access_key = ?;', ( service_id, sqlite3.Binary( hashlib.sha256( access_key ).digest() ) ) ).fetchone()
if result is None:
service_id = self._GetServiceId( c, service_key )
result = c.execute( 'SELECT 1 FROM registration_keys WHERE service_id = ? AND access_key = ?;', ( service_id, sqlite3.Binary( access_key ) ) ).fetchone()
( account_key, ) = c.execute( 'SELECT account_key FROM accounts, account_map USING ( account_id ) WHERE service_id = ? AND access_key = ?;', ( service_id, sqlite3.Binary( hashlib.sha256( access_key ).digest() ) ) ).fetchone()
if result is None: return False
return True
except: return False
return True
class DB( ServiceDB ):
def __init__( self ):
self._local_shutdown = False
self._loop_finished = False
self._db_path = HC.DB_DIR + os.path.sep + 'server.db'
self._jobs = Queue.PriorityQueue()
@ -2087,18 +2047,16 @@ class DB( ServiceDB ):
c.execute( 'CREATE TABLE services ( service_id INTEGER PRIMARY KEY, service_key BLOB_BYTES, type INTEGER, options TEXT_YAML );' )
c.execute( 'CREATE TABLE accounts ( account_id INTEGER PRIMARY KEY, account_key BLOB_BYTES, access_key BLOB_BYTES );' )
c.execute( 'CREATE TABLE accounts( account_id INTEGER PRIMARY KEY, service_id INTEGER REFERENCES services ON DELETE CASCADE, account_key BLOB_BYTES, hashed_access_key BLOB_BYTES, account_type_id INTEGER, created INTEGER, expires INTEGER, used_bytes INTEGER, used_requests INTEGER );' )
c.execute( 'CREATE UNIQUE INDEX accounts_account_key_index ON accounts ( account_key );' )
c.execute( 'CREATE UNIQUE INDEX accounts_access_key_index ON accounts ( access_key );' )
c.execute( 'CREATE TABLE account_map ( service_id INTEGER REFERENCES services ON DELETE CASCADE, account_id INTEGER, account_type_id INTEGER, created INTEGER, expires INTEGER, used_bytes INTEGER, used_requests INTEGER, PRIMARY KEY( service_id, account_id ) );' )
c.execute( 'CREATE INDEX account_map_service_id_account_type_id_index ON account_map ( service_id, account_type_id );' )
c.execute( 'CREATE UNIQUE INDEX accounts_hashed_access_key_index ON accounts ( hashed_access_key );' )
c.execute( 'CREATE UNIQUE INDEX accounts_service_id_account_id_index ON accounts ( service_id, account_id );' )
c.execute( 'CREATE INDEX accounts_service_id_account_type_id_index ON accounts ( service_id, account_type_id );' )
c.execute( 'CREATE TABLE account_scores ( service_id INTEGER REFERENCES services ON DELETE CASCADE, account_id INTEGER, score_type INTEGER, score INTEGER, PRIMARY KEY( service_id, account_id, score_type ) );' )
c.execute( 'CREATE TABLE account_type_map ( service_id INTEGER REFERENCES services ON DELETE CASCADE, account_type_id INTEGER, PRIMARY KEY ( service_id, account_type_id ) );' )
c.execute( 'CREATE TABLE account_types ( account_type_id INTEGER PRIMARY KEY, title TEXT, account_type TEXT_YAML );' )
c.execute( 'CREATE TABLE account_types ( account_type_id INTEGER PRIMARY KEY, service_id INTEGER REFERENCES services ON DELETE CASCADE, title TEXT, account_type TEXT_YAML );' )
c.execute( 'CREATE UNIQUE INDEX account_types_service_id_account_type_id_index ON account_types ( service_id, account_type_id );' )
c.execute( 'CREATE TABLE bans ( service_id INTEGER REFERENCES services ON DELETE CASCADE, account_id INTEGER, admin_account_id INTEGER, reason_id INTEGER, created INTEGER, expires INTEGER, PRIMARY KEY( service_id, account_id ) );' )
c.execute( 'CREATE INDEX bans_expires ON bans ( expires );' )
@ -2195,11 +2153,7 @@ class DB( ServiceDB ):
server_admin_account_type = HC.AccountType( 'server admin', [ HC.MANAGE_USERS, HC.GENERAL_ADMIN, HC.EDIT_SERVICES ], ( None, None ) )
c.execute( 'INSERT INTO account_types ( title, account_type ) VALUES ( ?, ? );', ( 'server admin', server_admin_account_type ) )
server_admin_account_type_id = c.lastrowid
c.execute( 'INSERT INTO account_type_map ( service_id, account_type_id ) VALUES ( ?, ? );', ( server_admin_service_id, server_admin_account_type_id ) )
c.execute( 'INSERT INTO account_types ( service_id, title, account_type ) VALUES ( ?, ?, ? );', ( server_admin_service_id, 'server admin', server_admin_account_type ) )
c.execute( 'COMMIT' )
@ -2334,11 +2288,11 @@ class DB( ServiceDB ):
c.execute( 'CREATE TABLE registration_keys ( registration_key BLOB_BYTES PRIMARY KEY, service_id INTEGER REFERENCES services ON DELETE CASCADE, account_type_id INTEGER, account_key BLOB_BYTES, access_key BLOB_BYTES, expiry INTEGER );' )
c.execute( 'CREATE UNIQUE INDEX registration_keys_access_key_index ON registration_keys ( access_key );' )
for ( registration_key, service_id, account_type_id, access_key, expiry ) in old_r_k_info:
for ( registration_key, service_id, account_type_id, access_key, expires ) in old_r_k_info:
account_key = os.urandom( 32 )
c.execute( 'INSERT INTO registration_keys ( registration_key, service_id, account_type_id, account_key, access_key, expiry ) VALUES ( ?, ?, ?, ?, ?, ? );', ( sqlite3.Binary( registration_key ), service_id, account_type_id, sqlite3.Binary( account_key ), sqlite3.Binary( access_key ), expiry ) )
c.execute( 'INSERT INTO registration_keys ( registration_key, service_id, account_type_id, account_key, access_key, expiry ) VALUES ( ?, ?, ?, ?, ?, ? );', ( sqlite3.Binary( registration_key ), service_id, account_type_id, sqlite3.Binary( account_key ), sqlite3.Binary( access_key ), expires ) )
#
@ -2350,11 +2304,117 @@ class DB( ServiceDB ):
c.execute( 'BEGIN IMMEDIATE' )
if version == 131:
accounts_info = c.execute( 'SELECT * FROM accounts;' ).fetchall()
account_map_info = c.execute( 'SELECT * FROM account_map;' ).fetchall()
account_types_info = c.execute( 'SELECT * FROM account_types;' ).fetchall()
account_type_map_info = c.execute( 'SELECT * FROM account_type_map;' ).fetchall()
accounts_dict = { account_id : ( account_key, hashed_access_key ) for ( account_id, account_key, hashed_access_key ) in accounts_info }
account_types_dict = { account_type_id : ( title, account_type ) for ( account_type_id, title, account_type ) in account_types_info }
#
c.execute( 'DROP TABLE accounts;' )
c.execute( 'DROP TABLE account_map;' )
c.execute( 'DROP TABLE account_types;' )
c.execute( 'DROP TABLE account_type_map;' )
c.execute( 'CREATE TABLE accounts( account_id INTEGER PRIMARY KEY, service_id INTEGER REFERENCES services ON DELETE CASCADE, account_key BLOB_BYTES, hashed_access_key BLOB_BYTES, account_type_id INTEGER, created INTEGER, expires INTEGER, used_bytes INTEGER, used_requests INTEGER );' )
c.execute( 'CREATE UNIQUE INDEX accounts_account_key_index ON accounts ( account_key );' )
c.execute( 'CREATE UNIQUE INDEX accounts_hashed_access_key_index ON accounts ( hashed_access_key );' )
c.execute( 'CREATE UNIQUE INDEX accounts_service_id_account_id_index ON accounts ( service_id, account_id );' )
c.execute( 'CREATE INDEX accounts_service_id_account_type_id_index ON accounts ( service_id, account_type_id );' )
c.execute( 'CREATE TABLE account_types ( account_type_id INTEGER PRIMARY KEY, service_id INTEGER REFERENCES services ON DELETE CASCADE, title TEXT, account_type TEXT_YAML );' )
c.execute( 'CREATE UNIQUE INDEX account_types_service_id_account_type_id_index ON account_types ( service_id, account_type_id );' )
#
account_log_text = ''
existing_account_ids = set()
existing_account_type_ids = set()
next_account_id = max( accounts_dict.keys() ) + 1
next_account_type_id = max( account_types_dict.keys() ) + 1
account_tables = [ 'bans', 'contacts', 'file_petitions', 'mapping_petitions', 'mappings', 'messages', 'message_statuses', 'messaging_sessions', 'sessions', 'tag_parents', 'tag_siblings' ]
account_type_tables = [ 'accounts', 'registration_keys' ]
service_dict = { service_id : options[ 'port' ] for ( service_id, options ) in c.execute( 'SELECT service_id, options FROM services;' ) }
# have to do accounts first because account_types may update it!
for ( service_id, account_id, account_type_id, created, expires, used_bytes, used_requests ) in account_map_info:
( account_key, hashed_access_key ) = accounts_dict[ account_id ]
if account_id in existing_account_ids:
account_key = os.urandom( 32 )
access_key = os.urandom( 32 )
account_log_text += 'The account at port ' + str( service_dict[ service_id ] ) + ' now uses access key: ' + access_key.encode( 'hex' ) + os.linesep
hashed_access_key = hashlib.sha256( access_key ).digest()
new_account_id = next_account_id
next_account_id += 1
for table_name in account_tables:
c.execute( 'UPDATE ' + table_name + ' SET account_id = ? WHERE service_id = ? AND account_id = ?;', ( new_account_id, service_id, account_id ) )
c.execute( 'UPDATE bans SET admin_account_id = ? WHERE service_id = ? AND admin_account_id = ?;', ( new_account_id, service_id, account_id ) )
account_id = new_account_id
existing_account_ids.add( account_id )
c.execute( 'INSERT INTO accounts ( account_id, service_id, account_key, hashed_access_key, account_type_id, created, expires, used_bytes, used_requests ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ? );', ( account_id, service_id, sqlite3.Binary( account_key ), sqlite3.Binary( hashed_access_key ), account_type_id, created, expires, used_bytes, used_requests ) )
if len( account_log_text ) > 0:
with open( HC.BASE_DIR + os.path.sep + 'update to v132 new access keys.txt', 'wb' ) as f: f.write( account_log_text )
for ( service_id, account_type_id ) in account_type_map_info:
( title, account_type ) = account_types_dict[ account_type_id ]
if account_type_id in existing_account_type_ids:
new_account_type_id = next_account_type_id
next_account_type_id += 1
for table_name in account_type_tables:
c.execute( 'UPDATE ' + table_name + ' SET account_type_id = ? WHERE service_id = ? AND account_type_id = ?;', ( new_account_type_id, service_id, account_type_id ) )
account_type_id = new_account_type_id
existing_account_type_ids.add( account_type_id )
c.execute( 'INSERT INTO account_types ( account_type_id, service_id, title, account_type ) VALUES ( ?, ?, ?, ? );', ( account_type_id, service_id, title, account_type ) )
c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
def pub_after_commit( self, topic, *args, **kwargs ): self._pubsubs.append( ( topic, args, kwargs ) )
def LoopIsFinished( self ): return self._loop_finished
def MainLoop( self ):
def ProcessJob( c, job ):
@ -2364,7 +2424,8 @@ class DB( ServiceDB ):
if action == 'access_key': result = self._GetAccessKey( c, *args, **kwargs )
elif action == 'account': result = self._GetAccount( c, *args, **kwargs )
elif action == 'account_info': result = self._GetAccountInfo( c, *args, **kwargs )
elif action == 'account_key': result = self._GetAccountKey( c, *args, **kwargs )
elif action == 'account_key_from_access_key': result = self._GetAccountKeyFromAccessKey( c, *args, **kwargs )
elif action == 'account_key_from_identifier': result = self._GetAccountKeyFromIdentifier( c, *args, **kwargs )
elif action == 'account_types': result = self._GetAccountTypes( c, *args, **kwargs )
elif action == 'dirty_updates': result = self._GetDirtyUpdates( c, *args, **kwargs )
elif action == 'init': result = self._InitAdmin( c, *args, **kwargs )
@ -2471,7 +2532,7 @@ class DB( ServiceDB ):
( db, c ) = self._GetDBCursor()
while not HC.shutdown or not self._jobs.empty():
while not ( ( self._local_shutdown or HC.shutdown ) and self._jobs.empty() ):
try:
@ -2493,6 +2554,11 @@ class DB( ServiceDB ):
except: pass # no jobs this second; let's see if we should shutdown
c.close()
db.close()
self._loop_finished = True
def Read( self, action, priority, *args, **kwargs ):
@ -2507,6 +2573,8 @@ class DB( ServiceDB ):
return job.GetResult()
def Shutdown( self ): self._local_shutdown = True
def Write( self, action, priority, *args, **kwargs ):
job_type = 'write'
@ -2528,7 +2596,7 @@ def DAEMONClearBans(): HC.app.WriteDaemon( 'clear_bans' )
def DAEMONDeleteOrphans(): HC.app.WriteDaemon( 'delete_orphans' )
def DAEMONFlushRequestsMade( all_services_requests ): HC.app.WriteDaemon( 'flush_requests_made', all_services_requests )
def DAEMONFlushRequestsMade( all_requests ): HC.app.WriteDaemon( 'flush_requests_made', all_requests )
def DAEMONGenerateUpdates():

50
include/TestDB.py
View File

@ -5,6 +5,7 @@ import HydrusConstants as HC
import HydrusExceptions
import itertools
import os
import ServerDB
import shutil
import stat
import TestConstants
@ -1104,4 +1105,51 @@ class TestClientDB( unittest.TestCase ):
self.assertEqual( type( k ), int )
self.assertEqual( type( v ), int )
class TestServerDB( unittest.TestCase ):
def _read( self, action, *args, **kwargs ): return self._db.Read( action, HC.HIGH_PRIORITY, *args, **kwargs )
def _write( self, action, *args, **kwargs ): return self._db.Write( action, HC.HIGH_PRIORITY, True, *args, **kwargs )
@classmethod
def setUpClass( self ):
self._old_db_dir = HC.DB_DIR
self._old_server_files_dir = HC.SERVER_FILES_DIR
self._old_server_thumbnails_dir = HC.SERVER_THUMBNAILS_DIR
HC.DB_DIR = HC.TEMP_DIR + os.path.sep + os.urandom( 32 ).encode( 'hex' )
HC.SERVER_FILES_DIR = HC.DB_DIR + os.path.sep + 'server_files'
HC.SERVER_THUMBNAILS_DIR = HC.DB_DIR + os.path.sep + 'server_thumbnails'
if not os.path.exists( HC.TEMP_DIR ): os.mkdir( HC.TEMP_DIR )
if not os.path.exists( HC.DB_DIR ): os.mkdir( HC.DB_DIR )
self._db = ServerDB.DB()
threading.Thread( target = self._db.MainLoop, name = 'Database Main Loop' ).start()
@classmethod
def tearDownClass( self ):
self._db.Shutdown()
while not self._db.LoopIsFinished(): time.sleep( 0.1 )
def make_temp_files_deletable( function_called, path, traceback_gumpf ):
os.chmod( path, stat.S_IWRITE )
function_called( path ) # try again
if os.path.exists( HC.DB_DIR ): shutil.rmtree( HC.DB_DIR, onerror = make_temp_files_deletable )
HC.DB_DIR = self._old_db_dir
HC.SERVER_FILES_DIR = self._old_server_files_dir
HC.SERVER_THUMBNAILS_DIR = self._old_server_thumbnails_dir

29
include/TestHydrusServer.py
View File

@ -37,13 +37,14 @@ class TestServer( unittest.TestCase ):
permissions = [ HC.GET_DATA, HC.POST_DATA, HC.POST_PETITIONS, HC.RESOLVE_PETITIONS, HC.MANAGE_USERS, HC.GENERAL_ADMIN, HC.EDIT_SERVICES ]
account_id = 1
account_key = os.urandom( 32 )
account_type = HC.AccountType( 'account', permissions, ( None, None ) )
created = HC.GetNow() - 100000
expiry = None
used_data = ( 0, 0 )
expires = None
used_bytes = 0
used_requests = 0
self._account = HC.Account( account_id, account_type, created, expiry, used_data )
self._account = HC.Account( account_key, account_type, created, expires, used_bytes, used_requests )
self._access_key = os.urandom( 32 )
self._file_hash = os.urandom( 32 )
@ -384,7 +385,7 @@ class TestServer( unittest.TestCase ):
HC.app.SetRead( 'service', service )
HC.app.SetRead( 'account_key', os.urandom( 32 ) )
HC.app.SetRead( 'account_key_from_access_key', os.urandom( 32 ) )
HC.app.SetRead( 'account', self._account )
# account
@ -398,12 +399,9 @@ class TestServer( unittest.TestCase ):
account_info = { 'message' : 'hello' }
HC.app.SetRead( 'account_info', account_info )
HC.app.SetRead( 'account_key_from_identifier', os.urandom( 32 ) )
response = service.Request( HC.GET, 'account_info', { 'subject_account_id' : 1 } )
self.assertEqual( response[ 'account_info' ], account_info )
response = service.Request( HC.GET, 'account_info', { 'subject_access_key' : os.urandom( 32 ).encode( 'hex' ) } )
response = service.Request( HC.GET, 'account_info', { 'subject_account_key' : os.urandom( 32 ).encode( 'hex' ) } )
self.assertEqual( response[ 'account_info' ], account_info )
@ -643,15 +641,16 @@ class TestAMP( unittest.TestCase ):
permissions = [ HC.GET_DATA, HC.POST_DATA, HC.POST_PETITIONS, HC.RESOLVE_PETITIONS, HC.MANAGE_USERS, HC.GENERAL_ADMIN, HC.EDIT_SERVICES ]
account_id = 1
account_key = os.urandom( 32 )
account_type = HC.AccountType( 'account', permissions, ( None, None ) )
created = HC.GetNow() - 100000
expiry = None
used_data = ( 0, 0 )
expires = None
used_bytes = 0
used_requests = 0
account = HC.Account( account_id, account_type, created, expiry, used_data )
account = HC.Account( account_key, account_type, created, expires, used_bytes, used_requests )
HC.app.SetRead( 'account_key', os.urandom( 32 ) )
HC.app.SetRead( 'account_key_from_access_key', os.urandom( 32 ) )
HC.app.SetRead( 'account', account )
deferred = protocol.callRemote( HydrusServerAMP.IMSessionKey, access_key = access_key, name = name )

61
include/TestHydrusSessions.py
View File

@ -18,19 +18,19 @@ class TestSessions( unittest.TestCase ):
permissions = [ HC.GET_DATA, HC.POST_DATA, HC.POST_PETITIONS, HC.RESOLVE_PETITIONS, HC.MANAGE_USERS, HC.GENERAL_ADMIN, HC.EDIT_SERVICES ]
account_id = 1
access_key = os.urandom( 32 )
account_key = os.urandom( 32 )
account_type = HC.AccountType( 'account', permissions, ( None, None ) )
created = HC.GetNow() - 100000
expiry = HC.GetNow() + 300
used_data = ( 0, 0 )
expires = HC.GetNow() + 300
used_bytes = 0
used_requests = 0
account_key = os.urandom( 32 )
account = HC.Account( account_key, account_type, created, expires, used_bytes, used_requests )
account = HC.Account( account_id, account_type, created, expiry, used_data )
expires = HC.GetNow() - 10
expiry = HC.GetNow() - 10
HC.app.SetRead( 'sessions', [ ( session_key_1, service_key, account_key, account, expiry ) ] )
HC.app.SetRead( 'sessions', [ ( session_key_1, service_key, account, expires ) ] )
session_manager = HydrusSessions.HydrusSessionManagerServer()
@ -41,9 +41,9 @@ class TestSessions( unittest.TestCase ):
# test fetching a session already in db, after bootup
expiry = HC.GetNow() + 300
expires = HC.GetNow() + 300
HC.app.SetRead( 'sessions', [ ( session_key_1, service_key, account_key, account, expiry ) ] )
HC.app.SetRead( 'sessions', [ ( session_key_1, service_key, account, expires ) ] )
session_manager = HydrusSessions.HydrusSessionManagerServer()
@ -53,24 +53,22 @@ class TestSessions( unittest.TestCase ):
# test adding a session
expiry = HC.GetNow() + 300
expires = HC.GetNow() + 300
account_key_2 = os.urandom( 32 )
account_2 = HC.Account( 2, account_type, created, expiry, used_data )
account_2 = HC.Account( account_key_2, account_type, created, expires, used_bytes, used_requests )
HC.app.SetRead( 'account_key', account_key_2 )
HC.app.SetRead( 'account_key_from_access_key', account_key_2 )
HC.app.SetRead( 'account', account_2 )
account_identifier = HC.AccountIdentifier( access_key = os.urandom( 32 ) )
( session_key_2, expiry_2 ) = session_manager.AddSession( service_key, account_identifier )
( session_key_2, expires_2 ) = session_manager.AddSession( service_key, access_key )
[ ( args, kwargs ) ] = HC.app.GetWrite( 'session' )
( written_session_key, written_service_key, written_account_key, written_expiry ) = args
( written_session_key, written_service_key, written_account_key, written_expires ) = args
self.assertEqual( ( session_key_2, service_key, account_key_2, expiry_2 ), ( written_session_key, written_service_key, written_account_key, written_expiry ) )
self.assertEqual( ( session_key_2, service_key, account_key_2, expires_2 ), ( written_session_key, written_service_key, written_account_key, written_expires ) )
read_account = session_manager.GetAccount( service_key, session_key_2 )
@ -78,18 +76,16 @@ class TestSessions( unittest.TestCase ):
# test adding a new session for an account already in the manager
HC.app.SetRead( 'account_key', account_key )
HC.app.SetRead( 'account_key_from_access_key', account_key )
HC.app.SetRead( 'account', account )
account_identifier = HC.AccountIdentifier( access_key = os.urandom( 32 ) )
( session_key_3, expiry_3 ) = session_manager.AddSession( service_key, account_identifier )
( session_key_3, expires_3 ) = session_manager.AddSession( service_key, access_key )
[ ( args, kwargs ) ] = HC.app.GetWrite( 'session' )
( written_session_key, written_service_key, written_account_key, written_expiry ) = args
( written_session_key, written_service_key, written_account_key, written_expires ) = args
self.assertEqual( ( session_key_3, service_key, account_key, expiry_3 ), ( written_session_key, written_service_key, written_account_key, written_expiry ) )
self.assertEqual( ( session_key_3, service_key, account_key, expires_3 ), ( written_session_key, written_service_key, written_account_key, written_expires ) )
read_account = session_manager.GetAccount( service_key, session_key_3 )
@ -101,16 +97,13 @@ class TestSessions( unittest.TestCase ):
# test individual account refresh
expiry = HC.GetNow() + 300
expires = HC.GetNow() + 300
updated_account = HC.Account( account_id, account_type, created, expiry, ( 1, 1 ) )
updated_account = HC.Account( account_key, account_type, created, expires, 1, 1 )
HC.app.SetRead( 'account_key', account_key )
HC.app.SetRead( 'account', updated_account )
account_identifier = HC.AccountIdentifier( access_key = os.urandom( 32 ) )
session_manager.RefreshAccounts( service_key, [ account_identifier ] )
session_manager.RefreshAccounts( service_key, [ account_key ] )
read_account = session_manager.GetAccount( service_key, session_key_1 )
@ -122,13 +115,11 @@ class TestSessions( unittest.TestCase ):
# test all account refresh
expiry = HC.GetNow() + 300
expires = HC.GetNow() + 300
updated_account_key = os.urandom( 32 )
updated_account_2 = HC.Account( account_key, account_type, created, expires, 2, 2 )
updated_account_2 = HC.Account( 1, account_type, created, expiry, ( 2, 2 ) )
HC.app.SetRead( 'sessions', [ ( session_key_1, service_key, updated_account_key, updated_account_2, expiry ), ( session_key_2, service_key, account_key_2, account_2, expiry ), ( session_key_3, service_key, account_key_2, updated_account_2, expiry ) ] )
HC.app.SetRead( 'sessions', [ ( session_key_1, service_key, updated_account_2, expires ), ( session_key_2, service_key, account_2, expires ), ( session_key_3, service_key, updated_account_2, expires ) ] )
session_manager.RefreshAllAccounts()