Commit Graph

54 Commits

Author SHA1 Message Date
WithoutPants 94d192b833
Don't scan zero length files or directories (#1779)
* Don't scan zero length files or directories
2021-09-27 16:49:30 +10:00
InfiniteTF 9cb1eccadb
Improve image scanning performance and thumbnail generation (#1655)
* Improve image scanning performance and thumbnail generation
* Add vips-tools to build image
* Add option to write generated thumbnails to disk
* Fallback to image if thumbnail generation fails

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-09-23 15:22:14 +10:00
SmallCoccinelle 4a0c4c4847
Reorder waitgroup completion (#1748)
Rather than passing a pointer to a waitgroup into task.Start(..)
functions, handle the waitgroup.Done() at the callsite.

This makes waitgroup handling local to its definition rather than it
being spread out over multiple files. Tasks now simply execute, and
the policy of waiting on them is handled by the caller.
2021-09-22 13:22:59 +10:00
SmallCoccinelle 87709fd018
Errcheck phase 1 (#1715)
* Avoid redundant logging in migrations

Return the error and let the caller handle the logging of the error if
needed.

While here, defer m.Close() to the function boundary.

* Treat errors as values

Use %v rather than %s and pass the errors directly.

* Generate a wrapped error on stat-failure

* Log 3 unchecked errors

Rather than ignore errors, log them at
the WARNING log level.

The server has been functioning without these, so assume they are not at
the ERROR level.

* Propagate errors upward

Failure in path generation was ignored. Propagate the errors upward the
call stack, so it can be handled at the level of orchestration.

* Warn on errors

Log errors rather than quenching them.

Errors are logged at the Warn-level for now.

* Check error when creating test databases

Use the builtin log package and stop the program fatally on error.

* Add warnings to uncheck task errors

Focus on the task system in a single commit, logging unchecked
errors as warnings.

* Warn-on-error in API routes

Look through the API routes, and make sure errors are being logged if
they occur. Prefer the Warn-log-level because none of these has proven
to be fatal in the system up until now.

* Propagate error when adding Util API

* Propagate error on adding util API

* Return unhandled error

* JS log API: propagate and log errors

* JS Plugins: log GQL addition failures.

* Warn on failure to write to stdin

* Warn on failure to stop task

* Wrap viper.BindEnv

The current viper code only errors if no name is provided, so it should
never fail. Rewrite the code flow to factor through a panic-function.

This removes error warnings from this part of the code.

* Log errors in concurrency test

If we can't initialize the configuration, treat the test as a failure.

* Warn on errors in configuration code

* Plug an unchecked error in gallery zip walking

* Warn on screenshot serving failure

* Warn on encoder screenshot failure

* Warn on errors in path-handling code

* Undo the errcheck on configurations for now.

* Use one-line initializers where applicable

rather than using

  err := f()
  if err!= nil { ..

prefer the shorter

  if err := f(); err != nil { ..

If f() isn't too long of a name, or wraps a function with a body.
2021-09-21 09:34:25 +10:00
gitgiggety 82e4ad4130
Execute Gallery.Create.Post plugin hook during scan (#1731)
* Execute Gallery.Create.Post plugin hook during scan

Fix issue where Gallery.Create.Post hook is not executed when a new
gallery is created during scan, when the gallery is created based on the
folder.

* Fix Gallery.Create.Post hook being invoked in transaction

Invoke the Gallery.Create.Post hook during zip scan after the
transaction has been committed. This is necessary to allow the plugin to
access the gallery (using GraphQL API). Otherwise the API obviously uses
a different database transaction which can't find the gallery as it
isn't committed yet.
2021-09-19 10:28:34 +10:00
WithoutPants 46bbede9a0
Plugin hooks (#1452)
* Refactor session and plugin code
* Add context to job tasks
* Show hooks in plugins page
* Refactor session management
2021-06-11 17:24:58 +10:00
WithoutPants dde361f9f3
Handle case sensitive file moves (#1427) 2021-06-11 15:25:09 +10:00
WithoutPants 0e01374537
Job queueing (#1379) 2021-05-24 14:24:18 +10:00
UnluckyChemical765 547f6d79ad
Add Handy / Funscript support (#1377)
* Add funscript route to scenes

Adds a /scene/:id/funscript route which serves a funscript file, if present.

Current convention is that these are files stored with the same path, but with the extension ".funscript".

* Look for funscript during scan

This is stored in the Scene record and used to drive UI changes for funscript support.

Currently, that's limited to a funscript link in the Scene's file info.

* Add filtering and sorting for interactive
* Add Handy connection key to interface config
* Add Handy client and placeholder component.

Uses defucilis/thehandy, but not thehandy-react as I had difficulty integrating the context with the existing components.

Instead, the expensive calculation for the server time offset is put in localStorage for reuse.

A debounce was added when scrubbing the video, as otherwise it spammed the Handy API with updates to the current offset.
2021-05-24 13:34:28 +10:00
WithoutPants 3df7ee06eb
Shortcut scan walk for excluded folders (#1382)
* Shortcut scan walk for excluded folders
* Add note to manual
* Add changelog entry
2021-05-17 15:46:00 +10:00
bnkai 2ab42e9cd3
Populate image/gallery title during scan (#1359) 2021-05-03 14:21:51 +10:00
WithoutPants f6ffda7504
Setup and migration UI refactor (#1190)
* Make config instance-based
* Remove config dependency in paths
* Refactor config init
* Allow startup without database
* Get system status at UI initialise
* Add setup wizard
* Cache and Metadata optional. Database mandatory
* Handle metadata not set during full import/export
* Add links
* Remove config check middleware
* Stash not mandatory
* Panic on missing mandatory config fields
* Redirect setup to main page if setup not required
* Add migration UI
* Remove unused stuff
* Move UI initialisation into App
* Don't create metadata paths on RefreshConfig
* Add folder selector for generated in setup
* Env variable to set and create config file.
Make docker images use a fixed config file.
* Set config file during setup
2021-04-12 09:31:33 +10:00
InfiniteTF c38660d209
Add phash generation and dupe checking (#1158) 2021-04-12 09:04:40 +10:00
bnkai fe990e00c1
Check if gallery is already associated during scanning (#1154) 2021-03-01 16:37:55 +11:00
WithoutPants f7a8899d90
Add rescan option to overflow dropdown (#1119)
* Make scan options optional
* Add scene rescan
* Add image rescan
* Add gallery rescan
* Add changelog
2021-02-23 12:56:01 +11:00
WithoutPants bbc34bd1bf
Exclude media in generated directory (#1118) 2021-02-11 11:06:04 +11:00
InfiniteTF 8d8a8530e8
Migrate generated files when a scene is rescanned (#1106) 2021-02-10 10:50:34 +11:00
InfiniteTF 4fd022a93b
Decouple galleries from scenes (#1057) 2021-02-02 07:56:54 +11:00
WithoutPants 1e04deb3d4
Data layer restructuring (#997)
* Move query builders to sqlite package
* Add transaction system
* Wrap model resolvers in transaction
* Add error return value for StringSliceToIntSlice
* Update/refactor mutation resolvers
* Convert query builders
* Remove unused join types
* Add stash id unit tests
* Use WAL journal mode
2021-01-18 12:23:20 +11:00
bnkai 1882b44951
Strip file extension from scene title when scanning (#1022) 2021-01-07 11:38:30 +11:00
InfiniteTF e84c92355e
Fix integer overflow for scene size on 32bit systems (#994)
* Fix integer overflow for scene size on 32bit systems
* Cast to double in sqlite to prevent potential overflow
* Add migration to reset scene sizes and scan logic to repopulate if empty
2020-12-22 10:29:53 +11:00
JoeSmithStarkers e3eb550a7d
Parallel scanning/generation, and combined scanning/preview/sprite (#820)
* Implement parallel scanning and generation, and combined scanning/preview/sprite generation.
* Added UI component for preview/sprite generation during scan, and configurable number of parallel tasks.
* Add v050 changelog entry
2020-11-25 12:45:10 +11:00
WithoutPants 5f482b7b8a
Handle zip file modification (#877)
* Rescan zip if updating mod time
* Use inequality for mod time comparison
* Add sort by file_mod_time (fixes #469)
2020-11-05 10:26:51 +11:00
JoeSmithStarkers 47468fe122
Cache generated regex for each path (#891) 2020-10-26 15:57:58 +11:00
WithoutPants 872bb70f6e
Fix scan issue when encountering invalid symlinks (#871)
* Implement fixed symwalk algorithm
* Remove dependency
2020-10-20 17:00:23 +11:00
InfiniteTF 482f8cbd92
Fix clean and scan bugs (#846) 2020-10-14 10:51:36 +11:00
WithoutPants aca2c7c5f4
Images section (#813)
* Add new configuration options
* Refactor scan/clean
* Schema changes
* Add details to galleries
* Remove redundant code
* Refine thumbnail generation
* Gallery overhaul
* Don't allow modifying zip gallery images
* Show gallery card overlays
* Hide zoom slider when not in grid mode
2020-10-13 10:12:46 +11:00
WithoutPants 98dda782aa
Prevent invalid date tag in video file from aborting scan (#836)
* Give more context when ffprobe fails
* Suppress JSONTime unmarshal error
* Tidy scan logging
2020-10-11 12:02:41 +11:00
JoeSmithStarkers 85aa1d8790
Replace os.Rename with util.SafeMove to allow cross device moving to not fail. (#745)
Fixed annoyingly noisy transcoding progress log messages.
Fixed minor minor issue with directories than are named "blah.mp4" being detected as files to be scanned.
2020-08-21 17:57:07 +10:00
WithoutPants 0874852fa8
Improve oshash collision detection and logging (#713)
* Log colliding file when setting hash
* Check for existing using both hashes
2020-08-08 11:22:25 +10:00
WithoutPants 5992ff8706
Add oshash support (#667) 2020-08-06 11:21:14 +10:00
WithoutPants d3ababf0a1
Gallery list improvement (#622)
* Add grid view to galleries
* Show scene in gallery card
* Add is missing scene gallery filter
* Don't store galleries with no images
2020-06-21 21:43:57 +10:00
bnkai bd45daacf3
Add a cache for gallery thumbnails (#496) 2020-05-11 17:20:08 +10:00
bnkai d5617307f1
Add detection of container/video_codec/audio_codec compatibility for live file streaming or transcoding (#384)
* add forceMKV, forceHEVC config options
* drop audio stream instead of trying to transcode for ffmpeg unsupported/unknown audio codecs
2020-04-10 08:38:34 +10:00
bnkai 9dacad70a1
Autoassociate galleries to scenes when scanning (#405) 2020-03-19 12:36:00 +11:00
WithoutPants 3de6955a9e
Generate cover image (#376)
* Make mutating metadata ops mutation
* Implement scene generate screenshot
* Remove fetch policy on metadata mutations
* Port UI changes to v2.5
* Set generated image in database
2020-03-12 08:34:04 +11:00
WithoutPants c05496a724 Make scene metadata from file metadata optional (#259) 2019-12-13 01:18:02 -05:00
bnkai 5c0ec39db1 Calculate and print job totals for scan and generate tasks (#188)
* Calculate and print job totals for scan and generate tasks

* Cosmetic print fixes
2019-11-15 12:23:58 -05:00
bill 2693a2b22f print message when recreating images for the scan process 2019-10-18 20:35:53 +03:00
bill 8d2df527b7 makeScreenshots cleanup 2019-10-18 02:17:51 +03:00
bill ab1690d5b1 makeScreenshots can be called for a new or already existing file 2019-10-17 16:50:30 +03:00
bill 02f485e40e refactoring 2019-10-16 18:27:38 +03:00
bill 40b270eb49 fix issue #144 2019-10-16 04:17:04 +03:00
Leopere 57e954d4e7
Merge pull request #137 from WithoutPants/nullable_scene_props
Allow unsetting of rating, studio, gallery
2019-10-15 10:09:26 -04:00
WithoutPants 0852199e27 Only update non-nil database fields 2019-10-15 08:57:53 +11:00
WithoutPants afcadd941b Make title from file metadata optional 2019-10-12 19:20:27 +11:00
StashAppDev 32a4f34c7c
Merge pull request #106 from bnkai/scan_ign_dupes
fix for my own added bug ....
2019-08-20 13:08:37 -07:00
bill aa6420e9e9 bug fix 2019-08-20 16:46:05 +03:00
StashAppDev 6e46b103d7
Merge pull request #103 from WithoutPants/metadata_from_file
Set title, details and date from scene file metadata
2019-08-19 16:29:25 -07:00
WithoutPants 79f2902e26 Parse title, details, date from file metadata 2019-08-16 08:47:35 +10:00