Commit Graph

15 Commits

Author SHA1 Message Date
WithoutPants 1a3a2f1f83
Scrape scene by name (#1712)
* Support scrape scene by name in configs
* Initial scene querying
* Add to manual
2021-09-14 14:54:53 +10:00
SmallCoccinelle 4b00d24248
Remove unused (#1709)
* Remove stuff which isn't being used

Some fields, functions and structs aren't in use by the project. Remove
them for janitorial reasons.

* Remove more unused code

All of these functions are currently not in use. Clean up the code by
removal, since the version control has the code if need be.

* Remove unused functions

There's a large set of unused functions and variables in the code base.
Remove these, so it clearer what code to support going forward.

Dead code has been eliminated.

Where applicable, comment const-sections in tests, so reserved
identifiers are still known.

* Fix use-def of tsURL

The first def of tsURL doesn't matter because there's no use before
we hit the 2nd def.

* Remove dead code assignment

Setting logFile = "" is effectively dead code, because there's no use
of it later.

* Comment out found

The variable 'found' is dead in the function (because no post-process
action is following it). Comment it for now.

* Comment dead code in tests

These might provide hints as to what isn't covered at the moment.

* Dead code removal

In the case of constants where iota is involved, move the iota so it
matches the current key values.

This avoids problems with persistently stored key IDs.
2021-09-09 14:10:08 +10:00
SmallCoccinelle e7f6cb22b7
Error strings noncapitalized (#1704)
* Fix error string capitalization

Error strings often follow another string. Hence, they should not be
capitalized, unless referencing a name.

* Uncapitalize more error strings

While here, use %v on the error directly, which makes it easier to wrap
the error later with %w if need be.

* Uncapitalize more error strings

While here, rename Url to URL as a nitpick.
2021-09-08 11:23:10 +10:00
bnkai 4c05535a13
Fix potential race condintion in CDP (#1536) 2021-06-28 10:36:51 +10:00
bnkai cd6b6b74eb
Add http headers support to scraper (#1273) 2021-04-16 15:42:56 +10:00
WithoutPants f6ffda7504
Setup and migration UI refactor (#1190)
* Make config instance-based
* Remove config dependency in paths
* Refactor config init
* Allow startup without database
* Get system status at UI initialise
* Add setup wizard
* Cache and Metadata optional. Database mandatory
* Handle metadata not set during full import/export
* Add links
* Remove config check middleware
* Stash not mandatory
* Panic on missing mandatory config fields
* Redirect setup to main page if setup not required
* Add migration UI
* Remove unused stuff
* Move UI initialisation into App
* Don't create metadata paths on RefreshConfig
* Add folder selector for generated in setup
* Env variable to set and create config file.
Make docker images use a fixed config file.
* Set config file during setup
2021-04-12 09:31:33 +10:00
bnkai 68d4a4fe42
Add User Agent to image download reqs (#1222) 2021-03-24 08:12:11 +11:00
bnkai 144cd6e4f2
Skip insecure certificates check when scraping (#1120)
* Ignore insecure certificates when scraping
* add ScraperCertCheck to scraper config options
2021-03-01 11:47:39 +11:00
bnkai e883e5fe27
Add Mouse Click support for the CDP scraper (#827) 2020-12-22 09:42:31 +11:00
bnkai a96ab9ce6f
Add support for setting cookies in the scraper (#934) 2020-12-01 16:34:09 +11:00
WithoutPants 109e55a25a
Query url parameters (#878) 2020-10-22 11:56:04 +11:00
SpedNSFW 147d0067f5
Add gallery scraping (#862) 2020-10-21 09:24:32 +11:00
WithoutPants 7158e83b75
Add JSON scrape support (#717)
* Add support for scene fragment scrape in xpath
2020-08-10 14:21:50 +10:00
WithoutPants b166abfa7b
Fix scraping error (#704) 2020-08-04 20:43:56 +10:00
bnkai 4373f9bf01
Add cdp support for xpath scrapers (#625)
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2020-08-04 10:42:40 +10:00