Rooba
/
stash
mirror of https://github.com/stashapp/stash.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
stash/pkg/scraper/json_test.go

101 lines
6.9 KiB
Go
Raw Normal View History

Add JSON scrape support (#717) * Add support for scene fragment scrape in xpath
2020-08-10 04:21:50 +00:00
package scraper
import (
Scraper refactor middle (#2043) * Push scrapeByURL into scrapers Replace ScrapePerfomerByURL, ScrapeMovie..., ... with ScrapeByURL in the scraperActionImpl interface. This allows us to delete a lot of repeated code in the scrapers and replace the central part with a switch on the scraper type. * Fold name scraping into one call Follow up on scraper refactoring. Name scrapers use the same code path. This allows us to restructure some code and kill some functions, adding variance to the name scraping code. It allows us to remove some code repetition as well. * Do not export loop refs. * Simplify fragment scraping Generalize fragment scrapers into ScrapeByFragment. This simplifies fragment code flows into a simpler pathing which should be easier to handle in the future. * Eliminate more context.TODO() In a number of cases, we have a context now. Use the context rather than TODO() for those cases in order to make those operations cancellable. * Pass the context for the stashbox scraper This removes all context.TODO() in the path of the stashbox scraper, and replaces it with the context that's present on each of the paths. * Pass the context into subscrapers Mostly a mechanical update, where we pass in the context for subscraping. This removes the final context.TODO() in the scraper code. * Warn on unknown fields from scripts A common mistake for new script writers are that they return fields not known to stash. For instance the name "description" is used rather than "details". Decode disallowing unknown fields. If this fails, use a tee-reader to fall back to the old behavior, but print a warning for the user in this case. Thus, we retain the old behavior, but print warnings for scripts which fails the more strict unknown-fields detection. * Nil-check before running the postprocessing chain Fixes panics when scraping returns nil values. * Lift nil-ness in post-postprocessing If the struct we are trying to post-process is nil, we shouldn't enter the postprocessing flow at all. Pass the struct as a value rather than a pointer, eliminating nil-checks as we go. Use the top-level postProcess call to make the nil-check and then abort there if the object we are looking at is nil. * Allow conversion routines to handle values If we have a non-pointer type in the interface, we should also convert those into ScrapedContent. Otherwise we get errors on deprecated functions.
2021-11-26 00:20:06 +00:00
"context"
Add JSON scrape support (#717) * Add support for scene fragment scrape in xpath
2020-08-10 04:21:50 +00:00
"testing"
"gopkg.in/yaml.v2"
)
func TestJsonPerformerScraper(t *testing.T) {
const yamlStr = `name: Test
jsonScrapers:
performerScraper:
common:
$extras: data.extras
performer:
Name: data.name
Gender: $extras.gender
Birthdate: $extras.birthday
Ethnicity: $extras.ethnicity
Height: $extras.height
Measurements: $extras.measurements
Tattoos: $extras.tattoos
Piercings: $extras.piercings
Aliases: data.aliases
Image: data.image
added details, deathdate, hair color, weight to performers and added details to studios (#1274) * added details to performers and studios * added deathdate, hair_color and weight to performers * Simplify performer/studio create mutations * Add changelog and recategorised Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-04-16 06:06:35 +00:00
Details: data.bio
HairColor: $extras.hair_colour
Weight: $extras.weight
Add JSON scrape support (#717) * Add support for scene fragment scrape in xpath
2020-08-10 04:21:50 +00:00
`
const json = `
{
"data": {
"id": "2cd4146b-637d-49b1-8ff9-19d4a06947bb",
"name": "Mia Malkova",
"bio": "Some girls are so damn hot that they can get you bent out of shape, and you will not even be mad at them for doing so. Well, tawny blonde Mia Malkova can bend her body into any shape she pleases, and that’s sure to satisfy all of the horny cocks and wet pussies out there. This girl has acrobatic and contortionist abilities that could even twist a pretzel into a new knot, which can be very helpful in the ... arrow_drop_down Some girls are so damn hot that they can get you bent out of shape, and you will not even be mad at them for doing so. Well, tawny blonde Mia Malkova can bend her body into any shape she pleases, and that’s sure to satisfy all of the horny cocks and wet pussies out there. This girl has acrobatic and contortionist abilities that could even twist a pretzel into a new knot, which can be very helpful in the VR Porn movies – trust us. Ankles behind her neck and feet over her back so she can kiss her toes, turned, twisted and gyrating, she can fuck any which way she wants (and that ass!), will surely make you fall in love with this hot Virtual Reality Porn slut, as she is one of the finest of them all. Talking about perfection, maybe it’s all the acrobatic work that keeps it in such gorgeous shape? Who cares really, because you just want to take a big bite out of it and never let go. But it’s not all about the body. Mia’s also got a great smile, which might not sound kinky, but believe us, it is a smile that will heat up your innards and drop your pants. Is it her golden skin, her innocent pink lips or that heart-shaped face? There is just too much good stuff going on with Mia Malkova, which is maybe why these past few years have heaped awards upon awards on this Southern California native. Mia came to VR Bangers for her first VR Porn video, so you know she’s only going for top-notch scenes with top-game performers, men, and women. Better hit up that yoga studio if you ever dream of being able to bang a flexible and talented chick like lady Malkova. arrow_drop_up",
"extras": {
"gender": "Female",
"birthday": "1992-07-01",
"birthday_timestamp": 709948800,
"birthplace": "Palm Springs, California, United States",
"active": 1,
"astrology": "Cancer (Jun 21 - Jul 22)",
"ethnicity": "Caucasian",
"nationality": "United States",
"hair_colour": "Blonde",
added details, deathdate, hair color, weight to performers and added details to studios (#1274) * added details to performers and studios * added deathdate, hair_color and weight to performers * Simplify performer/studio create mutations * Add changelog and recategorised Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-04-16 06:06:35 +00:00
"weight": 57,
Add JSON scrape support (#717) * Add support for scene fragment scrape in xpath
2020-08-10 04:21:50 +00:00
"height": "5'6\" (or 167 cm)",
"measurements": "34-26-36",
"cupsize": "34C (75C)",
"tattoos": "None",
"piercings": "Navel",
"first_seen": null
},
"aliases": [
"Mia Bliss",
"Madison Clover",
"Madison Swan",
"Mia Mountain",
"Mia M.",
"Mia Malvoka",
"Mia Molkova",
"Mia Thomas"
],
"image": "https:\/\/thumb.metadataapi.net\/unsafe\/1000x1500\/smart\/filters:sharpen():upscale()\/https%3A%2F%2Fcdn.metadataapi.net%2Fperformer%2F49%2F05%2F30%2Fade2255dc065032a89ebb23f0e038fa%2Fposter%2Fmia-malkova.jpg%3Fid1582610531"
}
}
`
c := &config{}
err := yaml.Unmarshal([]byte(yamlStr), &c)
if err != nil {
t.Fatalf("Error loading yaml: %s", err.Error())
}
// perform scrape using json string
performerScraper := c.JsonScrapers["performerScraper"]
q := &jsonQuery{
doc: json,
}
Scraper refactor middle (#2043) * Push scrapeByURL into scrapers Replace ScrapePerfomerByURL, ScrapeMovie..., ... with ScrapeByURL in the scraperActionImpl interface. This allows us to delete a lot of repeated code in the scrapers and replace the central part with a switch on the scraper type. * Fold name scraping into one call Follow up on scraper refactoring. Name scrapers use the same code path. This allows us to restructure some code and kill some functions, adding variance to the name scraping code. It allows us to remove some code repetition as well. * Do not export loop refs. * Simplify fragment scraping Generalize fragment scrapers into ScrapeByFragment. This simplifies fragment code flows into a simpler pathing which should be easier to handle in the future. * Eliminate more context.TODO() In a number of cases, we have a context now. Use the context rather than TODO() for those cases in order to make those operations cancellable. * Pass the context for the stashbox scraper This removes all context.TODO() in the path of the stashbox scraper, and replaces it with the context that's present on each of the paths. * Pass the context into subscrapers Mostly a mechanical update, where we pass in the context for subscraping. This removes the final context.TODO() in the scraper code. * Warn on unknown fields from scripts A common mistake for new script writers are that they return fields not known to stash. For instance the name "description" is used rather than "details". Decode disallowing unknown fields. If this fails, use a tee-reader to fall back to the old behavior, but print a warning for the user in this case. Thus, we retain the old behavior, but print warnings for scripts which fails the more strict unknown-fields detection. * Nil-check before running the postprocessing chain Fixes panics when scraping returns nil values. * Lift nil-ness in post-postprocessing If the struct we are trying to post-process is nil, we shouldn't enter the postprocessing flow at all. Pass the struct as a value rather than a pointer, eliminating nil-checks as we go. Use the top-level postProcess call to make the nil-check and then abort there if the object we are looking at is nil. * Allow conversion routines to handle values If we have a non-pointer type in the interface, we should also convert those into ScrapedContent. Otherwise we get errors on deprecated functions.
2021-11-26 00:20:06 +00:00
scrapedPerformer, err := performerScraper.scrapePerformer(context.Background(), q)
Add JSON scrape support (#717) * Add support for scene fragment scrape in xpath
2020-08-10 04:21:50 +00:00
if err != nil {
t.Fatalf("Error scraping performer: %s", err.Error())
}
verifyField(t, "Mia Malkova", scrapedPerformer.Name, "Name")
verifyField(t, "Female", scrapedPerformer.Gender, "Gender")
verifyField(t, "1992-07-01", scrapedPerformer.Birthdate, "Birthdate")
verifyField(t, "Caucasian", scrapedPerformer.Ethnicity, "Ethnicity")
verifyField(t, "5'6\" (or 167 cm)", scrapedPerformer.Height, "Height")
verifyField(t, "None", scrapedPerformer.Tattoos, "Tattoos")
verifyField(t, "Navel", scrapedPerformer.Piercings, "Piercings")
added details, deathdate, hair color, weight to performers and added details to studios (#1274) * added details to performers and studios * added deathdate, hair_color and weight to performers * Simplify performer/studio create mutations * Add changelog and recategorised Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-04-16 06:06:35 +00:00
verifyField(t, "Some girls are so damn hot that they can get you bent out of shape, and you will not even be mad at them for doing so. Well, tawny blonde Mia Malkova can bend her body into any shape she pleases, and that’s sure to satisfy all of the horny cocks and wet pussies out there. This girl has acrobatic and contortionist abilities that could even twist a pretzel into a new knot, which can be very helpful in the ... arrow_drop_down Some girls are so damn hot that they can get you bent out of shape, and you will not even be mad at them for doing so. Well, tawny blonde Mia Malkova can bend her body into any shape she pleases, and that’s sure to satisfy all of the horny cocks and wet pussies out there. This girl has acrobatic and contortionist abilities that could even twist a pretzel into a new knot, which can be very helpful in the VR Porn movies – trust us. Ankles behind her neck and feet over her back so she can kiss her toes, turned, twisted and gyrating, she can fuck any which way she wants (and that ass!), will surely make you fall in love with this hot Virtual Reality Porn slut, as she is one of the finest of them all. Talking about perfection, maybe it’s all the acrobatic work that keeps it in such gorgeous shape? Who cares really, because you just want to take a big bite out of it and never let go. But it’s not all about the body. Mia’s also got a great smile, which might not sound kinky, but believe us, it is a smile that will heat up your innards and drop your pants. Is it her golden skin, her innocent pink lips or that heart-shaped face? There is just too much good stuff going on with Mia Malkova, which is maybe why these past few years have heaped awards upon awards on this Southern California native. Mia came to VR Bangers for her first VR Porn video, so you know she’s only going for top-notch scenes with top-game performers, men, and women. Better hit up that yoga studio if you ever dream of being able to bang a flexible and talented chick like lady Malkova. arrow_drop_up", scrapedPerformer.Details, "Details")
verifyField(t, "Blonde", scrapedPerformer.HairColor, "HairColor")
verifyField(t, "57", scrapedPerformer.Weight, "Weight")
Add JSON scrape support (#717) * Add support for scene fragment scrape in xpath
2020-08-10 04:21:50 +00:00
}