Dusan Drevicky
96785b99df
Feature/4244 iou input expectations ( #4261 )
...
* =Add iou input checks
* =Add test for iou input checks
* =Update docstring for iou pred and target
2020-10-21 09:01:24 -04:00
Akihiro Nitta
d27ee8b5bf
docs: Add empty lines in docstring [ci skip] ( #4232 )
...
* Add empty lines in docstring for proper docs
* Remove Returns:
* Remove unnecessary Returns:
* Update pytorch_lightning/accelerators/ddp2_accelerator.py
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
* fix returns
Co-authored-by: William Falcon <waf2107@columbia.edu>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2020-10-21 09:00:39 -04:00
Dusan Drevicky
73863e9102
Fix unintended zero-only target in iou example ( #4262 )
2020-10-21 11:18:25 +02:00
Abhinav Gupta
5d1583d7f2
Corrected f_beta computation ( #4183 )
...
* Update f_beta.py
Added METRIC_EPS in the denominator to avoid nan values in f_beta score.
* Update f_beta.py
Made changes flake8 compliant
* Update f_beta.py
Makes use of class_reduce for macro f_beta computation to avoid nans
* Update f_beta.py
Made flake8 compliant
* Corrected F beta computation
* Removed offset to make the computation precise
2020-10-21 10:52:04 +02:00
Teddy Koker
827a557269
Add persistent flag to Metric.add_state ( #4195 )
...
* add persistant flag to add_state in metrics
* wrap register_buffer with try catch
* pep8
* use loose version
* test
* pep8
2020-10-16 14:36:03 -04:00
Akihiro Nitta
b45b57cc58
Use `Optional` for arguments set to `None` by default ( #4164 )
...
* Use `Optional` for variables set to `None` by default
* Use `Optional` instead of `Union[None, ...]` for consistency
2020-10-15 23:02:50 +02:00
William Falcon
09c2020a93
notices ( #4118 )
2020-10-13 07:18:07 -04:00
Ananya Harsh Jha
ae8772490d
classification metrics ( #4043 )
...
* docs + precision + recall + f_beta + refactor
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* rebase
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* fixes
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* added missing file
* docs
* docs
* extra import
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-10 12:31:00 -04:00
Nicki Skafte
8a3c800641
Update explained variance metric ( #4024 )
...
* update metrics
* pep8
* Update pytorch_lightning/metrics/regression/explained_variance.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* add typing for testing utils
* change from assert to raise exception
* add test for raised shape error
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-10-09 19:23:12 -04:00
Teddy Koker
b961e12f50
Implement Explained Variance Metric + metric fix ( #4013 )
...
* metric fix, explained variance
* one more test
* pep8
* remove comment
* fix add_state condition
Co-authored-by: ananyahjha93 <ananya@pytorchlightning.ai>
2020-10-08 22:58:33 -04:00
Nicki Skafte
342fc7d506
[skip ci] metric docs ( #3992 )
...
* update metric docs
* doc fix
* add assert desc
* update based on suggestion
2020-10-08 21:26:39 -04:00
Ananya Harsh Jha
6f1a2ce517
integrate metrics API with self.log ( #3961 )
...
* metrics integration into self.log
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* ddp and regualr test for self.log + metrics
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* pep8
* fix log tests
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* docs
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-07 22:54:32 -04:00
Jirka Borovec
616fe82c7d
latest restore func metrics ( #3949 )
...
* latest restore
* latest restore
2020-10-07 14:26:30 -04:00
Juan Pablo Carzolio
186fc130cf
Minor formatting & grammar fixes in docs [CI SKIP] ( #3952 )
...
* Minor formatting & grammar fixes in docs
* Few more tentative doc fixes
2020-10-07 14:25:52 -04:00
Jirka Borovec
b3d25262e9
restore functional metrics ( #3943 )
...
* restore functional metrics
* clean
* fix
2020-10-07 12:48:58 -04:00
Ananya Harsh Jha
4cd14c4237
fixes metrics pickle issue ( #3921 )
...
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-06 20:33:57 -04:00
Ananya Harsh Jha
f76bc5254e
revamp entire metrics ( #3868 )
...
* removed metric
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* added new metrics
Co-authored-by: Teddy Koker teddy.koker@gmail.com
* pep8
Co-authored-by: Teddy Koker teddy.koker@gmail.com
* pep8
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* docs
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* docs
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* win ddp tests skip
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* win ddp tests skip
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* win ddp tests skip
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* win ddp tests skip
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* reset in compute, cache compute
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* reduce_ops handling
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* sync -> sync_dist, type annotations
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* wip docs
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* mean squared error
* docstring
* added mean ___ error metrics
* added mean ___ error metrics
* seperated files
* accuracy doctest
* gpu fix
* remove unnecessary mixin
* metric and accuracy docstring
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* metric docs
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* pep8, changelog
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
* refactor dist utils, pep8
* refactor dist utils, pep8
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-06 17:03:24 -04:00
Nicki Skafte
fe290280be
Metric aggregation testing ( #3517 )
...
* aggregation testing
* add more tests
* mse
* more tests
* fix tests
* fix doctest
* fix codefactor
* fix import error
* fix doctest
* revert docfix
* test for model integration
* fix integration test
* added test cases
* fix rmsle
* aggregation testing
* add more tests
* mse
* more tests
* fix tests
* fix doctest
* fix codefactor
* fix import error
* fix doctest
* revert docfix
* test for model integration
* fix integration test
* fix psnr
* add warning/valueerror to embedding similarity
* fixed f scores
* disable some test
* fix tests
* fixing codefactor
* fix pep8
* changelog
* fix doctest
* cleaning test
* fix pickle error
* pickle fix
* fix pickle error
* Apply suggestions from code review
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* code cleanup + changes based on suggestions
* update based on suggestion
* update based on suggestions
* Apply suggestions from code review
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-10-01 15:37:51 +02:00
Nicki Skafte
9a7d1a1876
[metrics] Accuracy num_classes error fix ( #3764 )
...
* change accuracy error to warning
* changelog
2020-10-01 13:00:42 +02:00
Nicki Skafte
b1347c956a
[Metrics] AUROC error on multilabel + improved testing ( #3350 )
...
* error on multilabel
* fix tests
* fix pep8
* changelog
* update doc test
* fix doctest
* fix doctest
* update from suggestion
* Apply suggestions from code review
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* Update test_classification.py
* Update test_classification.py
* retrigger test
* 'pep8
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2020-09-21 11:46:48 +02:00
Adrian Wälchli
99f05ed23f
fix warnings on windows ( #3555 )
2020-09-19 18:29:06 -04:00
Yigit Ozen
67fed01a16
Fixed failed import warning for torch.distributed.group ( #3553 )
2020-09-19 18:52:30 +02:00
Abe Botros
76c4afb840
Fix IoU score for classes not present in target or pred ( #3098 )
...
* Fix IoU score for classes not present in target or pred
Fixes #3097
- Allow configurable not_present_score for IoU for classes
not present in target or pred. Defaults to 1.0.
- Also allow passing `num_classes` parameter through from iou
metric class down to its underlying functional iou
call.
* Changelog: move IoU not-present score fix to [unreleased]
* IoU: avoid recomputing class presence in target and pred
Use already-computed support, true positives, and false positives to
determine if a class is not present in either target or pred.
* Test IoU against sklearn jaccard_score
Also add TODO to test our IoU's not_present_score against sklearn's
jaccard_score's zero_division when it beecomes available.
* IoU: remove_bg -> ignore_index
Fixes #2736
- Rename IoU metric argument from `remove_bg` -> `ignore_index`.
- Accept an optional int class index to ignore, instead of a bool and
instead of always assuming the background class has index 0.
- If given, ignore the class index when computing the IoU output,
regardless of reduction method.
* Improve documentation for IoU not_present_score
* Update default IoU not_present_score to 0.0
* Add note about IoU division by zero
* Rename IoU not_present_score -> absent_score
* Update IoU absent score changelog wording
* Condense IoU absent_score argument docstring
* Remove unnecessary IoU ignore_index comment
* docstrings
* isort
* flake8
* Fix test of IoU against sklearn jaccard
Use macro instead of micro averaging in sklearn's jaccard score, to
match multi-class IoU, which conventionally takes per-class scores
before averaging.
Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
2020-09-17 10:37:49 +02:00
Nicki Skafte
28af34bc51
[Metrics] Class reduction similar to sklearn ( #3322 )
...
* new class reduce interface
* update docs
* pep8
* update_class_metrics
* fix doctest
* changelog
* fix docs
* fix codefactor
* fix codefactor
* formatting
* fix typo
* fix typo
* typo pr -> per
* update from suggestion
* fix error
* Apply suggestions from code review
* Update CHANGELOG.md
* formatting
* timeouts
* docstring formatting for reg metrics
* pep
* flake8
* revert workflow changes
* suggestions
Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
2020-09-15 14:36:14 +02:00
Justus Schock
4dc4c8cfa5
Metric aggregation ( #3321 )
...
* metric aggregation
* metric aggregation
* add at_least_1d
* fix output formatting
* add metric tests
* add missing test case
* remove reduce_op frm metric classes
* fix reduce_op stuff
* start test fixing
* fix tests due to aggregation
* fix faulty import
* Apply suggestions from code review
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
* remove reduce_op docstrings
* add compute
* remove import
* remove collection metric
* update base class
* update tests
* Update metric.py
* Update metric.py
* Apply suggestions from code review
* change default aggregate
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
2020-09-14 07:23:11 -04:00
Cookie_thief
a552d4a2d5
fix normalize mode at confusion matrix (replace nans with zeros) ( #3465 )
...
* replace nans to 0 at conf. matrix & update tests
* cm.isnan() -> torch.isnan(cm)
* fix row-wise division while normalize
* update tests
* pep8 fix
* Update tests/metrics/test_classification.py
add comment to test
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
* Update tests/metrics/functional/test_classification.py
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
* Update pytorch_lightning/metrics/functional/classification.py
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
* final update
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
2020-09-14 10:05:51 +02:00
Nicki Skafte
93cf6d0054
[Metrics] class based embedding similarity + tests ( #3358 )
...
* embedding similarity class + test
* fix tests
* fix pep8
* add docs
* noindex
* Update docs/source/metrics.rst
* Update pytorch_lightning/metrics/self_supervised.py
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
* Update pytorch_lightning/metrics/self_supervised.py
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
* suggestions
* changes to init
* move __all__
* fix imports
* Apply suggestions from code review
* assert typo
* change import
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Nicki Skafte <nugginea@gmail.com>
2020-09-11 12:11:50 +02:00
Cookie_thief
d05d4c78e1
add num_classes argument to confusion matrix ( #3450 )
...
* add num_classes arg to confusion matrix
* update ConfusionMatrix test
* final update)
2020-09-10 18:39:04 -04:00
William Falcon
227959b3df
added emb similarity ( #3349 )
2020-09-04 09:45:21 -04:00
HT Liu
d521c1b178
Fix: gather_all_tensors cross GPUs in DDP ( #3319 )
...
* Fix: gather_all_tensors cross GPUs in metrics
* add a test case for gather_all_tensors_ddp in #3253
2020-09-03 12:27:32 +02:00
Nicki Skafte
b66ce88f0d
[metrics] Renaming of precision recall metric ( #3308 )
...
* rename metrics
* update docs
2020-09-01 14:59:33 -04:00
Sordie
888340d17e
Fix RMSLE metric ( #3188 )
...
* fix rmsle
* Updated test to match rmsle fix
* Updated RMSLE example result to match functional
* chlog
* add randomized test
* fix pep8
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
2020-08-26 08:02:53 -04:00
Nicki Skafte
17d8773106
New modular metric interface ( #2528 )
...
* new base structure
* missing packages
* updated interface
* revert some changes
* fixes
* add changelog
* fix bug
* added description
* test for pickable
* fixing test
* fixing test
* fix pickle issue
* reduceop typehints back
* remove redundant module arg
* add save/load test
* add aggregate method
* text clarification
* fix doctest
* Apply suggestions from code review
* change test to results obj
* fix docs
* formatting
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
* formatting
* pep
* Update CHANGELOG.md
* suggestions
* fix tests
* fix pep8
* fix tests
Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2020-08-26 13:01:29 +02:00
Shiv Dhar
cb0c60bf7a
Fix typo ( #3174 )
2020-08-26 08:18:59 +02:00
William Falcon
f43028f3ae
added copyright notices ( #3062 )
2020-08-19 22:03:22 -04:00
Rohit Gupta
983c030326
fix reduction docstring and clean tests ( #2885 )
...
* fix reduction docstring
* Update docstring and some cleanup
* miss
* suggestion from code review
Co-authored-by: Ananya Harsh Jha <ananya@pytorchlightning.ai>
Co-authored-by: Ananya Harsh Jha <ananya@pytorchlightning.ai>
2020-08-09 06:03:24 -04:00
Krzysztof Woś
6ebe0d7266
Fix docstring ( #2884 )
...
* Fix docstring
"mean absolute loss" rather than "root mean absolute loss"
* minor docstring fix
Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
2020-08-08 11:58:18 +00:00
Younghun Roh
d9d7e91a3b
hotfix on classification metrics ( #2878 )
...
* Faster classfication stats
* Faster accuracy metric
* minor change on cls metric
* Add out-of-bound class clamping
* Add more tests and minor fixes
* Resolve code style warning
* Update for #2781
* hotfix
* Update pytorch_lightning/metrics/functional/classification.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update about conversation
* Add docstring on stat_scores_multiple_classes
* Fixing #2862
Co-authored-by: Younghun Roh <yhunroh@mindslab.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-08-08 06:01:38 -04:00
Jirka Borovec
b7d72706c3
clean imports ( #2867 )
...
* clean imports
* miss
2020-08-08 00:33:51 +02:00
Younghun Roh
ac4a215071
Faster Accuracy metric ( #2775 )
...
* Faster classfication stats
* Faster accuracy metric
* minor change on cls metric
* Add out-of-bound class clamping
* Add more tests and minor fixes
* Resolve code style warning
* Update for #2781
* hotfix
* Update pytorch_lightning/metrics/functional/classification.py
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
* Update about conversation
* Add docstring on stat_scores_multiple_classes
Co-authored-by: Younghun Roh <yhunroh@mindslab.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-08-06 11:40:35 +02:00
Jeff Yang
5bbcb8db1f
Improve SSIM ( #2833 )
...
* make ssim fast
* remove padding
* pep8
* add comments for readability
* plus -> coef
2020-08-05 13:40:11 -04:00
Nicki Skafte
e3732789d7
Add remaning sklearn metrics ( #2562 )
...
* added balanced accuracy
* added dcg score
* added mean absolute error
* added mean squared error
* fix
* added mean squared log error
* add median absolute error and r2 score
* switch arguments
* added mean poisson deviance
* add mean gamma deviance and mean tweedie deviance
* fix styling
* added explained variance score
* added cohen kappa score
* added hamming, hinge, jaccard
* fix styling
* update sklearn requirement to newer version
* update requirement
* fix doctest
* fix tests
* added balanced accuracy
* added dcg score
* added mean absolute error
* added mean squared error
* fix
* added mean squared log error
* add median absolute error and r2 score
* switch arguments
* added mean poisson deviance
* add mean gamma deviance and mean tweedie deviance
* fix styling
* added explained variance score
* added cohen kappa score
* added hamming, hinge, jaccard
* fix styling
* update sklearn requirement to newer version
* fix doctest
* fix tests
* fix doctest
* fix failing docs
* fix test
* trying to fix errors
* Apply suggestions from code review
* format
Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
2020-08-05 11:32:53 +02:00
Justus Schock
ad0f1194aa
Support Mean in DDP Sync ( #2568 )
...
* Update converters.py
* Update test_converters.py
* pep8
* pep8 tests
* Update test_datamodules.py
* Update test_converters.py
* Update converters.py
* Update test_datamodules.py
* Update test_converters.py
* Update test_converters.py
* fix tests
* fix ddp tests on windows
* chlog
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
2020-08-04 18:32:20 +02:00
Santiago Castro
471f2b80af
Fix import deprecation warning ( #2800 )
2020-08-02 15:20:11 -04:00
pwwang
c600ca65ae
Fix false num_classes warning in metrics ( #2781 )
...
* Fix num_classes warning
Put to_categorical before get_num_classes in metrics/functional/classification.py
* Update classification.py
Remove whitespaces in blank line.
2020-08-01 23:24:19 -04:00
Jeff Yang
bda7cf1653
metrics: add SSIM ( #2671 )
...
* metrics: add SSIM
* Update CHANGELOG.md
fix codefactor issue
fix doctest
fix doctest
fix test
* added test for raise Error
2020-07-23 12:13:52 -04:00
Jeff Yang
0a65826462
metrics: add BLEU ( #2535 )
...
* metrics: added bleu score and test bleu
* metrics: fixed type hints in bleu
* bleu score moved to metrics/functional/nlp.py
* refactor with torch.Tensor
* Update test_sequence.py
* refactor as Borda requests and nltk==3.2
* locked nltk==3.3
* nltk>=3.3, parametrized smooth argument for test
* fix bleu_score example
* added class BLEUScore metrics and test
* added class BLEUScore metrics and test
* update CHANGELOG
* refactor with torchtext
* torchtext changed to optional import
* fix E501 line too long
* add else: in optional import
* remove pragma: no-cover
* constants changed to CAPITALS
* remove class in tests
* List -> Sequence, conda -> pip, cast with tensor
* add torchtext in test.txt
* remove torchtext from test.txt
* bump torchtext to 0.5.0
* bump torchtext to 0.5.0
* Apply suggestions from code review
* ignore bleu score in doctest, renamed to nlp.py
* back to implementation with torch
* remove --ignore in CI test, proper reference format
* apply justus comment
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-07-22 09:58:24 -04:00
Adrian Wälchli
a5538af355
fix dtype/device property not getting updated in submodules ( #2657 )
...
* recursive dtype device apply
* simplify
* simple test
* submodule test
* rename
* explicit
* type hints
* test for dp backend
* fix test skip
* rename
* add ddp_spawn test
* fix None index in test
* try fix ddp_spawn test
* changelog
* move _dtype and _device to mixin
* additional doctest
2020-07-21 15:18:57 -04:00
Jirka Borovec
458bbad550
Avoid zeros in dice and iou ( #2567 )
...
* nones
* fix
* fix
* test
* test
* test
* fix
* eps
* tpu
* eps
* type
* test tpu
* Update __init__.py
Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-07-09 20:40:10 -04:00
Rohit Gupta
6f4a488bae
Add functional regression metrics ( #2492 )
...
* Add functional regression metrics
* add functional tests
* add docs
* changelog
* init
* pep8
* docs
* docs
* setup docs
* docs
* Apply suggestions from code review
* Apply suggestions from code review
* typo
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
2020-07-09 17:54:38 +02:00