Commit Graph

67 Commits

Author SHA1 Message Date
Dusan Drevicky 96785b99df
Feature/4244 iou input expectations (#4261)
* =Add iou input checks

* =Add test for iou input checks

* =Update docstring for iou pred and target
2020-10-21 09:01:24 -04:00
Akihiro Nitta d27ee8b5bf
docs: Add empty lines in docstring [ci skip] (#4232)
* Add empty lines in docstring for proper docs

* Remove Returns:

* Remove unnecessary Returns:

* Update pytorch_lightning/accelerators/ddp2_accelerator.py

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

* fix returns

Co-authored-by: William Falcon <waf2107@columbia.edu>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2020-10-21 09:00:39 -04:00
Dusan Drevicky 73863e9102
Fix unintended zero-only target in iou example (#4262) 2020-10-21 11:18:25 +02:00
Abhinav Gupta 5d1583d7f2
Corrected f_beta computation (#4183)
* Update f_beta.py

Added METRIC_EPS in the denominator to avoid nan values in f_beta score.

* Update f_beta.py

Made changes flake8 compliant

* Update f_beta.py

Makes use of class_reduce for macro f_beta computation to avoid nans

* Update f_beta.py

Made flake8 compliant

* Corrected F beta computation

* Removed offset to make the computation precise
2020-10-21 10:52:04 +02:00
Teddy Koker 827a557269
Add persistent flag to Metric.add_state (#4195)
* add persistant flag to add_state in metrics

* wrap register_buffer with try catch

* pep8

* use loose version

* test

* pep8
2020-10-16 14:36:03 -04:00
Akihiro Nitta b45b57cc58
Use `Optional` for arguments set to `None` by default (#4164)
* Use `Optional` for variables set to `None` by default

* Use `Optional` instead of `Union[None, ...]` for consistency
2020-10-15 23:02:50 +02:00
William Falcon 09c2020a93
notices (#4118) 2020-10-13 07:18:07 -04:00
Ananya Harsh Jha ae8772490d
classification metrics (#4043)
* docs + precision + recall + f_beta + refactor

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* rebase

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* fixes

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* added missing file

* docs

* docs

* extra import

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-10 12:31:00 -04:00
Nicki Skafte 8a3c800641
Update explained variance metric (#4024)
* update metrics

* pep8

* Update pytorch_lightning/metrics/regression/explained_variance.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* add typing for testing utils

* change from assert to raise exception

* add test for raised shape error

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-10-09 19:23:12 -04:00
Teddy Koker b961e12f50
Implement Explained Variance Metric + metric fix (#4013)
* metric fix, explained variance

* one more test

* pep8

* remove comment

* fix add_state condition

Co-authored-by: ananyahjha93 <ananya@pytorchlightning.ai>
2020-10-08 22:58:33 -04:00
Nicki Skafte 342fc7d506
[skip ci] metric docs (#3992)
* update metric docs

* doc fix

* add assert desc

* update based on suggestion
2020-10-08 21:26:39 -04:00
Ananya Harsh Jha 6f1a2ce517
integrate metrics API with self.log (#3961)
* metrics integration into self.log

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* ddp and regualr test for self.log + metrics

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* pep8

* fix log tests

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* docs

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-07 22:54:32 -04:00
Jirka Borovec 616fe82c7d
latest restore func metrics (#3949)
* latest restore

* latest restore
2020-10-07 14:26:30 -04:00
Juan Pablo Carzolio 186fc130cf
Minor formatting & grammar fixes in docs [CI SKIP] (#3952)
* Minor formatting & grammar fixes in docs

* Few more tentative doc fixes
2020-10-07 14:25:52 -04:00
Jirka Borovec b3d25262e9
restore functional metrics (#3943)
* restore functional metrics

* clean

* fix
2020-10-07 12:48:58 -04:00
Ananya Harsh Jha 4cd14c4237
fixes metrics pickle issue (#3921)
Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-06 20:33:57 -04:00
Ananya Harsh Jha f76bc5254e
revamp entire metrics (#3868)
* removed metric

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* added new metrics

Co-authored-by: Teddy Koker teddy.koker@gmail.com

* pep8

Co-authored-by: Teddy Koker teddy.koker@gmail.com

* pep8

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* docs

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* docs

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* win ddp tests skip

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* win ddp tests skip

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* win ddp tests skip

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* win ddp tests skip

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* reset in compute, cache compute

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* reduce_ops handling

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* sync -> sync_dist, type annotations

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* wip docs

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* mean squared error

* docstring

* added mean ___ error metrics

* added mean ___ error metrics

* seperated files

* accuracy doctest

* gpu fix

* remove unnecessary mixin

* metric and accuracy docstring

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* metric docs

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* pep8, changelog

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>

* refactor dist utils, pep8

* refactor dist utils, pep8

Co-authored-by: Teddy Koker <teddy.koker@gmail.com>
2020-10-06 17:03:24 -04:00
Nicki Skafte fe290280be
Metric aggregation testing (#3517)
* aggregation testing

* add more tests

* mse

* more tests

* fix tests

* fix doctest

* fix codefactor

* fix import error

* fix doctest

* revert docfix

* test for model integration

* fix integration test

* added test cases

* fix rmsle

* aggregation testing

* add more tests

* mse

* more tests

* fix tests

* fix doctest

* fix codefactor

* fix import error

* fix doctest

* revert docfix

* test for model integration

* fix integration test

* fix psnr

* add warning/valueerror to embedding similarity

* fixed f scores

* disable some test

* fix tests

* fixing codefactor

* fix pep8

* changelog

* fix doctest

* cleaning test

* fix pickle error

* pickle fix

* fix pickle error

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

* code cleanup + changes based on suggestions

* update based on suggestion

* update based on suggestions

* Apply suggestions from code review

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-10-01 15:37:51 +02:00
Nicki Skafte 9a7d1a1876
[metrics] Accuracy num_classes error fix (#3764)
* change accuracy error to warning

* changelog
2020-10-01 13:00:42 +02:00
Nicki Skafte b1347c956a
[Metrics] AUROC error on multilabel + improved testing (#3350)
* error on multilabel

* fix tests

* fix pep8

* changelog

* update doc test

* fix doctest

* fix doctest

* update from suggestion

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

* Update test_classification.py

* Update test_classification.py

* retrigger test

* 'pep8

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2020-09-21 11:46:48 +02:00
Adrian Wälchli 99f05ed23f
fix warnings on windows (#3555) 2020-09-19 18:29:06 -04:00
Yigit Ozen 67fed01a16
Fixed failed import warning for torch.distributed.group (#3553) 2020-09-19 18:52:30 +02:00
Abe Botros 76c4afb840
Fix IoU score for classes not present in target or pred (#3098)
* Fix IoU score for classes not present in target or pred

Fixes #3097

- Allow configurable not_present_score for IoU for classes
  not present in target or pred. Defaults to 1.0.
- Also allow passing `num_classes` parameter through from iou
  metric class down to its underlying functional iou
  call.

* Changelog: move IoU not-present score fix to [unreleased]

* IoU: avoid recomputing class presence in target and pred

Use already-computed support, true positives, and false positives to
determine if a class is not present in either target or pred.

* Test IoU against sklearn jaccard_score

Also add TODO to test our IoU's not_present_score against sklearn's
jaccard_score's zero_division when it beecomes available.

* IoU: remove_bg -> ignore_index

Fixes #2736

- Rename IoU metric argument from `remove_bg` -> `ignore_index`.
- Accept an optional int class index to ignore, instead of a bool and
  instead of always assuming the background class has index 0.
- If given, ignore the class index when computing the IoU output,
  regardless of reduction method.

* Improve documentation for IoU not_present_score

* Update default IoU not_present_score to 0.0

* Add note about IoU division by zero

* Rename IoU not_present_score -> absent_score

* Update IoU absent score changelog wording

* Condense IoU absent_score argument docstring

* Remove unnecessary IoU ignore_index comment

* docstrings

* isort

* flake8

* Fix test of IoU against sklearn jaccard

Use macro instead of micro averaging in sklearn's jaccard score, to
match multi-class IoU, which conventionally takes per-class scores
before averaging.

Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
2020-09-17 10:37:49 +02:00
Nicki Skafte 28af34bc51
[Metrics] Class reduction similar to sklearn (#3322)
* new class reduce interface

* update docs

* pep8

* update_class_metrics

* fix doctest

* changelog

* fix docs

* fix codefactor

* fix codefactor

* formatting

* fix typo

* fix typo

* typo pr -> per

* update from suggestion

* fix error

* Apply suggestions from code review

* Update CHANGELOG.md

* formatting

* timeouts

* docstring formatting for reg metrics

* pep

* flake8

* revert workflow changes

* suggestions

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
2020-09-15 14:36:14 +02:00
Justus Schock 4dc4c8cfa5
Metric aggregation (#3321)
* metric aggregation

* metric aggregation

* add at_least_1d

* fix output formatting

* add metric tests

* add missing test case

* remove reduce_op frm metric classes

* fix reduce_op stuff

* start test fixing

* fix tests due to aggregation

* fix faulty import

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

* remove reduce_op docstrings

* add compute

* remove import

* remove collection metric

* update base class

* update tests

* Update metric.py

* Update metric.py

* Apply suggestions from code review

* change default aggregate

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
2020-09-14 07:23:11 -04:00
Cookie_thief a552d4a2d5
fix normalize mode at confusion matrix (replace nans with zeros) (#3465)
* replace nans to 0 at conf. matrix & update tests

* cm.isnan() -> torch.isnan(cm)

* fix row-wise division while normalize

* update tests

* pep8 fix

* Update tests/metrics/test_classification.py

add comment to test

Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>

* Update tests/metrics/functional/test_classification.py

Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>

* Update pytorch_lightning/metrics/functional/classification.py

Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>

* final update

Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
2020-09-14 10:05:51 +02:00
Nicki Skafte 93cf6d0054
[Metrics] class based embedding similarity + tests (#3358)
* embedding similarity class + test

* fix tests

* fix pep8

* add docs

* noindex

* Update docs/source/metrics.rst

* Update pytorch_lightning/metrics/self_supervised.py

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

* Update pytorch_lightning/metrics/self_supervised.py

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

* suggestions

* changes to init

* move __all__

* fix imports

* Apply suggestions from code review

* assert typo

* change import

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Nicki Skafte <nugginea@gmail.com>
2020-09-11 12:11:50 +02:00
Cookie_thief d05d4c78e1
add num_classes argument to confusion matrix (#3450)
* add num_classes arg to confusion matrix

* update ConfusionMatrix test

* final update)
2020-09-10 18:39:04 -04:00
William Falcon 227959b3df
added emb similarity (#3349) 2020-09-04 09:45:21 -04:00
HT Liu d521c1b178
Fix: gather_all_tensors cross GPUs in DDP (#3319)
* Fix: gather_all_tensors cross GPUs in metrics

* add a test case for gather_all_tensors_ddp in #3253
2020-09-03 12:27:32 +02:00
Nicki Skafte b66ce88f0d
[metrics] Renaming of precision recall metric (#3308)
* rename metrics

* update docs
2020-09-01 14:59:33 -04:00
Sordie 888340d17e
Fix RMSLE metric (#3188)
* fix rmsle

* Updated test to match rmsle fix

* Updated RMSLE example result to match functional

* chlog

* add randomized test

* fix pep8

Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
2020-08-26 08:02:53 -04:00
Nicki Skafte 17d8773106
New modular metric interface (#2528)
* new base structure

* missing packages

* updated interface

* revert some changes

* fixes

* add changelog

* fix bug

* added description

* test for pickable

* fixing test

* fixing test

* fix pickle issue

* reduceop typehints back

* remove redundant module arg

* add save/load test

* add aggregate method

* text clarification

* fix doctest

* Apply suggestions from code review

* change test to results obj

* fix docs

* formatting

Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

* formatting

* pep

* Update CHANGELOG.md

* suggestions

* fix tests

* fix pep8

* fix tests

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
2020-08-26 13:01:29 +02:00
Shiv Dhar cb0c60bf7a
Fix typo (#3174) 2020-08-26 08:18:59 +02:00
William Falcon f43028f3ae
added copyright notices (#3062) 2020-08-19 22:03:22 -04:00
Rohit Gupta 983c030326
fix reduction docstring and clean tests (#2885)
* fix reduction docstring

* Update docstring and some cleanup

* miss

* suggestion from code review

Co-authored-by: Ananya Harsh Jha <ananya@pytorchlightning.ai>

Co-authored-by: Ananya Harsh Jha <ananya@pytorchlightning.ai>
2020-08-09 06:03:24 -04:00
Krzysztof Woś 6ebe0d7266
Fix docstring (#2884)
* Fix docstring

"mean absolute loss" rather than "root mean absolute loss"

* minor docstring fix

Co-authored-by: rohitgr7 <rohitgr1998@gmail.com>
2020-08-08 11:58:18 +00:00
Younghun Roh d9d7e91a3b
hotfix on classification metrics (#2878)
* Faster classfication stats

* Faster accuracy metric

* minor change on cls metric

* Add out-of-bound class clamping

* Add more tests and minor fixes

* Resolve code style warning

* Update for #2781

* hotfix

* Update pytorch_lightning/metrics/functional/classification.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Update about conversation

* Add docstring on stat_scores_multiple_classes

* Fixing #2862

Co-authored-by: Younghun Roh <yhunroh@mindslab.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-08-08 06:01:38 -04:00
Jirka Borovec b7d72706c3
clean imports (#2867)
* clean imports

* miss
2020-08-08 00:33:51 +02:00
Younghun Roh ac4a215071
Faster Accuracy metric (#2775)
* Faster classfication stats

* Faster accuracy metric

* minor change on cls metric

* Add out-of-bound class clamping

* Add more tests and minor fixes

* Resolve code style warning

* Update for #2781

* hotfix

* Update pytorch_lightning/metrics/functional/classification.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Update about conversation

* Add docstring on stat_scores_multiple_classes

Co-authored-by: Younghun Roh <yhunroh@mindslab.ai>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-08-06 11:40:35 +02:00
Jeff Yang 5bbcb8db1f
Improve SSIM (#2833)
* make ssim fast

* remove padding

* pep8

* add comments for readability

* plus -> coef
2020-08-05 13:40:11 -04:00
Nicki Skafte e3732789d7
Add remaning sklearn metrics (#2562)
* added balanced accuracy

* added dcg score

* added mean absolute error

* added mean squared error

* fix

* added mean squared log error

* add median absolute error and r2 score

* switch arguments

* added mean poisson deviance

* add mean gamma deviance and mean tweedie deviance

* fix styling

* added explained variance score

* added cohen kappa score

* added hamming, hinge, jaccard

* fix styling

* update sklearn requirement to newer version

* update requirement

* fix doctest

* fix tests

* added balanced accuracy

* added dcg score

* added mean absolute error

* added mean squared error

* fix

* added mean squared log error

* add median absolute error and r2 score

* switch arguments

* added mean poisson deviance

* add mean gamma deviance and mean tweedie deviance

* fix styling

* added explained variance score

* added cohen kappa score

* added hamming, hinge, jaccard

* fix styling

* update sklearn requirement to newer version

* fix doctest

* fix tests

* fix doctest

* fix failing docs

* fix test

* trying to fix errors

* Apply suggestions from code review

* format

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
2020-08-05 11:32:53 +02:00
Justus Schock ad0f1194aa
Support Mean in DDP Sync (#2568)
* Update converters.py

* Update test_converters.py

* pep8

* pep8 tests

* Update test_datamodules.py

* Update test_converters.py

* Update converters.py

* Update test_datamodules.py

* Update test_converters.py

* Update test_converters.py

* fix tests

* fix ddp tests on windows

* chlog

Co-authored-by: Jirka Borovec <jirka@pytorchlightning.ai>
2020-08-04 18:32:20 +02:00
Santiago Castro 471f2b80af
Fix import deprecation warning (#2800) 2020-08-02 15:20:11 -04:00
pwwang c600ca65ae
Fix false num_classes warning in metrics (#2781)
* Fix num_classes warning

Put to_categorical before get_num_classes in metrics/functional/classification.py

* Update classification.py

Remove whitespaces in blank line.
2020-08-01 23:24:19 -04:00
Jeff Yang bda7cf1653
metrics: add SSIM (#2671)
* metrics: add SSIM

* Update CHANGELOG.md

fix codefactor issue

fix doctest

fix doctest

fix test

* added test for raise Error
2020-07-23 12:13:52 -04:00
Jeff Yang 0a65826462
metrics: add BLEU (#2535)
* metrics: added bleu score and test bleu

* metrics: fixed type hints in bleu

* bleu score moved to metrics/functional/nlp.py

* refactor with torch.Tensor

* Update test_sequence.py

* refactor as Borda requests and nltk==3.2

* locked nltk==3.3

* nltk>=3.3, parametrized smooth argument for test

* fix bleu_score example

* added class BLEUScore metrics and test

* added class BLEUScore metrics and test

* update CHANGELOG

* refactor with torchtext

* torchtext changed to optional import

* fix E501 line too long

* add else: in optional import

* remove pragma: no-cover

* constants changed to CAPITALS

* remove class in tests

* List -> Sequence, conda -> pip, cast with tensor

* add torchtext in test.txt

* remove torchtext from test.txt

* bump torchtext to 0.5.0

* bump torchtext to 0.5.0

* Apply suggestions from code review

* ignore bleu score in doctest, renamed to nlp.py

* back to implementation with torch

* remove --ignore in CI test, proper reference format

* apply justus comment

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-07-22 09:58:24 -04:00
Adrian Wälchli a5538af355
fix dtype/device property not getting updated in submodules (#2657)
* recursive dtype device apply

* simplify

* simple test

* submodule test

* rename

* explicit

* type hints

* test for dp backend

* fix test skip

* rename

* add ddp_spawn test

* fix None index in test

* try fix ddp_spawn test

* changelog

* move _dtype and _device to mixin

* additional doctest
2020-07-21 15:18:57 -04:00
Jirka Borovec 458bbad550
Avoid zeros in dice and iou (#2567)
* nones

* fix

* fix

* test

* test

* test

* fix

* eps

* tpu

* eps

* type

* test tpu

* Update __init__.py

Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-07-09 20:40:10 -04:00
Rohit Gupta 6f4a488bae
Add functional regression metrics (#2492)
* Add functional regression metrics

* add functional tests

* add docs

* changelog

* init

* pep8

* docs

* docs

* setup docs

* docs

* Apply suggestions from code review

* Apply suggestions from code review

* typo

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
2020-07-09 17:54:38 +02:00