Rooba
/
lightning
mirror of https://github.com/Lightning-AI/lightning.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Use fsspec in load to resolve more paths/URLs from storage backends (#3692) 

* special case http for torch hub load

* Update CHANGELOG.md

* Update test.txt
This commit is contained in:
ananthsub 2020-10-03 10:29:03 -07:00 committed by GitHub
parent 0fb8c54fda
commit 8dd37e7c4a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 7 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CHANGELOG.md
View File

@ -51,6 +51,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
- Swap `torch.load` for `fsspec` load in DDP spawn backend ([#3787](https://github.com/PyTorchLightning/pytorch-lightning/pull/3787)) - Swap `torch.load` for `fsspec` load in DDP spawn backend ([#3787](https://github.com/PyTorchLightning/pytorch-lightning/pull/3787))
- Swap `torch.load` for `fsspec` load in cloud_io loading ([#3692](https://github.com/PyTorchLightning/pytorch-lightning/pull/3692))
### Deprecated ### Deprecated

8
pytorch_lightning/utilities/cloud_io.py
View File

@ -25,9 +25,11 @@ pathlike = Union[Path, str]
def load(path_or_url: str, map_location=None): def load(path_or_url: str, map_location=None):
if urlparse(path_or_url).scheme == "" or Path(path_or_url).drive: # no scheme or with a drive letter if path_or_url.startswith("http"):
return torch.load(path_or_url, map_location=map_location) return torch.hub.load_state_dict_from_url(path_or_url, map_location=map_location)
return torch.hub.load_state_dict_from_url(path_or_url, map_location=map_location) fs = get_filesystem(path_or_url)
with fs.open(path_or_url, "rb") as f:
return torch.load(f, map_location=map_location)
def get_filesystem(path: pathlike): def get_filesystem(path: pathlike):