riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity

Allow ray datasets to be used with trainer (#36699)

Browse Source 
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
This commit is contained in:
Fredrik Norén 2025-03-17 15:44:47 +01:00 committed by GitHub
parent 2256875a77
commit da7d64f4ff
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
src/transformers/trainer_utils.py
View File

@ -762,6 +762,9 @@ def has_length(dataset):
except TypeError:
# TypeError: len() of unsized object
return False
except AttributeError:
# Ray DataSets raises an AttributeError: https://github.com/ray-project/ray/blob/master/python/ray/data/dataset.py#L5616
return False
def denumpify_detensorize(metrics):