U Ph4@sddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddlm Z m Z ddlmZmZddlmZddlmZddlmZddlmZmZmZmZddlZddlZdd lmZdd l m!Z!dd l"m#Z$dd l"m%Z%dd l&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;mne:dde9d\Z=Z>e:d\Z?Z@e:d\ZAZ@e:d\ZBZ@e:d\ZCZ@Gddde$Z#Gddde#ZDGddde#ZEGd d!d!eEZFGd"d#d#eEZGGd$d%d%e#ZHGd&d'd'e.eHZIGd(d)d)e#ZJGd*d+d+e.e$ZKGd,d-d-e#ZLGd.d/d/e#ZMGd0d1d1eEZNdS)2) annotationsN)CallableSequence)copydeepcopy) ListProxy) ThreadPool)Path)IO TYPE_CHECKINGAnycast)Manager)DEFAULT_PROTOCOLDataset)Subset) MetaTensor)SUPPORTED_PICKLE_MODconvert_tables_to_dictspickle_hashing)Compose RandomizableRandomizableTrait Transformapply_transformconvert_to_contiguous reset_ops_id)MAX_SEEDconvert_to_tensorget_seedlook_up_option min_versionoptional_import)first)tqdmTr%z4.47.0cupylmdbpandasz kvikio.numpyc@sNeZdZdZdddddddZd d d d Zd d ddZdd ddZdS)ra. A generic dataset with a length property and an optional callable data transform when fetching a data sample. If passing slicing indices, will return a PyTorch Subset, for example: `data: Subset = dataset[1:4]`, for more details, please check: https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset For example, typical input data can be a list of dictionaries:: [{ { { 'img': 'image1.nii.gz', 'img': 'image2.nii.gz', 'img': 'image3.nii.gz', 'seg': 'label1.nii.gz', 'seg': 'label2.nii.gz', 'seg': 'label3.nii.gz', 'extra': 123 'extra': 456 'extra': 789 }, }, }] NrCallable | NoneNone)data transformreturncCs||_||_dS)z Args: data: input data to load and transform to generate dataset for model. transform: a callable data transform on input data. Nr+r,)selfr+r,r0G/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/data/dataset.py__init__PszDataset.__init__intr-cCs t|jSN)lenr+r/r0r0r1__len__ZszDataset.__len__indexcCs$|j|}|jdk r t|j|S|S)z: Fetch single data item from `self.data`. N)r+r,r)r/r:Zdata_ir0r0r1 _transform]s zDataset._transformzint | slice | Sequence[int]cCsZt|tr6|t|\}}}t|||}t||dSt|tjjrPt||dS| |S)z^ Returns a `Subset` if `index` is a slice or Sequence, a data item otherwise. )datasetindices) isinstanceslicer=r6ranger collectionsabcrr;)r/r:startstopstepr=r0r0r1 __getitem__ds    zDataset.__getitem__)N)__name__ __module__ __qualname____doc__r2r8r;rFr0r0r0r1r@s  rcs<eZdZdZddddfdd Zdd d d d d ZZS) DatasetFunca Execute function on the input dataset and leverage the output to act as a new Dataset. It can be used to load / fetch the basic dataset items, like the list of `image, label` paths. Or chain together to execute more complicated logic, like `partition_dataset`, `resample_datalist`, etc. The `data` arg of `Dataset` will be applied to the first arg of callable `func`. Usage example:: data_list = DatasetFunc( data="path to file", func=monai.data.load_decathlon_datalist, data_list_key="validation", base_dir="path to base dir", ) # partition dataset for every rank data_partition = DatasetFunc( data=data_list, func=lambda **kwargs: monai.data.partition_dataset(**kwargs)[torch.distributed.get_rank()], num_partitions=torch.distributed.get_world_size(), ) dataset = Dataset(data=data_partition, transform=transforms) Args: data: input data for the func to process, will apply to `func` as the first arg. func: callable function to generate dataset items. kwargs: other arguments for the `func` except for the first arg. r rr*)r+funcr-c s.tjddd||_||_||_|dS)Nr.)superr2srcrLkwargsreset)r/r+rLrO __class__r0r1r2s zDatasetFunc.__init__N Any | Noner))r+rLcKs<|dkr|jn|}|dkr*|j|f|jn ||f||_dS)aL Reset the dataset items with specified `func`. Args: data: if not None, execute `func` on it, default to `self.src`. func: if not None, execute the `func` with specified `kwargs`, default to `self.func`. kwargs: other arguments for the `func` except for the first arg. N)rNrLrOr+)r/r+rLrOrNr0r0r1rPs zDatasetFunc.reset)NN)rGrHrIrJr2rP __classcell__r0r0rQr1rKssrKc seZdZdZededdfddddd d d d d d fdd ZddddZddddZddZ ddZ ddZ d dddZ Z S) PersistentDataseta Persistent storage of pre-computed values to efficiently manage larger than memory dictionary format data, it can operate transforms for specific fields. Results from the non-random transform components are computed when first used, and stored in the `cache_dir` for rapid retrieval on subsequent uses. If passing slicing indices, will return a PyTorch Subset, for example: `data: Subset = dataset[1:4]`, for more details, please check: https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset The transforms which are supposed to be cached must implement the `monai.transforms.Transform` interface and should not be `Randomizable`. This dataset will cache the outcomes before the first `Randomizable` `Transform` within a `Compose` instance. For example, typical input data can be a list of dictionaries:: [{ { { 'image': 'image1.nii.gz', 'image': 'image2.nii.gz', 'image': 'image3.nii.gz', 'label': 'label1.nii.gz', 'label': 'label2.nii.gz', 'label': 'label3.nii.gz', 'extra': 123 'extra': 456 'extra': 789 }, }, }] For a composite transform like .. code-block:: python [ LoadImaged(keys=['image', 'label']), Orientationd(keys=['image', 'label'], axcodes='RAS'), ScaleIntensityRanged(keys=['image'], a_min=-57, a_max=164, b_min=0.0, b_max=1.0, clip=True), RandCropByPosNegLabeld(keys=['image', 'label'], label_key='label', spatial_size=(96, 96, 96), pos=1, neg=1, num_samples=4, image_key='image', image_threshold=0), ToTensord(keys=['image', 'label'])] Upon first use a filename based dataset will be processed by the transform for the [LoadImaged, Orientationd, ScaleIntensityRanged] and the resulting tensor written to the `cache_dir` before applying the remaining random dependant transforms [RandCropByPosNegLabeld, ToTensord] elements for use in the analysis. Subsequent uses of a dataset directly read pre-processed results from `cache_dir` followed by applying the random dependant parts of transform processing. During training call `set_data()` to update input data and recompute cache content. Note: The input data must be a list of file paths and will hash them as cache keys. The filenames of the cached files also try to contain the hash of the transforms. In this fashion, `PersistentDataset` should be robust to changes in transforms. This, however, is not guaranteed, so caution should be used when modifying transforms to avoid unexpected errors. If in doubt, it is advisable to clear the cache directory. Lazy Resampling: If you make use of the lazy resampling feature of `monai.transforms.Compose`, please refer to its documentation to familiarize yourself with the interaction between `PersistentDataset` and lazy resampling. pickleNTrSequence[Callable] | CallablePath | str | NoneCallable[..., bytes]strr3Callable[..., bytes] | Noneboolr*) r+r, cache_dir hash_func pickle_modulepickle_protocolhash_transformrr-c st|tst|}tj||d|dk r2t|nd|_||_||_||_|jdk r|j sn|jj ddd|j st dd|_ |dk r||||_dS)a Args: data: input data file paths to load and transform to generate dataset for model. `PersistentDataset` expects input data to be a list of serializable and hashes them as cache keys using `hash_func`. transform: transforms to execute operations on input data. cache_dir: If specified, this is the location for persistent storage of pre-computed transformed data tensors. The cache_dir is computed once, and persists on disk until explicitly removed. Different runs, programs, experiments may share a common cache dir provided that the transforms pre-processing is consistent. If `cache_dir` doesn't exist, will automatically create it. If `cache_dir` is `None`, there is effectively no caching. hash_func: a callable to compute hash from data items to be cached. defaults to `monai.data.utils.pickle_hashing`. pickle_module: string representing the module used for pickling metadata and objects, default to `"pickle"`. due to the pickle limitation in multi-processing of Dataloader, we can't use `pickle` as arg directly, so here we use a string name instead. if want to use other pickle module at runtime, just register like: >>> from monai.data import utils >>> utils.SUPPORTED_PICKLE_MOD["test"] = other_pickle this arg is used by `torch.save`, for more details, please check: https://pytorch.org/docs/stable/generated/torch.save.html#torch.save, and ``monai.data.utils.SUPPORTED_PICKLE_MOD``. pickle_protocol: can be specified to override the default protocol, default to `2`. this arg is used by `torch.save`, for more details, please check: https://pytorch.org/docs/stable/generated/torch.save.html#torch.save. hash_transform: a callable to compute hash from the transform information when caching. This may reduce errors due to transforms changing during experiments. Default to None (no hash). Other options are `pickle_hashing` and `json_hashing` functions from `monai.data.utils`. reset_ops_id: whether to set `TraceKeys.ID` to ``Tracekys.NONE``, defaults to ``True``. When this is enabled, the traced transform instance IDs will be removed from the cached MetaTensors. This is useful for skipping the transform instance checks when inverting applied operations using the cached content and with re-created transform instances. r.NTparentsexist_okzcache_dir must be a directory.)r>rrMr2r r]r^r_r`existsmkdiris_dir ValueErrortransform_hashset_transform_hashr) r/r+r,r]r^r_r`rarrQr0r1r2s .     zPersistentDataset.__init__)hash_xform_funcc Csg}|jjD]&}t|ts(t|ts,q8||qz ||}WnNtk r}z0dt|krf|d dd|D}||}W5d}~XYnX| d|_ dS)aGet hashable transforms, and then hash them. Hashable transforms are deterministic transforms that inherit from `Transform`. We stop at the first non-deterministic transform, or first that does not inherit from MONAI's `Transform` class.zis not JSON serializablerecss|]}|jjVqdSr5)rRrG).0trr0r0r1 ,sz7PersistentDataset.set_transform_hash..Nutf-8) r,flatten transformsr>rrappend TypeErrorrZjoindecoderj)r/rlZhashable_transformsZ_trrjtenamesr0r0r1rks   z$PersistentDataset.set_transform_hashr+cCs>||_|jdk r:|jr:tj|jdd|jjddddS)Q Set the input data and delete all the out-dated cache content. NT) ignore_errorsrb)r+r]rfshutilrmtreergr/r+r0r0r1set_data0szPersistentDataset.set_datacCsFt|jtstd|jdd}|j||dd}|jrBt||S)a Process the data from original state up to the first random element. Args: item_transformed: The data to be transformed Returns: the transformed element up to the first identified random transform object :transform must be an instance of monai.transforms.Compose.cSst|tpt|t Sr5r>rrtr0r0r1Jz2PersistentDataset._pre_transform..Tend threading)r>r,rriget_index_of_firstrr/item_transformed first_randomr0r0r1_pre_transform:s z PersistentDataset._pre_transformcCs>t|jtstd|jdd}|dk r:|j||d}|S)aD Process the data from before the first random transform to the final state ready for evaluation. Args: item_transformed: The data to be transformed (already processed up to the first random transform) Returns: the transformed element through the random transforms rcSst|tpt|t Sr5rrr0r0r1rarz3PersistentDataset._post_transform..NrC)r>r,rrirrr0r0r1_post_transformRs z!PersistentDataset._post_transformc Csd}|jdk r8||d}||j7}|j|d}|dk r|rz t|WStk r}ztj dkrt|W5d}~XYnNt k r}z0dt |krt d|d|n|W5d}~XYnX|t|}|dkr|Sztp}t||j}tj||t|jt|jd|rb|sbztt ||Wntk r`YnXW5QRXWntk rYnX|S) a A function to cache the expensive input data transform operations so that huge data sets (larger than computer memory) can be processed on the fly as needed, and intermediate results written to disk for future use. Args: item_transformed: The current data element to be mutated into transformed representation Returns: The transformed data_element, either from cache, or explicitly computing it. Warning: The current implementation does not encode transform information as part of the hashing mechanism used for generating cache names when `hash_transform` is None. If the transforms applied are changed in any way, the objects in the cache dir will be invalid. Nrp.ptwin32z"Invalid magic number; corrupt filezCorrupt cache file detected: z. Deleting and recomputing.objfr_r`)r]r^rvrjis_filetorchloadPermissionErrorsysplatform RuntimeErrorrZwarningswarnunlinkrrtempfileTemporaryDirectoryr namesaver!r_rr`r|moveFileExistsError)r/rhashfile data_item_md5e_item_transformed tmpdirnametemp_hash_filer0r0r1 _cachecheckgsH        zPersistentDataset._cachecheckr9cCs||j|}||Sr5)rr+r)r/r:Zpre_random_itemr0r0r1r;szPersistentDataset._transform)rGrHrIrJrrr2rkrrrrr;rTr0r0rQr1rUs<$? @rUc sTeZdZdZededdfddddd d dd d d d fdd ZddZddZZ S)CacheNTransDatasetz~ Extension of `PersistentDataset`, it can also cache the result of first N transforms, no matter it's random or not. rVNTrrWr3rXrYrZr[r\r*) r+r, cache_n_transr]r^r_r`rarr-c s&tj|||||||| d||_dS)a Args: data: input data file paths to load and transform to generate dataset for model. `PersistentDataset` expects input data to be a list of serializable and hashes them as cache keys using `hash_func`. transform: transforms to execute operations on input data. cache_n_trans: cache the result of first N transforms. cache_dir: If specified, this is the location for persistent storage of pre-computed transformed data tensors. The cache_dir is computed once, and persists on disk until explicitly removed. Different runs, programs, experiments may share a common cache dir provided that the transforms pre-processing is consistent. If `cache_dir` doesn't exist, will automatically create it. If `cache_dir` is `None`, there is effectively no caching. hash_func: a callable to compute hash from data items to be cached. defaults to `monai.data.utils.pickle_hashing`. pickle_module: string representing the module used for pickling metadata and objects, default to `"pickle"`. due to the pickle limitation in multi-processing of Dataloader, we can't use `pickle` as arg directly, so here we use a string name instead. if want to use other pickle module at runtime, just register like: >>> from monai.data import utils >>> utils.SUPPORTED_PICKLE_MOD["test"] = other_pickle this arg is used by `torch.save`, for more details, please check: https://pytorch.org/docs/stable/generated/torch.save.html#torch.save, and ``monai.data.utils.SUPPORTED_PICKLE_MOD``. pickle_protocol: can be specified to override the default protocol, default to `2`. this arg is used by `torch.save`, for more details, please check: https://pytorch.org/docs/stable/generated/torch.save.html#torch.save. hash_transform: a callable to compute hash from the transform information when caching. This may reduce errors due to transforms changing during experiments. Default to None (no hash). Other options are `pickle_hashing` and `json_hashing` functions from `monai.data.utils`. reset_ops_id: whether to set `TraceKeys.ID` to ``Tracekys.NONE``, defaults to ``True``. When this is enabled, the traced transform instance IDs will be removed from the cached MetaTensors. This is useful for skipping the transform instance checks when inverting applied operations using the cached content and with re-created transform instances. )r+r,r]r^r_r`rarN)rMr2r) r/r+r,rr]r^r_r`rarrQr0r1r2s0 zCacheNTransDataset.__init__cCs2t|jtstd|j||jdd}t||S)z Process the data from original state up to the N element. Args: item_transformed: The data to be transformed Returns: the transformed element up to the N transform object rTr)r>r,rrirrr/rr0r0r1rs z!CacheNTransDataset._pre_transformcCs$t|jtstd|j||jdS)a Process the data from before the N + 1 transform to the final state ready for evaluation. Args: item_transformed: The data to be transformed (already processed up to the first N transform) Returns: the final transformed result rr)r>r,rrirrr0r0r1rs z"CacheNTransDataset._post_transform) rGrHrIrJrrr2rrrTr0r0rQr1rs &<rc s|eZdZdZdeddejdddfdddd d d d d d dd fdd Zddfdd ZdddZ fddZ ddZ Z S) LMDBDataseta Extension of `PersistentDataset` using LMDB as the backend. See Also: :py:class:`monai.data.PersistentDataset` Examples: >>> items = [{"data": i} for i in range(5)] # [{'data': 0}, {'data': 1}, {'data': 2}, {'data': 3}, {'data': 4}] >>> lmdb_ds = monai.data.LMDBDataset(items, transform=monai.transforms.SimulateDelayd("data", delay_time=1)) >>> print(list(lmdb_ds)) # using the cached results cacheZ monai_cacheTNrrWz Path | strrYrZr\r[ dict | Noner*) r+r,r]r^db_nameprogressrar lmdb_kwargsr-c stj||||||| d||_|js.td|j|d|_| pFi|_|jddsbd|jd<d|_|j |jdt d |j d dS) aq Args: data: input data file paths to load and transform to generate dataset for model. `LMDBDataset` expects input data to be a list of serializable and hashes them as cache keys using `hash_func`. transform: transforms to execute operations on input data. cache_dir: if specified, this is the location for persistent storage of pre-computed transformed data tensors. The cache_dir is computed once, and persists on disk until explicitly removed. Different runs, programs, experiments may share a common cache dir provided that the transforms pre-processing is consistent. If the cache_dir doesn't exist, will automatically create it. Defaults to "./cache". hash_func: a callable to compute hash from data items to be cached. defaults to `monai.data.utils.pickle_hashing`. db_name: lmdb database file name. Defaults to "monai_cache". progress: whether to display a progress bar. pickle_protocol: pickle protocol version. Defaults to pickle.HIGHEST_PROTOCOL. https://docs.python.org/3/library/pickle.html#pickle-protocols hash_transform: a callable to compute hash from the transform information when caching. This may reduce errors due to transforms changing during experiments. Default to None (no hash). Other options are `pickle_hashing` and `json_hashing` functions from `monai.data.utils`. reset_ops_id: whether to set `TraceKeys.ID` to ``Tracekeys.NONE``, defaults to ``True``. When this is enabled, the traced transform instance IDs will be removed from the cached MetaTensors. This is useful for skipping the transform instance checks when inverting applied operations using the cached content and with re-created transform instances. lmdb_kwargs: additional keyword arguments to the lmdb environment. for more details please visit: https://lmdb.readthedocs.io/en/release/#environment-class )r+r,r]r^r`rarzcache_dir must be specified.z.lmdbmap_sizerlN show_progresszAccessing lmdb file: .) rMr2rr]ridb_filerget _read_env_fill_cache_start_readerprintabsolute) r/r+r,r]r^rrr`rarrrQr0r1r2 s&(   zLMDBDataset.__init__rycs"tj|d|j|jd|_dS)rzryrN)rMrrrrr~rQr0r1r_szLMDBDataset.set_datac CsFd|jd<tjf|jdd|j}|r8ts8td|jddz}trZ|rZt|j n|j D]X}| |}d\}}}|s|dkrzx| } | |}W5QRX|rWqz|dkr| t|}tj||jd }|jd d} | ||W5QRXd }Wqztjk rnd|d }}|d } | d } tdt| d?dt| d?d|| Yqztjk r|dYqzXqz|s`|d } |td| dq`W5QRX|d } |d |jd<| |jd <|jdddkr d|jd<|jdddkr*d|jd<tjf|jdd|jS)aF Check the LMDB cache and write the cache if needed. py-lmdb doesn't have a good support for concurrent write. This method can be used with multiple processes, but it may have a negative impact on the performance. Args: show_progress: whether to show the progress bar if possible. Freadonly)pathsubdirzHLMDBDataset: tqdm is not installed. not displaying the caching progress.write)FNrN)protocolTrz!Resizing the cache database from zMB to zMB.z;LMDB map size reached, increase size above current size of rlock readahead)rr'openrhas_tqdmrrbeginr%r+r^cursorZset_keyrrrVdumpsr`putZ MapFullErrorinfor3Z set_mapsizeZMapResizedErrorcloserir) r/renvZ search_txnitemkeydoneretryvalrtxnsizenew_sizer0r0r1rgsV             z$LMDBDataset._fill_cache_start_readerc s|jdkr|jdd|_|jjdd}|||}W5QRX|dkr`tdt|Sz t |WSt k r}zt d|W5d}~XYnXdS)zq if the item is not found in the lmdb file, resolves to the persistent cache default behaviour. NFrrz;LMDBDataset: cache key not found, running fallback caching.z)Invalid cache value, corrupted lmdb file?) rrrrr^rrrMrrVloads Exceptionr)r/rrr+errrQr0r1rs    zLMDBDataset._cachecheckcCsD|jdkr||_t|j}t|j|d<|j|d<|S)z4 Returns: dataset info dictionary. Nrfilename)rrdictrr6r+rr)r/outr0r0r1rs   zLMDBDataset.info)T) rGrHrIrJrrVHIGHEST_PROTOCOLr2rrrrrTr0r0rQr1rs&? 8 rcseZdZdZdejddddddedf ddd d d d d d d d ddd fdd ZdddddZd ddddZ d dddZ d dfdd Z Z S)! CacheDatasetau Dataset with cache mechanism that can load data and cache deterministic transforms' result during training. By caching the results of non-random preprocessing transforms, it accelerates the training data pipeline. If the requested data is not in the cache, all transforms will run normally (see also :py:class:`monai.data.dataset.Dataset`). Users can set the cache rate or number of items to cache. It is recommended to experiment with different `cache_num` or `cache_rate` to identify the best training speed. The transforms which are supposed to be cached must implement the `monai.transforms.Transform` interface and should not be `Randomizable`. This dataset will cache the outcomes before the first `Randomizable` `Transform` within a `Compose` instance. So to improve the caching efficiency, please always put as many as possible non-random transforms before the randomized ones when composing the chain of transforms. If passing slicing indices, will return a PyTorch Subset, for example: `data: Subset = dataset[1:4]`, for more details, please check: https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset For example, if the transform is a `Compose` of:: transforms = Compose([ LoadImaged(), EnsureChannelFirstd(), Spacingd(), Orientationd(), ScaleIntensityRanged(), RandCropByPosNegLabeld(), ToTensord() ]) when `transforms` is used in a multi-epoch training pipeline, before the first training epoch, this dataset will cache the results up to ``ScaleIntensityRanged``, as all non-random transforms `LoadImaged`, `EnsureChannelFirstd`, `Spacingd`, `Orientationd`, `ScaleIntensityRanged` can be cached. During training, the dataset will load the cached results and run ``RandCropByPosNegLabeld`` and ``ToTensord``, as ``RandCropByPosNegLabeld`` is a randomized transform and the outcome not cached. During training call `set_data()` to update input data and recompute cache content, note that it requires `persistent_workers=False` in the PyTorch DataLoader. Note: `CacheDataset` executes non-random transforms and prepares cache content in the main process before the first epoch, then all the subprocesses of DataLoader will read the same cache content in the main process during training. it may take a long time to prepare cache content according to the size of expected cache data. So to debug or verify the program before real training, users can set `cache_rate=0.0` or `cache_num=0` to temporarily skip caching. Lazy Resampling: If you make use of the lazy resampling feature of `monai.transforms.Compose`, please refer to its documentation to familiarize yourself with the interaction between `CacheDataset` and lazy resampling. N?rTFr$Sequence[Callable] | Callable | Noner3float int | Noner\rYzbool | str | list | ListProxyr*) r+r, cache_num cache_rate num_workersr copy_cache as_contiguous hash_as_keyr^ runtime_cacher-c st|tst|}tj||d||_||_||_||_||_| |_ | |_ ||_ |j dk rnt t |j d|_ | |_d|_g|_g|_||dS)a Args: data: input data to load and transform to generate dataset for model. transform: transforms to execute operations on input data. cache_num: number of items to be cached. Default is `sys.maxsize`. will take the minimum of (cache_num, data_length x cache_rate, data_length). cache_rate: percentage of cached data in total, default is 1.0 (cache all). will take the minimum of (cache_num, data_length x cache_rate, data_length). num_workers: the number of worker threads if computing cache in the initialization. If num_workers is None then the number returned by os.cpu_count() is used. If a value less than 1 is specified, 1 will be used instead. progress: whether to display a progress bar. copy_cache: whether to `deepcopy` the cache content before applying the random transforms, default to `True`. if the random transforms don't modify the cached content (for example, randomly crop from the cached image and deepcopy the crop region) or if every cache item is only used once in a `multi-processing` environment, may set `copy=False` for better performance. as_contiguous: whether to convert the cached NumPy array or PyTorch tensor to be contiguous. it may help improve the performance of following logic. hash_as_key: whether to compute hash value of input data as the key to save cache, if key exists, avoid saving duplicated content. it can help save memory when the dataset has duplicated items or augmented dataset. hash_func: if `hash_as_key`, a callable to compute hash from data items to be cached. defaults to `monai.data.utils.pickle_hashing`. runtime_cache: mode of cache at the runtime. Default to `False` to prepare the cache content for the entire ``data`` during initialization, this potentially largely increase the time required between the constructor called and first mini-batch generated. Three options are provided to compute the cache on the fly after the dataset initialization: 1. ``"threads"`` or ``True``: use a regular ``list`` to store the cache items. 2. ``"processes"``: use a ListProxy to store the cache items, it can be shared among processes. 3. A list-like object: a users-provided container to be used to store the cache items. For `thread-based` caching (typically for caching cuda tensors), option 1 is recommended. For single process workflows with multiprocessing data loading, option 2 is recommended. For multiprocessing workflows (typically for distributed training), where this class is initialized in subprocesses, option 3 is recommended, and the list-like object should be prepared in the main process and passed to all subprocesses. Not following these recommendations may lead to runtime errors or duplicated cache across processes. r.Nrr)r>rrMr2set_numset_raterrrrr^rmaxr3rr_cache _hash_keysr) r/r+r,rrrrrrrr^rrQr0r1r2s$7  zCacheDataset.__init__r+r-cs|_ddfdd }jrnfddtjD}|t|t|dj_t|dj}n|tjttj}j dkr |_ dSt j t rd j krtdgj_ dSj d kst j t r d j kr dgj_ dSj _ dS) aA Set the input data and run deterministic transforms to generate cache content. Note: should call this func after an entire epoch and must set `persistent_workers=False` in PyTorch DataLoader, because it needs to create new worker processes based on new generated cache content. r3data_lencs"ttjt|j|_dSr5)minr3rrrrr7r0r1_compute_cache_numJsz1CacheDataset.set_data.._compute_cache_numcsi|]\}}||qSr0)r^)rmivr7r0r1 Osz)CacheDataset.set_data..N)FNprocessTthread)r+r enumerater6listrrvaluesr@r _fill_cacherr>rZr)r/r+rmappingr=r0r7r1r?s(    $zCacheDataset.set_datarr4c Cs|jdkrgS|dkr$tt|j}|jr8ts8tdt|jV}|jrztrztt | |j |t |ddW5QRSt| |j |W5QRSQRXdS)z Compute and fill the cache content from data source. Args: indices: target indices in the `self.data` source to compute cache. if None, use the first `cache_num` items. rNz>tqdm is not installed, will not show the caching progress bar.zLoading dataset)totaldesc) rrr@rrrrrrr%imap_load_cache_itemr6)r/r=pr0r0r1rds     ,zCacheDataset._fill_cache)idxcCsB|j|}|jdd}|j||dd}|jr>t|tjd}|S)zN Args: idx: the index of the input data sequence. cSst|tpt|t Sr5rrr0r0r1rrz/CacheDataset._load_cache_item..Tr) memory_format)r+r,rrrrcontiguous_format)r/rrrr0r0r1rxs zCacheDataset._load_cache_itemr9csd}|jr2||j|}||jkrH|j|}n|t||jkrH|}|dkr\t|S|j dkrnt d|j |}|dkr| |}|j |<t |j tstd|j dd}|dk r|jdkrt|n|}|j ||d}|S)Nz@cache buffer is not initialized, please call `set_data()` first.rcSst|tpt|t Sr5rrr0r0r1rrz)CacheDataset._transform..Tr)rr^r+rr:r6rrMr;rrrr>r,rrirrr)r/r:Z cache_indexrr+rrQr0r1r;s.     zCacheDataset._transform)N) rGrHrIrJrmaxsizerr2rrrr;rTr0r0rQr1rs 9*J%rcseZdZdZddejdddddddddf d d d d d d d ddd dddd fdd Zd dfdd Zd ddddZddddZ ddZ ddZ d d!Z d"d#Z d$d%Zd&d'Zd(d)Zd d*d+d,Zd-d.Zd/d0Zddd1d2Zd3d4ZZS)5SmartCacheDataseta; Re-implementation of the SmartCache mechanism in NVIDIA Clara-train SDK. At any time, the cache pool only keeps a subset of the whole dataset. In each epoch, only the items in the cache are used for training. This ensures that data needed for training is readily available, keeping GPU resources busy. Note that cached items may still have to go through a non-deterministic transform sequence before being fed to GPU. At the same time, another thread is preparing replacement items by applying the transform sequence to items not in cache. Once one epoch is completed, Smart Cache replaces the same number of items with replacement items. Smart Cache uses a simple `running window` algorithm to determine the cache content and replacement items. Let N be the configured number of objects in cache; and R be the number of replacement objects (R = ceil(N * r), where r is the configured replace rate). For more details, please refer to: https://docs.nvidia.com/clara/clara-train-archive/3.1/nvmidl/additional_features/smart_cache.html If passing slicing indices, will return a PyTorch Subset, for example: `data: Subset = dataset[1:4]`, for more details, please check: https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset For example, if we have 5 images: `[image1, image2, image3, image4, image5]`, and `cache_num=4`, `replace_rate=0.25`. so the actual training images cached and replaced for every epoch are as below:: epoch 1: [image1, image2, image3, image4] epoch 2: [image2, image3, image4, image5] epoch 3: [image3, image4, image5, image1] epoch 3: [image4, image5, image1, image2] epoch N: [image[N % 5] ...] The usage of `SmartCacheDataset` contains 4 steps: 1. Initialize `SmartCacheDataset` object and cache for the first epoch. 2. Call `start()` to run replacement thread in background. 3. Call `update_cache()` before every epoch to replace training items. 4. Call `shutdown()` when training ends. During training call `set_data()` to update input data and recompute cache content, note to call `shutdown()` to stop first, then update data and call `start()` to restart. Note: This replacement will not work for below cases: 1. Set the `multiprocessing_context` of DataLoader to `spawn`. 2. Launch distributed data parallel with `torch.multiprocessing.spawn`. 3. Run on windows(the default multiprocessing method is `spawn`) with `num_workers` greater than 0. 4. Set the `persistent_workers` of DataLoader to `True` with `num_workers` greater than 0. If using MONAI workflows, please add `SmartCacheHandler` to the handler list of trainer, otherwise, please make sure to call `start()`, `update_cache()`, `shutdown()` during training. Args: data: input data to load and transform to generate dataset for model. transform: transforms to execute operations on input data. replace_rate: percentage of the cached items to be replaced in every epoch (default to 0.1). cache_num: number of items to be cached. Default is `sys.maxsize`. will take the minimum of (cache_num, data_length x cache_rate, data_length). cache_rate: percentage of cached data in total, default is 1.0 (cache all). will take the minimum of (cache_num, data_length x cache_rate, data_length). num_init_workers: the number of worker threads to initialize the cache for first epoch. If num_init_workers is None then the number returned by os.cpu_count() is used. If a value less than 1 is specified, 1 will be used instead. num_replace_workers: the number of worker threads to prepare the replacement cache for every epoch. If num_replace_workers is None then the number returned by os.cpu_count() is used. If a value less than 1 is specified, 1 will be used instead. progress: whether to display a progress bar when caching for the first epoch. shuffle: whether to shuffle the whole data list before preparing the cache content for first epoch. it will not modify the original input data sequence in-place. seed: random seed if shuffle is `True`, default to `0`. copy_cache: whether to `deepcopy` the cache content before applying the random transforms, default to `True`. if the random transforms don't modify the cache content or every cache item is only used once in a `multi-processing` environment, may set `copy=False` for better performance. as_contiguous: whether to convert the cached NumPy array or PyTorch tensor to be contiguous. it may help improve the performance of following logic. runtime_cache: Default to `False`, other options are not implemented yet. Ng?rrTrFrrrr3rr\r*) r+r, replace_raterrnum_init_workersnum_replace_workersrshuffleseedrrr-c s"| r|j| d| |_d|_t|_d|_d|_d|_| dk rHt dt j ||||||| | dd |j dkrz| |_ |jt|krtd|dkrtd||_|jdk rtt|jd|_t||_tt|j|t||j|_d d t|jD|_tt|j|_|dS) NrrrFz@Options other than `runtime_cache=False` is not implemented yet.) r+r,rrrrrrrz`cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.zSreplace_rate must be greater than 0, otherwise, please use monai.data.CacheDataset.cSsg|]}dqSr5r0rm_r0r0r1 *sz.SmartCacheDataset.__init__..) set_random_stater _start_posrLock _update_lock_round _replace_done _replace_mgrNotImplementedErrorrMr2rrrr6rrrirrr3 _total_numrmathceil _replace_numr@ _replacementsr_replace_data_idx_compute_data_idx)r/r+r,r rrrrrrrrrrrQr0r1r2sJ      "zSmartCacheDataset.__init__rycsB|rtd||jr2t|}||t|dS)z Set the input data and run deterministic transforms to generate cache content. Note: should call `shutdown()` before calling this func. zszSmartCacheDataset.randomizer4cCsBt|jD]2}|j|j|}||jkr2||j8}||j|<q dS)zJ Update the replacement data position in the total data. N)r@r!rrrr#)r/rposr0r0r1r$Ds   z#SmartCacheDataset._compute_data_idxcCs|jdkrdS|jS)zK Check whether the replacement thread is already started. NF)ris_aliver7r0r0r1r%OszSmartCacheDataset.is_startedcCs|s|dS)zY Start the background thread to replace training items for every epoch. N)r%_restartr7r0r0r1rCVszSmartCacheDataset.startcCs&d|_tj|jdd|_|jdS)zG Restart background thread if killed for some reason. rT)targetdaemonN)rrThreadmanage_replacementrrCr7r0r0r1r+^szSmartCacheDataset._restartc Cs|j|jsW5QRdS|jd|j=|j|j|j|j7_|j|jkrd|j|j8_||j d7_ d|_W5QRdSQRXdS)zQ Update the cache items with new replacement for current epoch. FNrT) rrrr!extendr"rrr$rr7r0r0r1_try_update_cachegs z#SmartCacheDataset._try_update_cachecCs ||stdqdS)z Update cache items for current epoch, need to call this function before every epoch. If the cache has been shutdown before, need to restart the `_replace_mgr` thread. {Gz?N)rCr1timesleepr7r0r0r1 update_cache~szSmartCacheDataset.update_cachec CsN|j>|jr6d|_d|_|d|_W5QRdSW5QRdSQRXdS)zK Wait for thread lock to shut down the background thread. rFTN)rrrrr$r7r0r0r1 _try_shutdownszSmartCacheDataset._try_shutdowncCs:|s dS|s tdq |jdk r6|jddS)zC Shut down the background thread for replacement. Nr2i,)r%r6r3r4rrur7r0r0r1r&s   zSmartCacheDataset.shutdownr9cCs|j|}|||j|<dS)zT Execute deterministic transforms on the new data for replacement. N)r#rr")r/r:r)r0r0r1_replace_cache_threads z'SmartCacheDataset._replace_cache_threadc Cs8t|j}||jtt|jW5QRXd|_dS)z Compute expected items for the replacement of next epoch, execute deterministic transforms. It can support multi-threads to accelerate the computation progress. TN)rrmapr7rr@r!r)r/rr0r0r1_compute_replacementss "z'SmartCacheDataset._compute_replacementsc CsX|jH|jdkr&d|_W5QRdS|j|kr8|d|jfW5QRSQRXdS)zX Wait thread lock and replace training items in the background thread. rT)TFN)rrrr9)r/ check_roundr0r0r1_try_manage_replacements  z)SmartCacheDataset._try_manage_replacementcCs*d}d}|s&||\}}tdqdS)z5 Background thread for replacement. r:Fr2N)r<r3r4)r/r;rr0r0r1r/s z$SmartCacheDataset.manage_replacementcCs|jS)zQ The dataset length is given by cache_num instead of len(data). )rr7r0r0r1r8szSmartCacheDataset.__len__)rGrHrIrJrr r2rr'r$r%rCr+r1r5r6r&r7r9r<r/r8rTr0r0rQr1r s:K,:     r csHeZdZdZdddddfdd Zd d d d Zd d ddZZS) ZipDatasetaS Zip several PyTorch datasets and output data(with the same index) together in a tuple. If the output of single dataset is already a tuple, flatten it and extend to the result. For example: if datasetA returns (img, imgmeta), datasetB returns (seg, segmeta), finally return (img, imgmeta, seg, segmeta). And if the datasets don't have same length, use the minimum length of them as the length of ZipDataset. If passing slicing indices, will return a PyTorch Subset, for example: `data: Subset = dataset[1:4]`, for more details, please check: https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset Examples:: >>> zip_data = ZipDataset([[1, 2, 3], [4, 5]]) >>> print(len(zip_data)) 2 >>> for item in zip_data: >>> print(item) [1, 4] [2, 5] Nrr)r*)datasetsr,r-cstjt||ddS)z Args: datasets: list of datasets to zip together. transform: a callable data transform operates on the zipped item from `datasets`. )r,N)rMr2r)r/r>r,rQr0r1r2szZipDataset.__init__r3r4cCstdd|jDS)Ncss|]}t|VqdSr5)r6)rmr<r0r0r1rosz%ZipDataset.__len__..)rr+r7r0r0r1r8szZipDataset.__len__r9cCsLdd}g}|jD]}||||q|jdk rDt|j|dd}t|S)NcSst|ttfrt|S|gSr5)r>tupler)xr0r0r1to_listsz&ZipDataset._transform..to_listF)Z map_items)r+r0r,rr?)r/r:rAr+r<r0r0r1r;s  zZipDataset._transform)NrGrHrIrJr2r8r;rTr0r0rQr1r=sr=c @sZeZdZdZddddddddddd Zd d d d ZddddddZd dddZdS) ArrayDataseta4 Dataset for segmentation and classification tasks based on array format input data and transforms. It ensures the same random seeds in the randomized transforms defined for image, segmentation and label. The `transform` can be :py:class:`monai.transforms.Compose` or any other callable object. For example: If train based on Nifti format images without metadata, all transforms can be composed:: img_transform = Compose( [ LoadImage(image_only=True), EnsureChannelFirst(), RandAdjustContrast() ] ) ArrayDataset(img_file_list, img_transform=img_transform) If training based on images and the metadata, the array transforms can not be composed because several transforms receives multiple parameters or return multiple values. Then Users need to define their own callable method to parse metadata from `LoadImage` or set `affine` matrix to `Spacing` transform:: class TestCompose(Compose): def __call__(self, input_): img, metadata = self.transforms[0](input_) img = self.transforms[1](img) img, _, _ = self.transforms[2](img, metadata["affine"]) return self.transforms[3](img), metadata img_transform = TestCompose( [ LoadImage(image_only=False), EnsureChannelFirst(), Spacing(pixdim=(1.5, 1.5, 3.0)), RandAdjustContrast() ] ) ArrayDataset(img_file_list, img_transform=img_transform) Examples:: >>> ds = ArrayDataset([1, 2, 3, 4], lambda x: x + 0.1) >>> print(ds[0]) 1.1 >>> ds = ArrayDataset(img=[1, 2, 3, 4], seg=[5, 6, 7, 8]) >>> print(ds[0]) [1, 5] Nrr)zSequence | Noner*)img img_transformseg seg_transformlabelslabel_transformr-c CsZ||f||f||fg}|jtddd|D}t|dkrF|dnt||_d|_dS)a Initializes the dataset with the filename lists. The transform `img_transform` is applied to the images and `seg_transform` to the segmentations. Args: img: sequence of images. img_transform: transform to apply to each element in `img`. seg: sequence of segmentations. seg_transform: transform to apply to each element in `seg`. labels: sequence of labels. label_transform: transform to apply to each element in `labels`. rcSs*g|]"}|ddk rt|d|dqS)rNrr)rmr@r0r0r1rUs z)ArrayDataset.__init__..rrN)rr r6r=r<_seed) r/rDrErFrGrHrIitemsr>r0r0r1r2=s zArrayDataset.__init__r3r4cCs t|jSr5)r6r<r7r0r0r1r8ZszArrayDataset.__len__rSrcCs|jjtdd|_dS)Nuint32)dtype)r(randintrrJr~r0r0r1r']szArrayDataset.randomizer9cCsv|t|jtrF|jjD](}t|dd}t|tr|j|jdqt|jdd}t|trl|j|jd|j|S)Nr,r) r'r>r<r=r+getattrrrrJ)r/r:r<r,r0r0r1rF`s     zArrayDataset.__getitem__)NNNNN)N)rGrHrIrJr2r8r'rFr0r0r0r1rC s4rCcsDeZdZdZddddddfd d Zd d Zd dddZZS)NPZDictItemDataseta= Represents a dataset from a loaded NPZ file. The members of the file to load are named in the keys of `keys` and stored under the keyed name. All loaded arrays must have the same 0-dimension (batch) size. Items are always dicts mapping names to an item extracted from the loaded arrays. If passing slicing indices, will return a PyTorch Subset, for example: `data: Subset = dataset[1:4]`, for more details, please check: https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset Args: npzfile: Path to .npz file or stream containing .npz file data keys: Maps keys to load from file to name to store in dataset transform: Transform to apply to batch dict other_keys: secondary data to load from file and store in dict `other_keys`, not returned by __getitem__ Nr0zstr | IOzdict[str, str]z$Callable[..., dict[str, Any]] | NoneSequence[str] | None)npzfilekeysr, other_keysc st|tr|nd|_t||_t|fdd|jD|_|jt tt |j j d|_ |dkrpinfdd|D|_|jD]:\}}|j d|j krtd|j d|d|j dqtg|dS) NZSTREAMcsi|]\}}||qSr0r0)rmZdatakZstoredkdatr0r1rsz/NPZDictItemDataset.__init__..rcsi|]}||qSr0r0)rmkrUr0r1rsz:All loaded arrays must have the same first dimension size z , array `z ` has size )r>rZrRrrSnprrKarraysr r$rshapelengthrTrirMr2)r/rRrSr,rTrWrrQrUr1r2}s  " zNPZDictItemDataset.__init__cCs|jSr5)r[r7r0r0r1r8szNPZDictItemDataset.__len__r3r9cs`fdd|jD}|js"|St|j|}t|tsPt|trTt|dtrT|StddS)Ncsi|]\}}||qSr0r0)rmrWrr9r0r1rsz1NPZDictItemDataset._transform..rzQWith a dict supplied to apply_transform, should return a dict or a list of dicts.)rYrKr,rr>rrAssertionError)r/r:r+resultr0r9r1r;s "zNPZDictItemDataset._transform)Nr0rBr0r0rQr1rPns rPc s4eZdZdZd ddddddd d fd d ZZS) CSVDataseta Dataset to load data from CSV files and generate a list of dictionaries, every dictionary maps to a row of the CSV file, and the keys of dictionary map to the column names of the CSV file. It can load multiple CSV files and join the tables with additional `kwargs` arg. Support to only load specific rows and columns. And it can also group several loaded columns to generate a new column, for example, set `col_groups={"meta": ["meta_0", "meta_1", "meta_2"]}`, output can be:: [ {"image": "./image0.nii", "meta_0": 11, "meta_1": 12, "meta_2": 13, "meta": [11, 12, 13]}, {"image": "./image1.nii", "meta_0": 21, "meta_1": 22, "meta_2": 23, "meta": [21, 22, 23]}, ] Args: src: if provided the filename of CSV file, it can be a str, URL, path object or file-like object to load. also support to provide pandas `DataFrame` directly, will skip loading from filename. if provided a list of filenames or pandas `DataFrame`, it will join the tables. row_indices: indices of the expected rows to load. it should be a list, every item can be a int number or a range `[start, end)` for the indices. for example: `row_indices=[[0, 100], 200, 201, 202, 300]`. if None, load all the rows in the file. col_names: names of the expected columns to load. if None, load all the columns. col_types: `type` and `default value` to convert the loaded columns, if None, use original data. it should be a dictionary, every item maps to an expected column, the `key` is the column name and the `value` is None or a dictionary to define the default value and data type. the supported keys in dictionary are: ["type", "default"]. for example:: col_types = { "subject_id": {"type": str}, "label": {"type": int, "default": 0}, "ehr_0": {"type": float, "default": 0.0}, "ehr_1": {"type": float, "default": 0.0}, "image": {"type": str, "default": None}, } col_groups: args to group the loaded columns to generate a new column, it should be a dictionary, every item maps to a group, the `key` will be the new column name, the `value` is the names of columns to combine. for example: `col_groups={"ehr": [f"ehr_{i}" for i in range(10)], "meta": ["meta_1", "meta_2"]}` transform: transform to apply on the loaded items of a dictionary data. kwargs_read_csv: dictionary args to pass to pandas `read_csv` function. kwargs: additional arguments for `pandas.merge()` API to join tables. Nzstr | Sequence[str] | NonezSequence[int | str] | NonerQz'dict[str, dict[str, Any] | None] | Nonezdict[str, Sequence[str]] | Noner)r)rN row_indices col_names col_types col_groupsr,kwargs_read_csvc  st|ttfs|fn|} g} | D]R} t| trR| |rDtj| f|nt| q t| tjrj| | q tdq t f| ||||d|} t j | |ddS)Nz.`src` must be file path or pandas `DataFrame`.)dfsr_r`rarbr.) r>r?rrZrspdread_csv DataFramerirrMr2) r/rNr_r`rarbr,rcrOZsrcsrdrr+rQr0r1r2s$  $   zCSVDataset.__init__)NNNNNNN)rGrHrIrJr2rTr0r0rQr1r^s1r^c sVeZdZdZeddfdddddd d d d d fdd ZddZddZddZZ S) GDSDataseta An extension of the PersistentDataset using direct memory access(DMA) data path between GPU memory and storage, thus avoiding a bounce buffer through the CPU. This direct path can increase system bandwidth while decreasing latency and utilization load on the CPU and GPU. A tutorial is available: https://github.com/Project-MONAI/tutorials/blob/main/modules/GDS_dataset.ipynb. See also: https://github.com/rapidsai/kvikio NTrrWrXr3rYr[r\r r*) r+r,r]devicer^rarrOr-c s0tjf||||||d|||_i|_dS)aM Args: data: input data file paths to load and transform to generate dataset for model. `GDSDataset` expects input data to be a list of serializable and hashes them as cache keys using `hash_func`. transform: transforms to execute operations on input data. cache_dir: If specified, this is the location for gpu direct storage of pre-computed transformed data tensors. The cache_dir is computed once, and persists on disk until explicitly removed. Different runs, programs, experiments may share a common cache dir provided that the transforms pre-processing is consistent. If `cache_dir` doesn't exist, will automatically create it. If `cache_dir` is `None`, there is effectively no caching. device: target device to put the output Tensor data. Note that only int can be used to specify the gpu to be used. hash_func: a callable to compute hash from data items to be cached. defaults to `monai.data.utils.pickle_hashing`. hash_transform: a callable to compute hash from the transform information when caching. This may reduce errors due to transforms changing during experiments. Default to None (no hash). Other options are `pickle_hashing` and `json_hashing` functions from `monai.data.utils`. reset_ops_id: whether to set `TraceKeys.ID` to ``Tracekys.NONE``, defaults to ``True``. When this is enabled, the traced transform instance IDs will be removed from the cached MetaTensors. This is useful for skipping the transform instance checks when inverting applied operations using the cached content and with re-created transform instances. )r+r,r]r^rarN)rMr2ri _meta_cache) r/r+r,r]rir^rarrOrQr0r1r2s$ zGDSDataset.__init__c Csd}|jdk r8||d}||j7}|j|d}|dk rz|rztj|jt |t ri}|D]z}|j |j d|dd}t j|d||dtdd ||<t|||d d |jd ||<|||d <qr|W5QRSt |tjtjfr|j |j dd}t j||dtdd }t||d d |jd }ttdd|} t| r||fW5QRS|W5QRSddtt|D}t|D]\} } | D]}|j |j d|d| d} t j|d|d| | dtdd } t|| | d d |jd } || || |d | iqΐq|W5QRSW5QRX|t|}|dkr|St |t r|D]Z}|d|}|j d|d}t ||tjtjfr| ||||n|Sqnt |tjtjfr:|}|j d}| |||nXt|D]N\} } | D]>}|d|d| }|j d|d| }| | ||qNqBt!|d"|S)a In order to enable direct storage to the GPU when loading the hashfile, rewritten this function. Note that in this function, it will always return `torch.Tensor` when load data from cache. Args: item_transformed: The current data element to be mutated into transformed representation Returns: The transformed data_element, either from cache, or explicitly computing it. Warning: The current implementation does not encode transform information as part of the hashing mechanism used for generating cache names when `hash_transform` is None. If the transforms applied are changed in any way, the objects in the cache dir will be invalid. Nrpr-z-meta)meta_hash_file_namerMr0)rMlikerZzcuda:)riZ _meta_dictcSs|dkS)N)rMrZr0)rr0r0r1rRrz(GDSDataset._cachecheck..cSsg|]}iqSr0r0rr0r0r1rWsz*GDSDataset._cachecheck..z-meta-a)#r]r^rvrjrcpcudaDevicerir>r_load_meta_cacher kvikio_numpyfromfileemptyrreshaperXndarrayrTensorrfilterrSr\r@r6rupdaterr_create_new_cacherr)r/rrrrrWZmeta_k_meta_dataZ filtered_keysrZ_itemZmeta_i_kZitem_kr data_hashfilerlr0r0r1r,sp   (& "$   zGDSDataset._cachecheckc Cst|trt|jni|j|<t|tr.|jn|}t|tjrF|}|j |j|d<t |j |j|d<t ||ztx}|j|}t||}tj|j||t|jt|jd|r|sztt ||Wntk rYnXW5QRXWntk rYnXdS)NrZrMr)r>rrmetarjarrayrrxnumpyrZrZrMrstofilerrr]r rr!r_rr`rr|rrr)r/r+r~rlZ_item_transformed_datarZmeta_hash_filerr0r0r1r{{s0      zGDSDataset._create_new_cachecCs(||jkr|j|St|j|SdSr5)rjrrr])r/rlr0r0r1rrs  zGDSDataset._load_meta_cache) rGrHrIrJrr2rr{rrrTr0r0rQr1rhs$0O rh)O __future__rcollections.abcrArrVr|rrrr3rrrrrZmultiprocessing.managersrZmultiprocessing.poolrpathlibr typingr r r r rrXrtorch.multiprocessingrZtorch.serializationrtorch.utils.datarZ _TorchDatasetrmonai.data.meta_tensorrmonai.data.utilsrrrmonai.transformsrrrrrrr monai.utilsrrr r!r"r#monai.utils.miscr$r%rrorr'rersrKrUrrrr r=rCrPr^rhr0r0r0r1 sh         $       32 d/n20c8K