U Ph @s ddlmZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl ZddlmZmZmZmZmZddlmZddlmZdd lmZmZmZmZmZmZdd l m!Z!m"Z"dd l#m$Z$d d ddgZ%Gdd d e"eZ&Gdd d e"eZ'Gddde"eZ(GdddZ)dS)) annotationsN)CallableSequence)Path)Any)DCM_FILENAME_REGEXdownload_tcia_series_instanceget_tcia_metadataget_tcia_ref_uidmatch_tcia_ref_uid_in_study)download_and_extract)PathLike) CacheDataset PydicomReaderload_decathlon_datalistload_decathlon_propertiespartition_datasetselect_cross_validation_folds) LoadImaged Randomizable) ensure_tupleMedNISTDatasetDecathlonDatasetCrossValidation TciaDatasetc@seZdZdZdZdZdZdZdddd d ej d d d d d df d dddddddddddddddddZ dddddZ ddddZ d dd d!d"Z d#S)$raQ The Dataset to automatically download MedNIST data and generate items for training, validation or test. It's based on `CacheDataset` to accelerate the training process. Args: root_dir: target directory to download and load MedNIST dataset. section: expected data section, can be: `training`, `validation` or `test`. transform: transforms to execute operations on input data. download: whether to download and extract the MedNIST from resource link, default is False. if expected file already exists, skip downloading even set it to True. user can manually copy `MedNIST.tar.gz` file or `MedNIST` folder to root directory. seed: random seed to randomly split training, validation and test datasets, default is 0. val_frac: percentage of validation fraction in the whole dataset, default is 0.1. test_frac: percentage of test fraction in the whole dataset, default is 0.1. cache_num: number of items to be cached. Default is `sys.maxsize`. will take the minimum of (cache_num, data_length x cache_rate, data_length). cache_rate: percentage of cached data in total, default is 1.0 (cache all). will take the minimum of (cache_num, data_length x cache_rate, data_length). num_workers: the number of worker threads if computing cache in the initialization. If num_workers is None then the number returned by os.cpu_count() is used. If a value less than 1 is specified, 1 will be used instead. progress: whether to display a progress bar when downloading dataset and computing the transform cache content. copy_cache: whether to `deepcopy` the cache content before applying the random transforms, default to `True`. if the random transforms don't modify the cached content (for example, randomly crop from the cached image and deepcopy the crop region) or if every cache item is only used once in a `multi-processing` environment, may set `copy=False` for better performance. as_contiguous: whether to convert the cached NumPy array or PyTorch tensor to be contiguous. it may help improve the performance of following logic. runtime_cache: whether to compute cache at the runtime, default to `False` to prepare the cache content at initialization. See: :py:class:`monai.data.CacheDataset`. Raises: ValueError: When ``root_dir`` is not a directory. RuntimeError: When ``dataset_dir`` doesn't exist and downloading is not selected (``download=False``). z]https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/MedNIST.tar.gzZ 0bc7306e7427e00ad1c5526a6677552dzMedNIST.tar.gzZMedNISTFrg??Tr strSequence[Callable] | Callableboolintfloatz int | NoneNone)root_dirsection transformdownloadseedval_frac test_frac cache_num cache_rate num_workersprogress copy_cache as_contiguous runtime_cachereturnc Cst|}|std||_||_||_|j|d||j}||j}d|_ |rlt |j |||j d| d|st d|d||}|dkrtd }tj||||| | | | | |d dS) N,Root directory root_dir must be a directory.r(rmd5urlfilepath output_dirZhash_valZ hash_typer.Cannot find dataset directory: *, please use download=True to download it.rimage datar&r+r,r-r.r/r0r1)ris_dir ValueErrorr%r)r*set_random_statecompressed_file_namedataset_folder_name num_classr resourcer5 RuntimeError_generate_data_listrr__init__)selfr$r%r&r'r(r)r*r+r,r-r.r/r0r1 tarfile_name dataset_dirr>rrH/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/apps/datasets.pyrH[sL      zMedNISTDataset.__init__ np.ndarrayr>r2cCs|j|dSNRshufflerIr>rrrL randomizeszMedNISTDataset.randomizer2cCs|jS)zGet number of classes.)rDrIrrrLget_num_classesszMedNISTDataset.get_num_classes list[dict]rKr2c sfttddDt|_fddt|jDfddt|jD}gggt|jD]>}||g|||g||qpt}t|}| |t ||j }t ||j }|j dkr|d|}nN|j dkr ||||}n0|j d kr>|||d}ntd |j d fd d|DS) zu Raises: ValueError: When ``section`` is not one of ["training", "validation", "test"]. css|]}|r|jVqdSrOr?name.0xrrrL sz5MedNISTDataset._generate_data_list..cs&g|]}dd|DqS)cSsg|] }|qSrrr\rrrL szAMedNISTDataset._generate_data_list...)iterdirr]i) class_namesrKrrLr`sz6MedNISTDataset._generate_data_list..csg|]}t|qSr)lenrb) image_filesrrLr`stestN validationtrainingzUnsupported section: z;, available options are ["training", "validation", "test"].cs$g|]}|||dqS))r<label class_namerrb)rk image_classimage_files_listrrLr`s)rsortedrarerDrangeextendnparangerTr!r*r)r%r@) rIrKZnum_eachrclengthindicesZ test_length val_lengthZsection_indicesr)rkrdrKrlrfrmrLrGs:       z"MedNISTDataset._generate_data_listN)__name__ __module__ __qualname____doc__rEr5rBrCsysmaxsizerHrTrWrGrrrrLr/s*&,9c@seZdZdZdddddddd d d d Zd dddddddddd Zddddejddddddf dddd d!d"d#d"d#d"d!d!d!d!d$d%d&d'Zd(d)d*d+Z d(d$d,d-d.Z dZ property_keysrrrLrH sd          zDecathlonDataset.__init__rMrUcCs|jSzD Get the indices of datalist used in this dataset. rtrVrrrL get_indicesgszDecathlonDataset.get_indicesrNcCs|j|dSrOrPrSrrrLrTnszDecathlonDataset.randomizeNzSequence[str] | str | Nonedict)rr2cs2|dkrjSjdk r.fddt|DSiS)z Get the loaded properties of dataset with specified keys. If no keys specified, return all the loaded properties. Ncsi|]}|j|qSr)r)r]keyrVrrL zsz3DecathlonDataset.get_properties..)rr)rIrrrVrLget_propertiesqs  zDecathlonDataset.get_propertiesrXrYcCs4t|}|jdkrdnd}t|dd|}||S)N)rirhrirgrT)rr%r_split_datalist)rIrKr%datalistrrrLrG}sz$DecathlonDataset._generate_data_listrr2cst|jdkrSt}t|}||t||j}|jdkrR||d|_n|d||_fdd|jDS)Nrgricsg|] }|qSrrrbrrrLr`sz4DecathlonDataset._split_datalist..r%rerqrrrTr!r)rtrIrrsrtrurrrLrs    z DecathlonDataset._split_datalist)N)rvrwrxryrEr5rzr{rHrrTrrGrrrrrLrsPB ,G c@seZdZdZdddddddd ed d ejd d ddddfdddddddddddddddddddddddddZddddZddddd Z dddd!d"d#Z dd$d%d&d'Z d$d$d(d)d*Z d+S),raA The Dataset to automatically download the data from a public The Cancer Imaging Archive (TCIA) dataset and generate items for training, validation or test. The Highdicom library is used to load dicom data with modality "SEG", but only a part of collections are supported, such as: "C4KC-KiTS", "NSCLC-Radiomics", "NSCLC-Radiomics-Interobserver1", " QIN-PROSTATE-Repeatability" and "PROSTATEx". Therefore, if "seg" is included in `keys` of the `LoadImaged` transform and loading some other collections, errors may be raised. For supported collections, the original "SEG" information may not always be consistent for each dicom file. Therefore, to avoid creating different format of labels, please use the `label_dict` argument of `PydicomReader` when calling the `LoadImaged` transform. The prepared label dicts of collections that are mentioned above is also saved in: `monai.apps.tcia.TCIA_LABEL_DICT`. You can also refer to the second example bellow. This class is based on :py:class:`monai.data.CacheDataset` to accelerate the training process. Args: root_dir: user's local directory for caching and loading the TCIA dataset. collection: name of a TCIA collection. a TCIA dataset is defined as a collection. Please check the following list to browse the collection list (only public collections can be downloaded): https://www.cancerimagingarchive.net/collections/ section: expected data section, can be: `training`, `validation` or `test`. transform: transforms to execute operations on input data. for further usage, use `EnsureChannelFirstd` to convert the shape to [C, H, W, D]. If not specified, `LoadImaged(reader="PydicomReader", keys=["image"])` will be used as the default transform. In addition, we suggest to set the argument `labels` for `PydicomReader` if segmentations are needed to be loaded. The original labels for each dicom series may be different, using this argument is able to unify the format of labels. download: whether to download and extract the dataset, default is False. if expected file already exists, skip downloading even set it to True. user can manually copy tar file or dataset folder to the root directory. download_len: number of series that will be downloaded, the value should be larger than 0 or -1, where -1 means all series will be downloaded. Default is -1. seg_type: modality type of segmentation that is used to do the first step download. Default is "SEG". modality_tag: tag of modality. Default is (0x0008, 0x0060). ref_series_uid_tag: tag of referenced Series Instance UID. Default is (0x0020, 0x000e). ref_sop_uid_tag: tag of referenced SOP Instance UID. Default is (0x0008, 0x1155). specific_tags: tags that will be loaded for "SEG" series. This argument will be used in `monai.data.PydicomReader`. Default is [(0x0008, 0x1115), (0x0008,0x1140), (0x3006, 0x0010), (0x0020,0x000D), (0x0010,0x0010), (0x0010,0x0020), (0x0020,0x0011), (0x0020,0x0012)]. fname_regex: a regular expression to match the file names when the input is a folder. If provided, only the matched files will be included. For example, to include the file name "image_0001.dcm", the regular expression could be `".*image_(\d+).dcm"`. Default to `"^(?!.*LICENSE).*"`, ignoring any file name containing `"LICENSE"`. val_frac: percentage of validation fraction in the whole dataset, default is 0.2. seed: random seed to randomly shuffle the datalist before splitting into training and validation, default is 0. note to set same seed for `training` and `validation` sections. cache_num: number of items to be cached. Default is `sys.maxsize`. will take the minimum of (cache_num, data_length x cache_rate, data_length). cache_rate: percentage of cached data in total, default is 0.0 (no cache). will take the minimum of (cache_num, data_length x cache_rate, data_length). num_workers: the number of worker threads if computing cache in the initialization. If num_workers is None then the number returned by os.cpu_count() is used. If a value less than 1 is specified, 1 will be used instead. progress: whether to display a progress bar when downloading dataset and computing the transform cache content. copy_cache: whether to `deepcopy` the cache content before applying the random transforms, default to `True`. if the random transforms don't modify the cached content (for example, randomly crop from the cached image and deepcopy the crop region) or if every cache item is only used once in a `multi-processing` environment, may set `copy=False` for better performance. as_contiguous: whether to convert the cached NumPy array or PyTorch tensor to be contiguous. it may help improve the performance of following logic. runtime_cache: whether to compute cache at the runtime, default to `False` to prepare the cache content at initialization. See: :py:class:`monai.data.CacheDataset`. Example:: # collection is "Pancreatic-CT-CBCT-SEG", seg_type is "RTSTRUCT" data = TciaDataset( root_dir="./", collection="Pancreatic-CT-CBCT-SEG", seg_type="RTSTRUCT", download=True ) # collection is "C4KC-KiTS", seg_type is "SEG", and load both images and segmentations from monai.apps.tcia import TCIA_LABEL_DICT transform = Compose( [ LoadImaged(reader="PydicomReader", keys=["image", "seg"], label_dict=TCIA_LABEL_DICT["C4KC-KiTS"]), EnsureChannelFirstd(keys=["image", "seg"]), ResampleToMatchd(keys="image", key_dst="seg"), ] ) data = TciaDataset( root_dir="./", collection="C4KC-KiTS", section="validation", seed=12345, download=True ) print(data[0]["seg"].shape) rFZSEG)`) )riU))ri)ri@)i0)r )rr)rr)r)rrr|grTr rrr r!tupler"r#)r$ collectionr%r&r' download_lenseg_type modality_tagref_series_uid_tagref_sop_uid_tag specific_tags fname_regexr(r)r+r,r-r.r/r0r1r2c CsNt|}|std||_||_||_||_| |_| |_|j | dt j ||}t | }||g7}||_|rtd|d|dd}|dkr|d|}t|dkrtd|d ||D]}|||qt j |std |d | |_tg|_|||_|d kr*td gd|jd}tj||j||||||||d dS)Nr3r4zgetSeries?Collection=z &Modality=ZSeriesInstanceUID)query attributerz"Cannot find data with collection: z seg_type: r:r~rr<r)rreaderrr=)rr?r@r%r)rrrrrAospathjoinr load_tagsr re_download_series_reference_datarrFrrqrrtrGrrrrH)rIr$rr%r&r'rrrrrrrr(r)r+r,r-r.r/r0r1 download_dirrZseg_series_list series_uidrrrLrHsX!        zTciaDataset.__init__rMrUcCs|jSrrrVrrrLrAszTciaDataset.get_indicesrNcCs|j|dSrOrPrSrrrLrTHszTciaDataset.randomize)rrr2cCstj|d|}t|||ddddtt|D}tj||d}td|jd|}|j rj|j n|j }|st d |d d }|j r|j n|j}|st d |d d}t|}tj||||j} tj|||d} g} |D]} tj|| }td|jd|}||jj|jkrt|d|j|jd} | dkrft|d|j|jd}t|j|} | dkr| | q| st d|dnt| d|| ddtj| st|| dS)z First of all, download a series from TCIA according to `series_uid`. Then find all referenced series and download. rawF)rrr9Z check_md5cSsg|]}|dr|qS)z.dcm)endswithr]frrrLr`Ts z?TciaDataset._download_series_reference_data..rT)Zstop_before_pixelsrz+unable to find patient name of dicom file: z, use 'patient' instead.Zpatientz,unable to find series number of dicom file: z, use '0' instead.r<)Zfind_soprrz.cSsg|]}|r|jqSrrZrrrrLr`sr<) rrscandirrrrrrrr) rIrKrZ patient_listrZ series_listrZseg_key image_pathZ mask_pathrrrLrGs  zTciaDataset._generate_data_listrcst|jdkrSt}t|}||t||j}|jdkrR||d|_n|d||_fdd|jDS)Nrgricsg|] }|qSrrrbrrrLr`sz/TciaDataset._split_datalist..rrrrrLrs    zTciaDataset._split_datalistN) rvrwrxryrrzr{rHrrTrrGrrrrrLrs2_ :R8c@s:eZdZdZdddddddd d Zd ddd d dZdS)raa Cross validation dataset based on the general dataset which must have `_split_datalist` API. Args: dataset_cls: dataset class to be used to create the cross validation partitions. It must have `_split_datalist` API. nfolds: number of folds to split the data for cross validation. seed: random seed to randomly shuffle the datalist before splitting into N folds, default is 0. dataset_params: other additional parameters for the dataset_cls base class. Example of 5 folds cross validation training:: cvdataset = CrossValidation( dataset_cls=DecathlonDataset, nfolds=5, seed=12345, root_dir="./", task="Task09_Spleen", section="training", transform=train_transform, download=True, ) dataset_fold0_train = cvdataset.get_dataset(folds=[1, 2, 3, 4]) dataset_fold0_val = cvdataset.get_dataset(folds=0, transform=val_transform, download=False) # execute training for fold 0 ... dataset_fold1_train = cvdataset.get_dataset(folds=[0, 2, 3, 4]) dataset_fold1_val = cvdataset.get_dataset(folds=1, transform=val_transform, download=False) # execute training for fold 1 ... ... dataset_fold4_train = ... # execute training for fold 4 ... robjectr!rr#) dataset_clsnfoldsr(dataset_paramsr2cKs.t|dstd||_||_||_||_dS)Nrz,dataset class must have _split_datalist API.)hasattrr@rrr(r)rIrrr(rrrrLrHs  zCrossValidation.__init__zSequence[int] | int)foldsrr2c sD|j|jt|j}||Gfddd|j}|f|S)a Generate dataset based on the specified fold indices in the cross validation group. Args: folds: index of folds for training or validation, if a list of values, concatenate the data. dataset_params: other additional parameters for the dataset_cls base class, will override the same parameters in `self.dataset_params`. cs$eZdZdddfdd ZdS)z4CrossValidation.get_dataset.._NsplitsDatasetrXrcst|dd}t|dS)NT)r>Znum_partitionsrRr() partitionsr)rr)rIrr>rrr(rrLrszDCrossValidation.get_dataset.._NsplitsDataset._split_datalistN)rvrwrxrrrrrL_NsplitsDatasetsr)rr(rrupdater)rIrrZdataset_params_rrrrL get_datasets   zCrossValidation.get_datasetN)rr)rvrwrxryrHrrrrrLrs%)* __future__rrrrzrcollections.abcrrpathlibrtypingrnumpyrqZmonai.apps.tciarrr r r Zmonai.apps.utilsr monai.config.type_definitionsr Z monai.datarrrrrrmonai.transformsrr monai.utilsr__all__rrrrrrrrL s.        P