o  i @sddlmZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl ZddlmZmZmZmZmZddlmZddlmZdd lmZmZmZmZmZmZdd l m!Z!m"Z"dd l#m$Z$gd Z%Gd dde"eZ&Gddde"eZ'Gddde"eZ(GdddZ)dS)) annotationsN)CallableSequence)Path)Any)DCM_FILENAME_REGEXdownload_tcia_series_instanceget_tcia_metadataget_tcia_ref_uidmatch_tcia_ref_uid_in_study)download_and_extract)PathLike) CacheDataset PydicomReaderload_decathlon_datalistload_decathlon_propertiespartition_datasetselect_cross_validation_folds) LoadImaged Randomizable) ensure_tuple)MedNISTDatasetDecathlonDatasetCrossValidation TciaDatasetc @sdeZdZdZdZdZdZdZdddd d ej d d d d d df d1d$d%Z d2d(d)Z d3d*d+Z d4d.d/Z d0S)5raQ The Dataset to automatically download MedNIST data and generate items for training, validation or test. It's based on `CacheDataset` to accelerate the training process. Args: root_dir: target directory to download and load MedNIST dataset. section: expected data section, can be: `training`, `validation` or `test`. transform: transforms to execute operations on input data. download: whether to download and extract the MedNIST from resource link, default is False. if expected file already exists, skip downloading even set it to True. user can manually copy `MedNIST.tar.gz` file or `MedNIST` folder to root directory. seed: random seed to randomly split training, validation and test datasets, default is 0. val_frac: percentage of validation fraction in the whole dataset, default is 0.1. test_frac: percentage of test fraction in the whole dataset, default is 0.1. cache_num: number of items to be cached. Default is `sys.maxsize`. will take the minimum of (cache_num, data_length x cache_rate, data_length). cache_rate: percentage of cached data in total, default is 1.0 (cache all). will take the minimum of (cache_num, data_length x cache_rate, data_length). num_workers: the number of worker threads if computing cache in the initialization. If num_workers is None then the number returned by os.cpu_count() is used. If a value less than 1 is specified, 1 will be used instead. progress: whether to display a progress bar when downloading dataset and computing the transform cache content. copy_cache: whether to `deepcopy` the cache content before applying the random transforms, default to `True`. if the random transforms don't modify the cached content (for example, randomly crop from the cached image and deepcopy the crop region) or if every cache item is only used once in a `multi-processing` environment, may set `copy=False` for better performance. as_contiguous: whether to convert the cached NumPy array or PyTorch tensor to be contiguous. it may help improve the performance of following logic. runtime_cache: whether to compute cache at the runtime, default to `False` to prepare the cache content at initialization. See: :py:class:`monai.data.CacheDataset`. Raises: ValueError: When ``root_dir`` is not a directory. RuntimeError: When ``dataset_dir`` doesn't exist and downloading is not selected (``download=False``). z]https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/MedNIST.tar.gzZ 0bc7306e7427e00ad1c5526a6677552dzMedNIST.tar.gzZMedNISTFrg??Troot_dirr sectionstr transformSequence[Callable] | Callabledownloadboolseedintval_fracfloat test_frac cache_num cache_rate num_workers int | Noneprogress copy_cache as_contiguous runtime_cachereturnNonec Cst|}|s td||_||_||_|j|d||j}||j}d|_ |r6t |j |||j d| d|sBt d|d||}|dkrOtd }tj||||| | | | | |d dS) N,Root directory root_dir must be a directory.r%rmd5urlfilepath output_dirZhash_valZ hash_typer.Cannot find dataset directory: *, please use download=True to download it.rimage datar!r*r+r,r.r/r0r1)ris_dir ValueErrorrr'r)set_random_statecompressed_file_namedataset_folder_name num_classr resourcer6 RuntimeError_generate_data_listrr__init__)selfrrr!r#r%r'r)r*r+r,r.r/r0r1 tarfile_name dataset_dirr?rrU/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/apps/datasets.pyrI[sL       zMedNISTDataset.__init__r? np.ndarraycC|j|dSNRshufflerJr?rrrM randomizezMedNISTDataset.randomizecC|jS)zGet number of classes.)rErJrrrMget_num_classesszMedNISTDataset.get_num_classesrL list[dict]c s`ttddDt|_fddt|jDfddt|jD}gggt|jD]}||g|||g||q8t}t|}| |t ||j }t ||j }|j dkr|d|}n%|j dkr||||}n|j d kr|||d}n td |j d fd d|DS) zu Raises: ValueError: When ``section`` is not one of ["training", "validation", "test"]. css |] }|r|jVqdSrPr@name.0xrrrM sz5MedNISTDataset._generate_data_list..cs&g|]}dd|DqS)cSsg|]}|qSrrr]rrrM szAMedNISTDataset._generate_data_list...)iterdirr^i) class_namesrLrrMras&z6MedNISTDataset._generate_data_list..csg|]}t|qSr)lenrc) image_filesrrMratestN validationtrainingzUnsupported section: z;, available options are ["training", "validation", "test"].cs$g|]}|||dqS))r=label class_namerrc)rm image_classimage_files_listrrMras)rsortedrbrfrErangeextendnparangerUr&r)r'rrA) rJrLZnum_eachrdlengthindicesZ test_length val_lengthZsection_indicesr)rmrerLrnrgrorMrHs:       z"MedNISTDataset._generate_data_listN)rr rr r!r"r#r$r%r&r'r(r)r(r*r&r+r(r,r-r.r$r/r$r0r$r1r$r2r3r?rNr2r3)r2r&rLr r2rZ)__name__ __module__ __qualname____doc__rFr6rCrDsysmaxsizerIrUrYrHrrrrMr/s,& 9 rc @seZdZdZdddddddd d d d Zd dddddddddd Zddddejddddddf dId4d5ZdJd7d8Z dKd:d;Z dLdMd@dAZ dNdDdEZ dOdGdHZ d)rr@rArr'rBrFlistkeysr r6existsrGrsarrayrvrHr _propertiesrrrI)rJrrrr!r#r%r'r*r+r,r.r/r0r1rLrKr?Z property_keysrrrMrI sR           zDecathlonDataset.__init__rNcCrWzD Get the indices of datalist used in this dataset. rvrXrrrM get_indicesgzDecathlonDataset.get_indicesr?cCrOrPrQrTrrrMrUnrVzDecathlonDataset.randomizeNrSequence[str] | str | Nonedictcs2|durjSjdurfddt|DSiS)z Get the loaded properties of dataset with specified keys. If no keys specified, return all the loaded properties. Ncsi|]}|j|qSr)r)r^keyrXrrM zrhz3DecathlonDataset.get_properties..)rr)rJrrrXrMget_propertiesqs  zDecathlonDataset.get_propertiesrLrZcCs4t|}|jdvr dnd}t|dd|}||S)N)rkrjrkrirT)rrr_split_datalist)rJrLrdatalistrrrMrH}s z$DecathlonDataset._generate_data_listrct|jdkrSt}t|}||t||j}|jdkr)||d|_n|d||_fdd|jDS)Nrirkcg|]}|qSrrrcrrrMraz4DecathlonDataset._split_datalist..rrfrsrtrUr&r'rvrJrrurvrwrrrMr    z DecathlonDataset._split_datalist)rr rr rr r!r"r#r$r%r&r'r(r*r&r+r(r,r&r.r$r/r$r0r$r1r$r2r3r2rNrxrP)rrr2rryrrZr2rZ)rzr{r|r}rFr6r~rrIrrUrrHrrrrrMrsRB  G   rc@steZdZdZdddddddd ed d ejd d ddddfdAd-d.ZdBd0d1ZdCd3d4Z dDd7d8Z dEd;d<Z dFd>d?Z d@S)GraA The Dataset to automatically download the data from a public The Cancer Imaging Archive (TCIA) dataset and generate items for training, validation or test. The Highdicom library is used to load dicom data with modality "SEG", but only a part of collections are supported, such as: "C4KC-KiTS", "NSCLC-Radiomics", "NSCLC-Radiomics-Interobserver1", " QIN-PROSTATE-Repeatability" and "PROSTATEx". Therefore, if "seg" is included in `keys` of the `LoadImaged` transform and loading some other collections, errors may be raised. For supported collections, the original "SEG" information may not always be consistent for each dicom file. Therefore, to avoid creating different format of labels, please use the `label_dict` argument of `PydicomReader` when calling the `LoadImaged` transform. The prepared label dicts of collections that are mentioned above is also saved in: `monai.apps.tcia.TCIA_LABEL_DICT`. You can also refer to the second example bellow. This class is based on :py:class:`monai.data.CacheDataset` to accelerate the training process. Args: root_dir: user's local directory for caching and loading the TCIA dataset. collection: name of a TCIA collection. a TCIA dataset is defined as a collection. Please check the following list to browse the collection list (only public collections can be downloaded): https://www.cancerimagingarchive.net/collections/ section: expected data section, can be: `training`, `validation` or `test`. transform: transforms to execute operations on input data. for further usage, use `EnsureChannelFirstd` to convert the shape to [C, H, W, D]. If not specified, `LoadImaged(reader="PydicomReader", keys=["image"])` will be used as the default transform. In addition, we suggest to set the argument `labels` for `PydicomReader` if segmentations are needed to be loaded. The original labels for each dicom series may be different, using this argument is able to unify the format of labels. download: whether to download and extract the dataset, default is False. if expected file already exists, skip downloading even set it to True. user can manually copy tar file or dataset folder to the root directory. download_len: number of series that will be downloaded, the value should be larger than 0 or -1, where -1 means all series will be downloaded. Default is -1. seg_type: modality type of segmentation that is used to do the first step download. Default is "SEG". modality_tag: tag of modality. Default is (0x0008, 0x0060). ref_series_uid_tag: tag of referenced Series Instance UID. Default is (0x0020, 0x000e). ref_sop_uid_tag: tag of referenced SOP Instance UID. Default is (0x0008, 0x1155). specific_tags: tags that will be loaded for "SEG" series. This argument will be used in `monai.data.PydicomReader`. Default is [(0x0008, 0x1115), (0x0008,0x1140), (0x3006, 0x0010), (0x0020,0x000D), (0x0010,0x0010), (0x0010,0x0020), (0x0020,0x0011), (0x0020,0x0012)]. fname_regex: a regular expression to match the file names when the input is a folder. If provided, only the matched files will be included. For example, to include the file name "image_0001.dcm", the regular expression could be `".*image_(\d+).dcm"`. Default to `"^(?!.*LICENSE).*"`, ignoring any file name containing `"LICENSE"`. val_frac: percentage of validation fraction in the whole dataset, default is 0.2. seed: random seed to randomly shuffle the datalist before splitting into training and validation, default is 0. note to set same seed for `training` and `validation` sections. cache_num: number of items to be cached. Default is `sys.maxsize`. will take the minimum of (cache_num, data_length x cache_rate, data_length). cache_rate: percentage of cached data in total, default is 0.0 (no cache). will take the minimum of (cache_num, data_length x cache_rate, data_length). num_workers: the number of worker threads if computing cache in the initialization. If num_workers is None then the number returned by os.cpu_count() is used. If a value less than 1 is specified, 1 will be used instead. progress: whether to display a progress bar when downloading dataset and computing the transform cache content. copy_cache: whether to `deepcopy` the cache content before applying the random transforms, default to `True`. if the random transforms don't modify the cached content (for example, randomly crop from the cached image and deepcopy the crop region) or if every cache item is only used once in a `multi-processing` environment, may set `copy=False` for better performance. as_contiguous: whether to convert the cached NumPy array or PyTorch tensor to be contiguous. it may help improve the performance of following logic. runtime_cache: whether to compute cache at the runtime, default to `False` to prepare the cache content at initialization. See: :py:class:`monai.data.CacheDataset`. Example:: # collection is "Pancreatic-CT-CBCT-SEG", seg_type is "RTSTRUCT" data = TciaDataset( root_dir="./", collection="Pancreatic-CT-CBCT-SEG", seg_type="RTSTRUCT", download=True ) # collection is "C4KC-KiTS", seg_type is "SEG", and load both images and segmentations from monai.apps.tcia import TCIA_LABEL_DICT transform = Compose( [ LoadImaged(reader="PydicomReader", keys=["image", "seg"], label_dict=TCIA_LABEL_DICT["C4KC-KiTS"]), EnsureChannelFirstd(keys=["image", "seg"]), ResampleToMatchd(keys="image", key_dst="seg"), ] ) data = TciaDataset( root_dir="./", collection="C4KC-KiTS", section="validation", seed=12345, download=True ) print(data[0]["seg"].shape) rFZSEG)`) )riU))ri)ri@)i0)r )rr)rr)r)rrrgrTrr collectionr rr!r"r#r$ download_lenr&seg_type modality_tagtupleref_series_uid_tagref_sop_uid_tag specific_tags fname_regexr%r'r(r*r+r,r.r/r0r1r2r3c CsLt|}|s td||_||_||_||_| |_| |_|j | dt j ||}t | }||g7}||_|rjtd|d|dd}|dkrO|d|}t|dkr_td|d ||D]}|||qat j |sxtd |d | |_tg|_|||_|d krtd gd|jd}tj||j||||||||d dS)Nr4r5zgetSeries?Collection=z &Modality=ZSeriesInstanceUID)query attributerz"Cannot find data with collection: z seg_type: r;rrr=r)rreaderrr>)rr@rArr'rrrrrBospathjoinr load_tagsr rf_download_series_reference_datarrGrrsrrvrHrrrrI)rJrrrr!r#rrrrrrrr%r'r*r+r,r.r/r0r1 download_dirrZseg_series_list series_uidrrrMrIsV!        zTciaDataset.__init__rNcCrWrrrXrrrMrArzTciaDataset.get_indicesr?cCrOrPrQrTrrrMrUHrVzTciaDataset.randomizerrcCstj|d|}t|||ddddtt|D}tj||d}td|jd|}|j r5|j n|j }|sEt d |d d }|j rK|j n|j}|s[t d |d d}t|}tj||||j} tj|||d} g} |D]C} tj|| }td|jd|}||jj|jkrt|d|j|jd} | dkrt|d|j|jd}t|j|} | dkr| | qx| st d|dn t| d|| ddtj| st|| dSdS)z First of all, download a series from TCIA according to `series_uid`. Then find all referenced series and download. rawF)rrr:Z check_md5cSsg|] }|dr|qS)z.dcm)endswithr^frrrMraTz?TciaDataset._download_series_reference_data..rT)Zstop_before_pixelsrz+unable to find patient name of dicom file: z, use 'patient' instead.Zpatientz,unable to find series number of dicom file: z, use '0' instead.r=)Zfind_soprrz.cSsg|] }|r|jqSrr[rrrrMrarr=) rrscandirrrrrrrr) rJrLrZ patient_listrZ series_listrZseg_key image_pathZ mask_pathrrrMrHs   zTciaDataset._generate_data_listrcr)NrirkcrrrrcrrrMrarz/TciaDataset._split_datalist..rrrrrMrrzTciaDataset._split_datalistN),rr rr rr r!r"r#r$rr&rr rrrrrrrrrr r%r&r'r(r*r&r+r(r,r&r.r$r/r$r0r$r1r$r2r3rrx)rr rr r2r3ryr) rzr{r|r}rr~rrIrrUrrHrrrrrMrs4_  R   8rc@s&eZdZdZddd dZdddZdS)raa Cross validation dataset based on the general dataset which must have `_split_datalist` API. Args: dataset_cls: dataset class to be used to create the cross validation partitions. It must have `_split_datalist` API. nfolds: number of folds to split the data for cross validation. seed: random seed to randomly shuffle the datalist before splitting into N folds, default is 0. dataset_params: other additional parameters for the dataset_cls base class. Example of 5 folds cross validation training:: cvdataset = CrossValidation( dataset_cls=DecathlonDataset, nfolds=5, seed=12345, root_dir="./", task="Task09_Spleen", section="training", transform=train_transform, download=True, ) dataset_fold0_train = cvdataset.get_dataset(folds=[1, 2, 3, 4]) dataset_fold0_val = cvdataset.get_dataset(folds=0, transform=val_transform, download=False) # execute training for fold 0 ... dataset_fold1_train = cvdataset.get_dataset(folds=[0, 2, 3, 4]) dataset_fold1_val = cvdataset.get_dataset(folds=1, transform=val_transform, download=False) # execute training for fold 1 ... ... dataset_fold4_train = ... # execute training for fold 4 ... r dataset_clsobjectnfoldsr&r%dataset_paramsrr2r3cKs.t|ds td||_||_||_||_dS)Nrz,dataset class must have _split_datalist API.)hasattrrArrr%r)rJrrr%rrrrMrIs  zCrossValidation.__init__foldsSequence[int] | intc sH|j|jt|j}||Gfddd|j}|di|S)a Generate dataset based on the specified fold indices in the cross validation group. Args: folds: index of folds for training or validation, if a list of values, concatenate the data. dataset_params: other additional parameters for the dataset_cls base class, will override the same parameters in `self.dataset_params`. cseZdZdfdd ZdS)z4CrossValidation.get_dataset.._NsplitsDatasetrrZr2cst|dd}t|dS)NT)r?Znum_partitionsrSr%) partitionsr)rr)rJrr?rrr%rrMrs zDCrossValidation.get_dataset.._NsplitsDataset._split_datalistNr)rzr{r|rrrrrM_NsplitsDatasetsrNr)rr%rrupdater)rJrrZdataset_params_rrrrM get_datasets    zCrossValidation.get_datasetN)rr) rrrr&r%r&rrr2r3)rrrrr2r)rzr{r|r}rIrrrrrMrs %r)* __future__rrrr~rcollections.abcrrpathlibrtypingrnumpyrsZmonai.apps.tciarrr r r Zmonai.apps.utilsr monai.config.type_definitionsr Z monai.datarrrrrrmonai.transformsrr monai.utilsr__all__rrrrrrrrMs0       P