o  iH@sLddlmZddlZddlmZddlmZmZddlZ ddl Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ddZ,e(ddrej-j./e&e,e(dde'd\Z0Z1ee2dZ3dgZ4GdddZ5dS)) annotationsN)path)Anycast) get_context)EnsureSameShaped) get_logger) SegSummarizer) datafold_read) config_parser) ConfigParser) DataLoaderDatasetpartition_dataset) no_collation)Compose EnsureTyped LoadImaged Orientationd) ImageMetaKeyStrEnum min_versionoptional_import) DataStatsKeysImageStatsKeyscCs|d|jS)Nztag:yaml.org,2002:str)represent_scalarvalue)dumperdatard/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/apps/auto3dseg/data_analyzer.pystrenum_representer#sr!yamltqdmz4.47.0) module_name DataAnalyzerc@sbeZdZdZ        d8d9d$d%Zed:d+d,Zd;d.d/Z 0 - dr&a The DataAnalyzer automatically analyzes given medical image dataset and reports the statistics. The module expects file paths to the image data and utilizes the LoadImaged transform to read the files, which supports nii, nii.gz, png, jpg, bmp, npz, npy, and dcm formats. Currently, only segmentation task is supported, so the user needs to provide paths to the image and label files (if have). Also, label data format is preferred to be (1,H,W,D), with the label index in the first dimension. If it is in onehot format, it will be converted to the preferred format. Args: datalist: a Python dictionary storing group, fold, and other information of the medical image dataset, or a string to the JSON file storing the dictionary. dataroot: user's local directory containing the datasets. output_path: path to save the analysis result. average: whether to average the statistical value across different image modalities. do_ccp: apply the connected component algorithm to process the labels/images device: a string specifying hardware (CUDA/CPU) utilized for the operations. worker: number of workers to use for loading datasets in each GPU/CPU sub-process. image_key: a string that user specify for the image. The DataAnalyzer will look it up in the datalist to locate the image files of the dataset. label_key: a string that user specify for the label. The DataAnalyzer will look it up in the datalist to locate the label files of the dataset. If label_key is NoneType or "None", the DataAnalyzer will skip looking for labels and all label-related operations. hist_bins: bins to compute histogram for each image channel. hist_range: ranges to compute histogram for each image channel. fmt: format used to save the analysis results. Currently support ``"json"`` and ``"yaml"``, defaults to "yaml". histogram_only: whether to only compute histograms. Defaults to False. extra_params: other optional arguments. Currently supported arguments are : 'allowed_shape_difference' (default 5) can be used to change the default tolerance of the allowed shape differences between the image and label items. In case of shape mismatch below the tolerance, the label image will be resized to match the image using nearest interpolation. Examples: .. code-block:: python from monai.apps.auto3dseg.data_analyzer import DataAnalyzer datalist = { "testing": [{"image": "image_003.nii.gz"}], "training": [ {"fold": 0, "image": "image_001.nii.gz", "label": "label_001.nii.gz"}, {"fold": 0, "image": "image_002.nii.gz", "label": "label_002.nii.gz"}, {"fold": 1, "image": "image_001.nii.gz", "label": "label_001.nii.gz"}, {"fold": 1, "image": "image_004.nii.gz", "label": "label_004.nii.gz"}, ], } dataroot = '/datasets' # the directory where you have the image files (nii.gz) DataAnalyzer(datalist, dataroot) Notes: The module can also be called from the command line interface (CLI). For example: .. code-block:: bash python -m monai.apps.auto3dseg \ DataAnalyzer \ get_all_case_stats \ --datalist="my_datalist.json" \ --dataroot="my_dataroot_dir" ./datastats.yamlTFcudaimagelabelrNr"datalist str | dictdatarootstr output_pathaveragebooldo_ccpdevicestr | torch.deviceworkerint image_key label_key str | None hist_binslist | int | None hist_range list | Nonefmthistogram_only extra_paramsrcKst|rtd|dt|d||_||_||_||_ ||_ t ||_ ||_ ||_| dkr7dn| |_| |_| durEddgn| |_| |_| |_||_dS)NzFile z( already exists and will be overwritten.z' will be overwritten by a new datastat.Nonei i)risfilewarningswarnloggerdebugr-r/r1r2r4torchr5r7r9r:r<r>r@rArB)selfr-r/r1r2r4r5r7r9r:r<r>r@rArBrrr __init__rs"   zDataAnalyzer.__init__keys list[str]resultdictreturncsvtjvs tjtjvrdSfdd|D}|D]}d|vr8t|dr8td|d|dddSqdS) ay Check data uniformity since DataAnalyzer provides no support to multi-modal images with different affine matrices/spacings due to monai transforms. Args: keys: a list of string-type keys under image_stats dictionary. Returns: False if one of the selected key values is not constant across the dataset images. Tcs g|] }tjtj|qSr)rSUMMARY IMAGE_STATS).0keyrNrr s z7DataAnalyzer._check_data_uniformity..stdevzsummary image_stats z has non-zero stdev .F)rrQrRnpanyrGrH)rLrNZconstant_propsproprrUr _check_data_uniformitysz#DataAnalyzer._check_data_uniformitytrainingc Csntjitjgi}tjitjgi}|jjdkrd}tdntj }td|d|dkrt d}| I}| }g} t |D]} |j|j| ||||fd} | | qC| D]} | qZ| D]} | qc|D] } |tj| tjqlWdn1swYn |d dd||}t|j|j|j|j|j|j|jd } t|tj}| tt |tj|tj<||tjd <|tj|tj<|t j!g|std |j"r!td |j"dt#j$||j"|j%ddd|j"&d|j%d|j%}||j"kr |d|j%7}td|dt#j$|||j%ddd|jjdkr-tj'|tj|tj<|S)a  Get all case stats. Caller of the DataAnalyser class. The function initiates multiple GPU or CPU processes of the internal _get_all_case_stats functions, which iterates datalist and call SegSummarizer to generate stats for each case. After all case stats are generated, SegSummarizer is called to combine results. Args: key: dataset key transform_list: option list of transforms before SegSummarizer Returns: A data statistics dictionary containing "stats_summary" (summary statistics of the entire datasets). Within stats_summary there are "image_stats" (summarizing info of shape, channel, spacing, and etc using operations_summary), "image_foreground_stats" (info of the intensity for the non-zero labeled voxels), and "label_stats" (info of the labels, pixel percentage, image_intensity, and each individual label in a list) "stats_by_cases" (List type value. Each element of the list is statistics of an image-label info. Within each element, there are: "image" (value is the path to an image), "label" (value is the path to the corresponding label), "image_stats" (summarizing info of shape, channel, spacing, and etc using operations), "image_foreground_stats" (similar to the previous one but one foreground image), and "label_stats" (stats of the individual labels ) Notes: Since the backend of the statistics computation are torch/numpy, nan/inf value may be generated and carried over in the computation. In such cases, the output dictionary will include .nan/.inf in the statistics. cpur#zUsing CPU for data analyzing!zFound z GPUs for data analyzing! forkserver)targetargsNrr2r4r<r>rAn_caseszVData spacing is not completely uniform. MONAI transforms may provide unexpected resultzWriting data stats to rXF)r@default_flow_style sort_keysz _by_case.z .by_case.zWriting by-case data stats to z, this may take a while.r))(rrQBY_CASEr5typerGinforIr) device_countrManagerlistrangeProcess_get_all_case_statsappendstartjoinextendr r9r:r2r4r<r>rAlen summarizerr\rSPACINGr1r export_config_filer@replace empty_cache)rJrTtransform_listrN result_bycasenprocsZtmp_ctxmanager manager_list processesrankp_ summarizerrcZ by_case_pathrrr get_all_case_statsst             zDataAnalyzer.get_all_case_statsr#r world_sizer}rTrycCsBt|j|j|j|j|j|j|jd}tt d|j|jg}|durMt |dddt |dt j dt|ddg}|jdurM|jd d }|t|j|j|d t|} t|j|jd |d \} } |t| krlt| |d|} n|t| kr|t| t| d|ng} t| | d} t| dd|jt|jjdkd} tj itj!gi}|jjdkr|jnt d|}|jdkrt j"#rt j"$dkst%&d|dt d}t'st()dt'r|dkrt*| n| D]5}|d}z;||j+|||j<d}|jdur||j}|j,ddkr t j-|ddn|d}d}|+|||j<||}Wnt.y}zd|/vr6|dt0j1}n ||jj2t0j1}t%&d|d|d||jjdkrt%&dz:||j+d||j<|jdur||j}|s|j,ddkrt j-|ddn|d}|+d||j<||}Wn,t.y}zt%&d|d|d|WYd}~WYd}~qd}~wwWYd}~qWYd}~nd}~wwtj3|tj3tj4|tj4i}|js|tj5|tj5<|jdkr|tj6|tj6<|jdur |7tj8|tj8tj9|tj9i|tj!|q|dur|S||dS)a Get all case stats from a partitioned datalist. The function can only be called internally by get_all_case_stats. Args: rank: GPU process rank, 0 for CPU process world_size: total number of GPUs, 1 for CPU process manager_list: multiprocessing manager list object, if using multi-GPU. key: dataset key transform_list: option list of transforms before SegSummarizer rbNT)rLensure_channel_first image_onlytensor)rL data_typedtypeRAS)rLaxcodesallowed_shape_difference)rL source_keyr)r-basedirfoldrT)rnum_partitions)r transformr#Fr)) batch_sizeshuffle num_workers collate_fn pin_memoryr^rzdevice=z5 but CUDA device is not available, using CPU instead.z;tqdm is not installed. not displaying the caching progress.)dimZimage_meta_dictzUnable to process data z on z. zSDataAnalyzer `device` set to GPU execution hit an exception. Falling back to `cpu`.):r r9r:r2r4r<r>rArkfilterrrrIfloatrrBpoprorrr r-r/rsrrr r7rr5rgrrQrfr) is_availablerirGrhhas_tqdmrErFr$toshapeargmax BaseExceptionrLrFILENAME_OR_OBJmetaBY_CASE_IMAGE_PATHBY_CASE_LABEL_PATHrRIMAGE_HISTOGRAMupdateFG_IMAGE_STATS LABEL_STATS)rJrrr}rTryrrLrrfilesrdataset dataloaderrzr5 batch_dataZ _label_argmaxr,derrfilenamestats_by_casesrrr rn s     $  "    &    &          z DataAnalyzer._get_all_case_stats) r'r(TFr)r*r+r,rNr"F)r-r.r/r0r1r0r2r3r4r3r5r6r7r8r9r0r:r;r<r=r>r?r@r0rAr3rBr)rLrMrNrOrPr3)r]N)rr#Nr]N) rr8rr8r}r?rTr0ryr?rPr) __name__ __module__ __qualname____doc__rK staticmethodr\rrnrrrr r&0s0D $  _)6 __future__rrEosrtypingrrnumpyrYrItorch.multiprocessingrZmonai.apps.auto3dseg.transformsrmonai.apps.utilsrmonai.auto3dsegr monai.auto3dseg.utilsr monai.bundler monai.bundle.config_parserr monai.datar rrmonai.data.utilsrmonai.transformsrrrr monai.utilsrrrrmonai.utils.enumsrrr!r" SafeDumperadd_multi_representerr$rrrG__all__r&rrrr s2