U PhH @sDddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z ddlZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZm Z m!Z!d ddddddddg Z"e dde\Z#Z$e d\Z%Z&ddddd Z'ddddddZ(dTdd d!d"d#dZ)dUd%d&d'd'd(dd)d*dZ*dVd%d&d+d d(d,d-d.dZ+dWd0d1d2d1d3d4d5dZ,d6d6d d7d8dZ-dXd9d:d(d1d;ddZ/d?d1d@dAdBZ0d6d1dCdDdEZ1dZd1dFd(d1dGdHdIZ2d1d(d1dJdKdLZ3d[d1dFd(d1dGdMdNZ4d1d(dOdJdPdQZ5d1d(dOdJdRdSZ6dS)\) annotationsN)deepcopy)Number)Anycast)Algo) ConfigParser) ID_SEP_KEY)PathLike MetaTensor)CropForegroundToCupy) min_versionoptional_importrun_cmdget_foreground_imageget_foreground_label get_label_ccpconcat_val_to_npconcat_multikeys_to_dict datafold_readverify_report_formatalgo_to_picklealgo_from_picklezskimage.measurez0.14.2cupyr z np.ndarray)imagereturncCs$tdddd}||}ttj|S)ay Get a foreground image by removing all-zero rectangles on the edges of the image Note for the developer: update select_fn if the foreground is defined differently. Args: image: ndarray image to segment. Returns: ndarray of foreground image by removing all-zero edges. Notes: the size of the output is smaller than the input. cSs|dkS)Nr)xrrJ/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/auto3dseg/utils.pyAz&get_foreground_image..T) select_fn allow_smaller)r rnpndarray)rcopperimage_foregroundrrr r2s)rlabelrcCst||dk}|S)a Get foreground image pixel values and mask out the non-labeled area. Args image: ndarray image to segment. label: ndarray the image input and annotated with class IDs. Returns: 1D array of foreground image with label > 0 rr )rr)Zlabel_foregroundrrr rFs Tboolztuple[list[Any], int]) mask_indexuse_gpurc s|td\}}g}|jjdkrtr|r|rt|}|j|}t |t |}|D]\}t ||k} tj | dd tj| dd fddttD} || qZt|} ~~~~ ~tntrltj|jddd \}} td | d D]^}t ||k} tj | dd tj| dd fd dttD} || q ntd || fS) a4 Find all connected components and their bounding shape. Backend can be cuPy/cuCIM or Numpy depending on the hardware. Args: mask_index: a binary mask. use_gpu: a switch to use GPU/CUDA or not. If GPU is unavailable, CPU will be used regardless of this setting. z cucim.skimagecudar)axiscs g|]}||dqSr.0iZ comp_idx_maxZ comp_idx_minrr lsz!get_label_ccp..T) background return_numr0cs g|]}||dqSr/rr1r4rr r5yszVCannot find one of the following required dependencies: {cuPy+cuCIM} or {scikit-image})rdevicetypehas_cprshortmeasurer)cpuniquenonzeroargwheremintolistmaxrangelenappendget_default_memory_poolfree_all_blocks has_measure measure_npdatacpunumpyr% RuntimeError) r+r,skimage has_cucim shape_listZ mask_cupyZlabeledvalsZncompZcomp_idxZ bbox_shape ncomponentsrr4r rVs2    Fz list[dict]zlist[str | int]z bool | Noner) data_list fixed_keysragged allow_missingkwargsrc KsFg}|D]}t|}t|D]\}} t| ||<q|t|} | dkrj|rZ|dqt|dqt| t r|t | qt| t j tfr|| qt| t jr|| qt| tr|t | qt| jdq|rdd|D}t|dkrt dgS|r2t j|f|St j|gf|SdS)a Get the nested value in a list of dictionary that shares the same structure. Args: data_list: a list of dictionary {key1: {key2: np.ndarray}}. fixed_keys: a list of keys that records to path to the value in the dict elements. ragged: if True, numbers can be in list of lists or ragged format so concat mode needs change. allow_missing: if True, it will return a None if the value cannot be found. Returns: nd.array of concatenated array. Nz is not nested in the dictionaryz concat is not supported.cSsg|]}|dk r|qS)Nr)r2rrrr r5sz$concat_val_to_np..r)r enumeratestrgetr joinrGAttributeError isinstancelistr%arraytorchTensorr rMrNr&rNotImplementedError __class__rF concatenate) rUrVrWrXrYZnp_listrLparserr3keyvalrrr rs4      z list[str]zdict[str, np.ndarray])rUrVkeys zero_insertrYrc KsDi}|D]6}|rd|gn|g}t|||f|}|||iq|S)a Get the nested value in a list of dictionary that shares the same structure iteratively on all keys. It returns a dictionary with keys with the found values in nd.ndarray. Args: data_list: a list of dictionary {key1: {key2: np.ndarray}}. fixed_keys: a list of keys that records to path to the value in the dict elements. keys: a list of string keys that will be iterated to generate a dict output. zero_insert: insert a zero in the list so that it can find the value in element 0 before getting the keys flatten: if True, numbers are flattened before concat. Returns: a dict with keys - nd.array of concatenated array pair. r)rupdate) rUrVrjrkrYret_dictrhZaddonrirrr rs trainingz str | dictr[intztuple[list, list])datalistbasedirfoldrhrc st|trt|}n|}t||}|D]|}|D]n\}}t||trhfdd||D||<q6t||tr6t||dkrtj ||n||||<q6q*g} g} |D].}d|kr|d|kr| |q| |q| | fS)a Read a list of data dictionary `datalist` Args: datalist: the name of a JSON file listing the data, or a dictionary. basedir: directory of image files. fold: which fold to use (0..1 if in training set). key: usually 'training' , but can try 'validation' or 'testing' to get the list data without labels (used in challenges). Returns: A tuple of two arrays (training, validation). csg|]}tj|qSr)ospathr])r2ivrqrr r5sz!datafold_read..rrr) r_r[rload_config_fileritemsr`rFrsrtr]rG) rprqrrrh json_dataZ dict_datadk_trrirrvr rs"   2  dict)report report_formatrcCs|D]z\}}||krdS||}t|trt|trt|dkrNtdt|dkr|t|dkr|t|d|dSdSqdS)z Compares the report and the report_format that has only keys. Args: report: dict that has real values. report_format: dict that only has keys and list-nested value. Fr0z%list length in report_format is not 1rT)rxr_r`rF UserWarningr)rrZk_fmtZv_fmtvrrr rs rzPathLike | None)algo template_pathalgo_meta_datarc Kstt|t|d}tj|d}|D]\}}|||iq.t|}t |d}| |W5QRX|S)a Export the Algo object to pickle file. Args: algo: Algo-like object. template_path: a str path that is needed to be added to the sys.path to instantiate the class. algo_meta_data: additional keyword to save into the dictionary, for example, model training info such as acc/best_metrics Returns: filename of the pickled Algo object ) algo_bytesrzalgo_object.pklwb) pickledumpsr[rsrtr]get_output_pathrxrlopenwrite) rrrrL pkl_filenamer{r data_bytesf_pirrr rs   )rrrYrc Ksht|d}|}W5QRXt|}t|tsDtd|jdd|kr\td|d|d}|dd}g}t j t |r| t j t || t j t j t |d t j t |r| t j || t j t j |d t j |} t j | d d } t j | r6| t j | t|d krVt|} d| _nt|D]\} } z"tj | t|} WqWnhtk r}zHtd | d tj | t|dkrtd|d||W5d}~XYnXq^| | _t j | t j | kr:t| d| d| | _i}|D]\}}|||iqF| |fS)a Import the Algo object from a pickle file. Args: pkl_filename: the name of the pickle file. template_path: a folder containing files to instantiate the Algo. Besides the `template_path`, this function will also attempt to use the `template_path` saved in the pickle file and a directory named `algorithm_templates` in the parent folder of the folder containing the pickle file. Returns: algo: the Algo object saved in the pickle file. algo_meta_data: additional keyword saved in the pickle file, for example, acc/best_metrics. Raises: ValueError if the pkl_filename does not contain a dict, or the dict does not contain `algo_bytes`. ModuleNotFoundError if it is unable to instantiate the Algo class. rbzthe data object is z. Dict is expected.rzkey [algo_bytes] not found in z. Unable to instantiate.rNz..algorithm_templatesrzFolder z; doesn't contain the Algo templates for Algo instantiation.r0zFailed to instantiate z with z5 is changed. Now override the Algo output_path with: .)rreadrloadsr_r~ ValueErrorrepoprsrtisdirr[rGabspathr]dirnamerFrrZsysModuleNotFoundErrorloggingdebugr output_pathrxrl)rrrYrrrLrZalgo_template_pathZtemplate_paths_candidatesZpkl_dirZalgo_template_path_fuzzyrr3p not_found_errrr{rrrr r,sZ            r`)argsrcCs ddd|D}d|dS)z Convert a list of arguments to a string that can be used in python-fire. Args: args: the list of arguments. Returns: the string that can be used in python-fire. ,cSsg|] }t|qSr)r[)r2argrrr r5sz/list_to_python_fire_arg_str..')r])rargs_strrrr list_to_python_fire_arg_strzs r)paramsrcCsTd}|D]B\}}t|tr(tdnt|tr:t|}|d|d|7}q |S)z;convert `params` into '--key_1=value_1 --key_2=value_2 ...'zNested dict is not supported.z --=)rxr_r~rr`r)rZ cmd_mod_optr{rrrr check_and_set_optional_argss   rz str | None)cmd cmd_prefixrYrcKs:|}|rd|krd}|ds*|d7}||t|S)a Prepare the command for subprocess to run the script with the given arguments. Args: cmd: the command or script to run in the distributed job. cmd_prefix: the command prefix to run the script, e.g., "python", "python -m", "python3", "/opt/conda/bin/python3.8 ". kwargs: the keyword arguments to be passed to the script. Returns: the command to run with ``subprocess``. Examples: To prepare a subprocess command "python train.py run -k --config 'a,b'", the function can be called as - _prepare_cmd_default("train.py run -k", config=['a','b']) - _prepare_cmd_default("train.py run -k --config 'a,b'") Nonepython )copyendswithr)rrrYrrrr _prepare_cmd_defaults   r)rrYrcKs|}|t|S)a Prepare the command for multi-gpu/multi-node job execution using torchrun. Args: cmd: the command or script to run in the distributed job. kwargs: the keyword arguments to be passed to the script. Returns: the command to append to ``torchrun`` Examples: For command "torchrun --nnodes=1 --nproc_per_node=8 train.py run -k --config 'a,b'", it only prepares command after the torchrun arguments, i.e., "train.py run -k --config 'a,b'". The function can be called as - _prepare_cmd_torchrun("train.py run -k", config=['a','b']) - _prepare_cmd_torchrun("train.py run -k --config 'a,b'") )rr)rrYrrrr _prepare_cmd_torchrunsrcKst|fd|i|S)a Prepare the command for distributed job running using bcprun. Args: script: the script to run in the distributed job. cmd_prefix: the command prefix to run the script, e.g., "python". kwargs: the keyword arguments to be passed to the script. Returns: The command to run the script in the distributed job. Examples: For command "bcprun -n 2 -p 8 -c python train.py run -k --config 'a,b'", it only prepares command after the bcprun arguments, i.e., "train.py run -k --config 'a,b'". the function can be called as - _prepare_cmd_bcprun("train.py run -k", config=['a','b'], n=2, p=8) - _prepare_cmd_bcprun("train.py run -k --config 'a,b'", n=2, p=8) r)r)rrrYrrr _prepare_cmd_bcprunsrzsubprocess.CompletedProcesscKsx|}|}dg}ddg}|D]8}||kr>td|d|d|t||g7}q"||7}t|fddi|S) a Run the command with torchrun. Args: cmd: the command to run. Typically it is prepared by ``_prepare_cmd_torchrun``. kwargs: the keyword arguments to be passed to the ``torchrun``. Return: the return code of the subprocess command. Ztorchrunnnodesnproc_per_nodeMissing required argument z for torchrun.z--run_cmd_verboseT)rsplitrr[rr)rrYrcmd_listZ torchrun_list required_argsrrrr _run_cmd_torchruns rcKsv|}dg}ddg}|D]8}||kr6td|d|d|t||g7}q|d|gt|fdd i|S) a Run the command with bcprun. Args: cmd: the command to run. Typically it is prepared by ``_prepare_cmd_bcprun``. kwargs: the keyword arguments to be passed to the ``bcprun``. Returns: the return code of the subprocess command. bcprunnrrz for bcprun.-z-crT)rrr[rextendr)rrYrrrrrrr _run_cmd_bcpruns r)T)FF)T)rrn)N)N)N)N)7 __future__rrrsr subprocessrrrnumbersrtypingrrrNr%rbZmonai.auto3dsegrmonai.bundle.config_parserrmonai.bundle.utilsr monai.configr monai.data.meta_tensorr monai.transformsr r monai.utilsrrr__all__rKrJr>r;rrrrrrrrrrrrrrrrrrrr  s`          .:'N