o  iBq@sddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZddlmZddlmZddlZddlmZdd lmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ee/dZ0e.1Z2ddgZ3GdddeZ4de2dZ5e6dde6dde6dde6dddZ7d'd"d#Z8d$d%Z9Gd&ddeZ:dS)() annotationsN)deepcopy)Path)TemporaryDirectory)Any)urlparse)download_and_extract) get_logger)AlgoAlgoGen)_prepare_cmd_bcprun_prepare_cmd_default_prepare_cmd_torchrun_run_cmd_bcprun_run_cmd_torchrunalgo_to_pickle) ConfigParser)PathLike) ensure_tuplelook_up_optionrun_cmd)AlgoKeys) MONAIEnvVars) module_name BundleAlgo BundleGenc@seZdZdZdCddZdDdEddZdFddZdGddZdHddZdIddZ dJd#d$Z dKd'd(Z dLdMd-d.Z dNdOd2d3Z )dPdQd5d6Zd7d8Zd9d:ZdLdRd?d@ZdAdBZd)S)Sra5 An algorithm represented by a set of bundle configurations and scripts. ``BundleAlgo.cfg`` is a ``monai.bundle.ConfigParser`` instance. .. code-block:: python from monai.apps.auto3dseg import BundleAlgo data_stats_yaml = "../datastats.yaml" algo = BundleAlgo(template_path="../algorithm_templates") algo.set_data_stats(data_stats_yaml) # algo.set_data_src("../data_src.json") algo.export_to_disk(".", algo_name="segresnet2d_1") This class creates MONAI bundles from a directory of 'bundle template'. Different from the regular MONAI bundle format, the bundle template may contain placeholders that must be filled using ``fill_template_config`` during ``export_to_disk``. Then created bundle keeps the same file structure as the template. template_pathrcCs||_d|_d|_d|_d|_d|_d|_d|_i|_d ddt t j Dtt j ttjddtjdd tjd dd |_dS) a0 Create an Algo instance based on the predefined Algo template. Args: template_path: path to a folder that contains the algorithm templates. Please check https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg/algorithm_templates N,cSsg|]}t|qS)str).0xrra/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/apps/auto3dseg/bundle_gen.py asz'BundleAlgo.__init__.. NUM_NODESMN_START_METHODbcprun CMD_PREFIX)CUDA_VISIBLE_DEVICES n_devicesr%r'r))rdata_stats_filesdata_list_filemlflow_tracking_urimlflow_experiment_name output_pathname best_metric fill_recordsjoinrangetorchcuda device_countintosenvirongetdevice_setting)selfrrrr#__init__Ks     zBundleAlgo.__init__Frskip_bundlegenbool skip_infor returntuple[bool, str]cCs||fS)a Analyse the data analysis report and check if the algorithm needs to be skipped. This function is overriden within algo. Args: skip_bundlegen: skip generating bundles for this algo if true. skip_info: info to print when skipped. r)r>r@rBrrr#pre_check_skip_algohszBundleAlgo.pre_check_skip_algor,NonecC ||_dS)z Set the data analysis report (generated by DataAnalyzer). Args: data_stats_files: path to the datastats yaml file N)r,)r>r,rrr#set_data_statsr zBundleAlgo.set_data_stats data_src_cfgcCrG)aN Set the data source configuration file Args: data_src_cfg: path to a configuration file (yaml) that contains datalist, dataroot, and other params. The config will be in a form of {"modality": "ct", "datalist": "path_to_json_datalist", "dataroot": "path_dir_data"} N)r-)r>rJrrr#set_data_source{ zBundleAlgo.set_data_sourcer. str | NonecCrGaI Set the tracking URI for MLflow server Args: mlflow_tracking_uri: a tracking URI for MLflow server which could be local directory or address of the remote tracking Server; MLflow runs will be recorded locally in algorithms' model folder if the value is None. Nr.r>r.rrr#set_mlflow_tracking_urirLz"BundleAlgo.set_mlflow_tracking_urir/cCrGz Set the experiment name for MLflow server Args: mlflow_experiment_name: a string to specify the experiment name for MLflow server. Nr/r>r/rrr#set_mlflow_experiment_namerIz%BundleAlgo.set_mlflow_experiment_namedata_stats_filename algo_pathkwargsrdictcKsiS)a The configuration files defined when constructing this Algo instance might not have a complete training and validation pipelines. Some configuration components and hyperparameters of the pipelines depend on the training data and other factors. This API is provided to allow the creation of fully functioning config files. Return the records of filling template config: {"": {"": value, ...}, ...}. Args: data_stats_filename: filename of the data stats report (generated by DataAnalyzer) Notes: Template filling is optional. The user can construct a set of pre-filled configs without replacing values by using the data analysis results. It is also intended to be re-implemented in subclasses of BundleAlgo if the user wants their own way of auto-configured template filling. r)r>rVrWrXrrr#fill_template_configszBundleAlgo.fill_template_configr0 algo_namecKs|ddr4tj|||_tj|jddtj|jr#t|jt tjt |j |j |jnt |j |_|ddrM|j |j|jfi||_td|jdS)a Fill the configuration templates, write the bundle (configs + scripts) to folder `output_path/algo_name`. Args: output_path: Path to export the 'scripts' and 'configs' directories. algo_name: the identifier of the algorithm (usually contains the name and extra info like fold ID). kwargs: other parameters, including: "copy_dirs=True/False" means whether to copy the template as output instead of inplace operation, "fill_template=True/False" means whether to fill the placeholders in the template. other parameters are for `fill_template_config` function. Z copy_dirsT)exist_okZ fill_templatez Generated:N)popr:pathr4r0makedirsisdirshutilrmtreecopytreer rr1rZr,r3loggerinfo)r>r0r[rXrrr#export_to_disks  "  zBundleAlgo.export_to_diskN train_params None | dicttuple[str, str]c Csb|duri}t|}tj|jdd}tj|jd}g}tj|rEtt|D]}|ds6|drD| t tj|| q*t |j ddkrz t|j d d gWntyp}z t|j d d |d}~wwt|d f|j d |d|dfSt |j ddkrt|d fd|i|dfSt|d f|j d |d|dfS)z: Create the command to execute training. Nscriptsztrain.pyconfigsyamljsonr%r&r'r(zI is not supported yet.Try modify BundleAlgo._create_cmd for your cluster.z runr)) cmd_prefix config_filerr+ro)rr:r^r4r0r`sortedlistdirendswithappendras_posixr9r=r ValueErrorNotImplementedErrorr rr )r>rgparamsZtrain_py config_dirZ config_filesfileerrrrr# _create_cmdsX    zBundleAlgo._create_cmdcmd devices_infosubprocess.CompletedProcessc Cs|r td|dtj}t|jd|d<tdd|}t |jddkrYz t |jdd gWnt yK}z t |jdd |d }~wwt ||jd|jd d St |jd dkrnt|d|jd |ddSt|d|ddS)zP Execute the training command with target devices information. zinput devices_info z is deprecated and ignored.r*z^\s*\w+=.*?\s+rr%r&r'r(zF is not supported yet.Try modify BundleAlgo._run_cmd for your cluster.Nr+)npT)nnodesZnproc_per_nodeenvcheck)run_cmd_verboserr)warningswarnr:r;copyr r=resubr9rrurvrrrsplit)r>r|r}Z ps_environrzrrr#_run_cmds, zBundleAlgo._run_cmdr=cCsn|dur|j|tt|jdd|jd<|dur+d|vr+td|d||\}}| |S)a Load the run function in the training script of each model. Training parameter is predefined by the algo_config.yaml file, which is pre-filled by the fill_template_config function in the same instance. Args: train_params: training parameters device_setting: device related settings, should follow the device_setting in auto_runner.set_device_info. 'CUDA_VISIBLE_DEVICES' should be a string e.g. '0,1,2,3' Nr*rr+z5CUDA_VISIBLE_DEVICES is deprecated from train_params!) r=updatelenr rrrr]r{r)r>rgr=r|Z_unused_returnrrr#trains    zBundleAlgo.traincOsRtj|jdd}t}|||jd|jd}ttj|d}|ddS)zU Returns validation scores of the model trained by the current Algo. rkzhyper_parameters.yaml ckpt_path)defaultz progress.yamlZbest_avg_dice_score)r:r^r4r0rZ read_configget_parsed_contentload_config_file)r>argsrXZ config_yamlparserrZ dict_filerrr# get_score.s   zBundleAlgo.get_scorecstj|jdd}tj|st|dtj|jdfddtD}tj d|}tj |}|t j d<|j ||j|g|Ri|S)a Load the InferClass from the infer.py. The InferClass should be defined in the template under the path of `"scripts/infer.py"`. It is required to define the "InferClass" (name is fixed) with two functions at least (``__init__`` and ``infer``). The init class has an override kwargs that can be used to override parameters in the run-time optionally. Examples: .. code-block:: python class InferClass def __init__(self, config_file: Optional[Union[str, Sequence[str]]] = None, **override): # read configs from config_file (sequence) # set up transforms # set up model # set up other hyper parameters return @torch.no_grad() def infer(self, image_file): # infer the model and save the results to output return output rjzinfer.pyz% is not found, please check the path.rkcsg|] }tj|qSr)r:r^r4r!frxrr#r$Ysz*BundleAlgo.get_inferer.. InferClass)r:r^r4r0isfilerurq importlibutilspec_from_file_locationmodule_from_specsysmodulesloader exec_moduler)r>rrXZinfer_pyZ configs_pathspecZ infer_classrrr# get_inferer;s    zBundleAlgo.get_inferer predict_fileslistpredict_params dict | Nonecs:|durint|}|jdi|fddt|DS)aL Use the trained model to predict the outputs with a given input image. Args: predict_files: a list of paths to files to run inference on ["path_to_image_1", "path_to_image_2"] predict_params: a dict to override the parameters in the bundle config (including the files to predict). Ncsg|]}|qSr)inferrinfererrr#r$lsz&BundleAlgo.predict..r)rrr)r>rrrwrrr#predictas zBundleAlgo.predictcC|jS)zCReturns the algo output paths to find the algo scripts and configs.)r0r>rrr#get_output_pathnzBundleAlgo.get_output_path)rr)Fr)r@rArBr rCrD)r,r rCrF)rJr rCrF)r.rMrCrF)r/rMrCrF)rVr rWr rXrrCrY)r0r r[r rXrrCrF)N)rgrhrCri)r)r|r r}r rCr~)NN)rgrhr=rhrCr~)rrrrrCr)__name__ __module__ __qualname____doc__r?rErHrKrQrUrZrfr{rrrrrrrrrr#r5s$     4  & zYhttps://github.com/Project-MONAI/research-contributions/releases/download/algo_templates/z.tar.gzz(segresnet2d.scripts.algo.Segresnet2dAlgo)_target_zdints.scripts.algo.DintsAlgoz$swinunetr.scripts.algo.SwinunetrAlgoz$segresnet.scripts.algo.SegresnetAlgo)Z segresnet2ddintsZ swinunetr segresneturlr at_pathrCdict[str, dict[str, str]]c Cstj|}t}tj|jd}d}t|D]J}z t||tj|dWn:t y`}z.d|d|dd|d}||dkrMt |t |n |t||WYd }~qd }~ww|tt}|D]} ||| d <ql|S) z Downloads the algorithm templates release archive, and extracts it into a parent directory of the at_path folder. Returns a dictionary of the algorithm templates. zalgo_templates.tar.gz)rfilepath output_dirzDownload and extract of z failed, attempt r&/.Nr)r:r^abspathrr4r1r5rdirname Exceptionrrtimesleepcleanuprur default_algos) rrZzip_download_dirZalgo_compressed_fileZdownload_attemptsiemsg algos_allr1rrr#_download_algos_urls.        rc Cstj|}tj|}||kr!tj|rt|t||i}t|D],}tjtj||ddrTt |d| d|d||<t d|d||q(|s^t d||S) zl Copies the algorithm templates folder to at_path. Returns a dictionary of algorithm templates. rjzalgo.pyz.scripts.algo.r )rrzCopying template: z -- zUnable to find any algos in )r:r^rexistsrarbrcrqr4rY capitalizerdreru)folderrrr1rrr#_copy_algos_folders      rc@seZdZdZ       d2d3ddZd4ddZddZddZddZddZ ddZ ddZ d d!Z d5d#d$Z  % &  'd6d7d0d1ZdS)8ra This class generates a set of bundles according to the cross-validation folds, each of them can run independently. Args: algo_path: the directory path to save the algorithm templates. Default is the current working dir. algos: If dictionary, it outlines the algorithm to use. If a list or a string, defines a subset of names of the algorithms to use, e.g. ('segresnet', 'dints') out of the full set of algorithm templates provided by templates_path_or_url. Defaults to None - to use all available algorithms. templates_path_or_url: the folder with the algorithm templates or a url. If None provided, the default template zip url will be downloaded and extracted into the algo_path. The current default options are released at: https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg. data_stats_filename: the path to the data stats file (generated by DataAnalyzer). data_src_cfg_name: the path to the data source config YAML file. The config will be in a form of {"modality": "ct", "datalist": "path_to_json_datalist", "dataroot": "path_dir_data"}. mlflow_tracking_uri: a tracking URI for MLflow server which could be local directory or address of the remote tracking Server; MLflow runs will be recorded locally in algorithms' model folder if the value is None. mlfow_experiment_name: a string to specify the experiment name for MLflow server. .. code-block:: bash python -m monai.apps.auto3dseg BundleGen generate --data_stats_filename="../algorithms/datastats.yaml" rNrWr algosdict | list | str | Nonetemplates_path_or_urlrMrVdata_src_cfg_namer.r/c sdus ttttfrq|durt}tjtj|d}tj |r2t d|t ||d} n t |jdvrHt d|t||d} n t|jd|durofdd | Dtd krntd | n| g|_ttrtD]?\} } | d d } t| d kr| tjvrtj| zt| } | | _|j| Wqty}zd}t||d}~wwntd||_||_||_ ||_!g|_"dS)NZalgorithm_templateszBundleGen from directory )rr)httphttpszBundleGen from )rrz) received invalid templates_path_or_url: cs"i|] \}}|tvr||qSr)r)r!kvrrr# s"z&BundleGen.__init__..rz!Unable to find provided algos in rruPlease make sure the folder structure of an Algo Template follows [algo_name] ├── configs │ ├── hyper_parameters.yaml # automatically generated yaml from a set of ``template_configs`` └── scripts ├── test.py ├── __init__.py └── validate.py z$Unexpected error algos is not a dict)# isinstancertupler default_algo_zipr:r^r4rr`rdrerrschemerru __class__itemsrrrYrpr<rrsrrr1 RuntimeErrorrVrr.r/history)r>rWrrrVrr.r/rrr[Z algo_paramsrZonealgorrrrr#r?sN         zBundleGen.__init__rCrFcCrG)zs Set the data stats filename Args: data_stats_filename: filename of datastats NrV)r>rVrrr#rHrIzBundleGen.set_data_statscCr)z"Get the filename of the data statsrrrrr#get_data_statsrzBundleGen.get_data_statscCrG)zy Set the data source filename Args: data_src_cfg_name: filename of data_source file Nr)r>rrrr# set_data_src rIzBundleGen.set_data_srccCr)zGet the data source filenamerrrrr# get_data_src)rzBundleGen.get_data_srccCrGrNrOrPrrr#rQ-rLz!BundleGen.set_mlflow_tracking_uricCrGrRrSrTrrr#rU8rIz$BundleGen.set_mlflow_experiment_namecCr)z&Get the tracking URI for MLflow serverrOrrrr#get_mlflow_tracking_uriArz!BundleGen.get_mlflow_tracking_uricCr)z)Get the experiment name for MLflow serverrSrrrr#get_mlflow_experiment_nameErz$BundleGen.get_mlflow_experiment_namercCr)zEGet the history of the bundleAlgo object with their names/identifiers)rrrrr# get_historyIrzBundleGen.get_historyFT output_foldernum_foldr9gpu_customizationrAgpu_customization_specsdict[str, Any] | None allow_skipc Cstt|}|jD]v}t|D]o}|} |} |} |} t|} | | | | | | | | | j d|}|rV| \}}|rVt|d|q|rc| j|||d|dn| j|||dt| |jd|jtj|tj| iqq dS)a0 Generate the bundle scripts/configs for each bundleAlgo Args: output_folder: the output folder to save each algorithm. num_fold: the number of cross validation fold. gpu_customization: the switch to determine automatically customize/optimize bundle script/config parameters for each bundleAlgo based on gpus. Custom parameters are obtained through dummy training to simulate the actual model training process and hyperparameter optimization (HPO) experiments. gpu_customization_specs: the dictionary to enable users overwrite the HPO settings. user can overwrite part of variables as follows or all of them. The structure is as follows. allow_skip: a switch to determine if some Algo in the default templates can be skipped based on the analysis on the dataset from Auto3DSeg DataAnalyzer. .. code-block:: python gpu_customization_specs = { 'ALGO': { 'num_trials': 6, 'range_num_images_per_batch': [1, 20], 'range_num_sw_batch_size': [1, 20] } } ALGO: the name of algorithm. It could be one of algorithm names (e.g., 'dints') or 'universal' which would apply changes to all algorithms. Possible options are - {``"universal"``, ``"dints"``, ``"segresnet"``, ``"segresnet2d"``, ``"swinunetr"``}. num_trials: the number of HPO trials/experiments to run. range_num_images_per_batch: the range of number of images per mini-batch. range_num_sw_batch_size: the range of batch size in sliding-window inferer. _z is skipped! T)foldrr)r)rN)rr5rrrrrrrrHrKrQrUr1rErdrerfrrrrsrIDALGO)r>rrrrrZfold_idxalgoZf_idZ data_statsrJr.r/Zgen_algor1r@rBrrr#generateMsB *       zBundleGen.generate)rNNNNNN)rWr rrrrMrVrMrrMr.rMr/rM)rVr rCrF)rCr)rrFNT) rr rr9rrArrrrArCrF)rrrrr?rHrrrrQrUrrrrrrrr#rs2 A     )rr rr rCr); __future__rrr:rra subprocessrrrrrpathlibrtempfilertypingr urllib.parserr6Z monai.appsrmonai.apps.utilsr Zmonai.auto3dseg.algo_genr r monai.auto3dseg.utilsr r rrrrZmonai.bundle.config_parserr monai.configr monai.utilsrrrmonai.utils.enumsrmonai.utils.miscrrrd algo_hashZ ALGO_HASH__all__rrrYrrrrrrrr#sP               B !