U PhBq@sddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZddlmZddlmZddlZddlmZdd lmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ee/dZ0e.1Z2ddgZ3GdddeZ4de2dZ5e6dde6dde6dde6dddZ7ddddd d!Z8d"d#Z9Gd$ddeZ:dS)%) annotationsN)deepcopy)Path)TemporaryDirectory)Any)urlparse)download_and_extract) get_logger)AlgoAlgoGen)_prepare_cmd_bcprun_prepare_cmd_default_prepare_cmd_torchrun_run_cmd_bcprun_run_cmd_torchrunalgo_to_pickle) ConfigParser)PathLike) ensure_tuplelook_up_optionrun_cmd)AlgoKeys) MONAIEnvVars) module_name BundleAlgo BundleGenc@seZdZdZddddZdd d d*d+d,d-Z d?d%d%d*d.d/d0Zd1d2Zd3d4Zd@d5d6d5d7d8d9Zd:d;Zd$S)Ara5 An algorithm represented by a set of bundle configurations and scripts. ``BundleAlgo.cfg`` is a ``monai.bundle.ConfigParser`` instance. .. code-block:: python from monai.apps.auto3dseg import BundleAlgo data_stats_yaml = "../datastats.yaml" algo = BundleAlgo(template_path="../algorithm_templates") algo.set_data_stats(data_stats_yaml) # algo.set_data_src("../data_src.json") algo.export_to_disk(".", algo_name="segresnet2d_1") This class creates MONAI bundles from a directory of 'bundle template'. Different from the regular MONAI bundle format, the bundle template may contain placeholders that must be filled using ``fill_template_config`` during ``export_to_disk``. Then created bundle keeps the same file structure as the template. r template_pathcCs||_d|_d|_d|_d|_d|_d|_d|_i|_d ddt t j Dtt j ttjddtjdd tjd dd |_dS) a0 Create an Algo instance based on the predefined Algo template. Args: template_path: path to a folder that contains the algorithm templates. Please check https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg/algorithm_templates N,cSsg|] }t|qS)str).0xr r T/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/apps/auto3dseg/bundle_gen.py asz'BundleAlgo.__init__.. NUM_NODESMN_START_METHODbcprun CMD_PREFIX)CUDA_VISIBLE_DEVICES n_devicesr&r(r*)rdata_stats_filesdata_list_filemlflow_tracking_urimlflow_experiment_name output_pathname best_metric fill_recordsjoinrangetorchcuda device_countintosenvirongetdevice_setting)selfrr r r$__init__Ks    zBundleAlgo.__init__Frboolr!ztuple[bool, str])skip_bundlegen skip_inforeturncCs||fS)a Analyse the data analysis report and check if the algorithm needs to be skipped. This function is overriden within algo. Args: skip_bundlegen: skip generating bundles for this algo if true. skip_info: info to print when skipped. r )r?rBrCr r r$pre_check_skip_algohszBundleAlgo.pre_check_skip_algoNone)r-rDcCs ||_dS)z Set the data analysis report (generated by DataAnalyzer). Args: data_stats_files: path to the datastats yaml file N)r-)r?r-r r r$set_data_statsrszBundleAlgo.set_data_stats) data_src_cfgrDcCs ||_dS)aN Set the data source configuration file Args: data_src_cfg: path to a configuration file (yaml) that contains datalist, dataroot, and other params. The config will be in a form of {"modality": "ct", "datalist": "path_to_json_datalist", "dataroot": "path_dir_data"} N)r.)r?rHr r r$set_data_source{s zBundleAlgo.set_data_source str | None)r/rDcCs ||_dSaI Set the tracking URI for MLflow server Args: mlflow_tracking_uri: a tracking URI for MLflow server which could be local directory or address of the remote tracking Server; MLflow runs will be recorded locally in algorithms' model folder if the value is None. Nr/r?r/r r r$set_mlflow_tracking_uris z"BundleAlgo.set_mlflow_tracking_uri)r0rDcCs ||_dSz Set the experiment name for MLflow server Args: mlflow_experiment_name: a string to specify the experiment name for MLflow server. Nr0r?r0r r r$set_mlflow_experiment_namesz%BundleAlgo.set_mlflow_experiment_namerdict)data_stats_filename algo_pathkwargsrDcKsiS)a The configuration files defined when constructing this Algo instance might not have a complete training and validation pipelines. Some configuration components and hyperparameters of the pipelines depend on the training data and other factors. This API is provided to allow the creation of fully functioning config files. Return the records of filling template config: {"": {"": value, ...}, ...}. Args: data_stats_filename: filename of the data stats report (generated by DataAnalyzer) Notes: Template filling is optional. The user can construct a set of pre-filled configs without replacing values by using the data analysis results. It is also intended to be re-implemented in subclasses of BundleAlgo if the user wants their own way of auto-configured template filling. r )r?rTrUrVr r r$fill_template_configszBundleAlgo.fill_template_config)r1 algo_namerVrDcKs|ddrhtj|||_tj|jddtj|jrFt|jt tjt |j |j |jn t |j |_|ddr|j |j|jf||_td|jdS)a Fill the configuration templates, write the bundle (configs + scripts) to folder `output_path/algo_name`. Args: output_path: Path to export the 'scripts' and 'configs' directories. algo_name: the identifier of the algorithm (usually contains the name and extra info like fold ID). kwargs: other parameters, including: "copy_dirs=True/False" means whether to copy the template as output instead of inplace operation, "fill_template=True/False" means whether to fill the placeholders in the template. other parameters are for `fill_template_config` function. Z copy_dirsT)exist_okZ fill_templatez Generated:N)popr;pathr5r1makedirsisdirshutilrmtreecopytreer!rr2rWr-r4loggerinfo)r?r1rXrVr r r$export_to_disks  "  zBundleAlgo.export_to_diskNz None | dictztuple[str, str]) train_paramsrDc Cst|dkr i}t|}tj|jdd}tj|jd}g}tj|rtt|D]4}|dsl|drT| t tj|| qTt |j ddkrzt|j d d gWn8tk r}zt|j d d |W5d}~XYnXt|d f|j d |d|dfSt |j ddkrHt|d fd|i|dfSt|d f|j d |d|dfSdS)z: Create the command to execute training. Nscriptsztrain.pyconfigsyamljsonr&r'r(r)zI is not supported yet.Try modify BundleAlgo._create_cmd for your cluster.z runr*) cmd_prefix config_filerr,rj)rr;r[r5r1r]sortedlistdirendswithappendras_posixr:r>r ValueErrorNotImplementedErrorr rr )r?rdparamsZtrain_py config_dirZ config_filesfileerrr r r$ _create_cmdsR    zBundleAlgo._create_cmdzsubprocess.CompletedProcess)cmd devices_inforDc Cs|rtd|dtj}t|jd|d<tdd|}t |jddkrzt |jdd gWn8t k r}zt |jdd |W5d }~XYnXt ||jd|jd d St |jd dkrt|d|jd |ddSt|d|ddSd S)zP Execute the training command with target devices information. zinput devices_info z is deprecated and ignored.r+z^\s*\w+=.*?\s+rr&r'r(r)zF is not supported yet.Try modify BundleAlgo._run_cmd for your cluster.Nr,)npT)nnodesZnproc_per_nodeenvcheck)run_cmd_verboser|r})warningswarnr;r<copyr!r>resubr:rrprqrrrsplit)r?rwrxZ ps_environrur r r$_run_cmds0 zBundleAlgo._run_cmd)rdr>rDcCsn|dk r2|j|tt|jdd|jd<|dk rVd|krVtd|d||\}}| |S)a Load the run function in the training script of each model. Training parameter is predefined by the algo_config.yaml file, which is pre-filled by the fill_template_config function in the same instance. Args: train_params: training parameters device_setting: device related settings, should follow the device_setting in auto_runner.set_device_info. 'CUDA_VISIBLE_DEVICES' should be a string e.g. '0,1,2,3' Nr+rr,z5CUDA_VISIBLE_DEVICES is deprecated from train_params!) r>updatelenr!rrrrZrvr)r?rdr>rwZ_unused_returnr r r$trains    zBundleAlgo.traincOsRtj|jdd}t}|||jd|jd}ttj|d}|ddS)zU Returns validation scores of the model trained by the current Algo. rfzhyper_parameters.yaml ckpt_path)defaultz progress.yamlZbest_avg_dice_score)r;r[r5r1rZ read_configget_parsed_contentload_config_file)r?argsrVZ config_yamlparserr dict_filer r r$ get_score.s  zBundleAlgo.get_scorecstj|jdd}tj|s,t|dtj|jdfddtD}tj d|}tj |}|t j d<|j ||j|f||S)a Load the InferClass from the infer.py. The InferClass should be defined in the template under the path of `"scripts/infer.py"`. It is required to define the "InferClass" (name is fixed) with two functions at least (``__init__`` and ``infer``). The init class has an override kwargs that can be used to override parameters in the run-time optionally. Examples: .. code-block:: python class InferClass def __init__(self, config_file: Optional[Union[str, Sequence[str]]] = None, **override): # read configs from config_file (sequence) # set up transforms # set up model # set up other hyper parameters return @torch.no_grad() def infer(self, image_file): # infer the model and save the results to output return output rezinfer.pyz% is not found, please check the path.rfcsg|]}tj|qSr )r;r[r5r"frsr r$r%Ysz*BundleAlgo.get_inferer.. InferClass)r;r[r5r1isfilerprl importlibutilspec_from_file_locationmodule_from_specsysmodulesloader exec_moduler)r?rrVZinfer_pyZ configs_pathspecZ infer_classr rr$ get_inferer;s    zBundleAlgo.get_infererlistz dict | None) predict_filespredict_paramsrDcs6|dkr int|}|jf|fddt|DS)aL Use the trained model to predict the outputs with a given input image. Args: predict_files: a list of paths to files to run inference on ["path_to_image_1", "path_to_image_2"] predict_params: a dict to override the parameters in the bundle config (including the files to predict). Ncsg|]}|qSr )inferrinfererr r$r%lsz&BundleAlgo.predict..)rrr)r?rrrrr rr$predictas  zBundleAlgo.predictcCs|jS)zCReturns the algo output paths to find the algo scripts and configs.)r1r?r r r$get_output_pathnszBundleAlgo.get_output_path)Fr)N)r)NN)N)__name__ __module__ __qualname____doc__r@rErGrIrNrRrWrcrvrrrrrrr r r r$r5s$     4  & zYhttps://github.com/Project-MONAI/research-contributions/releases/download/algo_templates/z.tar.gzz(segresnet2d.scripts.algo.Segresnet2dAlgo)_target_zdints.scripts.algo.DintsAlgoz$swinunetr.scripts.algo.SwinunetrAlgoz$segresnet.scripts.algo.SegresnetAlgo)Z segresnet2ddintsZ swinunetr segresnetr!zdict[str, dict[str, str]])urlat_pathrDc Cstj|}t}tj|jd}d}t|D]}zt||tj|dWnrt k r}zTd|d|dd|d}||dkrt |t |n|t||W5d }~XYq.Xqq.|tt}|D]} ||| d <q|S) z Downloads the algorithm templates release archive, and extracts it into a parent directory of the at_path folder. Returns a dictionary of the algorithm templates. zalgo_templates.tar.gz)rfilepath output_dirzDownload and extract of z failed, attempt r'/.Nr)r;r[abspathrr5r2r6rdirname Exceptionrrtimesleepcleanuprpr default_algos) rrZzip_download_dirZalgo_compressed_fileZdownload_attemptsiemsg algos_allr2r r r$_download_algos_urls(     rc Cstj|}tj|}||krBtj|r6t|t||i}t|D]X}tjtj||ddrPt |d| d|d||<t d|d||qP|st d||S) zl Copies the algorithm templates folder to at_path. Returns a dictionary of algorithm templates. rezalgo.pyz.scripts.algo.r )rrzCopying template: z -- zUnable to find any algos in )r;r[rexistsr^r_r`rlr5rS capitalizerarbrp)folderrrr2r r r$_copy_algos_folders      rc @seZdZdZd)dddddddddd Zdd d d d ZddZddZddZddZ ddZ ddZ ddZ ddddZ d*dd#d$d%d$d d&d'd(ZdS)+ra This class generates a set of bundles according to the cross-validation folds, each of them can run independently. Args: algo_path: the directory path to save the algorithm templates. Default is the current working dir. algos: If dictionary, it outlines the algorithm to use. If a list or a string, defines a subset of names of the algorithms to use, e.g. ('segresnet', 'dints') out of the full set of algorithm templates provided by templates_path_or_url. Defaults to None - to use all available algorithms. templates_path_or_url: the folder with the algorithm templates or a url. If None provided, the default template zip url will be downloaded and extracted into the algo_path. The current default options are released at: https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg. data_stats_filename: the path to the data stats file (generated by DataAnalyzer). data_src_cfg_name: the path to the data source config YAML file. The config will be in a form of {"modality": "ct", "datalist": "path_to_json_datalist", "dataroot": "path_dir_data"}. mlflow_tracking_uri: a tracking URI for MLflow server which could be local directory or address of the remote tracking Server; MLflow runs will be recorded locally in algorithms' model folder if the value is None. mlfow_experiment_name: a string to specify the experiment name for MLflow server. .. code-block:: bash python -m monai.apps.auto3dseg BundleGen generate --data_stats_filename="../algorithms/datastats.yaml" rNr!zdict | list | str | NonerJ)rUalgostemplates_path_or_urlrTdata_src_cfg_namer/r0c sdksttttfr|dkr$t}tjtj|d}tj |rdt d|t ||d} n@t |jdkrt d|t||d} nt|jd|dk rއfdd | Dtd krtd | n| g|_ttrtD]\} } | d d } t| d kr:| tjkr:tj| z"t| } | | _|j| Wn2tk r}zd}t||W5d}~XYnXqntd||_||_||_ ||_!g|_"dS)NZalgorithm_templateszBundleGen from directory )rr)httphttpszBundleGen from )rrz) received invalid templates_path_or_url: cs"i|]\}}|tkr||qSr )r)r"kvrr r$ s z&BundleGen.__init__..rz!Unable to find provided algos in rruPlease make sure the folder structure of an Algo Template follows [algo_name] ├── configs │ ├── hyper_parameters.yaml # automatically generated yaml from a set of ``template_configs`` └── scripts ├── test.py ├── __init__.py └── validate.py z$Unexpected error algos is not a dict)# isinstancertupler!default_algo_zipr;r[r5rr]rarbrrschemerrp __class__itemsrrrSrkr=rrnrrr2 RuntimeErrorrTrr/r0history)r?rUrrrTrr/r0rrrXZ algo_paramsrZonealgorrr rr$r@sF        "zBundleGen.__init__rF)rTrDcCs ||_dS)zs Set the data stats filename Args: data_stats_filename: filename of datastats NrT)r?rTr r r$rGszBundleGen.set_data_statscCs|jS)z"Get the filename of the data statsrrr r r$get_data_statsszBundleGen.get_data_statscCs ||_dS)zy Set the data source filename Args: data_src_cfg_name: filename of data_source file Nr)r?rr r r$ set_data_src szBundleGen.set_data_srccCs|jS)zGet the data source filenamerrr r r$ get_data_src)szBundleGen.get_data_srccCs ||_dSrKrLrMr r r$rN-s z!BundleGen.set_mlflow_tracking_uricCs ||_dSrOrPrQr r r$rR8sz$BundleGen.set_mlflow_experiment_namecCs|jS)z&Get the tracking URI for MLflow serverrLrr r r$get_mlflow_tracking_uriAsz!BundleGen.get_mlflow_tracking_uricCs|jS)z)Get the experiment name for MLflow serverrPrr r r$get_mlflow_experiment_nameEsz$BundleGen.get_mlflow_experiment_namer)rDcCs|jS)zEGet the history of the bundleAlgo object with their names/identifiers)rrr r r$ get_historyIszBundleGen.get_historyFTr:rAzdict[str, Any] | None) output_foldernum_foldgpu_customizationgpu_customization_specs allow_skiprDc Cstt|}|jD]}t|D]}|} |} |} |} t|} | | | | | | | | | j d|}|r| \}}|rt|d|q|r| j|||d|dn| j|||dt| |jd|jtj|tj| iqqdS)a0 Generate the bundle scripts/configs for each bundleAlgo Args: output_folder: the output folder to save each algorithm. num_fold: the number of cross validation fold. gpu_customization: the switch to determine automatically customize/optimize bundle script/config parameters for each bundleAlgo based on gpus. Custom parameters are obtained through dummy training to simulate the actual model training process and hyperparameter optimization (HPO) experiments. gpu_customization_specs: the dictionary to enable users overwrite the HPO settings. user can overwrite part of variables as follows or all of them. The structure is as follows. allow_skip: a switch to determine if some Algo in the default templates can be skipped based on the analysis on the dataset from Auto3DSeg DataAnalyzer. .. code-block:: python gpu_customization_specs = { 'ALGO': { 'num_trials': 6, 'range_num_images_per_batch': [1, 20], 'range_num_sw_batch_size': [1, 20] } } ALGO: the name of algorithm. It could be one of algorithm names (e.g., 'dints') or 'universal' which would apply changes to all algorithms. Possible options are - {``"universal"``, ``"dints"``, ``"segresnet"``, ``"segresnet2d"``, ``"swinunetr"``}. num_trials: the number of HPO trials/experiments to run. range_num_images_per_batch: the range of number of images per mini-batch. range_num_sw_batch_size: the range of batch size in sliding-window inferer. _z is skipped! T)foldrr)rrN)rr6rrrrrrrrGrIrNrRr2rErarbrcrrrrnrIDALGO)r?rrrrrZfold_idxalgoZf_idZ data_statsrHr/r0Zgen_algor2rBrCr r r$generateMs>*        zBundleGen.generate)rNNNNNN)rrFNT)rrrrr@rGrrrrNrRrrrrr r r r$rs0A    ); __future__rrr;rr^ subprocessrrrrrpathlibrtempfilertypingr urllib.parserr7Z monai.appsrmonai.apps.utilsr Zmonai.auto3dseg.algo_genr r monai.auto3dseg.utilsr r rrrrZmonai.bundle.config_parserr monai.configr monai.utilsrrrmonai.utils.enumsrmonai.utils.miscrrra algo_hashZ ALGO_HASH__all__rrrSrrrrr r r r$ sN              B !