U Ph@sddlmZddlZddlZddlZddlmZddlZddlm Z ddlm Z m Z m Z ddlmZddlmZmZddlmZed d d \ZZed d d \ZZed d d \ZZed\ZZejjeZdgZ GdddZ!dS)) annotationsN)Any) NNUNETMode) analyze_datacreate_new_data_copycreate_new_dataset_json) ConfigParser) ensure_tupleoptional_import)run_cmdz4batchgenerators.utilities.file_and_folder_operations load_pickle)namejointqdmnibabelnnUNetV2Runnerc@sveZdZdZdVdddddd d d Zd d ZdWdddddddZdXdddddddddZdYdddddddd d!Ze j e j e j fd"ddfd#d#dddd$d%d&Z dddddddddde j e j e j fd"df ddddddddddd#d#ddd'd(d)ZdZddd+ddd,d-d.Zd/d0Ze j e j e j e jfdfd1d2ddd3d4d5Ze j e j e j e jfdfd1d2dd6d3d7d8Ze j e j e j e jfdfd1d2ddd3d9d:Zddddd;dd?d@Zde j e j e j e jfdddddAdfd1d1dBddddCdddD dEdFZd[dJdKddLddddddddddddddddMdNdOZd\d#ddddddPdQdRZd]dddddddSdTdUZdS)^ra ``nnUNetV2Runner`` provides an interface in MONAI to use `nnU-Net` V2 library to analyze, train, and evaluate neural networks for medical image segmentation tasks. A version of nnunetv2 higher than 2.2 is needed for this class. ``nnUNetV2Runner`` can be used in two ways: #. with one line of code to execute the complete pipeline. #. with a series of commands to run each modules in the pipeline. The output of the interface is a directory that contains: #. converted dataset met the requirement of nnU-Net V2 #. data analysis results #. checkpoints from the trained U-Net models #. validation accuracy in each fold of cross-validation #. the predictions on the testing datasets from the final algorithm ensemble and potential post-processing Args: input_config: the configuration dictionary or the file path to the configuration in the form of YAML. The keys required in the configuration are: - ``"datalist"``: File path to the datalist for the train/testing splits - ``"dataroot"``: File path to the dataset - ``"modality"``: Imaging modality, e.g. "CT", ["T2", "ADC"] Currently, the configuration supports these optional keys: - ``"nnunet_raw"``: File path that will be written to env variable for nnU-Net - ``"nnunet_preprocessed"``: File path that will be written to env variable for nnU-Net - ``"nnunet_results"``: File path that will be written to env variable for nnU-Net - ``"nnUNet_trained_models"`` - ``"dataset_name_or_id"``: Name or Integer ID of the dataset If an optional key is not specified, then the pipeline will use the default values. trainer_class_name: the trainer class names offered by nnUNetV2 exhibit variations in training duration. Default: "nnUNetTrainer". Other options: "nnUNetTrainer_Xepoch". X could be one of 1,5,10,20,50,100, 250,2000,4000,8000. export_validation_probabilities: True to save softmax predictions from final validation as npz files (in addition to predicted segmentations). Needed for finding the best ensemble. Default: True. work_dir: working directory to save the intermediate and final results. Examples: - Use the one-liner to start the nnU-Net workflow .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner run --input_config ./input.yaml - Use `convert_dataset` to prepare the data to meet nnU-Net requirements, generate dataset JSON file, and copy the dataset to a location specified by ``nnunet_raw`` in the input config file .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner convert_dataset --input_config="./input.yaml" - `convert_msd_dataset` is an alternative option to prepare the data if the dataset is MSD. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner convert_msd_dataset \ --input_config "./input.yaml" --data_dir "/path/to/Task09_Spleen" - experiment planning and data pre-processing .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner plan_and_process --input_config "./input.yaml" - training all 20 models using all GPUs available. "CUDA_VISIBLE_DEVICES" environment variable is not supported. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" - training a single model on a single GPU for 5 epochs. Here ``config`` is used to specify the configuration. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train_single_model --input_config "./input.yaml" \ --config "3d_fullres" \ --fold 0 \ --gpu_id 0 \ --trainer_class_name "nnUNetTrainer_5epochs" \ --export_validation_probabilities True - training for all 20 models (4 configurations by 5 folds) on 2 GPUs .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" --gpu_id_for_all "0,1" - 5-fold training for a single model on 2 GPUs. Here ``configs`` is used to specify the configurations. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" \ --configs "3d_fullres" \ --trainer_class_name "nnUNetTrainer_5epochs" \ --export_validation_probabilities True \ --gpu_id_for_all "0,1" - find the best configuration .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner find_best_configuration --input_config "./input.yaml" - predict, ensemble, and post-process .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner predict_ensemble_postprocessing --input_config "./input.yaml" nnUNetTrainerwork_dirTrstrboolNone) input_configtrainer_class_namerexport_validation_probabilitiesreturncCsi|_||_||_||_||_t|jtr4|j|_n8t|jtr^tj |jr^t |j|_nt |d|jdtj d|jd|_|jdtj d|jd|_|jdtj d|jd|_tj |jst|jtj |jst|jtj |jst|j|jtjd <|jtjd<|jtjd <td tjd <t|jd d |_z ddlm}|t|j|_Wn*tk rtd|jdYnXddlm}||_d|_ i|_!dS)Nz is not a valid file or dict nnunet_raw.ZnnUNet_raw_data_basennunet_preprocessedZnnUNet_preprocessednnunet_resultsZnnUNet_trained_modelsZ nnUNet_rawZnnUNet_resultsZOMP_NUM_THREADSdataset_name_or_idrmaybe_convert_to_dataset_namezDataset with name/ID: z cannot be found in the record. Please ignore the message above if you are running the pipeline from a fresh start. But if the dataset is expected to be found, please check your input_config.)default_num_processes)" input_infoZinput_config_or_dictrrr isinstancedictrospathisfilerload_config_file ValueErrorpoprrrrexistsmakedirsenvironr -nnunetv2.utilities.dataset_name_id_conversionr"int dataset_name BaseExceptionloggerwarningZnnunetv2.configurationr# num_foldsbest_configuration)selfrrrrr"r#r:V/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/apps/nnunet/nnunetv2_runner.py__init__sR           znnUNetV2Runner.__init__c Cs`ztt|jd}d|dd}t|jd}dd|D}dd|D}||krltd WdS|jd }|d t j kr|dd }|d | t j d }t j |j|}t j |st |d dlm}|t|j|_t|jd}d|kr2t t j |dt t j |dntdWdSd}d|ksZd|krt t j |dd|kr|dnd}t||d trd||d krt t j |dt||d\} } |jd} t| ts| g} t| | | t|dt j |ddt|||| |dWn:tk rZ} ztd| WYdSd} ~ XYnXdS)zQConvert and make a copy the dataset to meet the requirements of nnU-Net workflow.iDatasetNz/*cSsg|]}|tjdqS))splitr(sep.0_itemr:r:r; sz2nnUNetV2Runner.convert_dataset..cSsg|]}|ddqS)_r)r@rBr:r:r;rEsz Dataset with the same ID exists!datarootr?rFrr!datalisttrainingZimagesTrZlabelsTrzHThe datalist file has incorrect format: the `training` key is not found.testtestingimagesTslabelZlabelsTs) datalist_jsondata_dirmodalityz dataset.json)rPnum_foreground_classesnum_input_channelsZnum_training_dataZoutput_filepath)test_keyrNrOrRZoutput_datafolderzIInput config may be incorrect. Detail info: error/exception message is: )rr2r globrr5r6r%r-r(rAr@r)rr.r/r1r"r3rr+errorr&r'rlistrlenrr4) r9Zraw_data_foldername_prefixsubdirsZ dataset_idsrOZraw_data_foldernamer"rNrSrRrQrPerrr:r:r;convert_datasetsf        &     znnUNetV2Runner.convert_datasetNr?z str | Noner2)rO overwrite_idn_procrcCs.ddlm}|dkrdn|j}||||dS)a Convert and make a copy the MSD dataset to meet requirements of nnU-Net workflow. Args: data_dir: downloaded and extracted MSD dataset folder. CANNOT be nnUNetv1 dataset! Example: "/workspace/downloads/Task05_Prostate". overwrite_id: Overwrite the dataset id. If not set then use the id of the MSD task (inferred from the folder name). Only use this if you already have an equivalently numbered dataset! n_proc: Number of processes used. r)convert_msd_datasetN)Z/nnunetv2.dataset_conversion.convert_MSD_datasetr]r#)r9rOr[r\r] num_processesr:r:r;r]s z"nnUNetV2Runner.convert_msd_datasetDatasetFingerprintExtractorF)fpenpfpverify_dataset_integritycleanverbosercCsFddlm}|dkr|jn|}td|t|jg|||||dS)aW Extracts the dataset fingerprint used for experiment planning. Args: fpe: [OPTIONAL] Name of the Dataset Fingerprint Extractor class that should be used. Default is "DatasetFingerprintExtractor". npfp: [OPTIONAL] Number of processes used for fingerprint extraction. verify_dataset_integrity: [RECOMMENDED] set this flag to check the dataset integrity. This is useful and should be done once for each dataset! clean: [OPTIONAL] Set this flag to overwrite existing fingerprints. If this flag is not set and a fingerprint already exists, the fingerprint extractor will not run. verbose: set this to print a lot of stuff. Useful for debugging. Will disable progress bar! Recommended for cluster environments. r)extract_fingerprintszFingerprint extraction...N)4nnunetv2.experiment_planning.plan_and_preprocess_apirer#r5infor2r )r9r`rarbrcrdrer:r:r;re,s  z#nnUNetV2Runner.extract_fingerprintsExperimentPlannerDefaultPreprocessor nnUNetPlansfloat)plgpu_memory_targetpreprocessor_nameoverwrite_target_spacingoverwrite_plans_namercCs4ddlm}td|t|jg|||||dS)at Generate a configuration file that specifies the details of the experiment. Args: pl: [OPTIONAL] Name of the Experiment Planner class that should be used. Default is "ExperimentPlanner". Note: There is no longer a distinction between 2d and 3d planner. It's an all-in-one solution now. gpu_memory_target: [OPTIONAL] DANGER ZONE! Sets a custom GPU memory target. Default: 8 [GB]. Changing this will affect patch and batch size and will definitely affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). preprocessor_name: [OPTIONAL] DANGER ZONE! Sets a custom preprocessor class. This class must be located in nnunetv2.preprocessing. Default: "DefaultPreprocessor". Changing this may affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). overwrite_target_spacing: [OPTIONAL] DANGER ZONE! Sets a custom target spacing for the 3d_fullres and 3d_cascade_fullres configurations. Default: None [no changes]. Changing this will affect image size and potentially patch and batch size. This will definitely affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). Changing the target spacing for the other configurations is currently not implemented. New target spacing must be a list of three numbers! overwrite_plans_name: [OPTIONAL] DANGER ZONE! If you used -gpu_memory_target, -preprocessor_name or -overwrite_target_spacing it is best practice to use -overwrite_plans_name to generate a differently named plans file such that the nnunet default plans are not overwritten. You will then need to specify your custom plan. r)plan_experimentszExperiment planning...N)rfrrr5rgr2r )r9rmrnrorprqrrr:r:r;rrIs!   znnUNetV2Runner.plan_experiments)ririrituple)cr\rqrdrcCs4ddlm}td|t|jg||||ddS)a Apply a set of preprocessing operations to the input data before the training. Args: overwrite_plans_name: [OPTIONAL] You can use this to specify a custom plans file that you may have generated. c: [OPTIONAL] Configurations for which the preprocessing should be run. Default: 2d 3f_fullres 3d_lowres. 3d_cascade_fullres does not need to be specified because it uses the data from 3f_fullres. Configurations that do not exist for some datasets will be skipped). n_proc: [OPTIONAL] Use this to define how many processes are to be used. If this is just one number then this number of processes is used for all configurations specified with -c. If it's a list of numbers this list must have as many elements as there are configurations. We then iterate over zip(configs, num_processes) to determine the number of processes used for each configuration. More processes are always faster (up to the number of threads your PC can support, so 8 for a 4-core CPU with hyperthreading. If you don't know what that is then don't touch it, or at least don't increase it!). DANGER: More often than not the number of processes that can be used is limited by the amount of RAM available. Image resampling takes up a lot of RAM. MONITOR RAM USAGE AND DECREASE -n_proc IF YOUR RAM FILLS UP TOO MUCH! Default: 8 4 8 (=8 processes for 2d, 4 for 3d_fullres and 8 for 3d_lowres if -c is at its default). verbose: Set this to print a lot of stuff. Useful for debugging. Will disable the progress bar! Recommended for cluster environments. r) preprocesszPreprocessing...)configurationsr^rdN)rfrur5rgr2r )r9rtr\rqrdrur:r:r;ruvs   znnUNetV2Runner.preprocess)r`rarbno_pprcrmrnrorprqrtr\rdrcCs<|||||| ||||| | |s8|| | | | dS)a_ Performs experiment planning and preprocessing before the training. Args: fpe: [OPTIONAL] Name of the Dataset Fingerprint Extractor class that should be used. Default is "DatasetFingerprintExtractor". npfp: [OPTIONAL] Number of processes used for fingerprint extraction. Default: 8. verify_dataset_integrity: [RECOMMENDED] set this flag to check the dataset integrity. This is useful and should be done once for each dataset! no_pp: [OPTIONAL] Set this to only run fingerprint extraction and experiment planning (no preprocessing). Useful for debugging. clean:[OPTIONAL] Set this flag to overwrite existing fingerprints. If this flag is not set and a fingerprint already exists, the fingerprint extractor will not run. REQUIRED IF YOU CHANGE THE DATASET FINGERPRINT EXTRACTOR OR MAKE CHANGES TO THE DATASET! pl: [OPTIONAL] Name of the Experiment Planner class that should be used. Default is "ExperimentPlanner". Note: There is no longer a distinction between 2d and 3d planner. It's an all-in-one solution now. gpu_memory_target: [OPTIONAL] DANGER ZONE! Sets a custom GPU memory target. Default: 8 [GB]. Changing this will affect patch and batch size and will definitely affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). preprocessor_name: [OPTIONAL] DANGER ZONE! Sets a custom preprocessor class. This class must be located in nnunetv2.preprocessing. Default: "DefaultPreprocessor". Changing this may affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). overwrite_target_spacing: [OPTIONAL] DANGER ZONE! Sets a custom target spacing for the 3d_fullres and 3d_cascade_fullres configurations. Default: None [no changes]. Changing this will affect image size and potentially patch and batch size. This will definitely affect your models performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). Changing the target spacing for the other configurations is currently not implemented. New target spacing must be a list of three numbers! overwrite_plans_name: [OPTIONAL] USE A CUSTOM PLANS IDENTIFIER. If you used -gpu_memory_target, -preprocessor_name or -overwrite_target_spacing it is best practice to use -overwrite_plans_name to generate a differently named plans file such that the nnunet default plans are not overwritten. You will then need to specify your custom plans file with -p whenever running other nnunet commands (training, inference, etc) c: [OPTIONAL] Configurations for which the preprocessing should be run. Default: 2d 3f_fullres 3d_lowres. 3d_cascade_fullres does not need to be specified because it uses the data from 3f_fullres. Configurations that do not exist for some datasets will be skipped. n_proc: [OPTIONAL] Use this to define how many processes are to be used. If this is just one number then this number of processes is used for all configurations specified with -c. If it's a list of numbers this list must have as many elements as there are configurations. We then iterate over zip(configs, num_processes) to determine the number of processes used for each configuration. More processes are always faster (up to the number of threads your PC can support, so 8 for a 4-core CPU with hyperthreading. If you don't know what that is then don't touch it, or at least don't increase it!). DANGER: More often than not the number of processes that can be used is limited by the amount of RAM available. Image resampling takes up a lot of RAM. MONITOR RAM USAGE AND DECREASE -n_proc IF YOUR RAM FILLS UP TOO MUCH! Default: 8 4 8 (=8 processes for 2d, 4 for 3d_fullres and 8 for 3d_lowres if -c is at its default). verbose: Set this to print a lot of stuff. Useful for debugging. Will disable progress bar! (Recommended for cluster environments). N)rerrru)r9r`rarbrwrcrmrnrorprqrtr\rdr:r:r;plan_and_processsDznnUNetV2Runner.plan_and_processrztuple | list | int)configfoldgpu_idkwargsrcKstd|kr|dtdd|kr8|dtdd|krT|dtd|||||}t|ddd S) a Run the training on a single GPU with one specified configuration provided. Note: this will override the environment variable `CUDA_VISIBLE_DEVICES`. Args: config: configuration that should be trained. Examples: "2d", "3d_fullres", "3d_lowres". fold: fold of the 5-fold cross-validation. Should be an int between 0 and 4. gpu_id: an integer to select the device to use, or a tuple/list of GPU device indices used for multi-GPU training (e.g., (0,1)). Default: 0. kwargs: this optional parameter allows you to specify additional arguments in ``nnunetv2.run.run_training.run_training_entry``. Currently supported args are: - p: custom plans identifier. Default: "nnUNetPlans". - pretrained_weights: path to nnU-Net checkpoint file to be used as pretrained model. Will only be used when actually training. Beta. Use with caution. Default: False. - use_compressed: True to use compressed data for training. Reading compressed data is much more CPU and (potentially) RAM intensive and should only be used if you know what you are doing. Default: False. - c: continue training from latest checkpoint. Default: False. - val: True to run the validation only. Requires training to have finished. Default: False. - disable_checkpointing: True to disable checkpointing. Ideal for testing things out and you don't want to flood your hard drive with checkpoints. Default: False. num_gpusz(please use gpu_id to set the GPUs to usetrzLplease specify the `trainer_class_name` in the __init__ of `nnUNetV2Runner`.npzzYplease specify the `export_validation_probabilities` in the __init__ of `nnUNetV2Runner`.T)shellN)r-r5r6train_single_model_commandr )r9ryrzr{r|cmdr:r:r;train_single_models      z!nnUNetV2Runner.train_single_modelc Cs&t|ttfrft|dkrVd}tt|D]}|||d7}q*d|dd}qpd|d}n d|}t|tst|dkrdnt|}|d|jd|d|dd |jd |} |jr| d 7} | D]D\} } | d ks| d kr | d| d| 7} q| d| d| 7} q| S)Nr,zCUDA_VISIBLE_DEVICES=r?rz nnUNetv2_train  z-tr z -num_gpus z --npzpZpretrained_weightsz -z --) r&rsrVrWranger2r rritems) r9ryrzr{r|Z gpu_ids_str_idevice_settingr}r_key_valuer:r:r;rs,  "z)nnUNetV2Runner.train_single_model_commandz tuple | strztuple | list | int | None)configsgpu_id_for_allr|rc Ks|dkrHtjddgtjd}|jd}t|d}tt |}nt |t rZt |}t dt|d|t|d kr|jft ||d |n6t |D],}t |jD]}|jf|||d |qqdS) a Run the training for all the models specified by the configurations. Note: to set the number of GPUs to use, use ``gpu_id_for_all`` instead of the `CUDA_VISIBLE_DEVICES` environment variable. Args: configs: configurations that should be trained. Default: ("2d", "3d_fullres", "3d_lowres", "3d_cascade_fullres"). gpu_id_for_all: a tuple/list/integer of GPU device ID(s) to use for the training. Default: None (all available GPUs). kwargs: this optional parameter allows you to specify additional arguments defined in the ``train_single_model`` method. Nz nvidia-smiz --list-gpus)stdoutzutf-8 znumber of GPUs is z, device ids are rrr)ryrzr{) subprocessrunPIPErdecoderWstripr@rsrr&r2r r5rgtrain_parallelr7r) r9rrr|resultoutputr}cfg_foldr:r:r;train.s    znnUNetV2Runner.trainrVc Ks@g}ttj|j|jD]&\}}}tdd|Dr||qddlm }|D](} t d| d|| dd|j d qT|pi}t |} t| } tjtjtjgtjgg} g} tt| D]|}| d d | Dd}| |D]V}|t |krt|jD]:}||| }|||||}| d |||d 7}qqq| S)a Create the line command for subprocess call for parallel training. Args: configs: configurations that should be trained. Default: ("2d", "3d_fullres", "3d_lowres", "3d_cascade_fullres"). gpu_id_for_all: a tuple/list/integer of GPU device ID(s) to use for the training. Default: None (all available GPUs). kwargs: this optional parameter allows you to specify additional arguments defined in the ``train_single_model`` method. css|]}|dVqdS)z.npzN)endswith)rCfiler:r:r; dsz4nnUNetV2Runner.train_parallel_cmd..r)unpack_datasetz unpacking 'z'...TF)folderZunpack_segmentationZoverwrite_existingr^cSsi|] }|gqSr:r:)rC_jr:r:r; ysz5nnUNetV2Runner.train_parallel_cmd..r?r)r(walkr)rrr3anyappendZ#nnunetv2.training.dataloading.utilsrr5rgr#r rWM N_3D_FULLRESN_2D N_3D_LOWRESN_3D_CASCADE_FULLRESrr7r)r9rrr|Z folder_namesrootrFfilesrZ folder_namedevices n_devicesZ_configsall_cmdsZ_stage_index_configrZ the_devicerr:r:r;train_parallel_cmdPs:"     z!nnUNetV2Runner.train_parallel_cmdcKs|jf||d|}t|D]V\}}|D]D\}}|s s"