o  i@sddlmZddlZddlZddlZddlmZddlZddlm Z ddlm Z m Z m Z ddlmZddlmZmZddlmZed d d \ZZed d d \ZZed d d \ZZed\ZZejjeZdgZ GdddZ!dS)) annotationsN)Any) NNUNETMode) analyze_datacreate_new_data_copycreate_new_dataset_json) ConfigParser) ensure_tupleoptional_import)run_cmdz4batchgenerators.utilities.file_and_folder_operations load_pickle)namejointqdmnibabelnnUNetV2Runnerc @seZdZdZ   ddd dZddZddddZ     ddd!d"Z # $ %  &ddd-d.Ze j e j e j fd/d&dfdd2d3Z dd$dddd#d$d%dd&e j e j e j fd/df dd5d6Zddd=d>Zd?d@Ze j e j e j e jfdfddEdFZe j e j e j e jfdfddHdIZe j e j e j e jfdfddJdKZddLdMZe j e j e j e jffddNdOZd&e j e j e j e jfdddddPdfddZd[Z  \       ]  ^ 7   7dddqdrZ P   dddvdwZ     ddd}d~ZdS)ra ``nnUNetV2Runner`` provides an interface in MONAI to use `nnU-Net` V2 library to analyze, train, and evaluate neural networks for medical image segmentation tasks. A version of nnunetv2 higher than 2.2 is needed for this class. ``nnUNetV2Runner`` can be used in two ways: #. with one line of code to execute the complete pipeline. #. with a series of commands to run each modules in the pipeline. The output of the interface is a directory that contains: #. converted dataset met the requirement of nnU-Net V2 #. data analysis results #. checkpoints from the trained U-Net models #. validation accuracy in each fold of cross-validation #. the predictions on the testing datasets from the final algorithm ensemble and potential post-processing Args: input_config: the configuration dictionary or the file path to the configuration in the form of YAML. The keys required in the configuration are: - ``"datalist"``: File path to the datalist for the train/testing splits - ``"dataroot"``: File path to the dataset - ``"modality"``: Imaging modality, e.g. "CT", ["T2", "ADC"] Currently, the configuration supports these optional keys: - ``"nnunet_raw"``: File path that will be written to env variable for nnU-Net - ``"nnunet_preprocessed"``: File path that will be written to env variable for nnU-Net - ``"nnunet_results"``: File path that will be written to env variable for nnU-Net - ``"nnUNet_trained_models"`` - ``"dataset_name_or_id"``: Name or Integer ID of the dataset If an optional key is not specified, then the pipeline will use the default values. trainer_class_name: the trainer class names offered by nnUNetV2 exhibit variations in training duration. Default: "nnUNetTrainer". Other options: "nnUNetTrainer_Xepoch". X could be one of 1,5,10,20,50,100, 250,2000,4000,8000. export_validation_probabilities: True to save softmax predictions from final validation as npz files (in addition to predicted segmentations). Needed for finding the best ensemble. Default: True. work_dir: working directory to save the intermediate and final results. Examples: - Use the one-liner to start the nnU-Net workflow .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner run --input_config ./input.yaml - Use `convert_dataset` to prepare the data to meet nnU-Net requirements, generate dataset JSON file, and copy the dataset to a location specified by ``nnunet_raw`` in the input config file .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner convert_dataset --input_config="./input.yaml" - `convert_msd_dataset` is an alternative option to prepare the data if the dataset is MSD. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner convert_msd_dataset \ --input_config "./input.yaml" --data_dir "/path/to/Task09_Spleen" - experiment planning and data pre-processing .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner plan_and_process --input_config "./input.yaml" - training all 20 models using all GPUs available. "CUDA_VISIBLE_DEVICES" environment variable is not supported. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" - training a single model on a single GPU for 5 epochs. Here ``config`` is used to specify the configuration. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train_single_model --input_config "./input.yaml" \ --config "3d_fullres" \ --fold 0 \ --gpu_id 0 \ --trainer_class_name "nnUNetTrainer_5epochs" \ --export_validation_probabilities True - training for all 20 models (4 configurations by 5 folds) on 2 GPUs .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" --gpu_id_for_all "0,1" - 5-fold training for a single model on 2 GPUs. Here ``configs`` is used to specify the configurations. .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" \ --configs "3d_fullres" \ --trainer_class_name "nnUNetTrainer_5epochs" \ --export_validation_probabilities True \ --gpu_id_for_all "0,1" - find the best configuration .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner find_best_configuration --input_config "./input.yaml" - predict, ensemble, and post-process .. code-block:: bash python -m monai.apps.nnunet nnUNetV2Runner predict_ensemble_postprocessing --input_config "./input.yaml" nnUNetTrainerwork_dirT input_configrtrainer_class_namestrexport_validation_probabilitiesboolreturnNonecCsi|_||_||_||_||_t|jtr|j|_nt|jtr/tj |jr/t |j|_nt |d|jdtj d|jd|_|jdtj d|jd|_|jdtj d|jd|_tj |jspt|jtj |js}t|jtj |jst|j|jtjd <|jtjd<|jtjd <td tjd <t|jd d |_zddlm}|t|j|_Wntytd|jdYnwddlm}||_d|_ i|_!dS)Nz is not a valid file or dict nnunet_raw.ZnnUNet_raw_data_basennunet_preprocessednnUNet_preprocessednnunet_resultsZnnUNet_trained_modelsZ nnUNet_rawnnUNet_resultsZOMP_NUM_THREADSdataset_name_or_idrmaybe_convert_to_dataset_namezDataset with name/ID: z cannot be found in the record. Please ignore the message above if you are running the pipeline from a fresh start. But if the dataset is expected to be found, please check your input_config.)default_num_processes)" input_infoZinput_config_or_dictrrr isinstancedictrospathisfilerload_config_file ValueErrorpoprrrrexistsmakedirsenvironr"-nnunetv2.utilities.dataset_name_id_conversionr$int dataset_name BaseExceptionloggerwarningZnnunetv2.configurationr% num_foldsbest_configuration)selfrrrrr$r%r<c/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/apps/nnunet/nnunetv2_runner.py__init__sP             znnUNetV2Runner.__init__c CsNz tt|jd}d|dd}t|jd}dd|D}dd|D}||vr6td WdS|jd }|d t j krI|dd }|d | t j d }t j |j|}t j |sht |d dlm}|t|j|_t|jd}d|vrt t j |dt t j |dntdWdSd}d|vsd|vrt t j |dd|vrdnd}t||d trd||d vrt t j |dt||d\} } |jd} t| ts| g} t| | | t|dt j |ddt|||| |dWdSty&} ztd| WYd} ~ dSd} ~ ww)zQConvert and make a copy the dataset to meet the requirements of nnU-Net workflow.iDatasetNz/*cSsg|] }|tjdqS))splitr*sep.0_itemr<r<r= sz2nnUNetV2Runner.convert_dataset..cSsg|] }|ddqS)_r)rBrDr<r<r=rGsz Dataset with the same ID exists!datarootrArHrr#datalisttrainingZimagesTrZlabelsTrzHThe datalist file has incorrect format: the `training` key is not found.testtestingimagesTslabelZlabelsTs) datalist_jsondata_dirmodalityz dataset.json)rRnum_foreground_classesnum_input_channelsZnum_training_dataZoutput_filepath)test_keyrPrQrTZoutput_datafolderzIInput config may be incorrect. Detail info: error/exception message is: )rr4r"globrr7r8r'r/r*rCrBr+rr0r1r3r$r5rr-errorr(r)rlistrlenrr6) r;Zraw_data_foldername_prefixsubdirsZ dataset_idsrQZraw_data_foldernamer$rPrUrTrSrRerrr<r<r=convert_datasetsj       "     znnUNetV2Runner.convert_datasetNrArQ overwrite_id str | Nonen_procr4cCs.ddlm}|dkr dn|j}||||dS)a Convert and make a copy the MSD dataset to meet requirements of nnU-Net workflow. Args: data_dir: downloaded and extracted MSD dataset folder. CANNOT be nnUNetv1 dataset! Example: "/workspace/downloads/Task05_Prostate". overwrite_id: Overwrite the dataset id. If not set then use the id of the MSD task (inferred from the folder name). Only use this if you already have an equivalently numbered dataset! n_proc: Number of processes used. r)convert_msd_datasetN)Z/nnunetv2.dataset_conversion.convert_MSD_datasetr`r%)r;rQr]r_r` num_processesr<r<r=r`s z"nnUNetV2Runner.convert_msd_datasetDatasetFingerprintExtractorFfpenpfpverify_dataset_integritycleanverbosecCsFddlm}|dkr |jn|}td|t|jg|||||dS)aW Extracts the dataset fingerprint used for experiment planning. Args: fpe: [OPTIONAL] Name of the Dataset Fingerprint Extractor class that should be used. Default is "DatasetFingerprintExtractor". npfp: [OPTIONAL] Number of processes used for fingerprint extraction. verify_dataset_integrity: [RECOMMENDED] set this flag to check the dataset integrity. This is useful and should be done once for each dataset! clean: [OPTIONAL] Set this flag to overwrite existing fingerprints. If this flag is not set and a fingerprint already exists, the fingerprint extractor will not run. verbose: set this to print a lot of stuff. Useful for debugging. Will disable progress bar! Recommended for cluster environments. r)extract_fingerprintszFingerprint extraction...N)4nnunetv2.experiment_planning.plan_and_preprocess_apirhr%r7infor4r")r;rcrdrerfrgrhr<r<r=rh,s  z#nnUNetV2Runner.extract_fingerprintsExperimentPlannerDefaultPreprocessor nnUNetPlansplgpu_memory_targetfloatpreprocessor_nameoverwrite_target_spacingoverwrite_plans_namecCs4ddlm}td|t|jg|||||dS)at Generate a configuration file that specifies the details of the experiment. Args: pl: [OPTIONAL] Name of the Experiment Planner class that should be used. Default is "ExperimentPlanner". Note: There is no longer a distinction between 2d and 3d planner. It's an all-in-one solution now. gpu_memory_target: [OPTIONAL] DANGER ZONE! Sets a custom GPU memory target. Default: 8 [GB]. Changing this will affect patch and batch size and will definitely affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). preprocessor_name: [OPTIONAL] DANGER ZONE! Sets a custom preprocessor class. This class must be located in nnunetv2.preprocessing. Default: "DefaultPreprocessor". Changing this may affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). overwrite_target_spacing: [OPTIONAL] DANGER ZONE! Sets a custom target spacing for the 3d_fullres and 3d_cascade_fullres configurations. Default: None [no changes]. Changing this will affect image size and potentially patch and batch size. This will definitely affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). Changing the target spacing for the other configurations is currently not implemented. New target spacing must be a list of three numbers! overwrite_plans_name: [OPTIONAL] DANGER ZONE! If you used -gpu_memory_target, -preprocessor_name or -overwrite_target_spacing it is best practice to use -overwrite_plans_name to generate a differently named plans file such that the nnunet default plans are not overwritten. You will then need to specify your custom plan. r)plan_experimentszExperiment planning...N)rirur7rjr4r")r;rorprrrsrtrur<r<r=ruIs !  znnUNetV2Runner.plan_experiments)rlrlrlctuplecCs4ddlm}td|t|jg||||ddS)a Apply a set of preprocessing operations to the input data before the training. Args: overwrite_plans_name: [OPTIONAL] You can use this to specify a custom plans file that you may have generated. c: [OPTIONAL] Configurations for which the preprocessing should be run. Default: 2d 3f_fullres 3d_lowres. 3d_cascade_fullres does not need to be specified because it uses the data from 3f_fullres. Configurations that do not exist for some datasets will be skipped). n_proc: [OPTIONAL] Use this to define how many processes are to be used. If this is just one number then this number of processes is used for all configurations specified with -c. If it's a list of numbers this list must have as many elements as there are configurations. We then iterate over zip(configs, num_processes) to determine the number of processes used for each configuration. More processes are always faster (up to the number of threads your PC can support, so 8 for a 4-core CPU with hyperthreading. If you don't know what that is then don't touch it, or at least don't increase it!). DANGER: More often than not the number of processes that can be used is limited by the amount of RAM available. Image resampling takes up a lot of RAM. MONITOR RAM USAGE AND DECREASE -n_proc IF YOUR RAM FILLS UP TOO MUCH! Default: 8 4 8 (=8 processes for 2d, 4 for 3d_fullres and 8 for 3d_lowres if -c is at its default). verbose: Set this to print a lot of stuff. Useful for debugging. Will disable the progress bar! Recommended for cluster environments. r) preprocesszPreprocessing...)configurationsrargN)rirxr7rjr4r")r;rvr_rtrgrxr<r<r=rxvs    znnUNetV2Runner.preprocessno_ppcCs@|||||| ||||| | |s|| | | | dSdS)a_ Performs experiment planning and preprocessing before the training. Args: fpe: [OPTIONAL] Name of the Dataset Fingerprint Extractor class that should be used. Default is "DatasetFingerprintExtractor". npfp: [OPTIONAL] Number of processes used for fingerprint extraction. Default: 8. verify_dataset_integrity: [RECOMMENDED] set this flag to check the dataset integrity. This is useful and should be done once for each dataset! no_pp: [OPTIONAL] Set this to only run fingerprint extraction and experiment planning (no preprocessing). Useful for debugging. clean:[OPTIONAL] Set this flag to overwrite existing fingerprints. If this flag is not set and a fingerprint already exists, the fingerprint extractor will not run. REQUIRED IF YOU CHANGE THE DATASET FINGERPRINT EXTRACTOR OR MAKE CHANGES TO THE DATASET! pl: [OPTIONAL] Name of the Experiment Planner class that should be used. Default is "ExperimentPlanner". Note: There is no longer a distinction between 2d and 3d planner. It's an all-in-one solution now. gpu_memory_target: [OPTIONAL] DANGER ZONE! Sets a custom GPU memory target. Default: 8 [GB]. Changing this will affect patch and batch size and will definitely affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). preprocessor_name: [OPTIONAL] DANGER ZONE! Sets a custom preprocessor class. This class must be located in nnunetv2.preprocessing. Default: "DefaultPreprocessor". Changing this may affect your models' performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). overwrite_target_spacing: [OPTIONAL] DANGER ZONE! Sets a custom target spacing for the 3d_fullres and 3d_cascade_fullres configurations. Default: None [no changes]. Changing this will affect image size and potentially patch and batch size. This will definitely affect your models performance! Only use this if you really know what you are doing and NEVER use this without running the default nnU-Net first (as a baseline). Changing the target spacing for the other configurations is currently not implemented. New target spacing must be a list of three numbers! overwrite_plans_name: [OPTIONAL] USE A CUSTOM PLANS IDENTIFIER. If you used -gpu_memory_target, -preprocessor_name or -overwrite_target_spacing it is best practice to use -overwrite_plans_name to generate a differently named plans file such that the nnunet default plans are not overwritten. You will then need to specify your custom plans file with -p whenever running other nnunet commands (training, inference, etc) c: [OPTIONAL] Configurations for which the preprocessing should be run. Default: 2d 3f_fullres 3d_lowres. 3d_cascade_fullres does not need to be specified because it uses the data from 3f_fullres. Configurations that do not exist for some datasets will be skipped. n_proc: [OPTIONAL] Use this to define how many processes are to be used. If this is just one number then this number of processes is used for all configurations specified with -c. If it's a list of numbers this list must have as many elements as there are configurations. We then iterate over zip(configs, num_processes) to determine the number of processes used for each configuration. More processes are always faster (up to the number of threads your PC can support, so 8 for a 4-core CPU with hyperthreading. If you don't know what that is then don't touch it, or at least don't increase it!). DANGER: More often than not the number of processes that can be used is limited by the amount of RAM available. Image resampling takes up a lot of RAM. MONITOR RAM USAGE AND DECREASE -n_proc IF YOUR RAM FILLS UP TOO MUCH! Default: 8 4 8 (=8 processes for 2d, 4 for 3d_fullres and 8 for 3d_lowres if -c is at its default). verbose: Set this to print a lot of stuff. Useful for debugging. Will disable progress bar! (Recommended for cluster environments). N)rhrurx)r;rcrdrerzrfrorprrrsrtrvr_rgr<r<r=plan_and_processs DznnUNetV2Runner.plan_and_processrconfigfoldgpu_idtuple | list | intkwargscKstd|vr|dtdd|vr|dtdd|vr*|dtd|||||}t|ddd S) a Run the training on a single GPU with one specified configuration provided. Note: this will override the environment variable `CUDA_VISIBLE_DEVICES`. Args: config: configuration that should be trained. Examples: "2d", "3d_fullres", "3d_lowres". fold: fold of the 5-fold cross-validation. Should be an int between 0 and 4. gpu_id: an integer to select the device to use, or a tuple/list of GPU device indices used for multi-GPU training (e.g., (0,1)). Default: 0. kwargs: this optional parameter allows you to specify additional arguments in ``nnunetv2.run.run_training.run_training_entry``. Currently supported args are: - p: custom plans identifier. Default: "nnUNetPlans". - pretrained_weights: path to nnU-Net checkpoint file to be used as pretrained model. Will only be used when actually training. Beta. Use with caution. Default: False. - use_compressed: True to use compressed data for training. Reading compressed data is much more CPU and (potentially) RAM intensive and should only be used if you know what you are doing. Default: False. - c: continue training from latest checkpoint. Default: False. - val: True to run the validation only. Requires training to have finished. Default: False. - disable_checkpointing: True to disable checkpointing. Ideal for testing things out and you don't want to flood your hard drive with checkpoints. Default: False. num_gpusz(please use gpu_id to set the GPUs to usetrzLplease specify the `trainer_class_name` in the __init__ of `nnUNetV2Runner`.npzzYplease specify the `export_validation_probabilities` in the __init__ of `nnUNetV2Runner`.T)shellN)r/r7r8train_single_model_commandr )r;r|r}r~rcmdr<r<r=train_single_models      z!nnUNetV2Runner.train_single_modelc Cs$t|ttfr3t|dkr+d}tt|D] }|||d7}qd|dd}n d|d}nd|}t|tsCt|dkrEdnt|}|d|jd|d|dd |jd |} |jrj| d 7} | D]!\} } | d ksz| d kr| d| d| 7} qn| d| d| 7} qn| S)Nr!,zCUDA_VISIBLE_DEVICES=rArz nnUNetv2_train  z-tr z -num_gpus z --npzpZpretrained_weightsz -z --) r(rwrXrYranger4r"rritems) r;r|r}r~rZ gpu_ids_str_idevice_settingrr_key_valuer<r<r=rs,  "z)nnUNetV2Runner.train_single_model_commandconfigs tuple | strgpu_id_for_alltuple | list | int | Nonec Ks|dur$tjddgtjd}|jd}t|d}tt |}n t |t r-t |}t dt|d|t|d krO|jd t ||d |dSt |D]}t |jD]}|jd |||d |qZqSdS) a Run the training for all the models specified by the configurations. Note: to set the number of GPUs to use, use ``gpu_id_for_all`` instead of the `CUDA_VISIBLE_DEVICES` environment variable. Args: configs: configurations that should be trained. Default: ("2d", "3d_fullres", "3d_lowres", "3d_cascade_fullres"). gpu_id_for_all: a tuple/list/integer of GPU device ID(s) to use for the training. Default: None (all available GPUs). kwargs: this optional parameter allows you to specify additional arguments defined in the ``train_single_model`` method. Nz nvidia-smiz --list-gpus)stdoutzutf-8 znumber of GPUs is z, device ids are r!rr)r|r}r~r<) subprocessrunPIPErdecoderYstriprBrwrr(r4r r7rjtrain_parallelr9r) r;rrrresultoutputrcfg_foldr<r<r=train.s    znnUNetV2Runner.trainrXc Ks@g}ttj|j|jD]\}}}tdd|Dr!||qddlm }|D]} t d| d|| dd|j d q*|pBi}t |} t| } tjtjtjgtjgg} g} tt| D]>}| d d | Dd}| |D]+}|t |vrt|jD]}||| }|||||}| d |||d 7}q~qqq_| S)a Create the line command for subprocess call for parallel training. Args: configs: configurations that should be trained. Default: ("2d", "3d_fullres", "3d_lowres", "3d_cascade_fullres"). gpu_id_for_all: a tuple/list/integer of GPU device ID(s) to use for the training. Default: None (all available GPUs). kwargs: this optional parameter allows you to specify additional arguments defined in the ``train_single_model`` method. css|]}|dVqdS)z.npzN)endswith)rEfiler<r<r= dsz4nnUNetV2Runner.train_parallel_cmd..r)unpack_datasetz unpacking 'z'...TF)folderZunpack_segmentationZoverwrite_existingracSsi|]}|gqSr<r<)rE_jr<r<r= ysz5nnUNetV2Runner.train_parallel_cmd..rAr!)r*walkr+rrr5anyappendZ#nnunetv2.training.dataloading.utilsrr7rjr%r rYM N_3D_FULLRESN_2D N_3D_LOWRESN_3D_CASCADE_FULLRESrr9r)r;rrrZ folder_namesrootrHfilesrZ folder_namedevices n_devicesZ_configsall_cmdsZ_stage_index_configrZ the_devicerr<r<r=train_parallel_cmdPs@"      z!nnUNetV2Runner.train_parallel_cmdcKs|jd||d|}t|D]+\}}|D]"\}}|sqtd|dd|d|dtj|j|j d qq|D]6} g} | D]&} | | sJqCd| | } td | d | d | t j | d t j d qC| D]} | qlq=dS)a Create the line command for subprocess call for parallel training. Note: to set the number of GPUs to use, use ``gpu_id_for_all`` instead of the `CUDA_VISIBLE_DEVICES` environment variable. Args: configs: configurations that should be trained. default: ("2d", "3d_fullres", "3d_lowres", "3d_cascade_fullres"). gpu_id_for_all: a tuple/list/integer of GPU device ID(s) to use for the training. Default: None (all available GPUs). kwargs: this optional parameter allows you to specify additional arguments defined in the ``train_single_model`` method. rztraining - stage r!z : for gpu z , commands: z log '.txt' inside ''z; z&Current running command on GPU device z: rT)rrNr<)r enumeraterr7rjr*r+rrr5rrPopenDEVNULLwait)r;rrrrscmdsr~Zgpu_cmdstage processes device_idZcmd_strrr<r<r=rs6  znnUNetV2Runner.train_parallelcKs|jd||dd|dS)al Perform validation on single model. Args: config: configuration that should be trained. fold: fold of the 5-fold cross-validation. Should be an int between 0 and 4. kwargs: this optional parameter allows you to specify additional arguments defined in the ``train_single_model`` method. T)r|r}only_run_validationNr<)r)r;r|r}rr<r<r=validate_single_models z$nnUNetV2Runner.validate_single_modelcKs8t|D]}t|jD] }|jd||d|q qdS)a Perform validation in all models defined by the configurations over 5 folds. Args: configs: configurations that should be trained. default: ("2d", "3d_fullres", "3d_lowres", "3d_cascade_fullres"). kwargs: this optional parameter allows you to specify additional arguments defined in the ``train_single_model`` method. )r|r}Nr<)r rr9r)r;rrrrr<r<r=validates znnUNetV2Runner.validaterr!planstrainerstuple | str | Noneallow_ensemblingra overwritefoldslist[int] | tuple[int, ...]strictc Cstddlm} m} t|}t|}|dur|j}t|}| |||} |dkr(|jn|}| t|j| |||||ddS)aH Find the best model configurations. Args: plans: list of plan identifiers. Default: nnUNetPlans. configs: list of configurations. Default: ["2d", "3d_fullres", "3d_lowres", "3d_cascade_fullres"]. trainers: list of trainers. Default: nnUNetTrainer. allow_ensembling: set this flag to enable ensembling. num_processes: number of processes to use for ensembling, postprocessing, etc. overwrite: if set we will overwrite already ensembled files etc. May speed up consecutive runs of this command (not recommended) at the risk of not updating outdated results. folds: folds to use. Default: (0, 1, 2, 3, 4). strict: a switch that triggers RunTimeError if the logging folder cannot be found. Default: False. r)0dumb_trainer_config_plans_to_trained_models_dictfind_best_configurationN)rrarrr)Z+nnunetv2.evaluation.find_best_configurationrrr rr%r4r") r;rrrrrarrrrrmodelsr<r<r=rs"  z&nnUNetV2Runner.find_best_configuration?checkpoint_final.pthr!list_of_lists_or_source_folderstr | list[list[str]] output_folderstr | None | list[str]model_training_output_dir use_foldstuple[int, ...] | str | Nonetile_step_size use_gaussian use_mirroringperform_everything_on_gpusave_probabilitiescheckpoint_name folder_with_segs_from_prev_stage num_partspart_idnum_processes_preprocessing!num_processes_segmentation_exportc Cs~|tjd<ddlm}|dkr|jn|}|dkr|jn|}|||||| d}|j||| d|j||| | ||| ||d dS)a6 Use this to run inference with nnU-Net. This function is used when you want to manually specify a folder containing a trained nnU-Net model. This is useful when the nnunet environment variables (nnUNet_results) are not set. Args: list_of_lists_or_source_folder: input folder. Remember to use the correct channel numberings for your files (_0000 etc). File endings must be the same as the training dataset! output_folder: Output folder. If it does not exist it will be created. Predicted segmentations will have the same name as their source images. model_training_output_dir: folder in which the trained model is. Must have subfolders fold_X for the different folds you trained. use_folds: specify the folds of the trained model that should be used for prediction Default: (0, 1, 2, 3, 4). tile_step_size: step size for sliding window prediction. The larger it is the faster but less accurate the prediction. Default: 0.5. Cannot be larger than 1. We recommend the default. use_gaussian: use Gaussian smoothing as test-time augmentation. use_mirroring: use mirroring/flipping as test-time augmentation. verbose: set this if you like being talked to. You will have to be a good listener/reader. save_probabilities: set this to export predicted class "probabilities". Required if you want to ensemble multiple configurations. overwrite: overwrite an existing previous prediction (will not overwrite existing files) checkpoint_name: name of the checkpoint you want to use. Default: checkpoint_final.pth. folder_with_segs_from_prev_stage: folder containing the predictions of the previous stage. Required for cascaded models. num_parts: number of separate nnUNetv2_predict call that you will be making. Default: 1 (= this one call predicts everything). part_id: if multiple nnUNetv2_predict exist, which one is this? IDs start with 0 can end with num_parts - 1. So when you submit 5 nnUNetv2_predict calls you need to set -num_parts 5 and use -part_id 0, 1, 2, 3 and 4. num_processes_preprocessing: out-of-RAM issues. num_processes_segmentation_export: Number of processes used for segmentation export. More is not always better. Beware of out-of-RAM issues. gpu_id: which GPU to use for prediction. CUDA_VISIBLE_DEVICESr)nnUNetPredictor)rrrZperform_everything_on_devicerg)rrr) rZ/output_folder_or_list_of_truncated_output_filesrrrrrrrN)r*r2(nnunetv2.inference.predict_from_raw_datarr%Z$initialize_from_trained_model_folderZpredict_from_files)r;rrrrrrrrrgrrrrrrrrr~rZn_processes_preprocessingZn_processes_segmentation_export predictorr<r<r=predicts6 7  znnUNetV2Runner.predict run_ensemble run_predictrun_postprocessingc Ksnddlm}ddlm}ddlm}t|j|jd} t|j |j} t t j |j |jd|_|o;t|jddd k}|} g} |jddD]3} t| d | d }| ||rz|t|j| d | d | d }|jd| ||| |ddd|qG|r|| t| ddd|rt| d}n|r| d}|rt|jdd\}}||t| d|||jddddSdS)a Run prediction, ensemble, and/or postprocessing optionally. Args: folds: which folds to use run_ensemble: whether to run ensembling. run_predict: whether to predict using trained checkpoints run_postprocessing: whether to conduct post-processing kwargs: this optional parameter allows you to specify additional arguments defined in the ``predict`` method. r)ensemble_folders)apply_postprocessing_to_folder)get_output_folderrNzinference_information.jsonZbest_model_or_ensembleZselected_model_or_modelsr!pred_ configurationtrainerplans_identifierFT)rrrrrrgrZensemble_predictions)Zsave_merged_probabilitiesZpostprocessing_fileZ"ensemble_predictions_postprocessedZsome_plans_file)Zplans_file_or_dictNr<)Znnunetv2.ensembling.ensemblerZ3nnunetv2.postprocessing.remove_connected_componentsrZ&nnunetv2.utilities.file_path_utilitiesrrrr5rrr-r*r+r:rYrr4r"rr )r;rrrrrrrr source_dirZtarget_dir_baseZ used_foldsZoutput_foldersim output_dir model_folderZ folder_for_ppZpp_fnsZ pp_fn_kwargsr<r<r=predict_ensemble_postprocessingRsd         z.nnUNetV2Runner.predict_ensemble_postprocessingrun_convert_datasetrun_plan_and_process run_trainrun_find_best_configuration#run_predict_ensemble_postprocessingcCs@|r||r ||r||r||r|dS)a Run the nnU-Net pipeline. Args: run_convert_dataset: whether to convert datasets, defaults to True. run_plan_and_process: whether to preprocess and analyze the dataset, defaults to True. run_train: whether to train models, defaults to True. run_find_best_configuration: whether to find the best model (ensemble) configurations, defaults to True. run_predict_ensemble_postprocessing: whether to make predictions on test datasets, defaults to True. N)r\r{rrr)r;rrrrrr<r<r=rsznnUNetV2Runner.run)rrT) rrrrrrrrrr)NrA)rQrr]r^r_r4rr)rbrAFFF) rcrrdr4rerrfrrgrrr)rkrlrmNrn) rorrprqrrrrsrrtrrr) rvrwr_rwrtrrgrrr)rcrrdr4rerrzrrfrrorrpr4rrrrsrrtrrvrwr_rwrgrrr)r) r|rr}r4r~rrrrr)rrrrrrrr)rrrrrrrrX)r|rr}r4rrrr)rrwrrrr)rrrrrrrrrar4rrrrrrrr)NrTTTTFTrNr!rrArAr)&rrrrrrrrrrqrrrrrrrgrrrrrrrrr^rr4rr4rr4rr4r~r4rr)rTTT) rrwrrrrrrrrrr)TTTTT) rrrrrrrrrrrr)__name__ __module__ __qualname____doc__r>r\r`rhrurrrrrxr{rrrrrrrrrrrrr<r<r<r=r%su @ D  / + J* $ 7 )  6 Y N)" __future__rrVr*rtypingrmonaiZmonai.apps.nnunet.utilsrrrrr monai.bundler monai.utilsr r monai.utils.miscr r rHrrhas_tqdmnibappsutils get_loggerrr7__all__rr<r<r<r=s$