o  i1h@sddlmZddlZddlmZmZddlmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z'ddl%m(Z)ddl*m+Z+erddl,m-Z-m.Z.ddl/m0Z0ne$de!j1e#d\Z-Z2e$de!j1e#d\Z0Z2e$de!j1e#d\Z.Z2gdZ3GdddeZ4Gddde4Z5Gddde4Z6dS)) annotationsN)IterableSequence) TYPE_CHECKINGAnyCallable) DataLoader)KeysCollection) MetaTensor)IterationEventsdefault_metric_cmp_fndefault_prepare_batch)Workflow)Inferer SimpleInferer) eval_mode train_mode) Transform) ForwardMode IgniteInfo ensure_tuple min_versionoptional_import) CommonKeys)EngineStatsKeys)look_up_option)Engine EventEnum)Metricz ignite.enginerzignite.metricsrr) EvaluatorSupervisedEvaluatorEnsembleEvaluatorcs^eZdZdZddeddddeddejdddddfd.fd%d& Zd/d0fd*d+ Z d,d-Z Z S)1ra9 Base class for all kinds of evaluators, inherits from Workflow. Args: device: an object representing the device on which to run. val_data_loader: Ignite engine use data_loader to run, must be Iterable or torch.DataLoader. epoch_length: number of iterations for one epoch, default to `len(val_data_loader)`. non_blocking: if True and this copy is between CPU and GPU, the copy may occur asynchronously with respect to the host. For other cases, this argument has no effect. prepare_batch: function to parse expected data (usually `image`, `label` and other network args) from `engine.state.batch` for every iteration, for more details please refer to: https://pytorch.org/ignite/generated/ignite.engine.create_supervised_trainer.html. iteration_update: the callable function for every iteration, expect to accept `engine` and `engine.state.batch` as inputs, return data will be stored in `engine.state.output`. if not provided, use `self._iteration()` instead. for more details please refer to: https://pytorch.org/ignite/generated/ignite.engine.engine.Engine.html. postprocessing: execute additional transformation for the model output data. Typically, several Tensor based transforms composed by `Compose`. key_val_metric: compute metric when every iteration completed, and save average value to engine.state.metrics when epoch completed. key_val_metric is the main metric to compare and save the checkpoint into files. additional_metrics: more Ignite metrics that also attach to Ignite Engine. metric_cmp_fn: function to compare current key metric with previous best key metric value, it must accept 2 args (current_metric, previous_best) and return a bool result: if `True`, will update `best_metric` and `best_metric_epoch` with current metric and epoch, default to `greater than`. val_handlers: every handler is a set of Ignite Event-Handlers, must have `attach` function, like: CheckpointHandler, StatsHandler, etc. amp: whether to enable auto-mixed-precision evaluation, default is False. mode: model forward mode during evaluation, should be 'eval' or 'train', which maps to `model.eval()` or `model.train()`, default to 'eval'. event_names: additional custom ignite events that will register to the engine. new events can be a list of str or `ignite.engine.events.EventEnum`. event_to_attr: a dictionary to map an event to a state attribute, then add to `engine.state`. for more details, check: https://pytorch.org/ignite/generated/ignite.engine.engine.Engine.html #ignite.engine.engine.Engine.register_events. decollate: whether to decollate the batch-first data to a list of data after model computation, recommend `decollate=True` when `postprocessing` uses components from `monai.transforms`. default to `True`. to_kwargs: dict of other args for `prepare_batch` API when converting the input data, except for `device`, `non_blocking`. amp_kwargs: dict of the args for `torch.autocast("cuda")` API, for more details: https://pytorch.org/docs/stable/amp.html#torch.autocast. NFTdevicetorch.device | strval_data_loaderIterable | DataLoader epoch_length int | None non_blockingbool prepare_batchriteration_update#Callable[[Engine, Any], Any] | NonepostprocessingTransform | Nonekey_val_metricdict[str, Metric] | Noneadditional_metrics metric_cmp_fn val_handlersSequence | NoneampmodeForwardMode | str event_names.list[str | EventEnum | type[EventEnum]] | None event_to_attr dict | None decollate to_kwargs amp_kwargsreturnNonecstjdid|ddd|d|d|d|d|d |d |d | d | d | d| d|d|d|d|d|t| t} | tjkrLt|_dS| tjkrVt|_dSt d| d)Nr" max_epochs data_loaderr&r(r*r+r- key_metricr1r2handlersr5r8r:r<r=r>zunsupported mode: z, should be 'eval' or 'train'.) super__init__rrEVALrr6TRAINr ValueError)selfr"r$r&r(r*r+r-r/r1r2r3r5r6r8r:r<r=r> __class__rFY/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/engines/evaluator.pyrHZsV           zEvaluator.__init__rB global_epochintcs0t|d|j_|d|j_d|j_tdS)z Execute validation/evaluation based on Ignite Engine. Args: global_epoch: the overall epoch if during a training. evaluator engine can get it from trainer. rBrN)maxstaterAepoch iterationrGrun)rLrPrMrFrOrVs z Evaluator.runcGsBtj|jjtj|jjtj|jji}|D] }t|j|d||<q|S)a Get the statistics information of the validation process. Default to return the `rank`, `best_validation_epoch` and `best_validation_metric`. Args: vars: except for the default stats, other variables name in the `self.state` to return, will use the variable name as the key and the state content as the value. if the variable doesn't exist, default value is `None`. N) ESKeysRANKrSrankBEST_VALIDATION_EPOCHZbest_metric_epochBEST_VALIDATION_METRIC best_metricgetattr)rLvarsstatskrFrFrO get_statss  zEvaluator.get_stats)&r"r#r$r%r&r'r(r)r*rr+r,r-r.r/r0r1r0r2rr3r4r5r)r6r7r8r9r:r;r<r)r=r;r>r;r?r@)rB)rPrQr?r@) __name__ __module__ __qualname____doc__r r rrIrHrVra __classcell__rFrFrMrOr,s*11rcsVeZdZdZddedddddeddejdddddddfd3fd+d, Zd4d1d2Z Z S)5r a Standard supervised evaluation method with image and label(optional), inherits from evaluator and Workflow. Args: device: an object representing the device on which to run. val_data_loader: Ignite engine use data_loader to run, must be Iterable, typically be torch.DataLoader. network: network to evaluate in the evaluator, should be regular PyTorch `torch.nn.Module`. epoch_length: number of iterations for one epoch, default to `len(val_data_loader)`. non_blocking: if True and this copy is between CPU and GPU, the copy may occur asynchronously with respect to the host. For other cases, this argument has no effect. prepare_batch: function to parse expected data (usually `image`, `label` and other network args) from `engine.state.batch` for every iteration, for more details please refer to: https://pytorch.org/ignite/generated/ignite.engine.create_supervised_trainer.html. iteration_update: the callable function for every iteration, expect to accept `engine` and `engine.state.batch` as inputs, return data will be stored in `engine.state.output`. if not provided, use `self._iteration()` instead. for more details please refer to: https://pytorch.org/ignite/generated/ignite.engine.engine.Engine.html. inferer: inference method that execute model forward on input data, like: SlidingWindow, etc. postprocessing: execute additional transformation for the model output data. Typically, several Tensor based transforms composed by `Compose`. key_val_metric: compute metric when every iteration completed, and save average value to engine.state.metrics when epoch completed. key_val_metric is the main metric to compare and save the checkpoint into files. additional_metrics: more Ignite metrics that also attach to Ignite Engine. metric_cmp_fn: function to compare current key metric with previous best key metric value, it must accept 2 args (current_metric, previous_best) and return a bool result: if `True`, will update `best_metric` and `best_metric_epoch` with current metric and epoch, default to `greater than`. val_handlers: every handler is a set of Ignite Event-Handlers, must have `attach` function, like: CheckpointHandler, StatsHandler, etc. amp: whether to enable auto-mixed-precision evaluation, default is False. mode: model forward mode during evaluation, should be 'eval' or 'train', which maps to `model.eval()` or `model.train()`, default to 'eval'. event_names: additional custom ignite events that will register to the engine. new events can be a list of str or `ignite.engine.events.EventEnum`. event_to_attr: a dictionary to map an event to a state attribute, then add to `engine.state`. for more details, check: https://pytorch.org/ignite/generated/ignite.engine.engine.Engine.html #ignite.engine.engine.Engine.register_events. decollate: whether to decollate the batch-first data to a list of data after model computation, recommend `decollate=True` when `postprocessing` uses components from `monai.transforms`. default to `True`. to_kwargs: dict of other args for `prepare_batch` API when converting the input data, except for `device`, `non_blocking`. amp_kwargs: dict of the args for `torch.autocast("cuda")` API, for more details: https://pytorch.org/docs/stable/amp.html#torch.autocast. compile: whether to use `torch.compile`, default is False. If True, MetaTensor inputs will be converted to `torch.Tensor` before forward pass, then converted back afterward with copied meta information. compile_kwargs: dict of the args for `torch.compile()` API, for more details: https://pytorch.org/docs/stable/generated/torch.compile.html#torch-compile. NFTr" torch.devicer$r%networktorch.nn.Moduler&r'r(r)r*rr+r,infererInferer | Noner-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>compilecompile_kwargsr?r@cstjdid|d|d|d|d|d|d| d| d | d | d | d |d |d|d|d|d|d||rP|durEin|}tj|fi|}||_||_|dur`t|_dS||_dS)Nr"r$r&r(r*r+r-r/r1r2r3r5r6r8r:r<r=r>rF)rGrHtorchrlrhrrj)rLr"r$rhr&r(r*r+rjr-r/r1r2r3r5r6r8r:r<r=r>rlrmrMrFrOrHsV      zSupervisedEvaluator.__init__engine batchdatadict[str, torch.Tensor]dictc Cs|durtd|j||jj|jfi|j}t|dkr&|\}}d}i}n|\}}}}|jr\d\}} } } t|t rKt d| |j |j}}} t|t r\| |j |j}} } tj|tj|i|j_||jK|jrtjd i|j|j||jg|Ri||jjtj<Wdn1swYn|j||jg|Ri||jjtj<Wdn1swY|jr|durt ||| d|jjtj<t |jjtj|| d|jjtj<| durt || | d|jjtj<|tj|tj|jjS) a callback function for the Supervised Evaluation processing logic of 1 iteration in Ignite Engine. Return below items in a dictionary: - IMAGE: image Tensor data for model input, already moved to device. - LABEL: label Tensor data corresponding to the image, already moved to device. - PRED: prediction result of model. Args: engine: `SupervisedEvaluator` to execute operation for an iteration. batchdata: input data for this iteration, usually can be dictionary or tuple of Tensor data. Raises: ValueError: When ``batchdata`` is None. N.Must provide batch data for current iteration.rF)NNNNzgWill convert to PyTorch Tensor if using compile, and casting back to MetaTensor after the forward pass.cuda)metaapplied_operationsru)rKr*rSr"r(r=lenrl isinstancer warningswarn as_tensorrvrwKeysIMAGELABELoutputr6rhr5rnautocastr>rjPRED fire_eventr FORWARD_COMPLETEDMODEL_COMPLETED) rLrorpbatchinputstargetsargskwargsZ inputs_metaZ targets_metaZinputs_applied_operationsZtargets_applied_operationsrFrFrO _iterations`      (&  zSupervisedEvaluator._iteration).r"rgr$r%rhrir&r'r(r)r*rr+r,rjrkr-r.r/r0r1r0r2rr3r4r5r)r6r7r8r9r:r;r<r)r=r;r>r;rlr)rmr;r?r@)ror rprqr?rr rbrcrdrer r rrIrHrrfrFrFrMrOr s.84r csTeZdZdZdddedddddeddejdddddfd3fd+d, Zd4d1d2Z Z S)5r!as Ensemble evaluation for multiple models, inherits from evaluator and Workflow. It accepts a list of models for inference and outputs a list of predictions for further operations. Args: device: an object representing the device on which to run. val_data_loader: Ignite engine use data_loader to run, must be Iterable, typically be torch.DataLoader. epoch_length: number of iterations for one epoch, default to `len(val_data_loader)`. networks: networks to evaluate in order in the evaluator, should be regular PyTorch `torch.nn.Module`. pred_keys: the keys to store every prediction data. the length must exactly match the number of networks. if None, use "pred_{index}" as key corresponding to N networks, index from `0` to `N-1`. non_blocking: if True and this copy is between CPU and GPU, the copy may occur asynchronously with respect to the host. For other cases, this argument has no effect. prepare_batch: function to parse expected data (usually `image`, `label` and other network args) from `engine.state.batch` for every iteration, for more details please refer to: https://pytorch.org/ignite/generated/ignite.engine.create_supervised_trainer.html. iteration_update: the callable function for every iteration, expect to accept `engine` and `engine.state.batch` as inputs, return data will be stored in `engine.state.output`. if not provided, use `self._iteration()` instead. for more details please refer to: https://pytorch.org/ignite/generated/ignite.engine.engine.Engine.html. inferer: inference method that execute model forward on input data, like: SlidingWindow, etc. postprocessing: execute additional transformation for the model output data. Typically, several Tensor based transforms composed by `Compose`. key_val_metric: compute metric when every iteration completed, and save average value to engine.state.metrics when epoch completed. key_val_metric is the main metric to compare and save the checkpoint into files. additional_metrics: more Ignite metrics that also attach to Ignite Engine. metric_cmp_fn: function to compare current key metric with previous best key metric value, it must accept 2 args (current_metric, previous_best) and return a bool result: if `True`, will update `best_metric` and `best_metric_epoch` with current metric and epoch, default to `greater than`. val_handlers: every handler is a set of Ignite Event-Handlers, must have `attach` function, like: CheckpointHandler, StatsHandler, etc. amp: whether to enable auto-mixed-precision evaluation, default is False. mode: model forward mode during evaluation, should be 'eval' or 'train', which maps to `model.eval()` or `model.train()`, default to 'eval'. event_names: additional custom ignite events that will register to the engine. new events can be a list of str or `ignite.engine.events.EventEnum`. event_to_attr: a dictionary to map an event to a state attribute, then add to `engine.state`. for more details, check: https://pytorch.org/ignite/generated/ignite.engine.engine.Engine.html #ignite.engine.engine.Engine.register_events. decollate: whether to decollate the batch-first data to a list of data after model computation, recommend `decollate=True` when `postprocessing` uses components from `monai.transforms`. default to `True`. to_kwargs: dict of other args for `prepare_batch` API when converting the input data, except for `device`, `non_blocking`. amp_kwargs: dict of the args for `torch.autocast("cuda")` API, for more details: https://pytorch.org/docs/stable/amp.html#torch.autocast. NFTr"rgr$r%networksSequence[torch.nn.Module] pred_keysKeysCollection | Noner&r'r(r)r*rr+r,rjrkr-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@cstjdid|d|d|d|d|d|d| d| d | d | d |d |d |d|d|d|d|d|t||_|durRddtt|jDnt||_t|jt|jkretd| durot|_ dS| |_ dS)Nr"r$r&r(r*r+r-r/r1r2r3r5r6r8r:r<r=r>cSsg|] }tjd|qS)_)r~r).0irFrFrO sz.EnsembleEvaluator.__init__..z?length of `pred_keys` must be same as the length of `networks`.rF) rGrHrrrangeryrrKrrj)rLr"r$rrr&r(r*r+rjr-r/r1r2r3r5r6r8r:r<r=r>rMrFrOrHsV       &zEnsembleEvaluator.__init__rorprqrrc Cs||durtd|j||jj|jfi|j}t|dkr&|\}}d}i}n|\}}}}tj|tj |i|j_ t |j D]r\}} | | a|jrtjdi|j&t|jj trp|jj |j||j|| g|Ri|iWdn1szwYnt|jj tr|jj |j||j|| g|Ri|iWdn1swYq;|tj|tj|jj S)a# callback function for the Supervised Evaluation processing logic of 1 iteration in Ignite Engine. Return below items in a dictionary: - IMAGE: image Tensor data for model input, already moved to device. - LABEL: label Tensor data corresponding to the image, already moved to device. - pred_keys[0]: prediction result of network 0. - pred_keys[1]: prediction result of network 1. - ... ... - pred_keys[N]: prediction result of network N. Args: engine: `EnsembleEvaluator` to execute operation for an iteration. batchdata: input data for this iteration, usually can be dictionary or tuple of Tensor data. Raises: ValueError: When ``batchdata`` is None. NrsrtrFrurx)rKr*rSr"r(r=ryr~rrr enumeraterr6r5rnrr>rzrrupdaterrjrr rr) rLrorprrrrridxrhrFrFrOrs<   "" zEnsembleEvaluator._iteration),r"rgr$r%rrrrr&r'r(r)r*rr+r,rjrkr-r.r/r0r1r0r2rr3r4r5r)r6r7r8r9r:r;r<r)r=r;r>r;r?r@)ror!rprqr?rrrrFrFrMrOr!^s,85r!)7 __future__rr{collections.abcrrtypingrrrrntorch.utils.datar monai.configr monai.datar Zmonai.engines.utilsr r r Zmonai.engines.workflowrZmonai.inferersrrmonai.networks.utilsrrmonai.transformsr monai.utilsrrrrrmonai.utils.enumsrr~rrWmonai.utils.moduler ignite.enginerrZignite.metricsrOPT_IMPORT_VERSIONr__all__rr r!rFrFrFrOs:         1