o  i>@sddlmZddlZddlZddlZddlmZddlmZm Z ddl m Z m Z m Z mZede je d\ZZerGddlmZdd lmZmZnede je d \ZZed e je d \ZZed e je d \ZZGdddZdS)) annotationsN)Mapping) TYPE_CHECKINGAny) IgniteInfo is_scalar min_versionoptional_importz ignite.engineEvents)Engine) Checkpoint DiskSaverr zignite.handlersr r c@s|eZdZdZ              d5d6d d!Zd7d#d$Zd8d'd(Zd)d*Zd8d+d,Zd9d/d0Z d8d1d2Z d8d3d4Z dS):CheckpointSaveraR CheckpointSaver acts as an Ignite handler to save checkpoint data into files. It supports to save according to metrics result, epoch number, iteration number and last model or exception. Args: save_dir: the target directory to save the checkpoints. save_dict: source objects that save to the checkpoint. examples:: {'network': net, 'optimizer': optimizer, 'lr_scheduler': lr_scheduler} name: identifier of logging.logger to use, if None, defaulting to ``engine.logger``. file_prefix: prefix for the filenames to which objects will be saved. save_final: whether to save checkpoint or session at final iteration or exception. If checkpoints are to be saved when an exception is raised, put this handler before `StatsHandler` in the handler list, because the logic with Ignite can only trigger the first attached handler for `EXCEPTION_RAISED` event. final_filename: set a fixed filename to save the final model if `save_final=True`. If None, default to `checkpoint_final_iteration=N.pt`. save_key_metric: whether to save checkpoint or session when the value of key_metric is higher than all the previous values during training.keep 4 decimal places of metric, checkpoint name is: {file_prefix}_key_metric=0.XXXX.pth. key_metric_name: the name of key_metric in ignite metrics dictionary. If None, use `engine.state.key_metric` instead. key_metric_n_saved: save top N checkpoints or sessions, sorted by the value of key metric in descending order. key_metric_filename: set a fixed filename to set the best metric model, if not None, `key_metric_n_saved` should be 1 and only keep the best metric model. key_metric_save_state: whether to save the tracking list of key metric in the checkpoint file. if `True`, then will save an object in the checkpoint file with key `checkpointer` to be consistent with the `include_self` arg of `Checkpoint` in ignite: https://pytorch.org/ignite/v0.4.5/generated/ignite.handlers.checkpoint.Checkpoint.html. typically, it's used to resume training and compare current metric with previous N values. key_metric_greater_or_equal: if `True`, the latest equally scored model is stored. Otherwise, save the first equally scored model. default to `False`. key_metric_negative_sign: whether adding a negative sign to the metric score to compare metrics, because for error-like metrics, smaller is better(objects with larger score are retained). default to `False`. epoch_level: save checkpoint during training for every N epochs or every N iterations. `True` is epoch level, `False` is iteration level. save_interval: save checkpoint every N epochs, default is 0 to save no checkpoint. n_saved: save latest N checkpoints of epoch level or iteration level, 'None' is to save all. Note: CheckpointHandler can be used during training, validation or evaluation. example of saved files: - checkpoint_iteration=400.pt - checkpoint_iteration=800.pt - checkpoint_epoch=1.pt - checkpoint_final_iteration=1000.pt - checkpoint_key_metric=0.9387.pt NFTrsave_dirstr save_dictdictname str | None file_prefix save_finalboolfinal_filenamesave_key_metrickey_metric_namekey_metric_n_savedintkey_metric_filenamekey_metric_save_statekey_metric_greater_or_equalkey_metric_negative_sign epoch_level save_intervaln_saved int | NonereturnNonec sH|durtd|_|durt|dkstd|_t|_|_|_d_ d_ d_ |_ |_ Gdddt}|rWdd d }tj|jj d ||d d_ |rdfdd }| durm| dkrmtdtj|j| d ||d| | | d_ |dkrdfdd }tj|jd||jrdnd|d_ dSdS)Nz/must provide directory to save the checkpoints.rz$must provide source objects to save.csBeZdZdZddfdd Zddfdd Zdfdd ZZS)z,CheckpointSaver.__init__.._DiskSaverzK Enhance the DiskSaver to support fixed filename. Ndirnamerfilenamercstj|ddd||_dS)NF)r)Z require_emptyatomic)super__init__r*)selfr)r* __class__a/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/handlers/checkpoint_saver.pyr-s z5CheckpointSaver.__init__.._DiskSaver.__init__ checkpointrmetadataMapping | Noner'r(cs&|jdur|j}tj|||ddS)N)r3r*r4)r*r,__call__)r.r3r*r4r/r1r2r6s z5CheckpointSaver.__init__.._DiskSaver.__call__cs"|jdur|j}tj|ddS)N)r*)r*r,remove)r.r*r/r1r2r7s z3CheckpointSaver.__init__.._DiskSaver.removeN)r)rr*r)r3rr*rr4r5r'r()r*rr'r()__name__ __module__ __qualname____doc__r-r6r7 __classcell__r1r1r/r2 _DiskSaver{s r>enginer r'rcSs|jjSr8)state iterationr?r1r1r2 _final_funcsz-CheckpointSaver.__init__.._final_func)r)r*Zfinal_iteration)to_save save_handlerfilename_prefixscore_function score_namecszttr}nt|jdr|jj}ntdd|jj|}t|s3t d|d|ddSr9d|Sd|S)Nrz>Incompatible values: save_key_metric=True and key_metric_name=.zkey metric is not a scalar value, skip metric comparison and don't save a model.please use other metrics as key metric, or change the `reduction` mode to 'mean'.got metric: =r) isinstancerhasattrr@r ValueErrormetricsrwarningswarn)r? metric_namemetric)rr"r1r2 _score_funcs$     z-CheckpointSaver.__init__.._score_funcrzSif using fixed filename to save the best metric model, we should only save 1 model. key_metric)rDrErFrGrHr% include_selfZgreater_or_equalcsjr|jjS|jjSr8)r#r@epochrArB)r.r1r2_interval_funcsz0CheckpointSaver.__init__.._interval_func)r)rWrA)rDrErFrGrHr%)r?r r'r)AssertionErrorrlenrlogging getLoggerloggerr#r$_final_checkpoint_key_metric_checkpoint_interval_checkpoint_name_final_filenamer r rN)r.rrrrrrrrrrr r!r"r#r$r%r>rCrTrXr1)rr"r.r2r-Ysb       zCheckpointSaver.__init__ state_dictcCs(|jdur |j|dStddS)a Utility to resume the internal state of key metric tracking list if configured to save checkpoints based on the key metric value. Note to set `key_metric_save_state=True` when saving the previous checkpoint. Example:: CheckpointSaver( ... save_key_metric=True, key_metric_save_state=True, # config to also save the state of this saver ).attach(engine) engine.run(...) # resumed training with a new CheckpointSaver saver = CheckpointSaver(save_key_metric=True, ...) # load the previous key metric tracking list into saver CheckpointLoader("/test/model.pt"), {"checkpointer": saver}).attach(engine) NzFno key metric checkpoint saver to resume the key metric tracking list.)r_load_state_dictrPrQ)r.rcr1r1r2rds zCheckpointSaver.load_state_dictr?r cCs|jdur |j|_|jdur|tj|j|tj|j|j dur+|tj |j |j durO|j rA|tj |jd|jdS|tj|jd|jdSdS)zg Args: engine: Ignite Engine, it can be a trainer, validator or evaluator. N)every)rar]r^add_event_handlerr Z COMPLETED completedZEXCEPTION_RAISEDexception_raisedr_EPOCH_COMPLETEDmetrics_completedr`r#r$interval_completedITERATION_COMPLETEDr.r?r1r1r2attachs    zCheckpointSaver.attachcCsX|jdur(|jj}t|dkr*|d}|jj|j|jd|jdSdSdS)Nrz)Deleted previous saved final checkpoint: ) r^Z_savedrZpoprEr7r*r]info)r.saveditemr1r1r2_delete_previous_final_ckpts   z+CheckpointSaver._delete_previous_final_ckptcCst|js td||||jdurtt|jds#td|jdur2tj |j |j}n|jj }|j d|dS)zCallback for train or validation/evaluation completed Event. Save final checkpoint if configure save_final is True. Args: engine: Ignite Engine, it can be a trainer, validator or evaluator. 0Error: _final_checkpoint function not specified.Nrp.Error, provided logger has not info attribute.z)Train completed, saved final checkpoint: callabler^rYrsr]rMrbospathjoinrZlast_checkpointrp)r.r?_final_checkpoint_pathr1r1r2rgs     zCheckpointSaver.completede ExceptioncCst|js td||||jdurtt|jds#td|jdur2tj |j |j}n|jj }|j d||)aCallback for train or validation/evaluation exception raised Event. Save current data as final checkpoint if configure save_final is True. This callback may be skipped because the logic with Ignite can only trigger the first attached handler for `EXCEPTION_RAISED` event. Args: engine: Ignite Engine, it can be a trainer, validator or evaluator. e: the exception caught in Ignite during engine.run(). rtNrpruz-Exception raised, saved the last checkpoint: rv)r.r?r|r{r1r1r2rhs     z CheckpointSaver.exception_raisedcCs t|js td||dS)zCallback to compare metrics and save models in train or validation when epoch completed. Args: engine: Ignite Engine, it can be a trainer, validator or evaluator. z5Error: _key_metric_checkpoint function not specified.N)rwr_rYrmr1r1r2rj2s z!CheckpointSaver.metrics_completedcCsxt|js td|||jdurtt|jdstd|jr/|jd|jjdS|jd|jj dS)zCallback for train epoch/iteration completed Event. Save checkpoint if configure save_interval = N Args: engine: Ignite Engine, it can be a trainer, validator or evaluator. z3Error: _interval_checkpoint function not specified.NrpruzSaved checkpoint at epoch: zSaved checkpoint at iteration: ) rwr`rYr]rMr#rpr@rWrArmr1r1r2rk<s    z"CheckpointSaver.interval_completed)NrFNFNrNFFFTrN)"rrrrrrrrrrrrrrrrrrrrr rr!rr"rr#rr$rr%r&r'r()rcrr'r()r?r r'r()r?r r|r}r'r() r9r:r;r<r-rdrnrsrgrhrjrkr1r1r1r2r!s0; v     r) __future__rr[rxrPcollections.abcrtypingrr monai.utilsrrrr OPT_IMPORT_VERSIONr _ ignite.enginer Zignite.handlersr r rr1r1r1r2s