U Phg@sjdZddlmZddlZddlZddlmZddlm Z ddl Z GdddZ dddd d d d d dZ dS)z This script is almost same with https://github.com/MIC-DKFZ/nnDetection/blob/main/nndet/evaluator/detection/coco.py The changes include 1) code reformatting, 2) docstrings. ) annotationsN)Sequence)Anyc@seZdZd3ddddddd d d Zd d d dddZdddddZddddZdddddZddd d!d"Zddd d#d$Z e d4d'd(d)d*d+d,d-d.Z e d5d'd)d)d*d+d,d/d0Z dddd1d2Z d%S)6 COCOMetric皙??g?rrg?dTz Sequence[str]zSequence[float]z Sequence[int]bool)classesiou_list iou_range max_detection per_classverbosec Cs||_||_||_t|}tj|d|dtt|d|d|dddd}t|||_ ||_ t |ddtj f|j tj kd|_ t |ddtj f|j tj kd|_|j |j |kr|j |j|kstdtjdd ttd ddd|_||_dS) a Class to compute COCO metrics Metrics computed includes, - mAP over the IoU range specified by `iou_range` at last value of `max_detection` - AP values at IoU thresholds specified by `iou_list` at last value of `max_detection` - AR over max detections thresholds defined by `max_detection` (over iou range) Args: classes (Sequence[str]): name of each class (index needs to correspond to predicted class indices!) iou_list (Sequence[float]): specific thresholds where ap is evaluated and saved iou_range (Sequence[float]): (start, stop, step) for mAP iou thresholds max_detection (Sequence[int]): maximum number of detections per image verbose (bool): log time needed for evaluation Example: .. code-block:: python from monai.data.box_utils import box_iou from monai.apps.detection.metrics.coco import COCOMetric from monai.apps.detection.metrics.matching import matching_batch # 3D example outputs of one image from detector val_outputs_all = [ {"boxes": torch.tensor([[1,1,1,3,4,5]],dtype=torch.float16), "labels": torch.randint(3,(1,)), "scores": torch.randn((1,)).absolute()}, ] val_targets_all = [ {"boxes": torch.tensor([[1,1,1,2,6,4]],dtype=torch.float16), "labels": torch.randint(3,(1,))}, ] coco_metric = COCOMetric( classes=['c0','c1','c2'], iou_list=[0.1], max_detection=[10] ) results_metric = matching_batch( iou_fn=box_iou, iou_thresholds=coco_metric.iou_thresholds, pred_boxes=[val_data_i["boxes"].numpy() for val_data_i in val_outputs_all], pred_classes=[val_data_i["labels"].numpy() for val_data_i in val_outputs_all], pred_scores=[val_data_i["scores"].numpy() for val_data_i in val_outputs_all], gt_boxes=[val_data_i["boxes"].numpy() for val_data_i in val_targets_all], gt_classes=[val_data_i["labels"].numpy() for val_data_i in val_targets_all], ) val_metric_dict = coco_metric(results_metric) print(val_metric_dict) rr T)endpointNzxRequire self.iou_thresholds[self.iou_list_idx] == iou_list_np and self.iou_thresholds[self.iou_range_idx] == _iou_range.gg?gY@)rrrnparraylinspaceintroundunion1diou_thresholdsrnonzeronewaxis iou_list_idx iou_range_idxall ValueErrorrecall_thresholdsmax_detections) selfrrrrrrZ iou_list_npZ _iou_ranger'V/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/apps/detection/metrics/coco.py__init__Ls.9 $**"zCOCOMetric.__init__rz5tuple[dict[str, float], dict[str, np.ndarray] | None])argskwargsreturncOs |j||S)a Compute metric. See :func:`compute` for more information. Args: *args: positional arguments passed to :func:`compute` **kwargs: keyword arguments passed to :func:`compute` Returns: dict[str, float]: dictionary with scalar values for evaluation dict[str, np.ndarray]: dictionary with arrays, e.g. for visualization of graphs )compute)r&r*r+r'r'r(__call__s zCOCOMetric.__call__ np.ndarrayNone)r*r,cGsHt|}|D]2}|jd|krtd|jdd|dqdS)z Check if shape of input in first dimension is consistent with expected IoU values (assumes IoU dimension is the first dimension) Args: args: array like inputs with shape function rzIRequire arg.shape[0] == len(self.get_iou_thresholds()). Got arg.shape[0]=z, self.get_iou_thresholds()=.N)lenget_iou_thresholdsshaper#)r&r*Znum_iousargr'r'r(check_number_of_ious  zCOCOMetric.check_number_of_iou)r,cCs t|jS)z Return IoU thresholds needed for this metric in an numpy array Returns: Sequence[float]: IoU thresholds [M], M is the number of thresholds )listr)r&r'r'r(r3szCOCOMetric.get_iou_thresholdsz&list[dict[int, dict[str, np.ndarray]]]ztuple[dict[str, float], None]) results_listr,cCs|jrtdt}|j|d}|jrJt}td||ddi}|||||||jrt}td||dd|dfS) a Compute COCO metrics Args: results_list (list[dict[int, dict[str, np.ndarray]]]): list with results per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored Returns: dict[str, float], dictionary with coco metrics z Start COCO metric computation...)r8z(Statistics for COCO metrics finished (t=z0.2fzs).zCOCO metrics computed in t=zs.N)rloggerinfotime_compute_statisticsupdate _compute_ap _compute_ar)r&r8ticdataset_statisticstocresultsr'r'r(r-s  zCOCOMetric.computezdict[str, np.ndarray | list]zdict[str, float])rAr,c Csdi}|jrd|jddd|jddd|jddd|jd}|j||jdd ||<|jrt|jD]^\}}|d |jddd|jddd|jddd|jd }|j||j|dd ||<qj|jD]}d |j|dd|jd}|j||gdd ||<|jrt|jD]D\}}|d |j|dd|jd}|j||g|dd ||<qq|S)a Compute AP metrics Args: dataset_statistics (list[dict[int, dict[str, np.ndarray]]]): list with result s per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored ZmAP_IoU_r.2f_r r_MaxDet_iou_idx max_det_idxZ _mAP_IoU_rIcls_idxrJZAP_IoU_Z_AP_IoU_) rr% _select_apr!r enumeraterr r)r&rArCkeyrLcls_stridxr'r'r(r>s.8<  "zCOCOMetric._compute_apc CsZi}t|jD]\}}d|jddd|jddd|jddd|}|j||d||<|jrt|jD]T\}}|d |jddd|jddd|jddd| }|j|||d ||<qlq|jD]}d |j|dd|jd }|j||d d ||<|jrt|jD]B\}}|d|j|dd|jd }|j|||d d||<qq|S)a Compute AR metrics Args: dataset_statistics (list[dict[int, dict[str, np.ndarray]]]): list with result s per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored ZmAR_IoU_rrDrEr rrF)rJZ _mAR_IoU_)rLrJZAR_IoU_rGrHZ_AR_IoU_rK)rNr%r _select_arrrr r) r&rArCrJmax_detrOrLrPrQr'r'r(r?s"46 "zCOCOMetric._compute_arNrGdictz#int | list[int] | np.ndarray | Nonezint | Sequence[int] | Nonerfloat)rArIrLrJr,cCsL|d}|dk r||}|dk r2|d|ddf}|d|f}tt|S)a Compute average precision Args: dataset_statistics (dict): computed statistics over dataset - `counts`: Number of thresholds, Number recall thresholds, Number of classes, Number of max detection thresholds - `recall`: Computed recall values [num_iou_th, num_classes, num_max_detections] - `precision`: Precision values at specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] - `scores`: Scores corresponding to specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] iou_idx: index of IoU values to select for evaluation(if None, all values are used) cls_idx: class indices to select, if None all classes will be selected max_det_idx (int): index to select max detection threshold from data Returns: np.ndarray: AP value precisionN.)rUrmean)rArIrLrJprecr'r'r(rMHs zCOCOMetric._select_apcCsl|d}|dk r||}|dk r2|d|ddf}|d|f}t||dkdkrVdStt||dkS)a Compute average recall Args: dataset_statistics (dict): computed statistics over dataset - `counts`: Number of thresholds, Number recall thresholds, Number of classes, Number of max detection thresholds - `recall`: Computed recall values [num_iou_th, num_classes, num_max_detections] - `precision`: Precision values at specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] - `scores`: Scores corresponding to specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] iou_idx: index of IoU values to select for evaluation(if None, all values are used) cls_idx: class indices to select, if None all classes will be selected max_det_idx (int): index to select max detection threshold from data Returns: np.ndarray: recall value recallN.rGrg)r2rUrrW)rArIrLrJrecr'r'r(rRks zCOCOMetric._select_arc szt|j}t|j}t|j}t|j}t||||f }t|||f }t||||f }t|jD]\} t|jD]\} fdd|D} t| dkrt d| qt fdd| D} tj | dd} | | }tj fdd| Dd d d d | f}tj fd d| Dd d d d | f}| ||t d d| D}t |dk}|dkrt d| qt|t|}tt|t|}tj|d d jtjd}tj|d d jtjd}tt||D]r\}\}}t|t|}}t||||j|\}}}|||| f<|||d d | f<|||d d | f<qqql||||g|||dS)a Compute statistics needed for COCO metrics (mAP, AP of individual classes, mAP@IoU_Thresholds, AR) Adapted from https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py Args: results_list (list[dict[int, dict[str, np.ndarray]]]): list with result s per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored Returns: dict: computed statistics over dataset - `counts`: Number of thresholds, Number recall thresholds, Number of classes, Number of max detection thresholds - `recall`: Computed recall values [num_iou_th, num_classes, num_max_detections] - `precision`: Precision values at specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] - `scores`: Scores corresponding to specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] csg|]}|kr|qSr'r'.0r)rLr'r( sz2COCOMetric._compute_statistics..rz4WARNING, no results found for coco metric for class csg|]}|ddqS)ZdtScoresrr'r[rSr'r(r^s mergesort)kindcs$g|]}|ddddfqS)Z dtMatchesNrr'r[r_r'r(r^sr )axisNcs$g|]}|ddddfqS)ZdtIgnoreNrr'r[r_r'r(r^scSsg|] }|dqS)ZgtIgnorer'r[r'r'r(r^sz/WARNING, no gt found for coco metric for class )dtype)countsrYrVscores)r2rr$rr%ronesrNr9warning concatenateargsortr6 count_nonzero logical_and logical_notcumsumastypefloat32zipr_compute_stats_single_threshold)r&r8Z num_iou_th num_recall_th num_classesZnum_max_detectionsrVrYreZcls_irJrCZ dt_scoresindsdt_scores_sortedZ dt_matchesZ dt_ignoresZ gt_ignorenum_gtZtpsfpstp_sumZfp_sumZth_indtpfpr]psr')rLrSr(r<sN     ((   zCOCOMetric._compute_statistics)rr r TT)NNrG)NNrG)__name__ __module__ __qualname__r)r.r6r3r-r>r? staticmethodrMrRr<r'r'r'r(rJs.T (/*"&rr/znp.ndarray | Sequence[float]rz$tuple[float, np.ndarray, np.ndarray])ryrzrur$rvr,cCs t|}||}|||td}t|r8|d}nd}t|f} t|f} |}| } tt|dddD](} || || dkrx|| || d<qxtj||dd} z.t| D] \} }||| | <||| | <qWntk rYnX|t | t | fS)a Compute recall value, precision curve and scores thresholds Adapted from https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py Args: tp (np.ndarray): cumsum over true positives [R], R is the number of detections fp (np.ndarray): cumsum over false positives [R], R is the number of detections dt_scores_sorted (np.ndarray): sorted (descending) scores [R], R is the number of detections recall_thresholds (Sequence[float]): recall thresholds which should be evaluated num_gt (int): number of ground truth bounding boxes (excluding boxes which are ignored) Returns: - float, overall recall for given IoU value - np.ndarray, precision values at defined recall values [RTH], where RTH is the number of recall thresholds - np.ndarray, prediction scores corresponding to recall values [RTH], where RTH is the number of recall thresholds r rGrleft)side) r2rspacingzerostolistrange searchsortedrN BaseExceptionr)ryrzrur$rvrrrcprrYrVZ th_scoresirtZsave_idxZ array_indexr'r'r(rqs*    rq)__doc__ __future__rloggingr9r;collections.abcrtypingrnumpyrrrqr'r'r'r(;s   !