o  ig@s\dZddlmZddlZddlZddlmZddlm Z ddl Z GdddZ dddZ dS)z This script is almost same with https://github.com/MIC-DKFZ/nnDetection/blob/main/nndet/evaluator/detection/coco.py The changes include 1) code reformatting, 2) docstrings. ) annotationsN)Sequence)Anyc@seZdZ     d:d;ddZdddZd?d!d"Zd@d&d'Zd@d(d)Z e * * +dAdBd4d5Z e * * +dAdCd6d7Z dDd8d9Z d*S)E COCOMetric皙??g?rrg?dTclasses Sequence[str]iou_listSequence[float] iou_range max_detection Sequence[int] per_classboolverbosec Cs||_||_||_t|}tj|d|dtt|d|d|dddd}t|||_ ||_ t |ddtj f|j tj kd|_ t |ddtj f|j tj kd|_|j |j |krt|j |j|ksxtdtjdd ttd ddd|_||_dS) a Class to compute COCO metrics Metrics computed includes, - mAP over the IoU range specified by `iou_range` at last value of `max_detection` - AP values at IoU thresholds specified by `iou_list` at last value of `max_detection` - AR over max detections thresholds defined by `max_detection` (over iou range) Args: classes (Sequence[str]): name of each class (index needs to correspond to predicted class indices!) iou_list (Sequence[float]): specific thresholds where ap is evaluated and saved iou_range (Sequence[float]): (start, stop, step) for mAP iou thresholds max_detection (Sequence[int]): maximum number of detections per image verbose (bool): log time needed for evaluation Example: .. code-block:: python from monai.data.box_utils import box_iou from monai.apps.detection.metrics.coco import COCOMetric from monai.apps.detection.metrics.matching import matching_batch # 3D example outputs of one image from detector val_outputs_all = [ {"boxes": torch.tensor([[1,1,1,3,4,5]],dtype=torch.float16), "labels": torch.randint(3,(1,)), "scores": torch.randn((1,)).absolute()}, ] val_targets_all = [ {"boxes": torch.tensor([[1,1,1,2,6,4]],dtype=torch.float16), "labels": torch.randint(3,(1,))}, ] coco_metric = COCOMetric( classes=['c0','c1','c2'], iou_list=[0.1], max_detection=[10] ) results_metric = matching_batch( iou_fn=box_iou, iou_thresholds=coco_metric.iou_thresholds, pred_boxes=[val_data_i["boxes"].numpy() for val_data_i in val_outputs_all], pred_classes=[val_data_i["labels"].numpy() for val_data_i in val_outputs_all], pred_scores=[val_data_i["scores"].numpy() for val_data_i in val_outputs_all], gt_boxes=[val_data_i["boxes"].numpy() for val_data_i in val_targets_all], gt_classes=[val_data_i["labels"].numpy() for val_data_i in val_targets_all], ) val_metric_dict = coco_metric(results_metric) print(val_metric_dict) rr T)endpointNzxRequire self.iou_thresholds[self.iou_list_idx] == iou_list_np and self.iou_thresholds[self.iou_range_idx] == _iou_range.g?gY@)rrrnparraylinspaceintroundunion1diou_thresholdsrnonzeronewaxis iou_list_idx iou_range_idxall ValueErrorrecall_thresholdsmax_detections) selfrrrrrrZ iou_list_npZ _iou_ranger+c/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/apps/detection/metrics/coco.py__init__Ls(9 2**" zCOCOMetric.__init__argsrkwargsreturn5tuple[dict[str, float], dict[str, np.ndarray] | None]cOs|j|i|S)a Compute metric. See :func:`compute` for more information. Args: *args: positional arguments passed to :func:`compute` **kwargs: keyword arguments passed to :func:`compute` Returns: dict[str, float]: dictionary with scalar values for evaluation dict[str, np.ndarray]: dictionary with arrays, e.g. for visualization of graphs )compute)r*r.r/r+r+r,__call__s zCOCOMetric.__call__ np.ndarrayNonecGsHt|}|D]}|jd|kr!td|jdd|dqdS)z Check if shape of input in first dimension is consistent with expected IoU values (assumes IoU dimension is the first dimension) Args: args: array like inputs with shape function rzIRequire arg.shape[0] == len(self.get_iou_thresholds()). Got arg.shape[0]=z, self.get_iou_thresholds()=.N)lenget_iou_thresholdsshaper')r*r.Znum_iousargr+r+r,check_number_of_ious zCOCOMetric.check_number_of_ioucCs t|jS)z Return IoU thresholds needed for this metric in an numpy array Returns: Sequence[float]: IoU thresholds [M], M is the number of thresholds )listr!)r*r+r+r,r8s zCOCOMetric.get_iou_thresholds results_list&list[dict[int, dict[str, np.ndarray]]]tuple[dict[str, float], None]cCs|jr tdt}|j|d}|jr%t}td||ddi}|||||||jrJt}td||dd|dfS) a Compute COCO metrics Args: results_list (list[dict[int, dict[str, np.ndarray]]]): list with results per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored Returns: dict[str, float], dictionary with coco metrics z Start COCO metric computation...)r=z(Statistics for COCO metrics finished (t=z0.2fzs).zCOCO metrics computed in t=zs.N)rloggerinfotime_compute_statisticsupdate _compute_ap _compute_ar)r*r=ticdataset_statisticstocresultsr+r+r,r2s  zCOCOMetric.computerHdict[str, np.ndarray | list]dict[str, float]c Csbi}|jred|jddd|jddd|jddd|jd}|j||jdd ||<|jret|jD]/\}}|d |jddd|jddd|jddd|jd }|j||j|dd ||<q5|jD]F}d |j|dd|jd}|j||gdd ||<|jrt|jD]!\}}|d |j|dd|jd}|j||g|dd ||<qqh|S)a Compute AP metrics Args: dataset_statistics (list[dict[int, dict[str, np.ndarray]]]): list with result s per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored ZmAP_IoU_r.2f_r r_MaxDet_iou_idx max_det_idxZ _mAP_IoU_rRcls_idxrSZAP_IoU_Z_AP_IoU_) rr) _select_apr%r enumeraterr$r!)r*rHrJkeyrUcls_stridxr+r+r,rEs>,     "zCOCOMetric._compute_apc CsXi}t|jD]Z\}}d|jddd|jddd|jddd|}|j||d||<|jrat|jD]*\}}|d |jddd|jddd|jddd| }|j|||d ||<q6q|jD]D}d |j|dd|jd }|j||d d ||<|jrt|jD] \}}|d|j|dd|jd }|j|||d d||<qqe|S)a Compute AR metrics Args: dataset_statistics (list[dict[int, dict[str, np.ndarray]]]): list with result s per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored ZmAR_IoU_rrMrNr rrO)rSZ _mAR_IoU_)rUrSZAR_IoU_rPrQZ_AR_IoU_rT)rWr)r _select_arrrr$r!) r*rHrJrSmax_detrXrUrYrZr+r+r,rFs64    "zCOCOMetric._compute_arNrPdictrR#int | list[int] | np.ndarray | NonerUint | Sequence[int] | NonerSrfloatcCsL|d}|dur ||}|dur|d|ddf}|d|f}tt|S)a Compute average precision Args: dataset_statistics (dict): computed statistics over dataset - `counts`: Number of thresholds, Number recall thresholds, Number of classes, Number of max detection thresholds - `recall`: Computed recall values [num_iou_th, num_classes, num_max_detections] - `precision`: Precision values at specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] - `scores`: Scores corresponding to specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] iou_idx: index of IoU values to select for evaluation(if None, all values are used) cls_idx: class indices to select, if None all classes will be selected max_det_idx (int): index to select max detection threshold from data Returns: np.ndarray: AP value precisionN.)r`rmean)rHrRrUrSprecr+r+r,rVHs zCOCOMetric._select_apcCsl|d}|dur ||}|dur|d|ddf}|d|f}t||dkdkr+dStt||dkS)a Compute average recall Args: dataset_statistics (dict): computed statistics over dataset - `counts`: Number of thresholds, Number recall thresholds, Number of classes, Number of max detection thresholds - `recall`: Computed recall values [num_iou_th, num_classes, num_max_detections] - `precision`: Precision values at specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] - `scores`: Scores corresponding to specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] iou_idx: index of IoU values to select for evaluation(if None, all values are used) cls_idx: class indices to select, if None all classes will be selected max_det_idx (int): index to select max detection threshold from data Returns: np.ndarray: recall value recallN.rPrg)r7r`rrb)rHrRrUrSrecr+r+r,r[ks zCOCOMetric._select_arc svt|j}t|j}t|j}t|j}t||||f }t|||f }t||||f }t|jD]\} t|jD]\} fdd|D} t| dkr[t d| q?t fdd| D} tj | dd} | | }tj fdd| Dd d d d | f}tj fd d| Dd d d d | f}| ||t d d| D}t t|dk}|dkrt d| q?t|t|}tt|t|}tj|d d jtjd}tj|d d jtjd}tt||D]8\}\}}t|t|}}t||||j|\}}}|||| f<|||d d | f<|||d d | f<qq?q6||||g|||dS)a Compute statistics needed for COCO metrics (mAP, AP of individual classes, mAP@IoU_Thresholds, AR) Adapted from https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py Args: results_list (list[dict[int, dict[str, np.ndarray]]]): list with result s per image (in list) per category (dict). Inner dict contains multiple results obtained by :func:`box_matching_batch`. - `dtMatches`: matched detections [T, D], where T = number of thresholds, D = number of detections - `gtMatches`: matched ground truth boxes [T, G], where T = number of thresholds, G = number of ground truth - `dtScores`: prediction scores [D] detection scores - `gtIgnore`: ground truth boxes which should be ignored [G] indicate whether ground truth should be ignored - `dtIgnore`: detections which should be ignored [T, D], indicate which detections should be ignored Returns: dict: computed statistics over dataset - `counts`: Number of thresholds, Number recall thresholds, Number of classes, Number of max detection thresholds - `recall`: Computed recall values [num_iou_th, num_classes, num_max_detections] - `precision`: Precision values at specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] - `scores`: Scores corresponding to specified recall thresholds [num_iou_th, num_recall_th, num_classes, num_max_detections] csg|] }|vr|qSr+r+.0r)rUr+r, z2COCOMetric._compute_statistics..rz4WARNING, no results found for coco metric for class csg|] }|ddqS)ZdtScoresrr+rfr\r+r,rirj mergesort)kindc$g|]}|ddddfqS)Z dtMatchesNrr+rfrkr+r,ri$r )axisNcrn)ZdtIgnoreNrr+rfrkr+r,rirocSsg|]}|dqS)ZgtIgnorer+rfr+r+r,risz/WARNING, no gt found for coco metric for class )dtype)countsrdrascores)r7r!r(rr)ronesrWr@warning concatenateargsortr;r count_nonzero logical_and logical_notcumsumastypefloat32zipr_compute_stats_single_threshold)r*r=Z num_iou_th num_recall_th num_classesZnum_max_detectionsrardrsZcls_irSrJZ dt_scoresindsdt_scores_sortedZ dt_matchesZ dt_ignoresZ gt_ignorenum_gtZtpsfpstp_sumZfp_sumZth_indtpfprhpsr+)rUr\r,rCsR     ((  'zCOCOMetric._compute_statistics)rr r TT) rrrrrrrrrrrr)r.rr/rr0r1)r.r4r0r5)r0r)r=r>r0r?)rHrKr0rL)NNrP) rHr]rRr^rUr_rSrr0r`) rHr]rRr_rUr_rSrr0r`)r=r>r0rK)__name__ __module__ __qualname__r-r3r;r8r2rErF staticmethodrVr[rCr+r+r+r,rJs0 T   ( /*"&rrr4rrr(np.ndarray | Sequence[float]rrr0$tuple[float, np.ndarray, np.ndarray]cCst|}||}|||td}t|r|d}nd}dg|} t|f} |}tt|dddD]} || || dkrK|| || d<q7tj||dd} zt| D]\} }||| | <||| | <qYWn tytYnw|t | t | fS)a Compute recall value, precision curve and scores thresholds Adapted from https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py Args: tp (np.ndarray): cumsum over true positives [R], R is the number of detections fp (np.ndarray): cumsum over false positives [R], R is the number of detections dt_scores_sorted (np.ndarray): sorted (descending) scores [R], R is the number of detections recall_thresholds (Sequence[float]): recall thresholds which should be evaluated num_gt (int): number of ground truth bounding boxes (excluding boxes which are ignored) Returns: - float, overall recall for given IoU value - np.ndarray, precision values at defined recall values [RTH], where RTH is the number of recall thresholds - np.ndarray, prediction scores corresponding to recall values [RTH], where RTH is the number of recall thresholds r rPrrleft)side) r7rspacingzerostolistrange searchsortedrW BaseExceptionr)rrrr(rrrcprrdraZ th_scoresirZsave_idxZ array_indexr+r+r,rs.     r) rr4rr4rr4r(rrrr0r)__doc__ __future__rloggingr@rBcollections.abcrtypingrnumpyrrrr+r+r+r,s:   !