Back to Study Material Deep Learning

Object detection

Object detection requires solving two problems simultaneously: identifying what objects are present in an image and determining where each one is located. ...

Object detection

The Core Task: Localization and Classification Together

Object detection requires solving two problems simultaneously: identifying what objects are present in an image and determining where each one is located. The location is typically expressed as a bounding box, a rectangle defined by the coordinates of its top-left corner and its width and height. A detector must predict both the class label and the bounding box for every object of interest in a scene, even when objects partially overlap.

This is more challenging than image classification, which only asks what is in the image without specifying where. A self-driving car system, for example, must not only recognize that a pedestrian is present but also determine the pedestrian's position and size to plan a safe trajectory. The real-time detection speed required by such applications makes object detection one of the most performance-demanding computer vision tasks.

Exam tip: Know the distinction between one-stage and two-stage detectors. Two-stage detectors like Faster R-CNN first generate region proposals and then classify each one, achieving high accuracy at the cost of speed. One-stage detectors like YOLO and SSD skip the proposal step and predict boxes and classes directly from feature maps, trading some accuracy for faster inference.

Anchor Boxes, Bounding Box Regression, and IoU

Most modern detectors use anchor boxes, which are predefined reference boxes of various scales and aspect ratios placed at each spatial location in a feature map. The network learns to predict offsets from these anchors rather than predicting absolute coordinates, which simplifies the regression problem because the anchors already cover likely bounding box shapes. Each anchor is also associated with a classification prediction and an objectness score indicating confidence that an object exists at that location.

Intersection over Union, abbreviated IoU, is the standard metric for evaluating bounding box quality. It is computed as the area of overlap between the predicted box and the ground-truth box divided by the area of their union. An IoU of one indicates a perfect match, and IoU below 0.5 is typically considered a false detection. During training, anchors with IoU above a threshold are treated as positive examples and trained to regress toward the matching ground-truth box.

Understanding IoU is fundamental for interpreting detection benchmarks like COCO mAP, which averages precision across multiple IoU thresholds.

Non-Maximum Suppression and Modern Detectors

Because anchors overlap spatially, a single object often produces multiple positive predictions from neighboring anchors. Non-maximum suppression removes redundant boxes by keeping only the highest-confidence prediction among overlapping boxes. The algorithm iteratively selects the box with the highest score, removes all boxes that overlap with it by more than a chosen IoU threshold, and repeats until no boxes remain.

Modern anchor-free detectors such as FCOS and CenterNet predict object centers and box dimensions without predefined anchors, simplifying training and removing the need to design anchor configurations for each dataset. Transformer-based detectors like DETR replace the handcrafted anchor and NMS pipeline entirely by framing detection as a set prediction problem solved with bipartite matching loss, producing a cleaner and more flexible architecture.

When discussing object detection in an interview, demonstrating awareness of the progression from anchor-based to anchor-free to transformer-based approaches shows an up-to-date understanding of the field's trajectory and the motivations behind each architectural shift.