Image segmentation
Image segmentation assigns a label to every pixel in an image rather than producing a single label for the whole image or a bounding box per object. Semant...
Pixel-Level Classification and Its Applications
Image segmentation assigns a label to every pixel in an image rather than producing a single label for the whole image or a bounding box per object. Semantic segmentation assigns a class label such as road, sky, or pedestrian to each pixel without distinguishing between separate instances of the same class. Instance segmentation goes further by assigning a unique identifier to each individual object instance, so two pedestrians standing next to each other receive different labels even though they share the same class.
Autonomous driving relies on semantic segmentation to understand drivable surfaces, lane markings, and the positions of obstacles at pixel-level precision. Medical imaging uses it to delineate tumor boundaries in MRI scans or measure organ volumes from CT data.
Exam tip: Know the difference between semantic and instance segmentation and be able to name an architecture for each. FCN and DeepLab are common answers for semantic segmentation; Mask R-CNN is the canonical example for instance segmentation. Panoptic segmentation unifies both into a single output, and demonstrating awareness of it signals familiarity with recent progress.
Encoder-Decoder Architectures and Skip Connections
Segmentation networks typically use an encoder-decoder structure. The encoder is a backbone network, often pretrained on ImageNet, that progressively reduces spatial resolution while increasing channel depth, building up abstract representations. The decoder then upsamples these representations back to the original input resolution to produce a per-pixel prediction map.
The challenge is that simple upsampling loses the fine spatial detail needed for precise boundary delineation. Skip connections address this by feeding feature maps from the encoder directly into the corresponding decoder stage, providing high-resolution detail that the decoder can use alongside the semantic context from deeper layers. U-Net popularized this approach for biomedical image segmentation and its architecture has been widely adopted across domains.
In practice, the quality of segmentation boundaries depends heavily on the resolution at which skip connections are merged and the capacity of the decoder heads. Architectures that maintain higher resolution feature maps throughout training at the cost of more memory, such as HRNet, consistently produce sharper segmentation masks than those that downsample aggressively early.
Loss Functions and Evaluation Metrics
Training a segmentation model requires a loss function that operates on per-pixel predictions. Cross-entropy loss computed independently over each pixel is the standard starting point, but class imbalance is a significant problem: in a street scene, the road and sky occupy far more pixels than rare categories like traffic signs or cyclists. Training with uniform cross-entropy causes the network to optimize for common classes at the expense of rare but important ones.
Weighted cross-entropy addresses this by up-weighting loss contributions from underrepresented classes. The Dice loss, derived from the Dice coefficient, directly optimizes the overlap between predicted and ground-truth masks and is less sensitive to class imbalance than cross-entropy. Many practitioners combine both, using a weighted sum that benefits from the properties of each.
The standard evaluation metric is mean Intersection over Union, the average IoU computed across all classes. A well-rounded interview answer on segmentation evaluation should mention that mIoU penalizes boundary errors and that it can mask poor performance on rare classes if the class-wise breakdown is not examined separately.