Semantic
Segmentation
Assign a class label to every pixel in an image. Semantic segmentation transforms an image into a detailed pixel-by-pixel map of the visual world.
What Is Semantic Segmentation?
Instead of predicting one label for the whole image or drawing boxes around objects, semantic segmentation classifies every individual pixel.
For example, in a street image, the model may classify every pixel as road, car, person, building, sky, or another category.
Pixel-Level Classification
Consider an image with a resolution of 640 × 480 pixels. That image contains 307,200 individual pixel locations. A semantic segmentation model predicts a class for each one.
Input Image
640 × 480 pixels
↓
Model Prediction
↓
Segmentation Map
640 × 480 labels
Pixel (100, 200) → Road
Pixel (250, 180) → Car
Pixel (400, 100) → Sky
Pixel (500, 350) → BuildingThe Segmentation Map
The output is usually represented as a mask or segmentation map. Every pixel contains a class ID.
Pixel 1
Class: Background
Pixel 2
Class: Road
Pixel 3
Class: Person
The final result is not simply a list of objects. It is a complete spatial interpretation of the image.
What Does “Semantic” Mean?
The word semantic refers to the meaning of the pixels. The model asks: What category does this pixel belong to?
All pixels belonging to the same semantic category receive the same class label, even if they belong to different physical objects.
How CNNs Learn Pixel-Level Features
Semantic segmentation is challenging because the model needs both high-level understanding and fine spatial detail. CNNs extract features at multiple levels of abstraction.
Low-Level Features
Edges, colors, gradients, and local textures.
Mid-Level Features
Shapes, patterns, and meaningful visual structures.
High-Level Features
Semantic concepts such as road, tumor, leaf, person, or building.
The main challenge is that deep layers understand what is present but may lose exact spatial details. Segmentation architectures solve this by combining deep semantic features with higher-resolution spatial features.
How a Semantic Segmentation Model Works
A common architecture uses an encoder to understand the image and a decoder to reconstruct a detailed pixel-level prediction.
Encoder
Extracts increasingly meaningful visual features while reducing spatial resolution.
Bottleneck
Contains a compact high-level representation of the image.
Decoder
Upsamples the features back to the original image resolution and predicts a class for each pixel.
Skip Connections
Many architectures use skip connections to bring high-resolution details from earlier layers into the decoder.
Input ↓ Encoder ───────────────┐ ↓ │ Deep Features │ Skip Connection ↓ │ Decoder ←──────────────┘ ↓ Pixel-Level Prediction
This helps the model preserve important boundaries and fine details.
How Semantic Segmentation Is Trained
Training requires a ground-truth mask. The annotation provides the correct class for every pixel in the image.
Ground Truth:
Pixel 1 → Road
Pixel 2 → Road
Pixel 3 → Car
Pixel 4 → Sky
Model Prediction:
Pixel 1 → Road ✓
Pixel 2 → Car ✗
Pixel 3 → Car ✓
Pixel 4 → Sky ✓
↓
Calculate Segmentation Loss
↓
BackpropagationThe model improves by learning to reduce the number of incorrectly classified pixels.
Semantic vs Instance Segmentation
| Concept | Semantic Segmentation | Instance Segmentation |
|---|---|---|
| Prediction | Class for every pixel | Mask for every object instance |
| Two Cars | Both are “car” pixels | Car 1 and Car 2 have separate masks |
| Primary Question | What class does this pixel belong to? | Which pixels belong to this specific object? |
| Object Identity | Not separated | Each instance is separated |
Three Cars in an Image: Semantic Segmentation: [Car] [Car] [Car] All pixels → Class: Car Instance Segmentation: Car 1 → Mask 1 Car 2 → Mask 2 Car 3 → Mask 3
Evaluating Semantic Segmentation
Because every pixel is a prediction, segmentation models require metrics that measure pixel-level accuracy and overlap.
Pixel Accuracy
The percentage of pixels classified correctly.
IoU
Measures the overlap between predicted and ground-truth regions for a class.
mIoU
Mean Intersection over Union averaged across all classes.
Dice Score
Measures similarity between predicted and ground-truth segmentation regions.
Precision
Measures how many pixels predicted as a class are actually that class.
Recall
Measures how many pixels belonging to a class were successfully found.
For One Class:
Prediction ∩ Ground Truth
IoU = ─────────────────────────────────────────
Prediction ∪ Ground Truth
mIoU = Average IoU Across All ClassesSemantic Segmentation in Vyrion Vision
Turn Every Pixel Into Meaning.
Vyrion Vision lets you work with segmentation models on real images and datasets. Run models, visualize pixel-level predictions, inspect class regions, and compare how different models segment the same image.
Upload and test a trained semantic segmentation model on new images. Visualize the predicted segmentation map and understand exactly how the model classifies each region.
See where the model is accurate, where boundaries are incorrect, and which classes are confused with each other.
Run multiple segmentation models on the same input and compare their masks, class regions, accuracy, and inference performance.
Input Image
│
├──→ Model A
│ └──→ Pixel-Level Map
│
└──→ Model B
└──→ Pixel-Level Map
Compare:
• Pixel Accuracy
• Class Boundaries
• mIoU
• Confused Regions
• Inference SpeedKey Takeaways
Every Pixel
Semantic segmentation assigns a class label to every pixel in an image.
Semantic Meaning
The model learns what each region of the image represents.
Encoder-Decoder
Deep features provide understanding while decoding restores spatial detail.
Class-Based
Objects from the same class are grouped together rather than separated individually.
mIoU
Mean Intersection over Union is one of the most important segmentation metrics.
Vyrion Vision
Run, visualize, test, and compare semantic segmentation models on your own data.