How do you tell a diffusion model exactly WHERE something should be in the image, or WHAT specific spatial structure to follow — not just what’s in the prompt? Standard text-to-image only gives you coarse control through prompts.
ControlNet is an additional neural network that runs in parallel with the diffusion model. It takes an extra input (edge map, depth map, pose skeleton, or glyph image) and adds spatial constraints to the generation process. The model must follow your control signal while still generating based on the text prompt.
ControlNet copies the UNet/DiT encoder layers and trains them separately on the control task. The trained ControlNet weights connect to the main model via “zero convolutions” that gradually merge the control signal.
- Canny Edge: Follow edges from an image
- Depth: Follow depth map structure
- Pose: Follow human pose skeleton
- Segmentation: Follow object layout
- Glyph: Follow exact text stroke positions (for text rendering)
When we render “Made by Lakme” as a black-on-white image and pass it to ControlNet, the model receives pixel-level spatial constraints: “there must be dark strokes at these coordinates.” It doesn’t need to understand letters — it just follows the stroke pattern.
- Pre-trained weights available for all control types
- Zero training required for glyph control
- controlnet_scale (0-1) controls strength
- Works with Flux, SDXL, other diffusers
- Built from: diffusion-models, vae
- Used in: glyph-injection, hybrid-pipeline
- Related to: flux-architecture
- Thin strokes = weak signal = drift
- Scale too high = looks pasted on
- Scale too low = doesn’t follow constraints
- Additional VRAM needed