A batch processing system that generates training captions for Flux LoRA models from marketing posters and images. It uses a multi-OCR approach combined with LLM-based text cleanup to produce high-quality, structured captions.

Purpose: Create training data for fine-tuning image generation models on specific domains (wedding invitations, festival posters, sale ads, etc.)

| Component | Description |

|-----------|-------------|

| caption_runner.py | Main batch processing script |

| caption_test.py | Interactive testing/debugging tool |

| split_batches.py | Image organization utility |

Models Used:

• Florence-2-base-ft - Visual description + OCR with region detection

• EasyOCR - Multi-pass OCR (English + Hindi)

• PaddleOCR - Additional OCR source (optional)

• Ollama Mistral 7B - LLM-based text cleanup and correction

flowchart TD

    A[Input Images] --> B{Image Analysis}

    B --> C[Preprocessing & Enhancement]

    C --> D[Florence-2 OCR<br/>With Region Detection]

    C --> E[EasyOCR Full Pass<br/>English + Hindi]

    C --> F[PaddleOCR Pass<br/>Optional]

    C --> G[Regional OCR<br/>12x Overlapping Crops]

    D --> H[Merge All Sources]

    E --> H

    F --> H

    G --> H

    H --> I[Fuzzy Deduplication<br/>RapidFuzz Matching]

    I --> J[Fragment Merging<br/>Context-Aware]

    J --> K[Ollama Cleanup<br/>Mistral 7B LLM]

    K --> L[Domain Detection<br/>Pattern Matching]

    L --> M[Style Tag Generation<br/>Auto-Tagging]

    M --> N[Final Caption Output<br/>Flux LoRA Format]

Our system uses four parallel OCR sources to maximize text detection coverage:

| OCR Source | Strengths | Weaknesses | Role in System |

|------------|-----------|------------|----------------|

| Florence-2 | Region-aware, visual context understanding | May miss small text | Primary OCR with spatial info |

| EasyOCR | Fast, supports English + Hindi | Less accurate on complex fonts | Full-image pass for coverage |

| PaddleOCR | High accuracy on dense text | Slower, optional dependency | Additional verification source |

| Regional OCR | Catches tiny text in corners/edges | Computationally expensive | Final safety net for missed text |

The regional OCR processes 12 overlapping regions to catch text that full-image passes miss:

 
regions = [
 
    # 3x3 grid with overlap
 
    (0.0,  0.0,  0.5,  0.38, "top-left"),
 
    (0.25, 0.0,  0.75, 0.38, "top-center"),
 
    (0.5,  0.0,  1.0,  0.38, "top-right"),
 
    (0.0,  0.3,  0.5,  0.7,  "middle-left"),
 
    (0.25, 0.3,  0.75, 0.7,  "center"),
 
    (0.5,  0.3,  1.0,  0.7,  "middle-right"),
 
    (0.0,  0.62, 0.5,  1.0,  "bottom-left"),
 
    (0.25, 0.62, 0.75, 1.0,  "bottom-center"),
 
    (0.5,  0.62, 1.0,  1.0,  "bottom-right"),
 
    # Footer strips — address, phone, website, T&C
 
    (0.0,  0.82, 0.5,  1.0,  "footer-left"),
 
    (0.5,  0.82, 1.0,  1.0,  "footer-right"),
 
    (0.0,  0.88, 1.0,  1.0,  "bottom-strip"),
 
]
 

Each region is 4x upscaled before OCR processing to catch tiny text that would be missed at full resolution.

Before any text is accepted, it passes through multiple validation layers:

flowchart LR

    A[Raw OCR Output] --> B{Length Check<br/>2-100 chars}

    B -->|Fail| Z[Discard]

    B -->|Pass| C{Valid Characters?}

    C -->|Fail| Z

    C -->|Pass| D{Real Words?}

    D -->|Fail| Z

    D -->|Pass| E{Lorem Ipsum?}

    E -->|Yes| Z

    E -->|No| F{Marketing Patterns?}

    F -->|Match| G[Always Keep]

    F -->|No Match| H[Accept]

Validation Rules:

1. Length Check: 2-100 characters (filters noise and fragments)

2. Character Validation: Only allows alphanumeric, Devanagari, and safe punctuation

3. Real Word Detection: Uses word frequency analysis to filter gibberish

4. Lorem Ipsum Filter: Removes placeholder text patterns

5. Marketing Pattern Preservation: Always keeps prices, dates, URLs, phone numbers

After OCR fusion and deduplication, an LLM cleans up common OCR errors:

Common OCR Errors Fixed:

| Error Type | Example | Fix |

|------------|---------|-----|

| Character confusion | “TlCKETS” → “TICKETS” | I/l/1, O/0, S/5, B/8 |

| Space errors | “TI CKETS” → “TICKETS” | Wrong splits and missing spaces |

| Unicode digits | ५ → 5 | Devanagari to ASCII conversion |

| Fragment merging | “GOLDEN SALE” from “GOLDENSALE” | Context-aware merging |

Cleanup Prompt Structure:

You are an OCR post-processor for ALL types of posters and designs.

Types include: marketing, wedding, festival, music concert, food menu, sports...


Fix these OCR text fragments:

1. TlCKETS

2. F0R 50% OFF

3. G0LDEN SALE


Rules:

1. Fix character confusion when result is a real word

2. Fix space errors (wrong splits and missing spaces)

3. Convert Unicode digits to ASCII

4. Keep as-is: Hindi/Devanagari text, brand names, prices, phones, URLs

5. Remove ONLY pure symbol garbage with zero real words


Return ONLY a valid JSON array of strings.

Automatically identifies poster type using pattern matching:

| Domain | Keywords | Output Format |

|--------|----------|---------------|

| Wedding | wedding, bride, groom, shaadi, mehndi | “wedding invitation card design” |

| Festival | diwali, holi, navratri, eid, christmas | “festival celebration poster design” |

| Sale | sale, discount, % off, offer, deal | “sale marketing promotional poster design” |

| Concert | concert, live music, band, tour | “music concert event poster design” |

| Food | restaurant, menu, cuisine, cafe | “restaurant food menu promotional design” |

| Real Estate | property, apartment, villa, bhk | “real estate property promotional poster” |

Auto-generates hashtags based on detected content:

 
STYLE_CHECKS = [
 
    (["sale", "discount", "% off"], "#sale #marketing #discount"),
 
    (["diwali", "festival", "holi"], "#indianfestival #festivedesign"),
 
    (["wedding", "bride", "groom"], "#wedding #invitation #elegant"),
 
    (["concert", "music", "band"], "#music #concert #livemusic"),
 
    # ... 15+ more patterns
 
]
 

The final caption follows a structured format optimized for Flux LoRA training:

domain description, visual description, text "content" at position, #tags

Example Output:

sale marketing promotional poster design, dark red festive background with gold bokeh particles and elegant typography,

text "DIWALI SALE" at top-center in bold serif font,

text "50% OFF" at top-right in large display font,

text "Shop Now" at bottom-center in sans-serif,

#sale #marketing #discount #promotional #indianfestival

Before OCR, images undergo adaptive preprocessing based on detected characteristics:

 
def get_preprocessed(img_bgr, has_skin, has_flame, has_split,
 
                     is_dark, is_colorful, is_low_contrast):
 
    if has_split:
 
        # Split image (two distinct halves) → CLAHE per half
 
        base = split_clahe(img_bgr)
 
    elif is_dark:
 
        # Dark background → Gamma correction + CLAHE
 
        base = enhance_dark(img_bgr)
 
    elif is_colorful:
 
        # Highly colorful → Saturation adjustment + CLAHE
 
        base = enhance_colorful(clahe(img_bgr))
 
    elif is_low_contrast:
 
        # Low contrast → Sharpening + aggressive CLAHE
 
        base = enhance_low_contrast(img_bgr)
 
    else:
 
        # Standard case → CLAHE
 
        base = clahe(img_bgr)
 
    return upscale_2x(sharpen(base))
 

Edge Case Detectors:

| Detector | Purpose | Threshold |

|----------|---------|-----------|

| detect_skin | Detects skin tones (people photos) | >25% skin area |

| detect_flame | Detects red/orange dominant images | >35% warm colors |

| detect_split | Detects split-screen layouts | >70 intensity difference |

| detect_devanagari | Detects Hindi text presence | Horizontal > Vertical edge strength |

| detect_dark_background | Detects dark backgrounds | Mean gray < 80 |

| detect_highly_colorful | Detects saturated images | Mean saturation > 140 |

| detect_low_contrast | Detects low contrast images | Std dev gray < 35 |

After merging OCR sources, fuzzy deduplication removes duplicates while keeping highest quality:

 
def fuzzy_deduplicate(texts, threshold=80):
 
    kept = []
 
    for candidate in texts:
 
        is_dup = False
 
        for i, existing in enumerate(kept):
 
            if fuzz.ratio(candidate.lower(), existing.lower()) >= threshold:
 
                # Keep the one with higher text score
 
                if text_score(candidate) > text_score(existing):
 
                    kept[i] = candidate
 
                is_dup = True
 
                break
 
        if not is_dup:
 
            kept.append(candidate)
 
    return kept
 

Text Scoring:

• Based on word frequency (real words scored higher)

• Space ratio (properly spaced text scored higher)

• Combined score determines which duplicate to keep

OCR often outputs fragmented text that needs merging:

 
def merge_fragments(texts):
 
    result, i = [], 0
 
    while i < len(texts):
 
        current = texts[i]
 
        # Hard stop conditions (don't merge further)
 
        if is_hard_stop(current) or len(current.split()) > 2:
 
            result.append(current)
 
            i += 1
 
            continue
 
        # Check if next fragment should be merged
 
        if i + 1 < len(texts):
 
            nxt = texts[i + 1]
 
            if (not is_hard_stop(nxt)
 
                    and len(nxt.split()) <= 2
 
                    and same_type(current, nxt)
 
                    and not current.rstrip().endswith('.', '!', '?')):
 
                result.append(current + ' ' + nxt)
 
                i += 2
 
                continue
 
        result.append(current)
 
        i += 1
 
    return result
 

Hard Stop Conditions:

• Phone numbers (3-digit patterns)

• Addresses (ST., APT., ZIP codes)

• Currency amounts (₹, $, €)

• URLs and emails

• Years (4 digits)

• Sentences ending with punctuation

Spatial information is preserved throughout processing:

 
POSITION_PRIORITY = {
 
    'top-center':    0, 'top-left':      1, 'top-right':     2,
 
    'center':        3,
 
    'middle-center': 4, 'middle-left':   5, 'middle-right':  6,
 
    'bottom-center': 7, 'bottom-left':   8, 'bottom-right': 9,
 
    None:           10,
 
}
 

Text is sorted by position in the final caption to maintain logical reading order.

Each generated caption receives a quality score:

 
def caption_quality_score(caption, florence_text, ocr_items):
 
    score = 0
 
    # Caption length scoring
 
    cap_len = len(caption)
 
    if cap_len >= 400:   score += 40
 
    elif cap_len >= 250: score += 30
 
    elif cap_len >= 150: score += 20
 
    elif cap_len >= 80:  score += 10
 
    # Florence visual description scoring
 
    flor_len = len(florence_text)
 
    if flor_len >= 200:   score += 30
 
    elif flor_len >= 100: score += 20
 
    elif flor_len >= 50:  score += 10
 
    # OCR coverage scoring
 
    n_ocr = len(ocr_items)
 
    if n_ocr >= 5:   score += 20
 
    elif n_ocr >= 3: score += 15
 
    elif n_ocr >= 1: score += 10
 
    # Sentence ending bonus
 
    if florence_text and florence_text[-1] in '.!?':
 
        score += 10
 
    return score
 

Quality Thresholds:

• Score ≥ 70: Excellent caption

• Score 50-69: Good caption

• Score 30-49: Acceptable, may need review

• Score < 30: Poor, should retry

Your prompt goes through a text encoder (typically CLIP - Contrastive Language-Image Pre-training):

"Made by Lakme" → [0.23, -0.87, 0.44, 0.11, ...]  (768-dimensional vector)

CLIP converts text into a numerical vector that represents its semantic meaning. But here’s the critical issue:

When you prompt: "write the word GOLD on the bottle"

The text encoder processes this, but the attention mechanism interprets it differently:

| What You Meant | What Model Generates |

|----------------|----------------------|

| G - O - L - D (4 specific letters) | ✨ golden shimmer, metallic sheen, luxury feel ✨ |

The model applies semantic meaning as visual form. It’s not generating letters—it’s generating the concept of gold.

Diffusion models don’t work directly with pixels. They operate in latent space:

Original Image (1024×1024 pixels)

        ↓

   VAE Encoder (compression)

        ↓

Latent Representation (128×128 × channels)  ← This is where generation happens

        ↓

   VAE Decoder (expansion)

        ↓

Final Image (1024×1024 pixels)

Processing 1024×1024 pixels directly would be computationally prohibitive. Latent space is faster and more efficient.

Diffusion starts from pure noise (like TV static) and gradually shapes it into an image through iterative denoising steps.

At each step, the model asks: “What was my prompt? What am I generating?”

Creates a soft-edged mask for smooth blending between text and background regions:

 
def create_soft_mask(schema, canvas_size=1024, blur_radius=15):
 
    """
 
    Creates a Gaussian-blurred mask covering all text regions.
 
    The blur radius controls how soft the transition is.
 
    """
 
    mask = Image.new("L", (canvas_size, canvas_size), 0)
 
    draw = ImageDraw.Draw(mask)
 
    for block in schema["texts"]:
 
        # Calculate bounding box with padding
 
        padding = 20
 
        x1 = max(0, block["x"] - padding)
 
        y1 = max(0, block["y"] - padding)
 
        # Approximate text width based on character count and font size
 
        text_width = len(block["content"]) * block["font_size"] * 0.6
 
        x2 = min(canvas_size, block["x"] + text_width + padding)
 
        y2 = min(canvas_size, block["y"] + block["font_size"] + padding)
 
        draw.rectangle([x1, y1, x2, y2], fill=255)
 
    # Apply Gaussian blur for soft edges
 
    mask_np = np.array(mask, dtype=np.float32)
 
    mask_np = cv2.GaussianBlur(mask_np, (0, 0), blur_radius)
 
    return Image.fromarray(np.clip(mask_np * 255, 0, 255).astype(np.uint8))
 

Why Soft Mask?

• Hard edges create visible seams between text and background

• Soft mask creates natural feathering that blends seamlessly

• Blur radius of 15px provides good balance between definition and smoothness

Adjusts conditioning strength based on smallest font size in the layout:

 
def _get_controlnet_scale(self, resolved_blocks: List[dict], canvas_size: int = 1024) -> float:
 
    """
 
    Calculate optimal ControlNet conditioning scale.
 
    Based on smallest text in layout because:
 
    - Smaller text requires stricter enforcement (fine details harder to render)
 
    - Scale must accommodate the most challenging text element
 
    Official InstantX recommendation: 0.3-0.8 for text enforcement
 
    """
 
    if not resolved_blocks:
 
        return 0.65
 
 
    min_font = min(b["font_size"] for b in resolved_blocks)
 
    longest  = max(len(b["content"]) for b in resolved_blocks)
 
 
    relative = min_font / canvas_size
 
 
    if relative < 0.05:          # very small text (< 51px)
 
        base = 0.75
 
    elif relative < 0.10:        # medium text (51-102px)
 
        base = 0.65
 
    else:                        # large headlines (> 102px)
 
        base = 0.55
 
 
    # Long text lines need slightly more enforcement
 
    if longest > 15:
 
        base += 0.05
 
 
    return round(min(base, 0.80), 2)
 

Key Implementation Details:

• Scale is based on the smallest font in the layout, not individual text blocks

• This ensures all text elements are properly enforced

• Values are rounded to 2 decimal places for consistency

• Official InstantX recommendation: 0.3-0.8 for text enforcement

Only enhances contrast when needed:

 
def boost_text_contrast(image: Image.Image,
 
                        soft_mask: Image.Image,
 
                        strength: float = 1.35) -> Image.Image:
 
    """
 
    Selectively boost contrast ONLY in text regions.
 
    Skips boost if text region already has high contrast (avoids over-sharpening).
 
    """
 
    # Measure existing contrast in text region
 
    img_gray  = np.array(image.convert("L"), dtype=np.float32)
 
    mask_arr  = np.array(soft_mask.convert("L"), dtype=np.float32) / 255.0
 
 
    text_pixels = img_gray[mask_arr > 0.5]
 
 
    if len(text_pixels) == 0:
 
        return image   # no text region found
 
 
    existing_contrast = text_pixels.std()
 
 
    # Already high contrast — skip (avoids over-sharpening)
 
    if existing_contrast > 65:
 
        return image
 
    enhanced  = ImageEnhance.Contrast(image).enhance(strength)
 
    mask_l    = soft_mask.convert("L")
 
 
    # Composite: text regions use enhanced, rest stays original
 
    result = Image.composite(enhanced, image, mask_l)
 
    return result
 

Contrast Boost Logic:

• Measures standard deviation of grayscale values in text regions

• Skips boost if existing contrast > 65 (avoids over-sharpening artifacts)

• Applies 1.35x contrast enhancement selectively to text regions only

• Uses soft mask for smooth blending between enhanced and original regions

• OS: Linux (tested on Ubuntu 22.04)

• GPU: NVIDIA GPU with ≥16GB VRAM recommended

• Python: 3.10+

• CUDA: 12.4+

| Operation | Minimum VRAM | Recommended VRAM | Notes |

|------------------|--------------|------------------|-------|

| Caption Generator (single image) | 8GB | 12GB | Florence-2 + EasyOCR |

| Caption Generator (batch processing) | 12GB | 16GB | Multiple models loaded |

| Image Generation Pipeline (Pass 1+2 shared) | 16GB | 24GB | Flux + ControlNet Union |

| Full pipeline with LoRA | 20GB | 24GB+ | Additional LoRA weights |

VRAM Optimization: The pipeline shares model components (VAE, text encoders, transformer) between Pass 1 and Pass 2 to save ~15GB VRAM.

Complete configuration reference for all pipeline parameters:

 
models:
 
  flux: "/home/aiteam/ai_team/Animesh-Diffusion/models/flux-dev"
 
  controlnet: "/home/aiteam/ai_team/Animesh-Diffusion/models/controlnet-union"
 
  lora: null                          # Optional LoRA weights path
 
  fonts_dir: "/home/aiteam/ai_team/Animesh-Diffusion/fonts"
 
 
generation:
 
  canvas_size: 1024                   # Output resolution (square)
 
  pass1_steps: 28                     # Background generation steps
 
  pass2_steps: 14                     # Text injection steps
 
  guidance_scale: 3.5                 # Flux CFG scale
 
  lora_scale: 0.85                    # LoRA weighting (if used)
 
  soft_mask_blur: 15                  # Soft mask blur radius in pixels
 
  min_font_size: 48                   # Minimum font size for text blocks
 
  margin: 60                          # Margin around canvas edges
 
  device: "cuda"                      # Device for computation
 
 
quality:
 
  ocr_ned_threshold: 0.85             # OCR verification threshold (0-1)
 
  max_pass2_retries: 3                # Maximum retry attempts
 
 
output:
 
  upscale_factor: 2                   # RealESRGAN upscaling factor
 
  jpeg_quality: 95                    # Output JPEG quality (1-100)
 
  output_dir: "/home/aiteam/ai_team/Animesh-Diffusion/pipeline/outputs"
 

Enable debug mode to save intermediate images:

 
python pipeline_runner.py --brief "test" --debug
 

Debug files saved:

• debug_glyph.png - Text glyph rendering

• debug_canny.png - Canny edge map for ControlNet

• debug_background.png - Pass 1 background output

• debug_pass2_attempt{N}.jpg - Each OCR retry attempt

| Priority | Enhancement | Description |

|----------|-------------|-------------|

| High | Add more OCR sources | Tesseract, Google Vision API |

| Medium | Hindi/Regional language support | Expand beyond English+Hindi |

| Medium | Configurable output format | JSON, CSV exports |

| Low | GPU acceleration for preprocessing | Faster image enhancement |

| Priority | Enhancement | Description |

|----------|-------------|-------------|

| High | LoRA fine-tuning support | Domain-specific style adaptation |

| High | Async batch processing | Generate multiple images concurrently |

| Medium | Better OCR retry logic | Adaptive seed selection based on failure patterns |

| Medium | Text position optimization | Auto-layout suggestions for better composition |

| Medium | Visual quality gate | Additional checks beyond OCR (color balance, artifacts) |

| Low | Web UI interface | Streamlit/Gradio dashboard |

For Production Deployment:

1. Generate 100-200 outputs to identify failure patterns:

   • Specific font weights where text drifts

   • Background colors where contrast boost is insufficient

   • LLM output positions that cause overlap

2. Tune thresholds based on observations:

   • OCR NED threshold (currently 0.85)

   • Contrast boost threshold (currently 65 std dev)

   • ControlNet scale ranges for different text sizes

3. Add domain-specific LoRA if generating consistent style is needed

| Purpose | Path |

|---------|------|

| Main caption script | caption_runner.py |

| Pipeline entry point | pipeline/scripts/pipeline_runner.py |

| Pipeline config | pipeline/config.yaml |

| Output directory | pipeline/outputs/ |

| Model storage | models/ |

| Fonts | fonts/ |