ECCV 2026

Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models

Yiyang Huang, Zhaowen Wang, Simon Jenni, Jing Shi, Yitian Zhang, Yizhou Wang, Yun Fu

Northeastern University | Adobe Research

Layout understanding, or the interpretation of element organization, is essential for document analysis, user interface (UI) creation, and graphic design. While recent vision-language models (VLMs) excel at interpreting atomic layouts composed of independent elements, they struggle with compositional layouts that require reasoning over visually entangled elements within hierarchical multi-layer structures.

Compositional layout understanding CoDeLayout MASON

Paper Supp Code Dataset BibTeX

Compositional layout understanding for design assistants — Compositional layout understanding is critical for design assistants, as handling such layouts often requires coordinated edits across interrelated elements to faithfully implement user intent.

Task

Compositional layout understanding aims to identify interacting elements within hierarchical layouts and interpret their design intent.

Each layout instance is represented by the rendered design image and the complete metadata of the design. The model is required to identify the corresponding compositional element and generate a textual answer explaining their design intent.

Motivation

Compositional designs involve visual entanglement among multiple elements within hierarchical structures.

Understanding compositional layouts is essential for enabling design assistants to perform editing operations correctly.

CoDeLayout

To support compositional layout understanding in graphic design, we construct the Compositional Design Layout Dataset.

CoDeLayout contains approximately 20K design instances spanning diverse resolutions and coherent multi-layer structures across four representative compositional types: Overlaying, Clipping, Blending, and Morphing.

Each instance includes: (1) a high-fidelity rendered design image; (2) a structured JSON file with element-level metadata (id, type, opacity, zIndex, position, etc.); and (3) QA-style annotations specifying compositional element pairs and their associated design intents.

CoDeLayout follows the natural distribution of real-world graphic designs without resampling for category balancing. It contains 20,009 training samples and 387 test samples. All test samples are manually verified to ensure annotation correctness and prevent data leakage.

Composition Categories

Figure 1 illustrates representative composition types, including clipping for inter-layer insertion, blending image crops into stylized shapes, overlaying an element with its outline, and morphing text for visual effects.

Overlaying: element plus outline or shadow — **Overlaying** Overlaying combines an element with its outline or shadow to emphasize visual hierarchy.

Clipping: cutout layer for insertion — **Clipping** Clipping extracts a region from a base image as a separate layer, allowing insertion of new elements between the base and the cutout.

Blending: merged image crops — **Blending** Blending integrates different crops of the same image into stylized shapes for aesthetic coherence.

Morphing: text replaced by graphics — **Morphing** Morphing replaces text with graphics to create expressive visual effects.

Representative Compositional Layouts

CoDeLayout spans diverse resolutions and coherent multi-layer structures across four representative compositional types.

Problem Analysis

Compositional layouts introduce structural complexity beyond atomic layouts.

Grounding performance versus compositional element identification accuracy — **Semantic Drift** Semantic drift arises from the visual entanglement of elements, as VLMs fail to align textual metadata with visual content, resulting in misinterpretation of element semantics and degraded compositional understanding.

Spatial relation perception accuracy versus compositional element identification accuracy — **Structural Ambiguity** Structural ambiguity arises from the hierarchical organization of compositional elements, where VLMs struggle to interpret layer-aware spatial structures, limiting their ability to model inter-element relationships.

MASON

MASON mitigates semantic drift and structural ambiguity through multimodal alignment and structural perception.

Multimodal Alignment

To address semantic drift, MA introduces an element grounding objective during VLM post-training, encouraging alignment between textual metadata and corresponding visual content under visual entanglement.

Structural Perception

To address structural ambiguity, SP incorporates layer-aware spatial relationships into VLM post-training, enabling accurate modeling of hierarchical inter-element relationships.

Results

Performance on CoDeLayout across four compositional categories.

Each cell reports Accuracy (GPT-Score/BLEU/ROUGE). Accuracy evaluates compositional element identification, while GPT-Score/BLEU/ROUGE measure explanation quality. Weighted Accuracy is computed using test sample proportions, and Average Accuracy is the unweighted mean across categories.

Model	Overlaying	Clipping	Blending	Morphing	Weighted Acc.	Average Acc.
Heuristic Baselines
Max-Overlap	61.74N/A	69.70N/A	23.08N/A	36.17N/A	44.27	47.67
Nearest-Neighbor	58.26N/A	62.12N/A	23.72N/A	38.30N/A	42.44	45.60
Open-Source VLMs
Qwen2.5-VL	66.093.51/3.65/23.94	37.882.85/1.16/18.94	53.853.37/2.48/22.28	36.173.22/2.16/21.42	52.60	48.49
Qwen3-VL	86.094.52/7.63/32.77	71.213.85/3.81/26.12	79.494.11/4.59/28.55	55.323.49/2.22/23.04	77.08	73.02
InternVL-3.5	84.354.02/5.66/28.84	54.552.93/1.29/19.49	70.513.52/2.74/23.38	42.553.11/1.70/20.48	68.48	62.99
LLaVA-OneVision	56.523.41/2.90/23.21	21.212.72/0.71/17.59	60.903.24/1.74/21.36	29.792.89/0.97/18.82	48.95	42.10
Closed-Source VLMs
GPT-4o	83.484.91/10.67/36.34	36.364.02/3.22/28.05	63.464.66/6.22/34.00	65.964.22/5.88/29.89	65.10	62.31
GPT-5	89.573.62/4.55/23.95	62.122.99/1.40/19.18	79.493.25/2.20/21.25	56.923.51/1.98/23.97	76.56	71.62
GPT-o3	93.913.35/3.26/22.08	63.642.81/1.25/18.46	79.493.12/2.25/19.93	68.093.46/1.87/23.73	79.68	76.28
Gemini-2.5-flash	81.744.36/6.88/31.18	53.033.54/2.96/23.87	72.443.78/3.61/25.14	55.323.81/2.85/25.66	69.79	65.63
Gemini-2.5-pro	90.434.02/4.92/28.48	45.453.77/2.99/25.79	75.003.89/3.53/26.33	59.573.79/3.15/25.70	72.65	67.61
Ours
Direct Finetune (Full data)	93.046.42/30.87/51.63	83.334.73/10.89/35.08	94.875.14/11.69/39.53	65.964.96/11.23/37.00	88.80	84.30
MASON (30% data)	92.176.29/29.92/50.22	86.364.81/10.69/35.68	93.594.86/9.80/36.34	72.344.69/9.95/34.44	89.32	86.12
MASON (Full data)	95.656.62/31.87/52.99	90.914.93/11.74/36.27	95.515.22/11.84/39.66	70.215.13/11.60/38.77	91.66	88.07

Data Scale Ablation — Performance of Direct Finetune and MASON trained on 10%, 15%, 30%, and 100% of the CoDeLayout training set. MASON consistently outperforms across all scales, with a clear advantage under low-resource settings.

+11.98

MASON (Full data) improves weighted accuracy from 79.68% to 91.66% and average accuracy from 76.28% to 88.07%.

30%

MASON also shows strong data efficiency by achieving 89.32% weighted and 86.12% average accuracy, surpassing Direct Finetune trained on the full dataset.

90.91

Compared with GPT-o3, MASON (Full data) shows especially large gains on structurally complex categories such as Clipping (90.91% vs. 63.64%).

Module Ablation

DF denotes Direct Finetune, MA denotes Multimodal Alignment, and SP denotes Structural Perception. Results show that MA and SP provide complementary improvements, with their combination achieving the strongest overall performance.

Model	Overlaying	Clipping	Blending	Morphing	Weighted	Average
DF	86.09	74.24	80.77	72.34	80.21	78.36
DF + MA	87.83	81.82	81.41	70.21	82.03	80.32
DF + SP	89.57	83.33	82.69	74.47	83.85	82.51
MASON	86.96	81.82	83.97	80.85	84.11	83.40

Case Study

DF often struggles to localize and associate the correct compositional elements when spatial or semantic cues are subtle, whereas MASON successfully identifies the underlying compositional pairs.

Release

Release Status

Paper PDF Download

Supplementary Download

Code Coming soon

Dataset Coming soon

Model To be announced

Citation

Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models

@inproceedings{huang2026beyond,
  title     = {Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models},
  author    = {Huang, Yiyang and Wang, Zhaowen and Jenni, Simon and Shi, Jing and Zhang, Yitian and Wang, Yizhou and Fu, Yun},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}