iLearn-Lab/CVPRW26-ChartLens

🚀 ChartLens @ CVPR 2026 DataMFM Chart Understanding Challenge

Hao Liu¹  Ruping Cao¹  Kun Wang¹  Zhiran Li¹  Fan Liu²  Yupeng Hu¹  Liqiang Nie³

¹Shandong University
²Southeast University
³Harbin Institute of Technology (Shenzhen)

These are the official implementation resources, model weights, and prediction files for ChartLens, our champion solution for DataMFM Challenge Track 2: Chart Understanding at CVPR 2026.

🔗 Paper: Arxiv
🔗 GitHub Repository: iLearnLab/CVPRW26-ChartLens
🔗 Challenge Page: DataMFM Challenge

---

📌 Model Information

1. Model Name

ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement

2. Task Type & Applicable Tasks

• Task Type: Chart Understanding / Multimodal Document Understanding
• Applicable Tasks: Chart-to-CSV extraction and chart-to-summary generation from chart images.

3. Project Introduction

Chart understanding requires models to recover structured chart data and generate faithful natural-language summaries from chart images. ChartLens addresses these complementary goals with a dual-branch, verification-guided correction framework.

💡 Method Highlight: ChartLens combines Granite-Vision-4.1-4B LoRA adaptation with two correction branches: Structure-Aware CSV Verification and Correction (SAVC) for reliable table recovery, and Text-Retention-Guided Summary Refinement (TRSR) for OCR-assisted factual summary repair. SAVC checks structure, completeness, and numerical accuracy, while TRSR preserves visible chart text such as titles, legends, annotations, sources, and numerical evidence.

4. Training Data Source

• Released ChartNet-based training data for LoRA adaptation.
• DataMFM Challenge chart understanding splits, including real and synthetic chart images.

5. Challenge Results

Method	CSV Numeric F1	CSV Structural Score	Summary ROUGE-L	Summary Numeric Fact F1	Overall
ChartLens (Ours)	80.62	75.66	45.57	74.55	69.10

ChartLens ranked 1st place on DataMFM Challenge Track 2.

---

🚀 Usage & Basic Inference

Step 1: Prepare the Environment

Clone the GitHub repository and set up the Conda environment:

git clone https://github.com/iLearnLab/CVPRW26-ChartLens.git
cd CVPRW26-ChartLens

conda create -n chartlens python=3.10 -y
conda activate chartlens
pip install -r requirements.txt

Step 2: Data & Weights Preparation

1. Challenge Data: Use the datasets and splits released by the DataMFM Challenge. The chart understanding track contains real and synthetic splits.
2. ChartLens Checkpoints: Download the model weights from this Hugging Face repository.
3. Granite Vision Backbone: Prepare the Granite-Vision-4.1-4B backbone and update the local --model_path argument when running inference.

To prepare ChartNet SFT data for LoRA training:

python code/load_chartnet_500.py \
  --out_dir Fine-tuning/Dataset/raw \
  --num_samples 500

python code/build_chartnet_sft.py \
  --gt_path Fine-tuning/Dataset/raw/gt.jsonl \
  --image_dir Fine-tuning/Dataset/raw/images \
  --out_dir Fine-tuning/Dataset/sft \
  --csv_repeat 2 \
  --summary_repeat 1

Step 3: Run Granite Vision + LoRA Inference

python code/infer_granite_with_lora.py \
  --image_root /path/to/data \
  --out_root /path/to/output \
  --model_path /path/to/granite-vision-4.1-4b \
  --lora_path /path/to/chartlens_lora \
  --gpu_id 0 \
  --splits real synthetic

Use code/infer_chartnet_granite.py for base Granite Vision inference without a LoRA adapter.

Step 4: SAVC CSV Correction

export OPENAI_API_KEY="..."

python code/calibrate_baseline_with_ai.py \
  --split all \
  --baseline_root /path/to/baseline_predictions \
  --image_root /path/to/data \
  --output_root /path/to/savc_output \
  --base_url "https://your-openai-compatible-endpoint" \
  --model gemini-3.5-flash \
  --threshold 85

--baseline_root should contain split directories such as real/ and synthetic/, each with chart2csv_predictions.jsonl and chart2summary_predictions.jsonl.

Step 5: TRSR Summary Refinement

python code/ocr.py \
  --real_images /path/to/data/real/images \
  --synthetic_images /path/to/data/synthetic/images \
  --real_summary /path/to/baseline/real/chart2summary_predictions.jsonl \
  --synthetic_summary /path/to/baseline/synthetic/chart2summary_predictions.jsonl \
  --output_dir /path/to/ocr_text_copy_coverage \
  --threshold 0.8

export AIGCBEST_API_KEY="..."

python code/repair_summary.py \
  --split all \
  --workers 20 \
  --ocr_eval_root /path/to/ocr_text_copy_coverage \
  --output_root /path/to/trsr_output

Step 6: Training (Optional)

Train the LoRA adapter on the prepared ChartNet SFT data:

python code/train_lora_chartnet.py \
  --model_path /path/to/granite-vision-4.1-4b \
  --train_jsonl Fine-tuning/Dataset/sft/train.jsonl \
  --val_jsonl Fine-tuning/Dataset/sft/val.jsonl \
  --output_dir Fine-tuning/FT/model/granite_chartnet_lora_bs2 \
  --gpu_id 0 \
  --epochs 2 \
  --batch_size 1 \
  --grad_accum 8

---

📦 Submission Format

For DataMFM Track 2, organize the final predictions as:

submission.zip
├── real/
│   ├── chart2csv_predictions.jsonl
│   └── chart2summary_predictions.jsonl
└── synthetic/
    ├── chart2csv_predictions.jsonl
    └── chart2summary_predictions.jsonl

Each CSV prediction line:

{"imagename": "example.png", "predicted_csv": "Header A,Header B\nA,1\nB,2"}

Each summary prediction line:

{"imagename": "example.png", "predicted_summary": "One paragraph summary grounded in the chart."}

---

⚠️ Limitations & Notes

Disclaimer: This framework and its model weights are intended for academic research purposes only.

• Chart-to-CSV extraction may still struggle with dense layouts, asymmetric legends, or adjacent semantic-column misalignment.
• Summary refinement depends on OCR quality; OCR errors can affect text-retention scoring and repair decisions.
• GPU execution is expected for Granite Vision inference and LoRA training.
• API-backed correction scripts require valid credentials and an OpenAI-compatible endpoint.

---

🤝 Acknowledgements & Contact

• Contact: If you have any questions or encounter issues, feel free to contact Hao Liu at [email protected] or Ruping Cao at [email protected].

---

📝⭐️ Citation

If you find this project useful for your research, please consider citing:

@article{liu2026chartlens,
  title={ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement},
  author={Liu, Hao and Cao, Ruping and Wang, Kun and Li, Zhiran and Liu, Fan and Hu, Yupeng and Nie, Liqiang},
  journal={arXiv preprint arXiv:2606.10640},
  year={2026}
}