GhostScientist/semanticwiki-coder-7b-v2

DeepWiki Coder 7B v2

Fine-tuned Qwen2.5-Coder-7B-Instruct for generating DeepWiki-style codebase wiki pages — structured technical documentation with source file references, Mermaid diagrams, markdown tables, and inline citations.

What It Does

Given source code files and a topic, this model generates a complete technical wiki page with:

• <details> source file block at the top listing all relevant files
• H1/H2/H3 markdown sections with introduction, detailed analysis, and conclusion
• Mermaid diagrams (flowchart TD, sequenceDiagram) showing architecture and data flow
• Markdown tables summarizing key information
• Source citations in Sources: [path:line]() format referencing specific files

Training Details

Parameter	Value
Base model	Qwen/Qwen2.5-Coder-7B-Instruct
Method	LoRA SFT (Supervised Fine-Tuning)
LoRA rank	64
LoRA alpha	128
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Epochs	4
Learning rate	2e-4
Effective batch size	16
Max sequence length	12288
Training examples	28
Training data	GhostScientist/deepwiki-sft-v2
Hardware	NVIDIA A100-SXM4-80GB
Final loss	0.2488
Token accuracy	92.4%

v2 Improvements over v1

Metric	v1 Training Data	v2 Training Data
Proper citation format	19%	100%
Mermaid diagrams	52%	100%
Markdown tables	0%	100%
Avg citations/example	0.4	6.8

The v2 training data was generated with few-shot prompting and post-processing to enforce DeepWiki format requirements (citations, diagrams, tables) that were missing or inconsistent in v1.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-Coder-7B-Instruct"
adapter_model_id = "GhostScientist/deepwiki-coder-7b-v2"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_model_id)
model = model.merge_and_unload()

messages = [
    {"role": "system", "content": "You are an expert code analyst..."},
    {"role": "user", "content": "<START_OF_CONTEXT>\n[source code here]\n<END_OF_CONTEXT>\n\n<query>\nGenerate a wiki page about...\n</query>"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=4096, temperature=0.7, top_p=0.9, do_sample=True)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training Infrastructure

• Training job: HF Jobs
• Trackio dashboard: GhostScientist/ml-intern-d8f3a2b1
• Training script: TRL SFTTrainer with LoRA via PEFT
• Monitoring: Trackio with alert callbacks for divergence/NaN/overfitting detection

Framework Versions

• TRL: 1.13.0
• Transformers: 5.17.0
• PyTorch: 2.14.0
• PEFT: 0.20.0
• Datasets: 5.0.1
• Tokenizers: 0.23.2

Related Resources

• v1 Model (previous iteration)
• v2 Training Dataset
• v1 Eval Results

Citations

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {vonwerra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern