Instructions to use google/gemma-4-31B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use google/gemma-4-31B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="google/gemma-4-31B")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("google/gemma-4-31B") model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-31B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use google/gemma-4-31B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "google/gemma-4-31B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-31B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/google/gemma-4-31B
- SGLang
How to use google/gemma-4-31B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "google/gemma-4-31B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-31B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "google/gemma-4-31B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-31B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use google/gemma-4-31B with Docker Model Runner:
docker model run hf.co/google/gemma-4-31B
Fine-Tuning Gemma 4 on Day Zero: 3 Bugs in 30 Minutes
Google released Gemma 4 today under Apache 2.0. We had a QLoRA training run stepping on a B200 within hours. Here's what broke.
Bug 1: "Transformers does not recognize this architecture"
Stable Transformers (5.4.0) doesn't include gemma4 model type yet.
Fix: pip install git+https://github.com/huggingface/transformers.git
Bug 2: "Target module Gemma4ClippableLinear is not supported"
Gemma 4 introduces Gemma4ClippableLinear for its vision encoder. PEFT doesn't recognize it - even from source. The layer wraps nn.Linear with input/output clamping, but inherits from nn.Module instead of nn.Linear, so PEFT's type check rejects it.
Fix: Monkey-patch it to inherit from nn.Linear. PEFT then treats it normally.
Bug 3: "mm_token_type_ids is required"
Gemma 3 needed token_type_ids. Gemma 4 adds mm_token_type_ids (multimodal). Both need to be in the data collator, even for text-only training.
Fix: Custom collator with both fields initialized to zeros.
Result: 31B model training at 4.5s/step on 1x B200. 534M trainable params (1.68% QLoRA). None of these issues are avoidable with experience - they're day-zero discovery problems.
Could you go into more detail about how you exactly resolved the issue with the Gemma4ClippableLinear, please?
Sure. @MetaMind42 The issue is that Gemma4ClippableLinear inherits from nn.Module instead of nn.Linear. PEFT checks the type of every module before applying LoRA, and it only recognizes nn.Linear, nn.Embedding, nn.Conv1d/2d/3d, and nn.MultiheadAttention. Since Gemma4ClippableLinear isn't any of those, PEFT rejects it - even if you use exclude_modules to skip it, because the type check runs first.
The fix is to replace the class definition before loading the model:
import torch
import torch.nn as nn
from transformers.models.gemma4 import modeling_gemma4
class PatchedClippableLinear(nn.Linear):
def init(self, config, in_features, out_features):
nn.Linear.init(self, in_features, out_features, bias=False)
self.use_clipped_linears = getattr(config, "use_clipped_linears", False)
if self.use_clipped_linears:
self.register_buffer("input_min", torch.tensor(-float("inf")))
self.register_buffer("input_max", torch.tensor(float("inf")))
self.register_buffer("output_min", torch.tensor(-float("inf")))
self.register_buffer("output_max", torch.tensor(float("inf")))
def forward(self, x):
if self.use_clipped_linears:
x = torch.clamp(x, self.input_min, self.input_max)
out = nn.Linear.forward(self, x)
if self.use_clipped_linears:
out = torch.clamp(out, self.output_min, self.output_max)
return out
modeling_gemma4.Gemma4ClippableLinear = PatchedClippableLinear
Place this before any AutoModelForCausalLM.from_pretrained() call. The patched class preserves the clamping behavior but now inherits from nn.Linear, so PEFT recognizes it.
Also use exclude_modules=["vision_tower", "multi_modal_projector", "audio_tower"] in your LoRA config to skip applying LoRA to the vision/audio layers entirely.
I filed this as huggingface/peft#3129 - Benjamin Bossan suggested targeting the inner .linear layer as an alternative approach, but I haven't tested that yet.
Will try! Thanks a lot.
@MetaMind42 let me know how it works out. Also check out Benjamin's .linear layer approach. I just havent had a moment to run it yet... π
No luck with Benjamin's workaround. But yours worked for me. Thanks!
Hello has anyone tried to deploy a fine tuned Gemma 4 Model?.Have you been successful?.
same but in different scenario: I was using gemma-4 2b it model for QloRA and at part of quantization one error : Target module Gemma4ClippableLinear(
(linear): Linear4bit(in_features=768, out_features=768, bias=False)
) is not supported. Currently, only the following modules are supported: torch.nn.Linear, torch.nn.Embedding, torch.nn.Conv1d, torch.nn.Conv2d, torch.nn.Conv3d, transformers.pytorch_utils.Conv1D, torch.nn.MultiheadAttention.. came for this I tryied to use : exclude_modules=["vision_tower", "audio_tower"] but still same can anyone help
Thanks for the field report β we recognise the same Bug 2 from the top of the thread, now on the gemma-4-2b-it model with QLoRA, and the detail that exclude_modules=["vision_tower", "audio_tower"] did not clear it is useful. The fix reported earlier in this thread was to patch Gemma4ClippableLinear to inherit from nn.Linear before loading the model, and to exclude the vision tower, the multi_modal_projector and the audio tower β the projector is the part missing from the exclude list here. MetaMind42 also reported that patch worked for them after the .linear-layer workaround did not.
ooh thanx for the help it worked. also one more thing on that solution part at :
def init(self, config, in_features, out_features):
nn.Linear.init(self, in_features, out_features, bias=False)
get type error. for that we have to use :
def init(self, config, in_features, out_features):
nn.Linear.init(self, in_features, out_features, bias=False)
diff of init insted of init
Thanks for reporting back β we're glad the fix worked, and thanks for catching that correction. You're right: plain init defines a new method rather than the constructor, so def __init__(...) and nn.Linear.__init__(self, in_features, out_features, bias=False) are the correct forms.