MiniCPM-V 4.5 - SGLang Documentation

1. Installing SGLang

Install SGLang from Source Code

git clone https://github.com/sgl-project/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

Installing flashinfer Dependencies

Method 1: pip installation (network speed may be insufficient)

pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

Method 2: whl file installation - Visit: https://flashinfer.ai/whl/cu121/torch2.4/flashinfer/ - Locate and download the whl file compatible with your server, e.g. flashinfer-0.1.6+cu121torch2.4-cp310-cp310-linux_x86_64.whl - Install using pip: cpp pip install flashinfer-0.1.6+cu121torch2.4-cp310-cp310-linux_x86_64.whl For any installation issues, please consult the official installation documentation

2. Launching Inference Service with sglang

By default, it downloads model files from Hugging Face Hub

python -m sglang.launch_server --model-path openbmb/MiniCPM-V-4_5 --port 30000

Alternatively, you can specify a local path after the --model-path parameter

python -m sglang.launch_server --model-path your_model_path --port 30000 --trust-remote-code

3. Service API Calls

• Bash call python curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniCPM-V-4.5", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What's in this image?" }, { "type": "image_url", "image_url": { "url": "https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/inference/assets/airplane.jpeg?raw=true" } } ] } ], "max_tokens": 300 }'

• Python call ```python from openai import OpenAI

  client = OpenAI(base_url=f"http://localhost:30000/v1", api_key="None")

  response = client.chat.completions.create( model="MiniCPM-V-4.5", messages=[ { "role": "user", "content": [ { "type": "text", "text": "What is in this image?", }, { "type": "image_url", "image_url": { "url": "https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/inference/assets/airplane.jpeg?raw=true", }, }, ], } ], max_tokens=300, )

  print(response.choices[0].message.content) ```

  If the image_url is inaccessible, it can be replaced with a local image path

  For more calling methods, please refer to the SGLang documentation