Video Understanding

Overview

Video understanding lets a model analyze video frames with text instructions and return descriptions, timestamps, or structured results. Typical uses include:

Describing scenes and people’s actions
Summarizing videos and answering questions about their content
Locating the start and end of events
Extracting information as JSON

This guide uses qwen3.7-plus to demonstrate MP4 input with Chat Completions and the Responses API.

APIs and Input Formats

Model and endpoint	Video content field	JSON output parameter
`qwen3.7-plus`, `/v1/chat/completions`	`video_url` in `messages[].content[]`	`response_format.type = "json_object"`
`qwen3.7-plus`, `/v1/responses`	`video_url` in `input[].content[]`	`text.format.type = "json_object"`

In Responses, video_url is an AIHubMix extension field. The examples below send it through HTTP requests. For multimedia input with Gemini models, see the Gemini guide.

Quick Start

The following example uses the built-in fetch in Node.js 22, with no SDK installation required. Set AIHUBMIX_API_KEY in your environment before running it. The sample URL is a public MP4 from Alibaba’s official documentation. For your application, replace url with a video address accessible to the model provider.

const response = await fetch("https://aihubmix.com/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.AIHUBMIX_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "qwen3.7-plus",
    enable_thinking: false,
    max_tokens: 800,
    response_format: { type: "json_object" },
    messages: [{
      role: "user",
      content: [
        {
          type: "video_url",
          video_url: {
            url: "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241115/cqqkru/1.mp4",
          },
        },
        {
          type: "text",
          text: 'Analyze the attached video. Return a JSON object with a scenes array. Each scene must contain start_seconds, end_seconds and description. If no video is available, return {"error":"no_video"}.',
        },
      ],
    }],
  }),
  signal: AbortSignal.timeout(120000),
});
const result = await response.json();
if (!response.ok) throw new Error(JSON.stringify(result));
const scenes = JSON.parse(result.choices[0].message.content);
console.log(JSON.stringify({
  request_id: response.headers.get("x-aihubmix-request-id"),
  result: scenes,
  usage: result.usage,
}, null, 2));

Save the code as an .mjs file and run it with Node.js. The successful result is in choices[0].message.content and requires another JSON parse. The example also prints the request ID and usage for checking in the console.

Local Video Input

Encode the local file as Base64 and pass it as a data:video/mp4;base64, data URL. The example reads the file path from the first command-line argument, or video.mp4 in the current directory if no argument is provided. A video URL in an HTTP request cannot be a local file path.

import { readFileSync } from "node:fs";

const video = readFileSync(process.argv[2] ?? "video.mp4").toString("base64");
const response = await fetch("https://aihubmix.com/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.AIHUBMIX_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "qwen3.7-plus",
    enable_thinking: false,
    max_tokens: 800,
    response_format: { type: "json_object" },
    messages: [{
      role: "user",
      content: [
        {
          type: "video_url",
          video_url: { url: `data:video/mp4;base64,${video}` },
        },
        {
          type: "text",
          text: 'Analyze the attached video. Return a JSON object with a scenes array. Each scene must contain start_seconds, end_seconds and description. If no video is available, return {"error":"no_video"}.',
        },
      ],
    }],
  }),
  signal: AbortSignal.timeout(120000),
});
const result = await response.json();
if (!response.ok) throw new Error(JSON.stringify(result));
const scenes = JSON.parse(result.choices[0].message.content);
console.log(JSON.stringify({
  request_id: response.headers.get("x-aihubmix-request-id"),
  result: scenes,
  usage: result.usage,
}, null, 2));

Responses API

Responses requests also use the video_url object, with input_text for text blocks. Set JSON output under text.format. This example reads the video from the first command-line argument or from video.mp4 in the current directory.

import { readFileSync } from "node:fs";

const video = readFileSync(process.argv[2] ?? "video.mp4").toString("base64");
const response = await fetch("https://aihubmix.com/v1/responses", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.AIHUBMIX_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "qwen3.7-plus",
    max_output_tokens: 800,
    text: { format: { type: "json_object" } },
    input: [{
      role: "user",
      content: [
        {
          type: "video_url",
          video_url: { url: `data:video/mp4;base64,${video}` },
        },
        {
          type: "input_text",
          text: 'Analyze the attached video. Return a JSON object with a scenes array. Each scene must contain start_seconds, end_seconds and description. If no video is available, return {"error":"no_video"}.',
        },
      ],
    }],
  }),
  signal: AbortSignal.timeout(120000),
});
const result = await response.json();
if (!response.ok || result.status !== "completed") {
  throw new Error(JSON.stringify(result));
}
const text = result.output
  .filter((item) => item.type === "message")
  .flatMap((item) => item.content)
  .filter((part) => part.type === "output_text")
  .map((part) => part.text)
  .join("");
console.log(JSON.stringify({
  request_id: response.headers.get("x-aihubmix-request-id"),
  result: JSON.parse(text),
  usage: result.usage,
}, null, 2));

Responses output can contain both reasoning and message items. Read the result by selecting output_text from message items. Output usage can include reasoning tokens, so the length of the final JSON alone does not determine the cost.

Structured Output

The examples use JSON mode to return a scenes array. Each scene contains:

Field	Type	Description
`start_seconds`	Number	Scene start time in seconds
`end_seconds`	Number	Scene end time in seconds
`description`	String	Description of the scene

Adjust the prompt and fields for your application. JSON mode ensures JSON output; your application should validate field types, required fields, and time ranges.

File and Duration Limits

Alibaba’s official vision documentation lists the following video limits for the Qwen3.7 series:

Video formats include MP4, AVI, MKV, MOV, FLV, and WMV.
Video duration ranges from 2 seconds to 2 hours.
The Base64-encoded string must be smaller than 10MB; an original file smaller than 7MB is recommended.
For public URLs, the provider’s file size limit is 2GB.

Base64 increases the request size. Prefer public video URLs for larger files, and ensure the link remains valid throughout the request.

Billing and Tokens

Qwen converts video frames into input tokens, which count toward input usage alongside the prompt. Generated content counts toward output usage. Video duration, resolution, and frame sampling affect token usage; MP4 file size or duration alone cannot determine the cost. Qwen3.7 Plus video input is charged at the model’s input token rate, and generated content at its output token rate. See the Qwen3.7 Plus model page for prices.

Chat Completions reports usage.prompt_tokens and usage.completion_tokens.
Responses reports usage.input_tokens and usage.output_tokens.
Reasoning tokens are already included in output tokens and should not be added again.
Cached input is charged at the applicable cache rate.

View token usage and charges for each request in the console. Keep the x-aihubmix-request-id response header when investigating a specific request.

Recommendations

Send video and text instructions together, specifying the information to extract and the output format.
Keep public video links valid during the request and accessible to the model provider.
Use Base64 for small local files and prefer public URLs for larger files.
Validate JSON fields, scene order, and time ranges in your application.
Timestamps are suitable for locating events and retrieving content. For precise editing, calibrate boundaries with video processing tools.

Updated: 2026-10-13

Getting Started

Console guides

Enterprise and teams

Gateway Features

Model Capabilities

Protocol Reference

Platform Management

Video Understanding

Overview

APIs and Input Formats

Quick Start

Local Video Input

Responses API

Structured Output

File and Duration Limits

Billing and Tokens

Recommendations

​Overview

​APIs and Input Formats

​Quick Start

​Local Video Input

​Responses API

​Structured Output

​File and Duration Limits

​Billing and Tokens

​Recommendations

Overview

APIs and Input Formats

Quick Start

Local Video Input

Responses API

Structured Output

File and Duration Limits

Billing and Tokens

Recommendations