Injecting small VLMs with strategies for tool-use on videos

Small VLMs are quite capable at tool use. I am working on a problem where we want to use tool-use agentic VLMs to study videos generated by T2V models, and identify errors in them. I provide the model with tools for viewing frames from videos at higher fps, showing frames side-by-side for comparisons, zooming into different frame regions, and even optical flow based frame motion comparison.

Here’s something interesting I found: models like Qwen 3.5 VL 9B are absolutely capable of tool-use, but they are hampered by their inability to reason about the problem well, come up with hypotheses to test, and strategically use tools.

Research Question

If a bigger model, say Opus 5.5 or GPT-6 Sol, could provide the Qwen model with a strategy, can this strategy be injected into Qwen’s multi-turn conversation to improve tool use?

Diagram titled “Large VLM Teacher strategy injected into Qwen 3.5 VL conversation”. A video and prompt lead into a Strategy step, produced by a large VLM and injected into the conversation, with three parts: set the scene, generate hypotheses to validate, and set up a tool-use plan. Strategy leads to a looping Tool Use step and then Answer Submission, along an arrow labelled multi-turn conversation. Listed Qwen-3.5-VL 9B tool-use outcomes: reasoning is succinct as it tries to follow the injected style, and tool use directly targets error regions based on the plan.

The short answer is YES! In fact, it produced some rather interesting outcomes… not only did Qwen become more accurate at tool-use, but its reasoning traces on the video were shorter, more succinct and more confident. This side effect is interesting: by modifying the expectation of how the Qwen model should think about the video, it started adapting that thinking setting to all future turns.