Compact (Sub-Billion) Multimodal Action Model for On-Device AI Agents
Octopus v3 represents a significant advancement in on-device multimodal AI and AI agent applications. While its predecessor, Octopus v2, focused on text-based interactions and outperformed GPT-4 in function-calling speed and accuracy, v3 expands capabilities to include visual inputs and multilingual support.
This blog explores Octopus v3's technical aspects and assesses its performance in real-world scenarios, comparing this compact, on-device model to larger, cloud-based alternatives like GPT-4V.
We developed Octopus v3 with these key techniques to add multimodal capabilities while keeping the model compact:
To assess Octopus v3's performance in multimodality and function-calling, we compared it with a combination of GPT-4V and GPT-4 across 10 common use cases in smartphone operating systems:
In all these cases, Octopus v3, with less than 1B parameters, demonstrated function-calling accuracy comparable to the cloud-based, large-scale GPT-4V and GPT-4 combination. It proficiently handled both English and Chinese inputs, effectively processing visual and textual information to generate accurate responses and actions.
The versatility across complex, multimodal tasks in various domains underscores Octopus v3's potential as a powerful AI agent for everyday smartphone use cases. Importantly, Octopus v3 runs entirely on-device, requiring no internet connection for its operations.