Microsoft’s new vision-language model outranks humans at image captioning
The company will integrate the new model into Azure Cognitive Services to support vision-language tasks
Microsoft researchers have developed a new object-attribute detection model for image encoding: VinVL (visual features in vision-language)
Vision-language (VL) systems make it possible to search relevant images for a text query (or vice versa). They also help describe an image’s content.
In most cases, the systems use two modules to achieve the VL understanding: an image encoding module to generate feature maps of an input image and a vision-language fusion module to map the encoded image and text into vectors in the same semantic space.
Microsoft’s new research focuses on improving the image-encoding module. When combined with VL fusion modules such as OSCAR and VIVO, Microsoft’s newest VL system scored big on the most competitive artificial intelligence (AI) benchmarks, including visual question answering (VQA), Microsoft COCO Image Captioning, and novel object captioning (nocaps).
The tech giant also highlighted that VinVL significantly surpasses human performance on the nocaps leaderboard for consensus-based image description evaluation (CIDEr).
Microsoft trained its VinVL object-attribute detection model using a large object detection dataset containing 2.49 million images ascribed to 1,848 object classes and 524 attribute classes to achieve the results mentioned above. Microsoft formed the dataset by merging four public object detection datasets (COCO, Open Images, Objects365, and VG).
Getting started with Azure Red Hat OpenShift
A developer’s guide to improving application building and deployment capabilities
“We first pretrained an object detection model on the merged dataset, and then fine-tuned the model with an additional attribute branch on VG, making it capable of detecting both objects and attributes,” said Microsoft.
Sign up today and you will receive a free copy of our Future Focus 2026 report - the leading resource for IT decision-maker insight on priorities and investment areas in AI, security and more.
“Our object-attribute detection model can detect 1,594 object classes and 524 visual attributes. As a result, the model can detect and encode nearly all the semantically meaningful regions in an input image, according to our experiments.”
Despite the promising results, Microsoft said its model is by no means close to the human-level VL understanding.
Microsoft also announced VinVL would be available to the public for general use. Additionally, it will integrate VinVL into Azure Cognitive Services to power a wide range of Microsoft services, including Image Captioning in Office and LinkedIn, and Seeing AI.
-
TfL takes Oyster photocard systems offline following breachNews Security experts warn the Oyster photocard breach could lead to follow-up attacks on victims
-
Irish credit unions hit by data breach after IT provider compromisedNews Wellington IT reportedly delayed alerting customers by multiple weeks
-
‘Businesses are being steered to Azure by financial penalties rather than by merit’: US IT leaders hit out at Microsoft licensing practices in call to action for regulatorsNews Technology leaders across the US say they're choosing Azure mainly to avoid fees, and are calling for regulators to step in
-
Microsoft is ending support for the Remote Desktop app – here are three alternatives you can try insteadNews Microsoft has announced plans to end support for its Remote Desktop application in just over two months.
-
Microsoft sharpens agentic AI focus with new ‘CoreAI’ divisionThe new department will bring together teams from across engineering and AI
-
Microsoft unveils Azure AI Foundry to help businesses consolidate AI across their tech stackNews The new AI application creation and management platform will integrate Azure AI Studio, and plug into popular IDEs such as Visual Studio to help developers and IT professionals drive AI adoption
-
Microsoft's huge AI spending has investors worried – now the company is changing its financial reporting to highlight successesNews The move comes as investors want more evidence that Microsoft’s AI investment will pay off
-
Hyperscaler AI spending is getting out of control — and Microsoft says it could take 15 years for it to make good on investmentsNews Tech giants' results show billions being poured into AI infrastructure, but big leaps in revenue remain elusive
-
Microsoft’s Mistral AI partnership has EU regulators concerned - here’s whyNews Microsoft has found its European champion in Mistral AI, but regulators worry the deal raises questions over the scope of control and influence the tech giant will have over the French startup
-
Generative AI has been the secret sauce for Microsoft as its market cap surpasses AppleAnalysis Microsoft has ramped up generative AI investment over the last year and its surging share price has been a reflection of its sharpened focus on the emerging technology