Apple Introduces Open Source Multimodal LLM, Ferret

- Advertisement -
- Advertisement -

The multimodal LLM can use parts of images as queries using the GRIT Dataset consists around 1.1Mn examples.

Apple Inc. in collaboration with Columbia University’s AI researchers has quietly introduced an open-source multimodal large language model named “Ferret.” This model, unveiled on GitHub in October, gained significant attention from the AI research community, despite no official announcement.

Ferret is trained on 8 A100 GPUs with 80GB memory. The dataset used in the project is governed by the CC BY NC 4.0 licence, which permits non-commercial use only. The key contributions of the project include the Ferret model, GRIT dataset and Ferret-Bench.

- Advertisement -

The Ferret model combines a hybrid region representation with a spatial-aware visual sampler to enable fine-grained and open-vocabulary referring and grounding within a multimodal large language model (MLLM). This capability enhances the model’s ability to understand and respond to complex queries that involve both text and images.

The project introduces the GRIT Dataset, which consists of approximately 1.1 million examples. This dataset is designed to support large-scale, hierarchical, and robust instruction tuning for grounding and referring tasks. It serves as a valuable resource for training and evaluating AI models in tasks related to understanding and responding to instructions.

Ferret-Bench is a multimodal evaluation benchmark created as part of the project. It is designed to assess the performance of AI models across various dimensions, including Referring/Grounding, Semantics, Knowledge, and Reasoning. This benchmark provides a comprehensive testing ground for evaluating the capabilities of models like Ferret in real-world scenarios.

Ferret is described as a model that can use parts of images as queries, making it a powerful multimodal AI system. Its working involves examination of a specific region of an image. It then identifies elements within that region that could be relevant to a query and draws bounding boxes around these elements. Then it uses the identified elements as part of a query to provide responses in a traditional language model manner.

This means if a user highlights an image of an animal within a larger image and asks what the animal is, Ferret identifies the species of the creature and can use context from other elements in the image to provide further information or context.

The release of Ferret is seen as significant because it represents an unexpected level of openness from Apple, a company known for its secrecy. This open-source approach contrasts with Apple’s traditional practices.

One reason for this openness may be Apple’s need to compete in the AI industry, where it faces challenges from rivals like Microsoft and Google. Apple’s infrastructure is not optimised for serving large language models (LLMs) at scale, which puts it at a disadvantage. To address this, Apple must choose between partnering with cloud hyperscalers for AI or sharing its work with the open-source community, a strategy similar to what Meta Platforms Inc. (formerly Facebook) has adopted.

Ferret’s release demonstrates Apple’s willingness to collaborate and contribute to the AI research community, reflecting a shift in its approach to AI development.

- Advertisement -

Related Artcles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Exclusive

How Effective Will The Global Biofuel Alliance Be?

0
From environmental benefits to economic opportunities, the potential of biofuels in revolutionising our energy landscape. India has made history by hosting the 18th G-20 summit...
Marina

Investments In Drone Industry Decline To US$1.7 billion

0
The 2023 data now separates drone company funding from AAM/eVTOL funding, which significantly boosted the total in 2020 and 2021. Drone companies received US$1.7 billion...

“Now’s A Great Time To Make In India”

0
In news and in current affairs relevant to the current geopolitical situation, India is emerging as one of the fastest-growing electronics manufacturing services nations...

Buzz

Quantum Circuits Logo

Quantum Circuits Appoints Ray Smets As CEO

0
His contributions to mobile networking technology have earned him over ten international patents. Quantum Circuits, Inc., a pioneer in fault-tolerant quantum computing, has announced the...

Jakson Green Signs Power Purchase Agreement With SECI

0
It is estimated to power 100,000 households annually, reduce 188,000 MT of carbon emissions each year, and boost job opportunities. Jakson Green, a leading player...

ZF Expands Aftermarket Commercial Vehicle Range With ADAS Sensors

0
Advanced Driver Assistance Systems (ADAS) are gaining traction in the commercial vehicle market. Since 2015, trucks have been required to have automatic emergency braking...

Important Sectors

Feb EV Sales: Electric Car Sales Decline, Startups Rise Once Again

0
Electric two-wheelers and electric three-wheelers continue to dominate the electric vehicle sales figures in India. Electric car adoption, on a larger scale, seems to...

Jakson Green Signs Power Purchase Agreement With SECI

0
It is estimated to power 100,000 households annually, reduce 188,000 MT of carbon emissions each year, and boost job opportunities. Jakson Green, a leading player...

ZF Expands Aftermarket Commercial Vehicle Range With ADAS Sensors

0
Advanced Driver Assistance Systems (ADAS) are gaining traction in the commercial vehicle market. Since 2015, trucks have been required to have automatic emergency braking...

Samsung Semiconductor India Research Launches New R&D Facility in Bangalore

0
The facility spans 1.6 lakh square feet across four floors and is designed to accommodate around 1,600 employees. Samsung Semiconductor India Research (SSIR) has inaugurated...
Intel, Digital India, CPU

Intel Eyes AI For Standalone Programmable Chip Unit Altera

0
The contract manufacturing customers, including Intel's own business units, may receive favorable pricing.  Intel's standalone programmable chip unit, Altera, is eyeing a significant opportunity in...

Manufacturing

Union Cabinet Approves CG-Led JV Proposal For OSAT Facility In Sanand

0
CG Power owns a 92.3% stake, while Renesas Electronics and Thai OSAT provider Stars Microelectronics will have 6.8% and 0.9% equity, respectively in the...

Govt Sanctions 3 Semiconductor Plants; Rs 1.26 Lakh Cr Investment

0
The Dholera (Gujarat) semiconductor fab will produce 50,000 wafers monthly. In contrast, the Morigaon (Assam) and Sanand (Gujarat) units will assemble, test, monitor, and...

“Now’s A Great Time To Make In India”

0
In news and in current affairs relevant to the current geopolitical situation, India is emerging as one of the fastest-growing electronics manufacturing services nations...

MeitY Secretary: Domestic Semiconductor Manufacturing Is The Next Big Thing

0
At the India Digital Summit (IDS) 2024, S Krishnan, Secretary of MeitY, highlighted that India is on the brink of a major move towards...

Tata Picks Somerset For UK Battery Plant

0
The Bridgewater factory, operated by the subsidiary Agratas, will initially produce batteries for JLR and Tata Motors; with a capacity of 40GWh, it will...