From Bard to Gemini: two years on, Google still cannot catch ChatGPT

Two years ago Google unveiled Bard, later rebranded as Gemini, as its answer to ChatGPT. The search giant had the makings of a worthy rival: access to up-to-date data, hefty computing power, in-house AI chips, hundreds of specialists and billions in investment. Yet the smaller OpenAI, albeit backed by Microsoft, has kept the lead, leaving Google unexpectedly in pursuit.

Why did that happen? ForkLog traced the development of both chatbots: their new features, day-to-day use and the key differences between the assistants today. Beyond numbers and benchmarks, we looked at real-world experience with each system.

A rocky start: Bard’s first steps

On the face of it, Bard did not stand out from rivals: the same prompt box and a dialogue-style output window. The notable distinction was its ability to tap Google Search for fresh information on the web.

From the outset Bard ran into a serious problem: critical inaccuracies in its answers. Other chatbots suffered similar lapses, but at Google the scale was unprecedented. The starkest example—a mistake in Bard’s own promotional materials—knocked $100bn off Alphabet’s market value within a day of the announcement.

The rush was to blame. Google hurried to ship a response to ChatGPT and Microsoft Copilot (which runs on OpenAI’s technology). The firm enlisted its own staff to test the bot, asking them to manually fix incorrect answers to smooth user interactions.

Matters were worsened by relying on the older LaMDA language model even though the company already had the more advanced PaLM.

At the time the service was in beta with a limited number of users. The public release arrived in July 2023 and, despite the rough start, Bard built an audience.

According to data from Coolest Gadgets, in 2023 the chatbot’s website drew more than 142m monthly visits. For comparison, ChatGPT’s peak in May that year reached 650m.

The road to refinement: from LaMDA to Gemini 2.0

Over two years Google’s chatbot travelled far. In April 2023 the company’s CEO Sundar Pichai announced Bard’s move to a more powerful model—from LaMDA (137bn parameters) to PaLM 2 (540bn). He said the initial choice of LaMDA reflected its modest compute demands, which allowed a quicker launch and faster collection of user feedback.

The switch to PaLM 2 markedly improved Bard’s capabilities. That is borne out by MMLU (Massive Multitask Language Understanding) tests—a standard set of 16,000 questions across 57 disciplines, from maths and physics to law. The benchmark gauges a model’s ability to tackle varied tasks: the higher the share of correct answers, the more versatile the system.

PaLM 2 scored 78.3%, handily beating GPT-3.5 (70%), which powered ChatGPT in early 2023. But in March OpenAI released GPT-4 with 86.4%, again putting Google behind.

The Gemini era: a new phase

The turning point came in December 2023 with the shift to Gemini, built by DeepMind. Three versions—Ultra, Pro and Nano—improved text, code and maths generation. It was Google’s first public multimodal model for analysing images, video and audio—a capability GPT-4 had added back in March 2023.

In MMLU tests Gemini Ultra narrowed the gap, posting 83.7%. Even so, by May 2024 OpenAI had introduced GPT-4o with 88.7%.

In February 2024 Gemini 1.5 arrived with a Mix of Experts approach, in which specialised networks tackle complex tasks in parts. The context window expanded to 1m tokens—the equivalent of one hour of video, 11 hours of audio or 700,000 words. In December of the same year Gemini 2.0 Flash Experimental was released with improved performance and new features, including real-time audio and video interaction.

Google lavished attention on mobile. In February 2024 it launched an Android app and integration with Google on iOS. Android users could replace Assistant with Gemini, although at first it lacked basics—it could not place calls, control smart-home devices or even set an alarm.

That changed after the Google I/O conference in May 2024. Gemini was integrated with Chrome, Photos, Workspace and other services via extensions. The smartphone version now can:

• work with Google Docs and Gmail;
• analyse screenshots;
• control smart-home devices;
• manage basic phone functions;
• search for travel information;
• control playback in YouTube, YouTube Music and Spotify.

One notable integration is in YouTube: while watching a video, users can ask the bot to summarise the content or find a specific moment. The feature remains temperamental, though.

In August, with the announcement of Pixel 9, Gemini Live appeared—an always-on voice mode with realistic delivery, similar to the ChatGPT feature introduced in September 2023.

In parallel with the technical progress, Google launched Gemini Advanced, a paid subscription that provides access to the most powerful Ultra model. The service lets users create personalised Gem chatbots tailored to specific professional or personal tasks, broadening the platform’s commercial appeal.

Despite a sizable expansion of everyday features, Gemini still trails competitors in information search and question answering.

On the AI front line: Gemini’s progress and problems

Information search remains the main use case for most users—ChatGPT was even dubbed a potential “killer” of Google. That has not come to pass, but OpenAI nonetheless integrated search functions into its product, underscoring demand for the format.

Chatbots have undoubtedly mastered many specialised tasks: writing copy, generating code, drafting plans and helping with schoolwork. These, however, appeal mainly to specific groups—copywriters, developers, managers and students. Searching for information unites everyone.

At first glance Gemini looks competent here. With deeper use, though, certain quirks emerge: the bot often redirects the user to Google Search instead of answering directly.

Gemini’s treatment of potentially contentious topics also stands out. The chatbot can be overly cautious, refusing to discuss even harmless internet memes or to provide factual information about legislative processes, mistakenly classifying them as unacceptable content.

Though ChatGPT has become more restrained than at launch, it still shows a more balanced approach. With delicate topics it more often provides contextual explanations rather than an outright refusal. That makes interactions more productive—users spend less time rephrasing prompts and get more useful feedback.

For all its quirks, Gemini handles most everyday tasks well. Like any AI assistant, it warrants a critical eye—advanced language models can still produce inaccurate information.

Gemini’s prospects

In two years the journey from Bard to Gemini has been an impressive evolution—from a simple browser chatbot to an integrated helper across Google’s ecosystem. In the near term the service will appear on Nest devices, Google TV sets and Wear OS watches. Given Android’s dominance in mobile—over 70% and 3.9bn devices—the potential audience for Gemini is vast.

Yet the initial problems persist: shakiness in information search despite integration with Google Search, excessive caution in answers and difficulty acknowledging its own mistakes. Even with formidable resources, Google has not yet caught rivals on the chatbot’s core jobs.

Even so, the breadth of Gemini’s integration with everyday technology is promising. If the current shortcomings are fixed, this assistant could become an indispensable tool in Google’s ecosystem.