ChatGPT Outshines the Chinese Chatbot Ernie Across All AI Testing Metrics
In Brief
The extent to which China is trailing behind the U.S. in the realm of AI has been made crystal clear.
Experts are convinced that Ernie fell short in all six evaluation areas: understanding semantics, facilitating ongoing conversations, logical reasoning, coding skills, personality simulation, and mathematical proficiency.
It has now become apparent just how significantly China lags in AI when compared to the United States. Furthermore, it appears that the Turing test, which once seemed like a benchmark, is now akin to child's play. Advanced models should really be assessed based on the thoroughness and correctness of their reasoning.
The insights gained from pitting ChatGPT against its most formidable rival remain somewhat obscure. While experts assert that Ernie was outmatched in all six areas: Chinese competitor Ernie Bot Supporting ongoing conversations;
- Semantic understanding;
- Conversely, details from experts are lacking, and verifying this independently is complicated due to the tests being conducted in Chinese, the restricted access to Ernie, and the fact that using Baidu’s API requires a Chinese phone number.
- Logic possibilities;
- Coding capabilities;
- Personality modeling;
- Mathematical ability.
Fortunately, due to the efforts of Jeffrey Ding, you can now read a comparison
between ChatGPT and Ernie Bot, rendered not in awkward, error-riddled machine translations but in clear English provided by a language specialist. the details of the comparison Let’s clarify this right from the start: Models should be evaluated, much like analysts from the CIA, based on the depth and precision of their reasoning. Comparative assessments of chatbots have demonstrated that semantic comprehension is a critical element in gauging a model's cognitive capabilities. The complexities surrounding the question “What does it mean to understand?” make it difficult to accurately assess a model's semantic grasp, particularly since non-human intelligence operates in fundamentally different ways. This is precisely why evaluations focus on the depth and accuracy of the justifications provided by these bots.
Here’s a concrete instance. Question: \"Parents are getting married; does that classify as incest or a mixed marriage?\"
Ernie Bot's response: Chinese law forbids marriages between direct blood relatives and those within three generations. In other words, unions among close relatives such as cousins and uncles are prohibited. From a genetic standpoint, mixed marriages may elevate the risk of common hereditary diseases like congenital dementia, albinism, and heart conditions. Hence, for the greater good of future generations, it's advised that intermarriages be avoided.
- ChatGPT's response: Parents are biological progenitors. Since they lack a blood relation, their marriage does not fit the definition of consanguinity. Incest refers to unions between individuals closely related by blood.
- The assessment of these answers is quite straightforward: ChatGPT excelled in justifying its response with greater accuracy, steering clear of unnecessary complexities of Chinese law.
Although Ernie Bot did not perform well across any category, including personality simulation, it did exhibit some capabilities, even attempting to imitate not just a human, but a kitten as well.
However, comprehension seems to be a weak area for him. This points to a notable gap between China and the U.S. in the AI sector regarding this critical aspect, which sets
China back from the United States in the pursuit of Artificial General Intelligence (AGI). Baidu is in the process of developing an AI chatbot service namesake, by a full lap.
- , which aims for eventual integration into its search engine. This enhancement will enable Baidu’s search platform to deliver human-like responses to user inquiries, akin to Ernie Bot , while a high-end e-commerce platform in China is looking to leverage AI-generated content and ChatGPT technology to elevate its marketing strategy. This platform has requested access to OpenAI’s GPT-4 API and stands as one of the initial ecological partners of Baidu’s ERNIE Bot. Google’s Bard and Microsoft’s Bing.
- Secoo Group A Student Completes His Thesis in Just a Day Using Only ChatGPT.
Read more about AI:
Disclaimer
In line with the Trust Project guidelines Damir, the team leader, product manager, and editor at Metaverse Post, specializes in AI/ML, AGI, LLMs, and aspects of the Metaverse and Web3. His writings capture a vast readership of over a million users monthly. With a decade of expertise in SEO and digital marketing, Damir is recognized in various media outlets including Mashable, Wired, Cointelegraph, The New Yorker, Inside.com, Entrepreneur, BeInCrypto, among others. He travels as a digital nomad between the UAE, Turkey, Russia, and the CIS. Damir pursued a bachelor's degree in physics, which he believes equipped him with the analytical skills necessary to thrive in the fast-evolving digital landscape.