Skip to main content

Siri answered 78% of questions correctly in latest test, up from 66% a year ago

Analysts at The Loup Ventures have published the results of the their latest voice assistant comparison test, which involves asking the leading digital assistants 800 questions spanning various domains. Whilst Siri still trails Google Assistant in accuracy, it has improved significantly.

With the same test conducted last year, Siri correctly answered 66% of the questions. The new testing shows Siri is getting smarter, jumping to a 78.5% accuracy rate.

The Loup Ventures report breaks down queries that were answered correctly, and queries that were parsed and understood correctly but the assistant could not deliver an appropriate response. At least on the 800 questions asked in this test, Siri apparently understood 99% of them, with a 78.5% correct answer rate.

Google Assistant had an even better record, understanding 100% of queries and answering 85% of them with an appropriate reply. Both Alexa and Cortana scores poorly on this survey, but to be fair to Amazon and Microsoft, their assistant tests were conducted using the client iOS apps, not the manufacturer’s own hardware.

The test includes a variety of questions including queries for mapping (‘Where is the nearest coffee shop?’), shopping (‘Can you order me more paper towels’), trivia and commands (like setting reminders or changing device settings).

 

Siri performed exceptionally well with questions in the Command category. It seems a lot of this comes from Siri’s handling of smart home and music questions, which Apple have obviously paid attention to recently because of the HomePod’s debut.

Obviously, these scores do not suggest that Siri can answer 80% of questions successfully first time. Scoring is heavily dependent on the questions asked, and The Loup Ventures do not publish a list of all 800 questions surveyed. They merely claim to represent a sampling of the most important tasks customers expect from voice assistants. It’s not difficult to invent esoteric questions that none of the leading smart assistants could understand, so there’s always a degree of subjectivity here.

The best takeaway is probably the headline that Siri’s recognition is going up, with a 13 percentage point increase year-over-year. That being said, ‘Siri sucks’ is still very much a meme in the community.

Apple has recently hired John Giannandrea as VP Machine Learning and AI Strategy, to oversee Siri and related projects. Giannandrea previously worked at Google on projects including the Assistant, so it’ll be interesting to see how the fresh talent can advance Apple’s efforts in the personal assistant field.

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Mac — experts who break news about Apple and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Mac on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel

Comments

Author

Avatar for Benjamin Mayo Benjamin Mayo

Benjamin develops iOS apps professionally and covers Apple news and rumors for 9to5Mac. Listen to Benjamin, every week, on the Happy Hour podcast. Check out his personal blog. Message Benjamin over email or Twitter.