Tuesday, 20 May 2014

Cortana vs. Siri vs. Google Now: An early look at how Cortana stacks up (hands-on)

WINDOWS PHONE 8.1

Like Siri and, to a lesser extent, Google's voice actions, Cortana is a personable (or vaguely person-like) voice-activated system for taking dictation, looking things up, and opening apps. (Yes, Halo fans -- Cortana is named for Master Chief's AI companion, and even voiced by the same talent: actress Jen Taylor.) Even in its early beta stage, Cortana mostly keeps pace with its rivals, and introduces one or two minor innovations that Apple and Google can learn from.

While Cortana isn't revolutionizing the field of voice assistants, it does give Windows phones a sorely-needed boost in the voice option department -- the previous generation, fueled byTellMe, was limited and impersonal. Out of all the additions to the Windows Phone 8.1, it's Cortana that pulls the most weight keeping the Windows Phone in the OS game.

Before we begin

There are two ways to start Cortana: by pressing the capacitive Search button below the screen, or by manually launching the app from the phone's start menu or app tray.

I tested Cortana on a Nokia Lumia Iconpreloaded with a pre-final version of Windows Phone 8.1. The OS update is designed to work on all version 8.0 phones, but update times will depend on individual vendors and carriers. Expect Lumia phones to receive the update over-the-air by summer, for instance. New devices like the Nokia Lumia 930630, and 635 will ship with 8.1 starting in late April and early May, and developers can get their hands on the new OS on April 14.

Now, Cortana gets it (or, if you're Microsoft, her) knowledge from your personal usage patterns, and also from a database of users. That means Cortana's ability to understand natural language (versus strict commands) and accents will improve as more people use it.

Since CNET got Cortana and Windows Phone 8.1 review units before even Microsoft's developer partners, the results of my testing may not be as sophisticated as they will be when more people start using Cortana to search.

Just one more qualification before we get going here: as I said before, this is an early summary and test, so we're reserving our final evaluation of Cortana until after it enters the mainstream. At that point, we also plan to revisit this battle and see if the circumstances change.

Oh, and also, this is a comparison of Cortana's voice capabilities with Siri and Google Now in a range of scenarios. Google Now has many predictive features that it displays in card form; however, that's not what we're evaluating today.

Versus Siri and Google Now

To get into the meat of the voice testing, I spoke the same commands to the most up-to-date versions of Cortana (on the Icon running Windows Phone 8.1), Siri (on an iPhone 5 running iOS 7.1), and Google Now (on the Samsung Galaxy S5 running Android 4.4.2), often with more than one syntax to see if phrasing makes a difference in the result. Below, I compare some of the more common activities you'll demand of a voice assistant, with a few outliers thrown in to see how each system behaves.

1. Calls, texts, and email

Launching calls, texts, and email is one of voice commands' most common actions. Cortana handily recognizes text and call commands. It can even take slightly more complex dictation, say: "Call Mom and Dad on speakerphone," just like Siri (Google Voice won't do this).

cortanadictation.jpg
Of the three voice assistants, Siri handled punctuation commands best.
Screenshot by Jessica Dolcourt/CNET

But Cortana doesn't compose email using either the terms "Send email to" or "Compose email to." That's a big hole for now, but hopefully one that Microsoft will plug with the next update. (Note: You can tell Cortana to open Outlook, then press the microphone button to dictate there, but it's a clumsier workaround.)

Another area that Microsoft needs to improve is its understanding of punctuation when composing messages by voice. Say: "Hey [comma] how's it going [question mark]," and the voice assistant should always spell out "Hey, how's it going?" Cortana, however, takes you literally, adding the words "comma" and "question mark," instead of inserting their respective symbols. Siri got both symbols; Google's system got one.

2. Alarms and appointments

All three voice assistants adeptly set alarms for specific times, say 6:00 a.m. ("Set alarm for 6:00 a.m."), as well as for spans of time, such as "Wake me up in 5 minutes." A little more interesting than that, Cortana and Siri will vocally cancel an alarm ("Cancel alarm for 6:00 a.m."), but Google Voice will not.

cortanaalarmcancel.jpg
Cortana and Siri could cancel an alarm.
Screenshot by Jessica Dolcourt/CNET

The same goes for calendar appointments; they're easy to schedule ("Schedule appointment for...", but you'll have to manually remove them in Cortana and Google Now. Siri alone will strike appointments from the record ("Cancel appointment for...").

3. Apps and system settings

Cortana, Siri, and Google's voice actions all open downloaded apps in response to your voice ("Open Yelp"), but only Cortana and Siri manipulate settings; for instance, toggling Wi-Fi and Bluetooth off and on ("Turn Wi-Fi on," or "Turn on Wi-Fi").

cortanawifi.jpg
Google Now can't manipulate system settings, but Cortana and Siri can.
Screenshot by Jessica Dolcourt/CNET

4. Facts and figures, flight status

Ask the three assistants for Genghis Khan's birthday, the height of Mt. Everest, sports scores, or the weather, and you'll get the same answers presented in an easy-to-read format that sets apart the response (above) from more detailed information or Web links (below).

They all also convert currency and can tell you the time in another city. Siri provided the most context with both of these, adding visual elements of a clock and a currency graph, respectively. Strangely, Cortana could tell me the time in Mumbai and London, but not Taipei (I got a list of search results instead.) This is likely a function of Cortana's beta status and something that Microsoft will build out.

cortanacurrency.jpg
Siri provided the most additional context when it comes to currency conversion.
Screenshot by Jessica Dolcourt/CNET

Depending on what you ask and how you ask it, the results may differ among the assistants. They can be a little finicky on your articulation and exact wording. Siri, for example, didn't understand when I asked "What time is Captain America playing tonight?" I got the MLB schedule for the Mets game instead (on more than once occasion). When I worded the phrase slightly -- "What are showtimes for Captain America tonight?" -- I was correctly shown movie times.

A similar oopsie occurred with Cortana, which displayed the film schedule when I searched for Captain America, but then presented a list of Bing results when I asked about "The Lego Movie."

cortanaflightstatus.jpg
Tracking a flight isn't Siri's forte, but it's something that Google Now and Cortana can do.
Screenshot by Jessica Dolcourt/CNET

Cortana's claim of flight status worked well when I asked it, "What's the status of United 931?" Google Now kept pace, with the same results posted clearly on a card, but Siri returned my query with a Web search, not with an at-a-glance flight status.

5. Directions

Voice-activated directions worked flawlessly for all three assistants ("Navigate me to Palo Alto.")

Looking up addresses (that you don't necessarily want directions for) is also something you can do. When asked, "What is the address for the Exploratorium?", all three returned the correct response...eventually. On my first attempt, which was outdoors, only Siri returned a specific result. The other two proffered up a list of links.

cortanaexploratorium.jpg
This time it's Cortana that offers up the greatest context, including a phone number and a prompt to get walking directions.
Screenshot by Jessica Dolcourt/CNET

There was one other on-demand command that only tripped up Google Now, and that was asking "How long will it take me to get home?" You'll get search results instead of a card response. That said, Google Now cards proactively predict your commute, so that information is available.

6. Reminders and relationships

Siri, Google Now, and Cortana all set simple reminders, like "Remind me to make a dentist appointment on Tuesday," but Cortana wants to nail down the time. Where Siri will set the reminder without a time commitment, Cortana -- and Google Now -- will prompt you to lock down your schedule.

One unique, clever Cortana trick links reminders to contacts. For instance, "The next time Dan and I talk, remind me to ask about the game." If more than one contact fits the description, Cortana will prompt you to pick from a list of contacts. The next time either of you texts or calls, a screen dominated by Cortana's trademark red circle pops up with the reminder, which you can complete or postpone for next time.

cortanareminderpopup.jpg
Link a reminder with a person or a place, and Cortana will pop up an alert tied to your location, or a call or text.
Screenshot by Jessica Dolcourt/CNET

Like Siri, Cortana lets you set relationships for people and places as well, which lets you refer to people as "Mom" or "Dad," "my brother," or simply "Amy," rather than speak the person's first and last name to place a call or text, for example.

In Cortana, you set these relationships manually through the Inner Circle area that's part of the settings menu that Microsoft calls Cortana's Notebook. With Siri and Google Now, however, you can just speak the relationship simply, like "Aidan is my brother-in-law," and then it is so. Microsoft should mimic this more convenient approach.

cortanarelationship.jpg
Cortana is the only voice assistant that requires you to manually add relationships.
Screenshot by Jessica Dolcourt/CNET

7. Music identification and playback

Windows Phone has had Shazam-style music identification rolled into the Bing search app for some time, so its placement in Cortana is inevitable. When music is playing, launch Cortana, then tap the music note to get it to listen. It works about the same in Google Search app. Siri doesn't have this capability yet; Apple's assistant just listens and does nothing.

I do think that Google is a little smarter about the info it returns, making it clear in the results that you can buy the tune from the Google Play Store. Microsoft's Cortana will also hook into the phone's music store, but only after you tap the entry. Neither provides the deep context around the artist, album, or lyrics the way that dedicated apps like SoundHound do.

cortana-musicid.jpg
Cortana and Google Now can identify songs.
Screenshot by Jessica Dolcourt/CNET

In addition to recognizing music, Cortana -- and Siri, and Google Now -- can play songs you already own when you simply say "Play Fun" or "Play Some Nights." You can say "Stop music" or "Pause music" to halt the tune. In theory, you should be able to tell Cortana "Stop [the name of the band or song]", but Cortana interpreted that command as a new Web search.

So what happens when you ask to see a song you don't own? Siri tells you it isn't in your library, but Google Now and Cortana at least provide links so you can play it yourself online. Google's search result lists the YouTube video first; Cortana serves up Last.fm.

8. Follow-up search commands

There are a few instances when you and Cortana can take turns to whittle down a search. Right now it works when you search for a business or look up the weather. For example, "Show me the weather," followed by "What's it like on Thursday?" -- no need to specify that you're still asking after the climate. Google's voice action and Siri do this as well.

What is unique to Cortana being able to follow up on questions related to looking up businesses. Say: "Show me Italian restaurants that are open in the Financial District tonight" and you'll see a list pulled from Bing's local search. Siri lists off restaurant names, too, but Google Voice returns URLs that are related to "restaurant."

cortanarestaurants.jpg
Remembering the context of certain searches is a Cortana specialty.
Screenshot by Jessica Dolcourt/CNET

Cortana alone remains contextually aware around this search, which lets you continue to request anything from distance (show me the ones within five miles), star ratings, even commands to call a business on the list or find out more about it. For example, you can say: "How long will it take me to get to Barbacco?" and then "How long will it take me to walk?"

Siri treats follow-up requests like this as a new query.

9. Cheeky personality

Microsoft bills Cortana as a cheeky monkey, sassier even than Siri. I'm not so sure of that. It really depends on what you ask. We had a lot of fun passing Cortana around the office to see what "she" would say.

Cortana will toss back a few witty remarks when you ask questions about the story of Halo, if she loves you, what she's wearing (a phone!), and who's her daddy. She also has a ready retort for questions about Siri and for questions about when she was born -- but not about where.

cortanapersonality1.jpg
Cortana was programmed with a little sass.
Screenshot by Jessica Dolcourt/CNET

Cortana will sing you song (actually, it's only a few lines long and it's kind of creepy) and tell you a joke (it did make me chuckle). Cortana will respond to your gratitude ("You're awesome.") and to vitriol as well ("You suck," but not "thanks for nothing").

So far, so good. However, when Cortana is finished amusing you, show's over. Your next comment will launch a new search. Siri, however, will play ball for awhile longer, allowing you to stretch out some semblance of a back-and-forth conversation. For example:

Me: You're awesome.

Siri: I am?

Me: Yeah, you are.

Siri: That's what I figured.

Me: You have no shame.

Siri: Is that so.

Me: Yeah, it is. Smartass.

Siri: That's what I thought.

cortanapersonality2.jpg
Some jokes are actually pretty humorous.
Screenshot by Jessica Dolcourt/CNET

Like Cortana, Siri is just pulling from a database of ready-made responses, but the exchange hangs together. Here's what happened when I riffed the script with Cortana:

Me: You're awesome.

Cortana: Thanks! I don't need to sleep so I have lots of time to practice.

Me: Very funny.

Cortana: [Pulls up Web search for "very funny" pictures and quotes.]

Google is the most staid of the three, by a long shot, and that's simply because Google hasn't chosen to anthropomorphize its search app. Tell Google Now it's awesome and it just creates a new Web search.

10. Cortana's own stuff

In addition to catching up to Siri and Google Now, Microsoft also wants to carve out its own style. Most of this takes place in the voice app's settings menu, called Cortana's Notebook.

Here you set your interests, look at reminders, name places you visit most often, and reference music searches. Quiet Hours is where you can set a no-call and no-texting zone, like when you're in a meeting or catching up on some Z's.

cortanasettings.jpg
With Cortana, Microsoft offers its own take on voice assistance.
Screenshot by Jessica Dolcourt/CNET

To designate an exception to the Quiet Hours rule, nip into the Inner Circle setting. This is also where you establish relationships, e.g., "Amy is my sister."

Cortana also gives you somewhat of a newsfeed. Even when Cortana is idle, you can swipe down to see suggestions for locations, headlines, the weather, and other items that Cortana thinks you might like -- this is based on your searches and also on interests you've set up.

Google will also preemptively give you location cards, but iOS's Siri keeps it simple with just its electronic ears.

How well does it work?

All told, Cortana kept pace with the more established voice assistants, both in terms of its abilities and in executing tasks correctly. Although not every command I tried worked perfectly well for any of the programs, Cortana, like Siri and Google Now, never crashed.

For the most part, Cortana understood my commands and complied, though (again, like the others), it struggled in noisier environments and couldn't always filter out cross-talk.

Ideally, voice assistants should act quicker than typing, and that's often the case when Cortana understands you the first time. In some instances, though, it would have been faster to manually perform a task than to repeat a similar request -- this goes for Siri and Google voice actions as well.

The Cortana rundown: Pros and cons

There's still much to explore with Cortana, especially after its understanding of casual language and accents grows as more people use it.

So far, Cortana blends Google's and Apple's voice actions capabilities, while bringing some of its own ideas to the table -- like the concept of quiet hours, and reminders that pop up when you link them with certain people.

cortanastill.jpg
Cortana's symbol? Its concentric red circle.
CNET

Microsoft does need to strengthen a few soft spots, however. At times, Cortana gave me a list of Bing search results when I really wanted my info displayed neatly in a card format. Cortana also needs to be able to cancel every task it can create, understand spoken punctuation, and let you edit searches in the written search field.

I'd also like a shortcut to launch Cortana's verbal commands without having to tap the on-screen microphone button first. Siri isn't completely hands-free, but it's the easiest protocol to launch, and depending on the Android phone in question, you can trigger Google's voice actions without tapping a thing.

As much as Microsoft touts the idea of using naturally phrased language instead of barking out a sequence of commands, the words you say often matter, as well as the order in which you say them.

For instance, you can say "The next time I talk to Mom, remind me to ask about what she wants to do for her birthday," but it won't understand if you say "Remind me to ask Mom what she wants for her birthday the next time we talk." Cortana isn't alone in this partial freedom of speech. Siri and Google Now also get particular at times about phraseology.

No comments:

Post a Comment