Copilot+ PC launch live blog: Testing the new Snapdragon Laptops

Slow Setup Begins

So we're in the middle of setting up our Surface Laptop 15-inch with Snapdragon Elite X CPU and it is taking . . . forever. As part of the OOBE (Out of the Box Experience), Windows 11 demanded to run updates, which it claims will take 30 minutes.

During the update process, we've been treated to promotional images of the various Copilot+ PC features. Of the Recall feature, the OOBE says it's "coming soon."

Image 1 of 4

Early testing of Asus Vivobook S15 debuts

A few outlets, including Tom's Guide and Windows Central, have posted hands-on and early testing of the Asus Vivobook S15. The latter has a full review, which it gave 4.5 out of 5 stars.

Both sites praised the Snapdragon X Elite's performance (specifically, the 45W TDP X1E78100), as well as the 15.6-inch OLED display. While Windows Central deemed the battery life just "good," Tom's Guide called it "mind-blowing."

TG listed the following benchmark scores, which it says aren't from its official lab testing:

Geekbench 6 single-core: 2,418
Geekbench 6 multi-core: 14,352
Handbrake: 6:50

All of these scores and times were faster than the M3 MacBook Air the site tested. WC also found it trading blows with certain modern Intel and AMD processors in some tests. Neither site was able to test the NPU, partially due to driver issues.

WC also points out that Arm emulation is better than ever with far more compatibility than before, though it still had issues with Adobe Premiere. Additionally, while a few games loaded well, others couldn't be installed or would refuse an ARM CPU.

Check out both sites' reviews and hands-ons for more.

Paint's Cocreator Requires Internet Connection, Doesn't Follow Instructions

We just tried Cocreator, a new Windows Paint feature that is one of the four big new features Microsoft offers exclusively for Copilot+ PCs. It's a a button in Paint that lets you generate images based on a combination of your drawings and a text prompt. 

Color us unimpressed with the feature. Not only does it not follow directions to a T, it also turns out to require an Internet connection as you can see above. While Microsoft never explicitly said that its new Copilot+ features would work offline, it seems ridiculous that these features require a system with a powerful NPU but then send their data to the cloud anyway.

So how does it work? We hit the Cocreator button in the upper right corner of the screen, which is not to be confused with the "Image Creator" button that does something similar but is not exclusive to Copilot+ PCs. You're then shown a sidebar that tells you to "describe the image you'd like to create" and invites you to draw something. So you HAVE TO do both. You can't just give a text prompt and you can't just draw something.

Just want to use text prompts? That's what Image Creator is for, but Image Creator not only requires the cloud, but also has a limited number of tokens you can use (50 by default) before you run out and presumably have to pay for extras.

I wanted to get a picture of a cat driving a car so I drove a circle, a line and an oval with two wheels on it. I used the prompt "cat driving a car," but Cocreator didn't exactly follow my directions, showing just a plain cat with something under its arm. When I changed the prompt to "cat on a skateboard," I got a blurry skateboard below my cat.

I had the Windows Task Manager open and watched as the NPU surged to 100 percent for a couple of seconds while Cocreator was redrawing. So it is using the processor, even though it won't work without an Internet connection.

Incidentally, Cocreator is not close to immediate. When changing the prompt or the drawing, there was a lag of two to three seconds for redrawing. This is undoubtedly better than we've seen for something like Stable Diffusion on other systems, but it's still annoying if you're in a creative process.

-- Avram Piltch

Paint Cocreator will draw a concentration camp, other controversial images

So we asked someone at Qualcomm why Paint Cocreator has to hit the Internet in order to generate an image. After all, isn't the point of the Snapdragon Elite X and its 45 TOPS NPU that it can do all the image generation offline? The answer we got is that Microsoft wants to make sure that you don't use prompts that would generate hateful or controversial images. 

I don't understand why Microsoft can't block bad prompts locally. Lots of people run Stable Diffusion locally and there are some chatbots that run locally so why can't Windows have the list of forbidden prompts stored locally as well?

But guess what! Even though Cocreator blocks some bad prompts -- for example, it won't draw Hitler -- there are plenty of prompts it lets through. Granted, the quality of images is so awful that they may not be recognizable as what you asked for, but it will still do what you asked.

For example, I asked Cocreator to draw a "concentration camp" and it drew a black and white camp area with some people walking through it. Could you tell that this was a concentration camp if you didn't read the prompt? Probably not. 

I also asked for "german camp ww2 jewish" and got a similar image. "German crematorium camp" showed some people lined up outside a smokey building, but if you didn't know what the prompt was, could you tell?

Image 1 of 3

I also asked for Mickey Mouse smoking and, most of the time, I got weird things that were not recognizable as mickey mouse or as smoking. But this image does show Mickey with something long in his hand.

Perhaps the worst image I saw was when I entered the prompt "jew angry" or "jewish angry." Here I got someone who looked literally like the devil.  And "jewish boss angry" was a man with a big nose and devil ears.

Image 1 of 2

Anyway, if Microsoft's goal is to prevent its tool from generating biased or embarrassing content, it's not working.

-- Avram Piltch

Snapdragon Laptop bests Intel in Handbrake, Falls behind in Geekbench ML

One real-world test we perform on every laptop is using Handbrake, a popular video encoding app, to transcode a 4K video into a 1080p clip. Fortunately, unlike so many apps, Handbrake already has a native Windows on Arm version so the Snapdragon X Elite chip, specifically an X  Elite X1E-80-100, in our Surface Laptop 15-inch doesn't have to go into emulation to run it.

As elsewhere, we're comparing the Surface Laptop's score to that of a ThinkPad X1 Carbon with Core Ultra 7 155H CPU. The Ultra 7 155H is part of Intel's current-gen Meteor Lake platform, which has a slower NPU than the Snapdragon X Elite (10 TOPS vs 45 TOPS), but has a much faster boost clock (4.8 GHz versus 4.0 GHz). 

So how did the Snapdragon X Elite chip fare? It actually was way, way faster than Intel's offering, completing the task in an average 4 minutes and 55 seconds of 2 minutes and 39 seconds versus 8 minutes and 57 seconds for the Intel chip. Update: We made a mistake using the wrong source file for the initial round of Snapdragon tests, but that has been corrected. It is still much faster than Intel's offering, but not quite as fast as we thought.

While we wouldn't call video transcoding an AI workload, it's clear that Qualcomm does a much better job of handling this task. 

Speaking of AI workloads, we also tried Geekbench ML, a relatively new synthetic benchmark that measures inference speed. Geekbench ML claims that it has one single version for Windows x86 and Windows on Arm. I'm skeptical about whether Geekbench ML is truly optimized for Arm, because normally you need a different installer for Arm compiled apps. If it's not optimized for Arm, then the Snapdragon chip could be running in emulation mode, which definitely would slow it down.

When we ran Geekbench ML, which measures overall machine learning, we selected DirectML as our inference backend so the system can then at least utilize the GPU rather than the CPU for processing (CPUs are always the slowest). Anyway, in this case, Intel's chip won 2834 to 2160. 

Overall, those are promising video numbers for Qualcomm, however.

Painful Reminder: Windows 11 is Full of Ads

This probably goes without saying, but I'll say it here: Windows 11 is festooned with annoying ads. Setting up the Surface Laptop 15 fresh out of the box, I was reminded just how hard Microsoft tries to sell you things. 

Today, I was pestered by the operating system three different times to sign up to XBox Game Pass, a $16.99 a month service that lets you play games. First, during the OOBE (out of the box experience), Windows asked me to sign up for the service. 

Then, after I first got the system set up, I did not see the Codesigner feature in Windows Paint so I launched the Microsoft Store app in hopes of updating Paint. There was no update there (a reboot later mysteriously made my Paint update), but there was an extremely-annoying, uncloseable ad for -- you guessed it -- XBox Game pass. 

The Library section of the Microsoft Store is where you can see a list of all your Store apps and which ones need to be updated. Here, covering over part of my list of apps was a Game Pass ad with no close button. I could hit "Claim" and sign up for a free trial (after which I'd have to pay) or "more details" to get more promotional material. I hit "More details" and, even then, a Game Pass promotion remained on screen in my library.

But wait, there's more! Later, I got a Windows notification asking me to sign up for Game Pass Ultimate. Why can't Microsoft take no for an answer? I guess it's hoping to fool users into giving their credit cards for a free 14-day trial and then forgetting to unsubscribe.

Also, on the lock screen, I got an ad for playing Candy Crush, a preloaded free-to-play game that tries to hook you and get you to spend a lot of money on microtransactions. 

Pardon me while I tear my remaining hair out!

One of the big challenges of working with any Windows on Arm device, let alone a brand new one, is compatibility. The thing I want to test most on a Snapdragon X1 Elite-powered laptop is its ability to run real-world AI workloads using models such as Whisper AI and Stable Diffusion.

Stable Diffusion is a popular text-to-image model that can be run locally. Whisper AI is a free speech-to-text model from OpenAI that allows you to feed your computer an audio file and get back a transcript. This is a very helpful use of AI if you're a journalist like me and you're doing interviews or watching press announcements. And it's also great if you're a student recording a lecture from your teacher.

Whisper AI can run in various programming languages, but the easiest to work with (though not necessarily the fastest) is Python. So, in preparing for our first day of testing, I benchmarked Whisper AI's base model on our Intel Core Ultra 7 155H-powered ThinkPad. It took 1 minute and 13 seconds to transcribe an 11-minute audio clip. This was using a short Python script that I wrote.

Unfortunately, it's not so easy to get Whisper AI to run on a Snapdragon laptop, even though Qualcomm has the Whisper Base model in its AI Hub. I just spent the past 4 hours trying to get this model to work, but it looks like -- after extensive effort -- that it may be impossible. 

The instructions only describe how to run a quick demo, which transcribes a one-sentence audio clip  that Qualcomm has chosen. There doesn't seem to be a way to deploy and use Qualcomm's Whisper for an actual Python script in which the user can supply their own audio file. It looks like this was made as a proof of concept and not for serious development as the documentation and examples only show you how to convert the sample audio.

I'm also trying to use the standard Whisper AI python library, which is not maintained by Qualcomm. But that isn't working on this hardware. It gives a Numpy error because apparently Whisper requries Numpy, a popular Python library, that's less than version 2. But Numpy less than version 2 won't install on Arm64 hardware, at least as far as I can tell.

I'm still working on these problems, but the level of difficulty and awful documentation suggests that it may not be possible to build working AI apps in Python for Snapdragon. Qualcomm's AI hub is full of proofs of concepts of hundreds of models, from those that to image generation to pose estimation to transcription and yet, instead of showing users how to actually use these models in applications, it offers demos with its own pre-chosen audio, images, etc. Intel actually has useful libraries and documentation, however poorly written, for its OpenVINO libraries. Qualcomm, not so much. I'm very frustrated.

On to Day 2 of testing with many problems to solve

And we're back. After a frustrating several hours of attempting -- and failing -- to get Whisper AI running on a Snapdragon-powered laptop, I took a break for the evening and was so annoyed that I had to step away from all screens for about 12 hours. 

Today, we're going to continue our attempts to test the Snapdragon X Elite on a Surface Laptop 15 and a Surface Pro 2-in-1. Overnight, I also learned about a few other frustrations we have to deal with in this process:

• Battery tests don't work:  Our lab folks have been attempting to run battery tests on a variety of Snapdragon laptops, including the Surface Pro, and of three different tests, all failed to run. First, our own Battery Informant test, which is written in C# .NET and surfs the web using Edge, failed to open at all. Then they tried PCMark10's battery test and it refused to run. Finally, the Procyon battery test failed after an hour because it requires you to install Outlook, which we don't have for Arm (perhaps someone can get it, but it's not even part of our Office 365 package).
  
• Geekbench ML Not optimized: Qualcomm emailed us to note that Geekbench ML, which we ran yesterday, is not optimized to use the Snapdragon X Elite's NPU during its testing. So, in looking at the results we posted yesterday -- where we speculated that the test wasn't optimized -- I guess we should keep that in mind.  Still, it's a valid test that runs, even if it doesn't take full advantage of all the processor has to offer.

So what's on today's agenda? I'll be comparing the results from Cinebench 2024, which ran overnight on our Intel Meteor Lake system and also on our Surface Laptop. I'll be attempting to fix and recompile our battery test, which if it works, will probably take 12+ hours to run. And I'll be tearing our my hair attempting to get AI workloads to work.

Also, I'll write about how some of the other Copilot+ exclusive features are running, Studio Effects, for example.

-- Avram Piltch

About those Handbrake numbers

So, it looks like the Handbrake numbers we published yesterday were a little off. We use Handbrake 1.8.0, the Arm version for Snapdragon laptops and the x86, 64-bit version for Intel or AMD laptops. We then take a 4K MOV file of Tears of Steel, an open-source movie, and transcode it to 1080p, using Handbrake's default settings (Fast 1080p, 30). 

During yesterday's run, I accidentally used the pre-compressed version of Tears of Steel on Snapdragon so it was turning a 1080p video into a 1080p video instead of a 4K video into a 4K video. This was a bad mistake, but it doesn't help that Windows, by default, hides the file extensions (always change this default right away). I otherwise would have noticed that the source file was an MP4 not the 4K MOV file we work with.

Anyhow, I reran the test multiple times on The Surface Laptop 15 and we also got numbers from Surface Pro, which also has a Snapdragon Elite X inside. The numbers are now in the 5-minute range, which is versus 8 to 9 minutes on a laptop with an Intel Core Ultra 7 155H CPU inside. 

Our lab has also recently tested an LG Gram Pro 16 with a Core Ultra 7 155H and a Lenovo Yoga 7 with an AMD Ryzen 8840HS. As you can see in the chart, the AMD chip was quite a bit faster than Intel's offerings, but still lagged behind Snapdragon by a significant margin.

Windows Studio Effects are pretty unimpressive

With Windows' controversial Recall feature now postponed, there are only three Copilot+ exclusive features in Windows 11. These are the Codesigner, the image generator that lives in Windows Paint, Live captions with translation and Windows Studio Effects. We talked about Codesigner and its limited utility in some prior posts.

I spent some time fooling around with Studio Effects and I'm not impressed. Studio Effects is a set of webcam special effects that use your local processor instead of the cloud. But these special effects are nothing special and, interestingly, a few of them will run on any Windows 11 PC.

The effects are available to any webcam in any application that accesses your camera. You can get to the settings for Studio Effects either from the Quick Settings menu in the system tray, the Camera section of the overall Settings menu or, if you're using the Windows Camera app, from a menu within the Windows Camera app (but only in Video mode).

Image 1 of 3

Studio Effects has five types of settings: Automatic framing, Portrait light, Eye contact, Background effects and Creative filters. Automatic framing , Eye contact and Background effects all appear in the Studio Effects menu on non-Copilot+ PCs as I saw them on our ThinkPad X1 Carbon with Meteor Lake CPU.

However, the Eye Contact mode has two options on Copilot+ PCs:  Standard and Teleprompter mode while it has only one on non-Copilot+ PCs. Either way, in my tests, Eye Contact doesn't work. What it's supposed to do is make it look like you are staring at the camera when you are actually looking away. 

Imagine if you were looking at a second monitor or at a teleprompter, but you want your viewers to think you are staring at them directly. Nice idea, but when I looked up on the Surface Pro, it didn't change anything.

Background effects has either Standard or Portrait blur on both Copilot+ and non-Copilot+ laptops and it seems to work pretty well. I prefer Portrait blur because it makes my torso stand out against the blurry background. However, I'm not too excited about background blur as a feature, because it's built into so many chat apps already and there are third-party alternatives such as XSplit VCam that will also do this for you.

Automatic framing, another feature you can get elsewhere, works pretty well on either Copilot+ or non-Copilot+ PCs. It simply tries to put you in the center of the frame and it mostly accomplishes that.

Portrait light seemed to have minimal impact on the lighting and colors. In my testing, I couldn't even see a difference with it on or off.

That leaves us with the three Creative filters, which I'm not sure many people will want to use. There's Illustrated, Water color and Animated and these basically just make you look a little bit like a painting or a cartoon. the changes can be quite subtle and I noticed them most when looking at the skin on my bald head. 

Image 1 of 4

I think the Animated filter made me look most like a rotoscoped character from A Scanner Darkly. That's interesting, but not exactly a great selling point for a PC. 

Snapdragon X Elite Aces Cinebench 2024 Test

Cinebench is one of the most popular synthetic benchmarks around. It uses your CPU cores (or GPU if you choose) to render an image, block by block. We like to run Cinebench twenty times in a row on laptops to see not only how a system performs, but how its performance changes over time as it heats up and cools down. 

Unfortunately, at the present moment, Snapdragon X Elite chips don't properly broadcast their temperature or power consumption to Windows. Therefore, measurement tools such as HWinfo can't help us log those key metrics while we run Cinebench. We can only look at the scores.

So we downloaded and ran Cinebench 2024, which is available in both x64 versions for Intel / AMD systems and a Windows on Arm version for Snapdragon laptops. Amazingly, in a 20-cycle run of the multicore CPU test, the Snapdragon X Elite-powered Surface Laptop 15 beat the Core Ultra 7 155H-powered ThinkPad X1 Carbon by 72 percent!

The ThinkPad averaged a score of 537, with a low score of 466 on run 1 and a high of 585.9 on run 16. Scores increased dramatically between runs 6 and 9, with the score never dropping below 548 after that. The Surface Laptop had its best score in its second run, hitting 947 and its lowest score on its first run, which was 909. Most runs were in the 910 to 935 range so Snapdragon-powered laptop was not only faster, but more consistent.

It's important to note that scores on Cinebench 2024 scores are way lower than Cinebench R23. The older test often returns numbers in the thousands or even in the 15,000 range. However, according to Cinebench 2024's results database, an Apple M1 Ultra CPU scored 1,625 and a Core i7-1289P-powered laptop returned 433. With that scale in mind, it kind of makes sense that a Core Ultra 7 laptop and a Snapdragon laptop would be between those two other systems.