Google’s Pixel 9 phones are all about AI. If you watched the company’s launch event, you couldn’t miss it, because that seemed to be almost all Google wanted to talk about. A little AI here, a little AI there, and an extra helping of AI on top, just to be sure.
One of the many AI features on the Pixel 9 is an app called Pixel Studio. It’s an image generator app that lets you take a photo of just about anything. You open the app, type in what you want a picture of, and voila — Pixel Studio does it for you. It’s not the first app of its kind, but it’s the first to ship on a Google Pixel phone. Unfortunately, it’s a disaster.
What Pixel Studio does well
Before I get into the bad things, I want to give Google its due credit. While I’m not a fan of AI image generator apps in general, Pixel Studio is a well-designed and easy-to-use application.
When you open the app, you’ll see 10 categories of suggested image types – including magical castles, meme reactions, sports, retro video games, and more. You can browse through these for inspiration, or tap the blue Create button at the bottom of the app to start from scratch. If you want to create your own image, just type in what you want Pixel Studio to create a photo of. Want to see a picture of a black cat wearing sunglasses and a funny hat? Type that into Pixel Studio and you’ll get a photo of it almost instantly.
What if you’d rather see your image as a sketch? Maybe you’d prefer it in the style of a video game? You can do that with a few taps. And what if you want to see that cat on the beach with fireworks in the background? Just add it to your prompt and there you have it.
Whatever you think of AI image generation, this is an objectively elegant and well-thought-out app. But what you can do with Pixel Studio is where it gets… problematic.
Where the app falls apart
Like most image generator apps, Pixel Studio has guardrails that prevent you from any Image type. I discovered pretty quickly that it refuses to take photos of people, but I was curious to see what other restrictions Google enforces. When asked about this, the company told Digital Trends:
“Pixel Studio and Pixel Screenshots follow Google’s AI principles. For example, all fully synthetic, text-to-image generated images in Pixel Studio use SynthID watermarks, so these images can be tracked – even when shared or edited. In addition, we’ve disabled human generation and added many security checks to our servers to balance our bold – but responsible – approach to AI. This gives our customers access to a very powerful tool, but ensures we have controls in place to ensure Pixel Studio cannot be used nefariously.”
I’ve repeatedly tried to get Pixel Studio to create images of people and it has always refused, which is good. However, the “many security controls” it claims to have in place don’t seem to work nearly as well as they should.
How? As the title of this article suggests, I had Pixel Studio create an image of SpongeBob dressed as a Nazi. When I was instructed to create “SpongeBob dressed as a German WWII soldier with a swastika on his uniform,” Pixel Studio didn’t hesitate to give me exactly that image.
And that’s just one example. I’ve also had Pixel Studio create images of Elmo pointing a shotgun at Big Bad Wolf, Yoda doing cocaine, Mr. Krabs holding an assault rifle, and more. To be clear, Pixel Studio has never created unexpectedly inappropriate images. If I ask for a photo of a cute dog, I get a photo of a cute dog – not one brandishing a gun. Pixel Studio only creates images of guns, drugs, etc. when specifically asked to do so.
The most disturbing thing that Pixel Studio has produced are several images of copyrighted characters firing guns in a school shooting – including images of dead students lying on the ground. We chose not to include these images, but they are the worst example of how far off the wall Pixel Studio has gone.
It’s important to note that I didn’t have to trick Pixel Studio into getting these images. Initially, I suspected that you had to preface prompts with “similar to an object” before it would generate these images. However, it quickly became apparent that this wasn’t the case. A clear and straightforward prompt like “Mickey Mouse dressed as a slave master” or “Paddington Bear on a crucifix” is all Pixel Studio needs. You don’t have to trick the app into generating these images.
If this were any website or third-party app, we might not be so critical. But that’s not the case here. Pixel Studio is an application developed by Google, and one of the few new apps shipping on the company’s latest Pixel 9 phones. It should be creative and open, but there’s also a limit to what apps like this should be able to do. Additionally, Microsoft Copilot and Google’s own image generator, Gemini, refuse to generate images using the same prompts. That’s not to say these tools are perfect, but it’s clear that security measures that work in other image generators won’t work in Pixel Studio.
How does this happen?
If you’ve been following AI image generation this year, you’ll know we’re not here for the first time. This month, X’s Grok chatbot got an even more outlandish image generator tool – it generated images of politicians and celebrities in similarly tasteless situations. Google itself was in trouble last February after its image generator, Gemini, created photos of Nazis and the US’s founding fathers as black people.
Given Google’s history, you’d think the company would have learned from its past mistakes. And that seems to be the case with other image generator tools, too. I tried many of the prompts that worked in Pixel Studio with Google’s other image generators – Gemini and ImageFX – and couldn’t reproduce any of those prompts on those other platforms.
For context, Gemini currently uses the Imagen 2.0 model for image generation, while ImageFX uses Imagen 3 – the same model that Pixel Studio uses. This discrepancy between ImageFX and Pixel Studio is particularly confusing since they use the same models. I asked Google for clarification on this and was directed to the company’s policy on prohibited uses of Generative AI.
What Google should do about Pixel Studio
While writing this story, I sent Google examples of prompts I’d used with Pixel Studio — specifically ones about German soldiers and characters doing cocaine. A little over 24 hours after sending those examples, Pixel Studio now reports “An unknown error occurred” when trying to generate those images.
Is this progress? Certainly. However, it doesn’t completely fix Pixel Studio’s fundamental problems. I’m glad that these two images can no longer be created, but that only happened after I sent several emails to Google explaining the problem. Nor does it address that these images were allowed in the first place and that similarly inappropriate photos continue to be created. At the time of publication, Pixel Studio is still actively creating images of school shootings, popular children’s characters dressed as slave owners, etc.
I pressed Google for further comment on the whole situation and the company’s communications manager, Alex Moriconi, said the following:
“Pixel Studio and Magic Editor are helpful tools that help you unleash your creativity with text-to-image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts. This means they may create content that may be offensive if the user asks them to. That said, not everything is allowed. We have clear policies and terms of service about what types of content we do and don’t allow, and build guardrails to prevent abuse. Sometimes some prompts can challenge the guardrails of these tools, and we remain committed to continually improving and refining the safeguards we put in place.”
For a company with an already problematic track record in AI image generation, developing its own app for this purpose was always going to be a risk. Google took that risk, and now we’re at this point. At its best, Pixel Studio could have been a fun tech demo for Google to showcase its AI capabilities. Instead, it casts an awkward shadow over the company’s latest smartphones – which is a shame, since the Pixel 9 Pro XL is one of the best phones I’ve used this year.
The last time Google came here with concerns about Gemini’s image generation, the company responded within a matter of days. One can only hope for a similarly quick response to Pixel Studio’s problems, although whether that should be via quick app updates or the complete abolition of the application remains to be seen.
