How to spot AI reviews

Photo by Lisa Fotios.

The most common question I get asked is “How can you possibly detect AI text?” After all, ChatGPT is so good. And when your only signal is the words on the page, how can you even begin to guess at the source? Well, as they say, style is character.

Spend enough time reading Brandon Sanderson and you’ll be able to recognize his writing even if you see an excerpt out of context. Spend enough time grading middle schooler’s essays and you’ll be able to tell apart the 7th graders from the 9th graders. And we’re in luck, because not only is ChatGPT more distinctive than either of those, and we also have a lot more examples to work off of.

Let’s start with an easy example - here we have a review on Google Maps:

Pouch Camporee Field holds a special place in the hearts of Scouts, and I can't speak highly enough about this fantastic Scouting venue. It's a true Scout's paradise, and I couldn't be more pleased with my experience there.

The campsite itself is exceptionally well-equipped, with ample space for camping, various activity areas, and well-maintained facilities. The beautiful natural surroundings make it the perfect place for Scouts to connect with nature and learn essential outdoor skills. The atmosphere is peaceful and fosters a sense of camaraderie among Scouts and leaders.

Pouch Camporee Field provides a wide range of activities and opportunities for skill-building. Whether it's orienteering, pioneering, or merit badge workshops, there's always something engaging to participate in. The programs are thoughtfully organized, and the staff members are not only knowledgeable but also deeply committed to the Scouting movement.

The sense of community and tradition at Pouch Camporee Field is remarkable. The spirit of Scouting is alive and well here, and it's truly inspiring to see so many young Scouts coming together to learn, grow, and have fun. The dedication of the staff and volunteers in ensuring a safe and memorable experience is highly commendable.

The location of Pouch Camporee Field is ideal, offering a serene and secluded environment while still being easily accessible. It's an excellent place for Troops to come together, bond, and create lasting memories.

In summary, Pouch Camporee Field is a place where Scouting dreams come true. With its well-equipped facilities, engaging programs, and a strong sense of Scouting community, it's a treasure for Scouts of all ages. I look forward to returning to this outstanding Scouting destination and continuing to build memories and skills with my Troop. Scout's honor!

What's the first thing you notice about this review? The length is way longer than your average review, and it's got way more adjectives and adverbs than your average review. Not only that, but most of these descriptors lack content besides being generically positive (fantastic, exceptionally, beautiful, perfect, thoughtful). I notice a lot of cliches such as “I can't speak highly enough” and “I couldn't be more pleased.”

Another thing to keep an eye out for is that ChatGPT often likes to repeat the prompt back - notice that the full name of the place is in the first sentence of the review. While a normal person knows that their reader already has the context from the page they’re on, ChatGPT makes sure the response fully answers the prompt.

One of the biggest tells for AI-generated writing is the highly structured writing. Notice that in this review, there is an intro, then one paragraph for each aspect of the review, and then a conclusion? This is a very common pattern that pops up all the time - humans vary their paragraph and sentence structure a lot more.

Now, let’s try to apply these learnings to a different domain. ChatGPT reviews are all over Amazon - look at a few top selling products and you’ll see that their review sections are rife with AI.

Let’s take a look at two reviews for the same product on Amazon, and rate them on the same scale as before.

Example 1 - Amazon review of a water bottle.

This is my favorite water bottle of the year! What I like about it; the option of sipping or taking a gulp.

I'm very satisfied with the built-in straw, it is very easy to sip through the straw. And this is coming from someone who has had multiple bottles with built-in straws, this Owalla bottle is the best in 2023.

Another great option is the option of taking a drink from the open mouth. Once again, really clever design, I never feel like I'll actually splash any water in my face or getting it up my nose when I'm taking a larger gulp of liquid.

Insulation is about average. It does the job of keeping my liquids chilled. We've had some incredibly hot weather here in California this year; so I've been putting more ice into my water. And this bottle will keep it chilled throughout the night.

There are cons, I don't like the weight. It is heavier than other insulated water bottles I've had. Also the cap release needs a hard push to close. I think this is something Owala needs to work on in their version 2.0.

Overall I'm very happy with this bottle. It is now my daily driver replacing my Takeya. I like the design of the bottle so much, I bought the (lighter) sports version! They could improve the cap design, and hopefully they'll do that with the next model.

1. Long - Yep
2. Empty descriptors - just one (really)
3. Cliches - None
4. Repeats the prompt back - No, just “water bottle.”
5. Highly structured writing - Intro, bullet points, and conclusion are here, but the cons paragraph is irregular and a little out of place

Overall, I would rate it 1.5/5 on this scale. Just being long is not enough of a signal for us to say it was written by AI. Highly likely this was written by a human.

Example 2 - Amazon review of the same water bottle.

I recently purchased the Owala 24-ounce insulated bottle and it has quickly become my go-to for staying hydrated throughout the day. The first thing that struck me was the design - sleek, modern, and available in a variety of colors that pop. The bottle feels durable yet lightweight, perfect for daily use.

Functionality-wise, it doesn’t disappoint. It’s leak-proof and insulated, which means my drinks stay cold for hours, even on the hottest days. The built-in straw is a game-changer for sipping on the move without the hassle of unscrewing a cap or tilting the bottle.

What I appreciate the most is the attention to detail. The straw has a cover that keeps it clean, and the bottle fits in most car cup holders and backpack side pockets, making it incredibly travel-friendly. Plus, the loop handle is perfect for carrying or clipping to a bag.

Cleaning is a breeze, which is always a concern with reusable bottles. The wide mouth accommodates a bottle brush and the straw components disassemble for a thorough wash.

In summary, the Owala bottle is a thoughtfully designed product that delivers on its promise of convenience and style. It’s a reliable companion for anyone looking to up their water intake without the fuss. Highly recommend!

1. Long - Yep
2. Empty descriptors - Sleek, modern, incredibly, thoughtfully
3. Cliches - “the first thing that struck me,” “game-changer”, “delivers on its promise” “without the fuss”
4. Repeats the prompt back - Yes, the full product name “Owala 24-ounce insulated bottle”
5. Highly structured writing - Yes, intro, 3 bullet points and a summary.

This looks like it’s 5 for 5 on AI-specific tells. I would say it is almost certainly AI-written.

Great to see that our internal "model" has generalized! None of these points are smoking guns on their own, but when all signs point in the same direction, you can add them up and draw your own conclusion. Let me know what you think! You can find me Twitter at @max_spero_ or email me at max [at] pangram.com.