How Quora uses Pangram to handle AI-written answers

In April 2024, Pangram Labs partnered with Quora to help them take on spammers using ChatGPT to respond to posts with inauthentic, AI-generated answers.

Quora is one of the largest websites in the world, ranking as the 33rd most trafficked website in the world and pulling in over 1 billion page visits per month as of August 2024 (Source: Semrush). Quora allows users to post questions, which will be seen and answered by others browsing the site.

"Our mission is sharing and growing the world’s knowledge. A big belief of ours is that a lot of knowledge is stuck in people's heads, and if we match the right questions to the right people we can extract that knowledge."

Lexie Wu, Group Product Manager leading moderation at Quora

What's the problem with AI answers?

An example AI-written answer

Generative AI allows spammers to generate hundreds or thousands of authentic-looking answers with little effort. One could make the argument that an AI answer is still a net positive. Users get to see an answer that is perhaps low- or medium-quality but often directionally correct. That's better than no answer at all, right?

In a vacuum, perhaps – but there are several undesirable effects of having AI answers on Quora.

1. When a question is already answered, this dissuades others from putting in the time to write their own answer. So even though a question may have gotten an okay AI answer, the AI answer makes it less likely that someone chooses to share their own real-world experience.
2. AI crowds out authentic posts. As with many platforms, views and engagement are zero-sum. Each time a user sees AI content on their feed or digest, other real creators are missing out on engagement that they deserve.
3. Reputational risk. People can tell when an answer they're reading is AI-written. This creates the question: why visit Quora at all if you're just going to be reading AI answers? Why not just go to ChatGPT? Quora presents a different value proposition than ChatGPT – authentic answers from real people – and curates their platform to ensure that this remains true.

Why use Pangram?

Sometimes it's hard to tell with the human eye if something is AI-written. Other times, it is simply a time-consuming task, requiring a moderator to read carefully for a while before they are certain. Automating this process frees up what is otherwise a costly moderation job, saving time and money in the long run.

While there are a couple open-source solutions that aim to solve the problem of AI detection for GPT-2, no solution worked well on GPT-4, the most-commonly used large language model (LLM). Platforms like Quora want a solution that can classify outputs of even advanced models like GPT-4. Ideally, a solution would continue to work with new LLMs, as new frontier-level language models are released every couple of months.

Pangram, with robust evaluations and over 100x accuracy over competitors like GPTZero, was one of the only options that could reliably detect GPT-4 written content in April 2024 and remains the most accurate AI detection model by a large margin today.

Additionally, Pangram's data pipeline includes built-in robustness to future LLM releases. It is capable of generating synthetic training data and training a new model within 24 hours of an LLM becoming available. In July 2024, Pangram expanded language support to over 20 languages and continues to make modeling improvements to ensure high accuracy for customers.

Impact

As of September 2024, Quora has identified more than 1 million AI-generated posts, improving content quality across the site and maintaining their reputation as an authentic and trustworthy source of information.

Pangram continues to serve as a force multiplier for Trust & Safety teams, providing them with the tools they need to confidently set policies around AI content.

Have a use case for AI detection? Contact us at info@pangram.com!