Alarming Results: AI Text Detectors Fail on 65.93% of German Texts!

AI text detectors are tools programmed to detect whether a text was partially or fully created with AI text generators like ChatGPT. Using an AI detector is helpful whenever there is a suspicion that a text may have been generated by AI. Teachers in particular can benefit from it, as they can check whether their students are writing texts on their own or not.

ChatGPT is calling…

What if I told you that the previous paragraph was written by an AI?
In fact, I asked ChatGPT to write a short introduction for this post, and that is the result. Looks pretty convincing, doesn't it?
And here lies the dilemma: AI texts are sometimes so convincing that someone could pass them off as their own work!
As an SEO agency, we often work with content writers and therefore need to make sure the texts they deliver were created by humans. That motivated us to investigate whether there are ways to detect AI-written texts.
Detecting AI-generated content is a challenging task, and I don't want to give you false hope that detection with 100 percent accuracy is possible. With a bit of linguistics, detective work and support from AI content detector tools, however, you can track down most AI-generated content.
This post is about how AI detector tools work and how reliable they are. Beyond that, it tries to find out whether there are ways to trick the detector tools.

How do AI text detectors work?

Detecting AI writing is essentially based on reverse-engineering language patterns. In simpler terms, they determine how high the probability is that word B follows word A.
This means the machine takes a passage of text apart and then uses an algorithm to detect a pattern in the words. If a pattern is easier to detect, the chances increase that it was written by AI.
The young scientist Edward Tian from Princeton demonstrates the best approach to AI text detection in his project. He uses evaluation dimensions such as:

• Perplexity (confusion)
• Burstiness (variation)

Perplexity

The term perplexity describes the degree of confusion in a given text:
A high value usually points to a human author and is characterized by higher complexity. Lower values, on the other hand, suggest that AI text generators have taken over the authorship and focus more on clarity and readability.
AI text generators work by calculating the probabilities for the next word in a sentence. The resulting texts are usually clearly structured and easy to read, but can also come across as boring or repetitive.

• High perplexity = text is confusing → human-generated
• Low perplexity= text is not confusing → AI-generated

For example, for the sentence "Milana works at Buzz­matic to …" there are more plausible and less plausible continuations, such as:

Burstiness

Burstiness applies this predictability to an entire text. The lower the complexity of a text, the higher the probability that an AI wrote the content

• A low burstiness therefore means that sentence structure and length barely vary, which suggests the text was created by an AI.
• A high burstiness, on the other hand, indicates that sentence structure and length vary, which suggests the text was written by a human.

Which AI detector tools are there?

Various tools are available to identify AI-generated content. However, the accuracy of these tools can vary and often depends on the type of content and its length.
Below you'll find the tools that delivered the best results in my analysis and are the easiest to use:
Originality AI → Works as a Google Chrome extension that lets you easily check in real time whether a page was created by an AI.
Copyleaks → Detects AI-generated text, including newer models like GPT-4, Jasper, and Bard. Also offers a free Chrome extension.
Writer → Users can add a URL or copy and paste text into the window. After you click "Analyze text", the platform shows the percentage of AI-generated text detected in the article.
Crossplag → Features a thermometer scale that shows how much of the article is fake. If the thermometer stays green, the article is fine and was mostly created by a human.
GPT Zero → Frequently used by teachers, but also useful for business purposes.

How were the AI detector tools tested?

I tested the tools mentioned above with different types of content generated by ChatGPT. In short, 100 different texts were collected that were written by ChatGPT.
Each text was then run through the following content detector tools. Here is a short summary of my results. In the end, I only chose tools with an accuracy above 70% per 100 texts.

As you can see, Originality.ai was able to detect AI-generated texts with a 98% detection level.
Since Originality.ai delivered the best results, I decided to continue my detective work with this tool only. I was curious to see how reliable this tool is and whether there are ways to trick it.
And here is what happened next…
To validate my results, I decided to use texts that all of the tools mentioned above had recognized as 100% AI-generated.
First, I pasted the exact completion from ChatGPT into the Originality.ai tool. As you can see below, Originality.ai recognized ChatGPT very well:

However, it only takes a few small changes for the tool to deliver completely different results:

Can you spot what I changed?

With a few imperfections such as extra spaces or typos, you can confuse the AI detector. The tool now identifies the entire paragraph as human-written.

Comparison: true positives & false positives

The same test was also carried out with human-written texts. It was interesting to analyze how the detector tools perform on texts created by humans.
I uploaded 100 human-written texts into the tool. For the English texts, everything looked promising. Originality.ai was able to identify 94.65% of the texts as true positives. For the German texts, however, the picture looks quite different. Only 34.07% of the content was recognized correctly:

As you can see in the example below, the tool struggles to recognize original German texts. Even an original historical text from 1933 was identified as 79% AI-generated:

In our upcoming posts, we plan to conduct a thorough analysis of why certain German texts are identified as AI-created even though they show clear indicators of human authorship.

Can you bypass an AI detector tool?

AI content detectors may be effective and popular tools, but they also have their downsides. Their accuracy depends on the data they were trained on.
This insight made me wonder whether you can give an AI like ChatGPT a specific prompt so that the detector tools cannot identify the created content as AI-written. So I asked ChatGPT to write a paragraph about Buzz­matic, which Originality.ai could easily identify as AI-generated.

Then I asked ChatGPT to rephrase it. I used specific prompts for this, which I will cover in my next post.
And this is what came out of it…

If you use precise instructions in ChatGPT, you can actually generate AI texts that tools like Originality.ai classify as human-written.
Until I write my next post, you can see a comparison of both texts below. On the left you see an original ChatGPT text, on the right you see a text that was reworked by ChatGPT. Feel free to try to analyze the differences.