ChatGPT study shows that training data does not match actual usage

A study by the Data Provenance Initiative, a collective of independent and academic researchers committed to data transparency, reveals a discrepancy between ChatGPT’s training data and its typical use cases.

The study, which analyzed 14,000 web domains, found that ChatGPT’s training data consists primarily of news articles, encyclopedias, and social media content.

However, the most common practical applications of the tool are in creative writing, brainstorming and the search for explanations.

As the study states,

“While news websites account for nearly 40% of all tokens, less than 1% of ChatGPT requests appear to be related to news or current events.”

To examine usage patterns in more detail, researchers analyzed a dataset called WildChat, which contains 1 million user conversations using ChatGPT. They found that over 30% of these conversations involve creative compositions such as writing fictional stories or role-playing.

This discrepancy suggests that ChatGPT’s performance may vary depending on the specific task and alignment with the tool’s training data.

Marketers should be aware that ChatGPT may struggle to create content based on current events, industry-specific knowledge, or niche topics.

Adapting to the strengths and limitations of ChatGPT

Knowing what ChatGPT is trained to do will allow you to tailor the prompts to the tool’s strengths and limitations.

This means you may need to add more context, specify the tone and style you want, and break complex tasks down into smaller steps.

Use ChatGPT for AI-powered content creation for tasks like brainstorming social media posts or email subject lines. Leverage human expertise for complex, industry-specific content.

Optimize output through effective prompt engineering. Always review AI-generated content for accuracy and edit to ensure quality.

AI tools can speed up ideation and content creation, but don’t expect perfection. Human review is essential for accuracy, brand consistency, and channel-specific copy.

outlook

This research highlights that marketers need to be cautious when using AI tools like ChatGPT.

Understand what AI can and cannot do and combine it with human expertise. This combination can improve content strategies and help achieve KPIs.

As this field continues to evolve, we may see AI tools that are better tailored to real-world usage patterns.

Until then, remember that while it is helpful, it is no substitute for the judgment of an expert.

Featured image: Emil Kazaryan/Shutterstock