Accuracy of our sentiment AI

created on: 2021-10-20

We took 453 crypto news articles from last week and measured the accuracy of our sentiment analyzer.

What is Sentiment?

When press or users on the web talk about something, it could be good news or bad news or if they found it good, interesting etc. They could predict good things or bad things will happen to it. We measure sentiment on a scale of -100% (really bad news/review), 0 (neutral news or opinions) and 100% (extremely good news or opinion). Sentiments drive the crypto market.

How did we measure?

We first let the AI classify the news as good or bad and give it a sentiment score. Then, we manually audited the results and gave our own sentiment score. We marked all the news headlines that were wrongly scored.

Results

Accuracy MetricValue
Correct classifications 91.37 %
Mean deviation 0.1 %
Significant disagreements 11.28 %

The accuracy is more than what we expected. We will come up with better tests and reduce the disagreements. One improvement could be not being able to see our AI's sentiment scores while entering sentiment manually and a larger dataset. Nevertheless, you should always expect more than 75-80% accuracy. We carefully crafted our AI for that. It is enough to catch news and social trends.

Our test data is available for download. There are a few duplicate news headlines. They are the same news sourced from different websites. We keep them as a popularity measure.

Koinsentiment aggregates news and social media talk around crypto and analyzes the sentiment of each digital asset. Visit our Dashboard for real-time sentiment analysis and please support us on Producthunt.