Think! X captures online sentiment from social media discussion for destination marketers with their unique Tourism Sentiment Index (TSI); enabling them to understand the online perception of their destination and tailor their messaging for maximum impact. Their process wasn’t simple though. To produce these insights, Think! X had to manually perform imports, calculations, and reports at scale for each individual destination they served. This lengthy process inhibited real-time insights, restricted the number of clients Think! X could serve, and only allowed them to provide client-specific data.

This led Think! X to partner with Invoke to automate this process with TSI Live — enabling destination marketers to immediately react to ever-evolving public perceptions. We may have been creating the product from scratch, but the business already existed. Their clients had expectations on what data and insights they were paying for. In order to deliver value, our automated solution had to provide everything they already had and more.

To successfully analyze data, it must first be captured. Working with their data pipeline, we utilized a collection of serverless functions and technologies that import data, segment it, provide post analysis, and leverage the enterprise-grade natural language processing of AWS Comprehend for text classification.

Then we store it, but not in a traditional database. It’s a database of indexed documents containing information like sentiment scores, text excerpts, emotions, locations, and more. From these documents, the product builds queries to aggregate and extract data and determine an analysis based on continual segmentations of the data. That could mean starting with a query like “Whistler”, then aggregating and analyzing that bucket of data to extract statistics and other analytics such as sentiment ranges and trending topics.

The biggest challenge for this product was the scale — the platform is processing a huge amount of data from sources like Twitter, Reddit, Instagram, news sources, and more. All in, it processes between 100,000 and 170,000 posts per day. In a month, that’s approximately three million posts.

Tackling this challenge required an approach that maintained live, accurate information for end users while keeping the cost of machine learning at that scale under control. Our multi-faceted approach relied on the use of cutting-edge serverless technologies that allow us to ingest, queue, rate throttle, and serve data as soon as it was available.

In order to be a thorough representation of word of mouth, TSI Live pulls in data from a wide variety of social media sources. This leads to huge variations in the content itself; there are posts with text ranging from 100 to 5000 characters, posts with images or videos, posts with hashtags and emojis, posts with colloquial grammar and nicknames, and even posts with raw HTML. Handling inconsistent data and extracting valuable entities is vital to the value of the platform.

However, it’s not just extracting the subject from a sentence. It’s being able to differentiate them. How can the product consistently recognize the difference between London, Ontario and London, England? Or understand that Myrtle Beach is also a city, not just a beach. With a thorough process that applies broad heuristics and scrutinized targeting, TSI Live is now able to consistently classify items correctly.

Part of our engagement with Think! X involved extending TSI Live to cover the specific use case of sentiment regarding how a location is addressing the COVID-19 pandemic.

The TSI is reborn as TSI Live, an interactive dashboard that covers both aggregated tourism and COVID-indexed data. By uncovering useful insights, these marketers can understand changing trends, what is impacting positive or negative sentiment, and what the overall conversation volume is towards both COVID-19 and general travel. As the travel industry evolves and changes going forward, this is the type of invaluable information that can help ensure thoughtful and successful messaging from destination marketers around the world.