Market Intelligence Pipeline

The Problem

Finding real demand signals takes too long to do by hand

Understanding what a market actually struggles with usually means manually reading through forums, reviews, and comment threads, then guessing which complaints matter most. That process is slow, subjective, and easy to get wrong. Most decisions end up based on a handful of posts someone happened to read that week.

We built a system to remove the guesswork entirely and replace it with a repeatable, scored, and visualised pipeline.

What We Built

An automated pipeline that mines, scores, and clusters real pain points

The system runs across multiple Reddit communities and Trustpilot review sets in parallel, pulling raw posts, comments, and reviews relevant to a target market. Each piece of content is filtered against a pain keyword set, then scored for urgency based on language patterns such as frustration, repeated daily friction, or explicit mentions of lost business.

An AI layer runs alongside the scoring logic to extract a clean summary of each pain point, so every data row carries both a structured score and a human readable explanation. All sources are merged, deduplicated, and sorted by urgency before being logged.

Turning Data Into Decisions

Clustering, relevance scoring, and visualisation

Once the raw data is collected, the system groups it into bottleneck clusters such as booking issues, payment friction, or missed reminders. Each cluster carries an average urgency score, a maximum urgency score, and a content angle suggestion generated directly from the data.

The cluster output connects directly into Google Sheets, which feeds Looker Studio for visualisation. We calculated a relevance and virality score for each content type by combining comment volume, upvotes, and the urgency score together, giving a single ranked view of which pain points are both real and worth acting on first.

The Result

A repeatable system instead of a one time guess

The pipeline now runs as a single trigger and produces a complete, ranked view of market pain points without manual reading or subjective judgment. Every session produces fresh, sorted, and visualised data ready to act on immediately.