Building a Keyword Clustering Tool in Python: TF-IDF Plus HDBSCAN

Keyword grouping is one of those SEO tasks that sounds simple until you're staring at a spreadsheet with 12,000 rows and a "group by intent" column you're filling in by hand.

Manual clustering doesn't scale, and rule-based grouping misses the semantic overlap between phrases that don't share a single word but clearly belong together.

The framework here uses TF-IDF vectorization to generate feature vectors, then clusters them with HDBSCAN, a density-based clustering algorithm.

Why cluster keywords at all

Keyword clustering is the catalyst for topic generation.

Rather than briefing content writers with hundreds of individual keywords, clustering groups semantically related queries into coherent topics, making it much easier to produce content that satisfies a wider range of search intent.

The result is a set of topics that can strengthen semantic relationships, topical authority, internal linking, and visibility across related queries.

There are two key challenges: preprocessing the data and clustering the topics.

Challenge 1: preprocessing

SEO keyword exports are often noisy, especially from your own databases.

Removing stopwords and non-ASCII characters can't be done manually, hence the need to automate cleaning at scale. Python provides the flexibility to build a pipeline that can clean, normalize and transform large keyword datasets with minimal manual intervention.

Challenge 2: you don't know how many clusters exist

You rarely know how many topical groups exist until you've explored the dataset.

This makes algorithms such as k-means a poor fit, as they require you to specify the number of clusters in advance.

That's why TF-IDF plus HDBSCAN proves particularly effective.

How TF-IDF + HDBSCAN works

TF-IDF transforms each keyword into a numerical vector by assigning greater weight to terms that are distinctive within the dataset while down-weighting those that appear frequently across many keywords.

Those vectors feed HDBSCAN, a density-based algorithm that identifies natural groupings without requiring the number of clusters to be defined in advance.

HDBSCAN's key advantage: identifying noise

Rather than forcing every keyword into a cluster, it assigns outliers to a -1 label, excluding keywords that don't belong to any topic.

This is especially valuable for SEO datasets, where exports often contain highly specific long-tail queries that don't naturally fit into broader thematic groups.

Instead of degrading cluster quality by assigning these terms arbitrarily, HDBSCAN isolates them, producing cleaner and more coherent topical clusters.

Sourcing the keyword list from BigQuery

Before any clustering happens, you need a keyword list.

If your Search Console property is already exporting to BigQuery, that's a better source than the UI export, since it isn't capped at 1,000 rows and isn't sampled.

Run a simple pull against the standard GSC BigQuery export schema, then:

1. Export the result as a CSV
2. Strip it down to a single query column
3. Save it as a .txt file with one keyword per line

That's your clustering input.

Without BigQuery export you can still work with Search Console data, just a more limited dataset. Either way, the notebook only cares that you feed it a list of keywords.

Prompting the AI

A less polished version of this script worked years ago. AI made two things easier when rebooting and fine-tuning it.

Ask for tunable parameters, not hardcoded ones

Cluster sensitivity and minimum cluster size behave very differently depending on whether you're clustering 50 keywords or 50,000.

Requesting these as adjustable variables up top meant retuning without touching the logic.

Specify the environment

The first version was a plain Python script meant to run from a terminal. But the actual workflow is "pull keywords, run notebook, hand a client a file."

Asking for a Google Colab-specific version changes the shape of the code — try/except blocks around Colab-only imports, no argparse — more than a small tweak would suggest. It also enabled better visualization by leveraging Plotly to the fullest.

Running the code

The gist remains simple: it takes a flat list of keywords and automatically groups them into topic clusters.

1. Upload a .txt file with one keyword per line
2. It cleans the text — stripping special characters, stopwords and non-English entries
3. Tune sensitivity and min_cluster_size based on the size of your keyword list — using tunable parameters is the key to the entire framework
4. Once you're happy with the number of clusters, TF-IDF scores how important each word is within the set
5. HDBSCAN finishes the process
6. Each cluster gets an auto-generated label based on its most distinctive terms
7. Results export to Excel with both a grouped cluster view and a full keyword-by-keyword breakdown

Where AI adds value

The decisions that actually matter — from choosing TF-IDF over word embeddings to selecting HDBSCAN because the number of topics is unknown — still require an understanding of how keyword clustering works.

What AI removed was the repetitive work of rebuilding boilerplate code, wiring libraries together, and refining the notebook into something reusable.

The final result is a lightweight clustering tool that can process thousands of keywords in minutes, producing a sensible first draft of your topical taxonomy.

It won't replace editorial judgment, but it will eliminate hours of manual grouping and give your content team a far more structured starting point.

If you already export Search Console data into BigQuery, this becomes a natural step in your workflow: extract queries, run the notebook, review the clusters, and start planning content based on topics rather than isolated keywords.

Practical takeaways

Understand the algorithm choice, not just the tool. Avoiding k-means (predefined cluster count) and choosing HDBSCAN (noise labelled -1) is what determines output quality. Even when AI writes the code, a human makes that call.

Hoist parameters to the top. The same settings do not work at 50 and 50,000 keywords. Making retuning nearly free is the condition for real use.

Fix the data source first. The UI export's 1,000-row cap and sampling set a hard ceiling on cluster quality. Configuring the BigQuery export matters more than tuning the algorithm.

Connect clusters to a topical authority strategy — the goal is category-level share, as covered in Topical Authority Still Decides Who Wins in AI Search.

Translate clusters into entities. AI search looks for the source of a concept, not a keyword — see Entity SEO, which reframes clusters as entity coverage rather than page counts.

Use the output as a cleanup plan too. The noise (-1) group and duplicate clusters overlap with the pruning candidates in The Data-Backed Case for Content Pruning.