Project: text-analysis tool
From: Mei Lin, Product
"We got a batch of survey comments in comments.txt, one per line. I don't have time to read them all before planning. Can you tell me the most common words and phrases, which of our themes come up most, and give me one real quote for each of the top two themes?
Our themes and the words that signal them are in THEMES. Ignore filler words; there's a STOP list."
Done looks like: a ranked list of themes with counts, and a quote per top theme.
Free-text comments are among the most common "messy" inputs an analyst gets. You don't need a machine learning library to get useful answers from a few hundred of them; the tools from the text unit go a long way.
The pipeline is: load and drop blank lines, normalize (lowercase, strip punctuation), split into words, remove stop words, then count. Pairs of neighboring words (bigrams) often say more than single words: "delivery late" is a clearer signal than "late".
Themes are tagged with regular expressions so that "log in", "login" and "logging" all count, matched without caring about capitals. Keep the original comments around: quotes are what convince people.
from collections import Counter
words = ["delivery", "late", "box", "damaged", "delivery", "late"]
Counter(zip(words, words[1:])).most_common(1)
Out: [(('delivery', 'late'), 2)]. zip with a shifted copy makes neighboring pairs.