Quick Success Data Science

Learn graphical text evaluation with NLTK

The Natural Language Tool Kit (NLTK) ships with a fun feature called a dispersion plot that helps you to post the placement of a word in a text. More specifically, it plots the occurrences of a word versus the variety of words from the start of the corpus.

Here’s an example dispersion plot for the primary characters within the Sherlock Holmes novel, The Hound of the Baskervilles:

The vertical blue tick marks represent the locations of the goal words within the text. Each row covers the corpus from starting to finish.

For those who’re aware of The Hound of the Baskervilles — and I won’t spoil it should you’re not — then you definately’ll appreciate the sparse occurrence of Holmes in the center, the late return of Mortimer, and the overlap of Barrymore, Selden, and the hound.

Dispersion plots can have more practical applications. For instance, imagine you’re an information scientist working with paralegals on a criminal case involving insider trading. To search out out whether the accused contacted board members just before making the illegal trades, you’ll be able to load the subpoenaed emails of the accused as a continuous string and generate a dispersion plot to ascertain for the juxtapositions of names.

Social scientists analyze dispersion plots to review language trends related to specific topics. By tracking the occurrence of terms like “climate change” or “gun control” in news articles, they’ll gain insights into priorities which are vital to society over specific timeframes.

On this Quick Success Data Science project, we’ll write the Python code that generated The Hound of the Baskervilles dispersion plot shown previously.

We’ll use a replica of the novel stored on this Gist. It originally got here from Project Gutenberg, an awesome source for public domain literature. As beneficial for natural language processing, I’ve stripped it of…