Text Preprocessing for Sentiment Analysis

NLP

Sentiment Analysis

Python

Author

Solomon Eshun

Published

April 11, 2024

Sentiment analysis is a natural language processing (NLP) technique used to identify and classify the sentiment expressed in text. In public health, sentiment analysis can inform health communication, patient engagement, and policy decisions by revealing community responses to health advisories, vaccination campaigns, and other public health interventions. However, the effectiveness of sentiment analysis in these sensitive areas is heavily reliant on the quality of preprocessing applied to text data.

This post details my approach to preprocessing texts to enhance sentiment analysis. Through a series of carefully designed steps, I aim to transform raw text data into a clean, analyzable format, setting the stage for accurate classification.

Here’s a breakdown of the preprocessing techniques I use to refine the raw text data:

URL Removal: I begin by stripping URLs from the texts. URLs usually don’t contribute to sentiment and can add irrelevant noise to the text, so removing them helps in focusing on meaningful content.
Tokenization: Next, I tokenize the texts by splitting the them into individual words. This is important because it helps to process the text at the word level, which is essential for the subsequent cleaning and analysis steps.
Removing Stopwords: I then remove stopwords - common words such as “the,” “and,” “is,” etc., which are abundant in English but do not carry significant sentiment. Eliminating these words helps reduce textual noise and focus the analysis on more impactful words that convey sentiment.
Eliminating Special Characters: I also scrubb the text of special characters like punctuation marks, symbols and emojis. These characters do not typically contribute to sentiment and can impede the performance of sentiment analysis models.
Lemmatization: Finally, I apply lemmatization to consolidate similar words to their base or dictionary forms. Unlike stemming, which simply chops off the ends of words to reach a common base, lemmatization considers the context and morphological analysis of words to accurately transform them to their lemma. This reduces the number of unique words the model must handle, enhancing its ability to generalize across different variations of the same word.

Note

These preprocessing steps are not always necessary when using modern transformer-based models such as BERT. Unlike traditional bag-of-words or machine-learning approaches, BERT models are designed to work with relatively raw text and rely on their own tokenization and contextual representations to capture meaning. Aggressive preprocessing, such as removing stopwords, punctuation, or emojis, can even remove information that may be useful for understanding sentiment. Therefore, when working with BERT and other modern language models, preprocessing is typically kept to a minimum, with steps such as handling missing text, and addressing obvious formatting issues being sufficient in many applications.

Python Implementation

Before creating the preprocessing function, it’s essential to set up the environment with the required packages. This includes loading libraries and downloading necessary data for text manipulation.

!pip install emoji
import pandas as pd
pd.set_option('display.max_colwidth', None)
import numpy as np
import nltk, re, emoji, spacy
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords

# Download necessary NLTK resources
nltk.download('stopwords')
nltk.download('punkt')
nltk.download('wordnet')

The preprocess_text function below incorporates all the preprocessing steps discussed above. It is structured to be reusable for any text dataset that requires cleaning for sentiment analysis.

# Load the spaCy model
nlp = spacy.load("en_core_web_sm")

def preprocess_text(data):
    processed_data = []
    # Set of English stopwords
    stop = set(stopwords.words('english'))
    for sentence in data:
        # Remove URLs
        text = re.sub(r'https?://\S+|www\.\S+', '', sentence)

        # Remove emojis using the emoji library
        text = emoji.replace_emoji(text, replace='')

        # Create spaCy doc object
        doc = nlp(text)

        lem_list = []
        for token in doc:
            if token.is_alpha and token.text.lower() not in stop:  # Check if token is alphabetic and not a stopword
                if token.pos_ != "PRON":  # Exclude all pronouns
                    lem_list.append(token.lemma_.lower())

        # Rejoin words to form the final cleaned text
        final_text = ' '.join(lem_list)
        processed_data.append(final_text)

    return processed_data

To demonstrate the functionality, the preprocessing function is applied to a sample dataset containing reviews. This will output the cleaned versions of the input reviews, showing how punctuation, common words, and case have been normalized.

# Sample input text
sample_text = ["The pandemic has affected the world.. @100% https://example.com!! :)"]
cleaned_text = preprocess_text(sample_text)
cleaned_text

['pandemic affect world']

Next, let’s consider a dataframe with 20 sample reviews, ratings, and IDs.

# Creating a sample DataFrame with texts

data = {
    'Review': [
        "Lockdown was necessary for our safety, I fully support it! 😊",
        "I understand the need but it was too long. 😕",
        "Lockdown completely disrupted my life, it was terrible. 😡",
        "It was well-handled and necessary for public health. 🏥",
        "Lockdown helped us control the virus spread. 😷",
        "It was too restrictive and unnecessary for so long. 😒",
        "Not sure if lockdown was the best solution, but we had no choice. 🤷",
        "It saved lives, but at a great personal cost to many. 😔",
        "The government did what was necessary during the lockdown. 👍",
        "Lockdown was too harsh and not managed well. 😠",
        "It was a good measure, but I hated being stuck at home. 🏠😖",
        "Lockdown was essential but mental health suffered a lot. 🧠💔",
        "The benefits outweigh the negatives of the lockdown. ➕➖",
        "Poor execution made it harder than it should be. 😞",
        "Lockdown was over before it was truly safe to do so. 😨",
        "We needed the lockdown to protect vulnerable populations. 🛡️",
        "The economic impact of lockdown was devastating. 💸",
        "Lockdown showed we can take collective action when needed. 🤝",
        "It was necessary, but the government support was insufficient. 💔",
        "Lockdown should have been stricter to be more effective. ⚠️"
    ],
    'Sentiment_Rating': [5, 3, 1, 4, 5, 2, 3, 3, 4, 1, 3, 2, 4, 2, 3, 5, 1, 4, 2, 4]
}

reviews_df = pd.DataFrame(data)

# Applying the text preprocessing function
reviews_df['Cleaned_Review'] = preprocess_text(reviews_df['Review'].tolist())
reviews_df

	Review	Sentiment_Rating	Cleaned_Review
0	Lockdown was necessary for our safety, I fully support it! 😊	5	lockdown necessary safety fully support
1	I understand the need but it was too long. 😕	3	understand need long
2	Lockdown completely disrupted my life, it was terrible. 😡	1	lockdown completely disrupt life terrible
3	It was well-handled and necessary for public health. 🏥	4	well handle necessary public health
4	Lockdown helped us control the virus spread. 😷	5	lockdown help control virus spread
5	It was too restrictive and unnecessary for so long. 😒	2	restrictive unnecessary long
6	Not sure if lockdown was the best solution, but we had no choice. 🤷	3	sure lockdown good solution choice
7	It saved lives, but at a great personal cost to many. 😔	3	save life great personal cost many
8	The government did what was necessary during the lockdown. 👍	4	government necessary lockdown
9	Lockdown was too harsh and not managed well. 😠	1	lockdown harsh manage well
10	It was a good measure, but I hated being stuck at home. 🏠😖	3	good measure hat stick home
11	Lockdown was essential but mental health suffered a lot. 🧠💔	2	lockdown essential mental health suffer lot
12	The benefits outweigh the negatives of the lockdown. ➕➖	4	benefit outweigh negative lockdown
13	Poor execution made it harder than it should be. 😞	2	poor execution make hard
14	Lockdown was over before it was truly safe to do so. 😨	3	lockdown truly safe
15	We needed the lockdown to protect vulnerable populations. 🛡️	5	need lockdown protect vulnerable population
16	The economic impact of lockdown was devastating. 💸	1	economic impact lockdown devastating
17	Lockdown showed we can take collective action when needed. 🤝	4	lockdown show take collective action need
18	It was necessary, but the government support was insufficient. 💔	2	necessary government support insufficient
19	Lockdown should have been stricter to be more effective. ⚠️	4	lockdown strict effective

To gain insights into the most frequently mentioned words in the reviews, we use a word cloud, where the size of each word indicates its frequency or importance. Here, I have used the most common words from the cleaned review data. It offers a quick and intuitive understanding of the main themes and sentiments expressed. Larger words appear more often in the reviews.

from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plt
from nltk.probability import FreqDist

all_reviews = " ".join(review for review in reviews_df['Cleaned_Review'])
words = all_reviews.split()
top_words = FreqDist(words).most_common()
top_words_text = " ".join(word for word, _ in top_words)

stopwords = set(STOPWORDS)
wordcloud = WordCloud(collocations=False, width=1000, height=600,
                      stopwords=stopwords, max_words=300,
                      mode="RGBA", background_color=None).generate(top_words_text)

fig = plt.figure(figsize=(22, 6.35))
fig.patch.set_alpha(0)
plt.tight_layout(pad=2)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.savefig("wordcloud.png", transparent=True, bbox_inches="tight")
plt.show()

In this post, I covered some important steps involved in preprocessing text for sentiment analysis. These preprocessing decisions can influence the performance of sentiment analysis models by helping ensure that the input text is appropriately structured.

It is important to remember that the extent of preprocessing should depend on the model being used. Traditional NLP approaches often benefit from more extensive preprocessing, whereas modern transformer-based models such as BERT typically require much less preprocessing to preserve the contextual information contained in the original text.

I hope this guide provides a useful starting point for your own sentiment analysis projects.