I recently read Andrew Regan (et al.)’s 2016 influential paper on a "sentiment analysis tool to extract the reader-perceived emotional content of written stories."
I also recently wrote a short story and wanted to try it out. While Andrew has made his research code available on GitHub, I wasn't quite ready to take on that project.
It was an excellent opportunity to test the ChatGPT-o1 preview's updated code generation capability. I asked ChatGPT to write a simple tool to perform some of the tasks described in Andrew's 2016 paper, but in a more straightforward manner. A key difference in my approach was that I used OpenAI's embedding model to estimate how the sentiment changed in my short story. Changes in sentiments or features estimate the changing “mood” or emotional impact throughout the 11,000 words of the story. The mood changes are reflected in the plot shown in Figure 1.
sentiment_texts = {
'dramatic': "I am feeling dramatic, excited and on an adrenaline rush.",
# 'non-dramatic': "I am feeling bored, lack-luster, without energy.",
'positive': "I am feeling very happy and joyful today!",
# 'neutral': "The weather is average, and nothing special is happening.",
'negative': "I am feeling very sad and depressed today."
}
Through an iterative "chat" process, I guided ChatGPT to generate a tool that I could use without any code clean-up or debugging. I provided initial guidance to ChatGPT, but that was only the start. I then looked at its output and provided feedback, repeated, etc.
There are a couple of important caveats. First, I am familiar with the algorithms and concepts, so I could steer ChatGPT precisely to what I wanted. Second, I haven't yet vetted the algorithm in detail. The code runs and produces an expected result. However, I need to sit down and decide whether all algorithm choices were correct - but that is a different task than the one I’m interested in here. For example, should Cosine or Euclidean distance (or another) be used to compute text similarity to the sentiment anchors? Sounds like a science experiment!
Discussion
Every few months, I conduct a ChatGPT code generation "fire drill." This helps me better assess the quality of large language models when it comes to generating code. Experiments from a couple of years ago are laughable compared to what I can now do — just in terms of its ability to generate code from a single prompt. However, what's notable is how a collaborative, multi-prompt process with ChatGPT leads to quicker and more effective outcomes. Additionally, the more you understand or can figure out along the way—whether from previous output or other sources—the faster you can refine your prompts and converge on a good outcome.
Appendix A illustrates my collaboration with ChatGPT-01 preview to produce Python code, which I used to analyze my short story and generate Figure 1.
Reagan, A.J., Mitchell, L., Kiley, D. et al. The emotional arcs of stories are dominated by six basic shapes. EPJ Data Sci. 5, 31 (2016). https://doi.org/10.1140/epjds/s13688-016-0093-1
Appendix A
This appendix lists all the prompts I used in this experiment. They were presented to ChatGPT in the sequence given below. I have not included the responses for purposes of readability. Note the mix of high-level conceptual prompting and specific feedback about code/APIs and algorithms.
Propose a sentiment analysis algorithm based on LLM embeddings and sliding text windows.
===
Propose a simple sentiment classification layer implementation.
===
Instead of labeled sentiment data and a feed-forward classification layer, can you compute the distance of the embedding from embeddings for key sentiments?
====
Propose a sentiment analysis Python program based on using OpenAI embeddings and a sliding text window applied over text documents.
For example, it may want to obtain the embeddings for the documents in the following manner:
embeddings_model = OpenAIEmbeddings(openai_api_key=openai_api_key)
texts = [doc.page_content for doc in documents]
embeddings = embeddings_model.embed_documents(texts)
You should use the distance between the embeddings of the input text and predefined embeddings that represent key sentiments (e.g., positive, neutral, and negative). This approach leverages semantic similarity to classify sentiment based on how close the input text's embedding is to these predefined "sentiment anchor" embeddings.
The building block algorithm steps are, per document:
Tokenize text in the document.
Create sliding windows.
Generate embeddings for each window using LLM.
Classify sentiment for each window.
Optional smoothing of sentiments across windows.
Visualize the sentiments as a graph representing the entire document.
====
Can you update the code to use the openai 1.0.0 api, example below. def generate_embeddings(documents, openai_api_key):
embeddings_model = OpenAIEmbeddings(openai_api_key=openai_api_key)
texts = [doc.page_content for doc in documents]
embeddings = embeddings_model.embed_documents(texts)
for i, document in enumerate(documents):
document.metadata['embedding'] = embeddings[i]
return documents
====
Can you rework the code so it uses this text reader? def do_read_text_file(file_path):
loader = TextLoader(file_path, encoding='utf-8')
raw_documents = loader.load()
for document in raw_documents:
content = document.page_content
metadata = document.metadata
print(f'length of page_content={len(content)}')
text_splitter = NLTKTextSplitter(chunk_size=1000, chunk_overlap=100)
text_content = " ".join([doc.page_content for doc in raw_documents])
text_sections = text_splitter.split_text(text_content)
print(f'number of text sections chunked={len(text_sections)}')
return text_sections
====
can you update this code to read and plot output for multiple documents (multiple file paths).
====
If I want to extend the sentiments to include more than the three currently in place include some that are overlapping, how might you display it? Consider this example list. sentiment_texts = {
'dramatic' : "I am feeling dramatic, excited and on an adrenaline rush.",
'non-dramatic' : "I am feeling bored, lack-luster, without energy.",
'cormac' : "In the style of Cormac McCarthy",
'margaret' : "In the style of Margaret Atwood",
'king' : "In the style of Stephen King",
'positive': "I am feeling very happy and joyful today!",
'neutral': "The weather is average, and nothing special is happening.",
'negative': "I am feeling very sad and depressed today."
}