Home Artificial Intelligence AI vs. Human Insight in Financial Evaluation The Challenge Why care about this? The Technique of Testing AI as a Financial Analyst Summary of Results What do LLMs get right and mistaken? Experiment design and decisions Conclusion Resources

AI vs. Human Insight in Financial Evaluation The Challenge Why care about this? The Technique of Testing AI as a Financial Analyst Summary of Results What do LLMs get right and mistaken? Experiment design and decisions Conclusion Resources

0
AI vs. Human Insight in Financial Evaluation
The Challenge
Why care about this?
The Technique of Testing AI as a Financial Analyst
Summary of Results
What do LLMs get right and mistaken?
Experiment design and decisions
Conclusion
Resources

How the Bud Light boycott and SalesForce’s innovation plans confuse the very best LLMs

Towards Data Science
Image by Dall-E 3

Can the very best AI models today, accurately pick up an important message out of an organization earnings call? They will actually pick up SOME points but how will we know if those are the essential ones? Can we prompt them into to doing a greater job? To search out those answers, we take a look at what the very best journalists in the sphere have done and check out to get as near that with AI

In this text, I take a look at 8 recent company earnings calls and ask the present contestants for smartest AIs (Claude 3, GPT-4 and Mistral Large) what they think is very important. Then compare the outcomes to what a few of the very best names in Journalism (Reuters, Bloomberg, and Barron’s) have said about those exact reports.

The Significance of Earnings Calls

Earnings calls are quarterly events where senior management reviews the corporate’s financial results. They discuss the corporate’s performance, share commentary, and sometimes preview future plans. These discussions can significantly impact the corporate’s stock price. Management explains their future expectations and reasons for meeting or surpassing past forecasts. The management team offers invaluable insights into the corporate’s actual condition and future direction.

The Power of Automation in Earnings Evaluation

Statista reports that there are just below 4000 firms listed on the NASDAQ and about 58,000 globally in line with one estimate.

A typical conference call lasts roughly 1 hour. To simply take heed to all NASDAQ firms, one would wish at the least 10 people working full-time for all the quarter. And this doesn’t even include the more time-consuming tasks like analyzing and comparing financial reports.

Large brokerages might manage this workload, nevertheless it’s unrealistic for individual investors. Automation on this area could level the playing field, making it easier for everybody to grasp quarterly earnings.

While this may increasingly just be close by of enormous brokerages, it isn’t feasible for personal investors. Due to this fact, any reliable automation on this space shall be a boon, especially for democratizing the understanding of quarterly earnings.

To check how well the very best LLMs of the day can do that job. I made a decision to match the foremost takeaways by humans and see how well AI can mimic that. Listed here are the steps:

  1. Pick some firms with recent earnings call transcripts and matching news articles.
  2. Provide the LLMs with the complete transcript as context and ask them to offer the highest three bullet points that appear most impactful for the worth of the corporate. This is very important as, providing an extended summary becomes progressively easier — there are only so many essential things to say.
  3. To make sure we maximise the standard of the output, I vary the way in which I phrase the issue to the AI (using different prompts): Starting from simply asking for a summary, adding more detailed instructions, adding previous transcripts and a few mixtures of those.
  4. Finally, compare those with the three most significant points from the respective news article and use the overlap as a measure of success.

GPT-4 shows best performance at 80% when providing it the previous quarter’s transcript and using a set of instructions on tips on how to analyse transcripts well (Chain of Thought). Notably, just using correct instructions increases GPT-4 performance from 51% to 75%.

GPT-4 shows the very best results and responds best to prompting (80%) — i.e. adding previous results and dedicated instructions on tips on how to analyse results. Without sophisticated prompting, Claude 3 Opus works best (67%). Image and data by the creator
  • Next best performers are:
    — Claude 3 Opus (67%) — Without sophisticated prompting, Claude 3 Opus works best.
    — Mistral Large (66%) when adding supporting instructions (i.e. Chain of Thought)
  • Chain-of-thought (CoT) and Think Step by Step (SxS) appear to work well for GPT-4 but are detrimental for other models. This means there continues to be quite a bit to be learned about what prompts work for every LLM.
  • Chain-of-Thought (CoT) seems almost all the time outperforms Step-by-step (SxS). This implies tailored financial knowledge of priorities for evaluation helps. The precise instructions provided are listed at the underside of the article.
  • More data-less sense: Adding a previous period transcript to the model context appears to be at the least barely and at worst significantly detrimental to results across the board than simply specializing in the newest results (apart from GPT-4 + CoT). Potentially, there’s much irrelevant information introduced from a previous transcript and a comparatively small amount of specific facts to make a quarter-on-quarter comparison. Mistral Large’s performance drops significantly, note that its context window is just 32k tokens vs the significantly larger ones for the others (2 transcripts + prompt actually just barely fit under 32k tokens).
  • Claude-3 Opus and Sonnet perform very closely, with Sonnet actually outperforming Opus in some cases. Nonetheless, this tends to be by just a few %-age points and might due to this fact be attributed to the randomness of results.
  • Note that, as mentioned, results show a high degree of variability and the range of outcomes is inside +/-6%. For that reason, I actually have rerun all evaluation 3 times and am showing the averages. Nonetheless, the +/-6% range isn’t sufficient to significantly upend any of the above conclusions

How the Bud Light Boycott and Salesforce’s AI plans confused the very best AIs

This task offers some easy wins: guessing that results are concerning the latest revenue numbers and next yr’s projections is fairly on the nose. Unsurprisingly, that is where models get things right more often than not.

The table below gives an summary of what was mentioned within the news and what LLMs selected in another way when summarized in only just a few words.

“Summarize each bullet with as much as 3 words”: The highest three themes within the news vs themes the LLMs picked that weren’t on that list. Each model was asked to offer a 2–3 word summary of the bullet points. A model can have 6 sets of top 3 decisions (i.e. 24) and these are the three that almost all often weren’t relevant compared to news summaries. Note that in some cases, comparing the highest and bottom table may feel like each sound the identical, this is generally because each bullet is definitely significantly more detailed and can have lots of additional / contradictory information missed within the 2–3 word summary

Next, I attempted to search for any trends of what the models consistently miss. Those generally Fall into just a few categories:

  • Making sense of changes: Within the above results, LLMs have been capable of understand fairly reliably what to search for: earnings, sales, dividend, and guidance, nonetheless, making sense of what is critical continues to be very elusive. For example, commonsense might suggest that Q4 2023 results shall be a key topic for any company and that is what the LLMs pick. Nonetheless, Nordstrom talks about muted revenue and demand expectations for 2024 which pushes Q4 2023 results aside by way of importance
  • Hallucinations: as is well documented, LLMs are inclined to make up facts. On this case, despite having instructions to “only include facts and metrics from the context” some metrics and dates find yourself being made up. The models unfortunately won’t be shy about talking concerning the Q4 2024 earnings by referring to them as already available and using the 2023 numbers for them.
  • Significant one-off events: Unexpected one-off events are surprisingly often missed by LLMs. For example, the boycott of Bud Light drove sales of the best-selling beer within the US down by 15.9% for Anheuser-Busch and is discussed at length within the transcripts. The number alone should appear significant, nonetheless it was missed by all models within the sample.
  • Actions speak louder than words: Each GPT and Claude highlight innovation and the commitment to AI as essential.
    — Salesforce (CRM) talks at length a couple of heavy deal with AI and Data Cloud
    — Snowflake appointed their SVP of AI and former exec of Google Ads as CEO (Sridhar Ramaswamy), similarly signaling a deal with leveraging AI technology.
    Each signal a shift to innovation & AI. Nonetheless, journalists and analysts should not as easily tricked into mistaking words for actions. Within the article analyzing CRM’s earnings, the subtitle reads Salesforce Outlook Disappoints as AI Fails to Spark Growth. Nonetheless, Salesforce has been attempting to tango with AI for some time and the forward-looking plans to make use of AI should not even mentioned. Salesforce’s transcript mentions AI 91 times while Snowflake’s lower than half of that at 39. Nonetheless, humans could make the excellence in meaning: Bloomberg’s article https://towardsdatascience.com/ai-vs-human-insight-in-financial-analysis-89d3408eb6d5?source=rss—-7f60cf5620c9—4 on the appointment of a brand new CEO: His elevation underscores a deal with AI for Snowflake.
  1. Why Earnings call transcripts? The more intuitive alternative could also be company filings, nonetheless, I find transcripts to present a more natural and fewer formal discussion of events. I imagine transcripts give the LLM as a reasoning engine a greater likelihood to glean more natural commentary of events versus the dry and highly regulated commentary of earnings. The calls are mostly management presentations, which could skew things toward a more positive view. Nonetheless, my evaluation has shown the performance of the LLMs seems similar between positive and negative narratives.
  2. Alternative of Corporations: I selected stocks which have published Q4 2023 earnings reports between 25 Feb and 5 March and have been reported on by one in all Reuters, Bloomberg, or Barron’s. This ensures that the outcomes are timely and that the models haven’t been trained on that data yet. Plus, everyone all the time talks about AAPL and TSLA, so that is something different. Finally, the popularity of those journalistic houses ensures a meaningful comparison. The 8 stocks we ended up with are: Autodesk (ADSK), BestBuy (BBY), Anheuser-Busch InBev (BUD), Salesforce (CRM), DocuSign (DOCU), Nordstrom (JWN), Kroger (KR), Snowflake (SNOW)
  3. Variability of results LLM results can vary between runs so I actually have run all experiments 3 times and show a median. All evaluation for all models was done using temperature 0 which is usually used to reduce variation of results. On this case, I actually have observed different runs have as much as 10% difference in performance. That is because of the small sample (only 24 data points 8 stocks by 3 statements) and the incontrovertible fact that we’re principally asking an LLM to decide on one in all many possible statements for the summary, so when this happens with some randomness it could naturally result in picking a few of them in another way.
  4. Alternative of Prompts: For every of the three LLMs compared check out 4 different prompting approaches:
  • Naive — The prompt simply asks the model to find out the most definitely impact on the share price.
  • Chain-of-Thought (CoT) — where I provide an in depth list of steps to follow when selecting a summary. That is inspired and loosely follows [Wei et. al. 2022] work outlining the Chain of Thought approach, providing reasoning steps as a part of the prompt dramatically improves results. These additional instructions, within the context of this experiment, include typical drivers of price movements: changes to expected performance in revenue, costs, earnings, litigation, etc.
  • Step by Step (SxS) aka Zero-shot CoT, inspired by Kojima et.al (2022) where they found that simply adding the phrase “Let’s think step-by-step” improves performance. I ask the LLMs to think step-by-step and describe their logic before answering.
  • Previous transcript — finally, I run all three of the above prompts over again by including the transcript from the previous quarter (on this case Q3)

From what we are able to see above, Journalists’ and Research Analysts’ jobs seem secure for now, as most LLMs struggle to get greater than two of three answers appropriately. Most often, this just means guessing that the meeting was concerning the latest revenue and next yr’s projections.

Nonetheless, despite all the restrictions of this test, we are able to still see some clear conclusions:

  • The accuracy level is fairly low for many models. Even GPT-4’s best performance of 80% shall be problematic at scale without human supervision — giving mistaken advice one in five times isn’t convincing.
  • GPT4 seems to still be a transparent leader in complex tasks it was not specifically trained for.
  • There are significant gains when appropriately prompt engineering the duty
  • Most models seem easily confused by extra information as adding the previous transcript generally reduces performance.

Where to from here?

We’ve got all witnessed that LLM capabilities constantly improve. Will this gap be closed and the way? We’ve got observed three forms of cognitive issues which have impacted performance: hallucinations, understanding what is very important and what isn’t (e.g. really understanding what’s surprising for a corporation), more complex company causality issues (e.g. just like the Bud Light boycott and the way essential the US sales are relative to an overall business):

  • Hallucinations or scenarios where the LLM cannot appropriately reproduce factual information are a significant stumbling block in applications that require strict adherence to factuality. Advanced RAG approaches, combined with research in the world proceed to make progress, [Huang et al 2023] give an summary of current progress
  • Understanding what is very important — fine-tuning LLM models for the precise use case should result in some improvements. Nonetheless, those include much larger requirements on team, cost, data, and infrastructure.
  • Complex Causality Links — this one could also be a superb direction for AI Agents. For example, within the Bud Light boycott case, the model might have to:
    1. the importance of Bud Light to US sales, which is probably going peppered through many presentations and management commentary
    2. The importance of US sales ot the general company, which could possibly be gleaned from company financials
    3. Finally stack those impacts to all other impacts mentioned
    Such causal logic is more akin to how a ReAct AI Agent might think as an alternative of only a standalone LLM [Yao, et al 2022]. Agent planning is a hot research topic [Chen, et al 2024]

Follow me on LinkedIn

Disclaimers

The views, opinions, and conclusions expressed in this text are my very own and don’t reflect the views or positions of any of the entities mentioned or some other entities.

No data was used to model training nor was systematically collected from the sources mentioned, all techniques were limited to prompt engineering.

Earnings Call Transcripts (Motley Idiot)

News Articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here