Study Finds AI Models Unprepared to Tackle Financial Problems, Far from Replacing Professionals

A new study by researchers from Clemson University and Washington State University finds that ChatGPT isn't completely reliable to give financial advice yet. The researchers analyzed 10,000 responses from AI after asking it different questions about finance and the results revealed that AI still has a long way to go until it can take over finance jobs. ChatGPT and other AI models are good enough to specify broader concepts of finance and summarize the issues and their solutions but it still cannot work on it thoroughly and in detail.

Many people like Goldman Sachh predicted that 15% to 35% finance jobs will be soon assisted by AI and it may take over the jobs of asset and wealth managers. But this study highlights that there is no need to panic because AI still doesn't have adequate knowledge of finance that is needed to handle the large systems. Even though AI can pass finance certification exams, it still isn't enough and we need to find what these AI models can actually do.

The researchers of the study assembled 1083 MCQs from different finance licensing exams and tested AI models like ChatGPT, Bard and LLaMA. These exams are required to be passed by human financial professionals in order for them to get a license. The researchers evaluated the responses of AI models based on how well they could explain the reasoning behind choosing a certain answer and how much sophisticated language they used while giving explanations.

ChatGPT-4 showed the best results, with 18% to 28% more accurate answers than other AI models. When researchers gave ChatGPT-3.5 some examples of responses, it nearly matched the responses of ChatGPT-4 showing that it can also give accurate answers which can resemble human professionals. There was a 73.4% accuracy on answers about customer accounts, trading and prohibited activities by both these models, but both of them showed less accuracy(56.6%) on answers about investment objectives and client financial profiles. There were also inaccurate answers by both models on specified problems related to tax status and insurance coverage.

The team of researchers also asked ChatGPT to evaluate potential merger deals and are now exploring how well this AI model can do if they test it on the deals that came after September 2021. Overall, the limitations of AI when it comes to finance are still there are AI models are not ready at all to do high-profile financial tasks as they can have serious legal and financial consequences.

Image: DIW-Aigen

Read next: iPhone 16 Named Best Small Phone, Samsung S24 Ultra Wins Best Smartphone of 2024
by Arooj Ahmed via Digital Information World