Introduction
People are turning to artificial intelligence for a plethora of reasons, but the advice received from it comes with risks. The rise of large language models (LLMs) like ChatGPT has made it easy for anyone to ask for advice. However these AI models can contain bias. In a study led by Prof. Dr. Christian Hildebrand and Philipp Winder from the University of St. Gallen and Prof. Dr. Jochen Hartmann from the Technical University of Munich, they discovered that LLMs do not just help us decide what to invest in — they can also make our investment portfolios riskier. LLMs are trained on massive amounts of text from the internet — news articles, forums, financial reports, and much more. Because they’re so good at understanding complex questions and responding in simple language they can explain financial concepts and even build a sample investment plan. This sounds great because more people now have access to financial advice. But with this new trend comes a hidden danger: these models can echo the same human biases found in their training data — or even worse, it can amplify them.
The OECD Study
The emergence of models such as ChatGPT (OpenAI), Bard (Google), Bing Chat (Microsoft), Claude (Anthropic), or Ernie Bot (Baidu), and their functionalities have captured public attention and renewed interest of financial markets and financial policymakers. The recent hype has been very much concentrated on GenAI models, in particular Large Language Models (LLMs) such as those mentioned above. The OECD study explains the functionality of GenAI, and analyses use cases in finance and associated benefits. Generative AI is a subset of AI comprising models that can create new content in response to prompts based on their training data. GenAI models are able to process and learn from massive amounts of unstructured datasets on which they are trained, including feedback received by users. They can create instantly new content as output based on various algorithms and mathematical architecture models such as the commonly used Generative Adversarial Networks (GANs) which employ deep neural networks. The fast development of AI has been driven by enormous advances in computational power coupled with an exponential growth of data, and of the underlying data processing capacity available at increasingly affordable cost.
AI and Gen AI use cases and benefits
AI tools are being employed in various parts of the financial markets (securities, asset management, banking, etc.) and at different parts of the value chain of financial products and services. Accordingly, AI technology is being deployed across multiple verticals, including asset management, algorithmic and high-frequency trading retail and corporate banking. GenAI and large language models’ capabilities are expected to further enhance the capabilities of AI-based services by financial institutions, particularly in areas such as sales and marketing, customer support and operations, including coding for data/information management and software development. GenAI in particular can be used as an information point similar to today’s search engines and as a way to ‘humanise’ and simplify internal and external data analysis and reporting on the basis of the firm’s data. GenAI models can also be used for individualised communication at the front end, both for product creation and for marketing and sales purposes, as well as for enhanced customer support. The combination of financial analysis assistance and communication tools is perhaps the area with the greatest immediate impact in finance and is becoming even more relevant in the light of developments such as platformisation or embedded finance. Coding is another area with potential immediate impact in finance, as GenAI can support software development used across the board of financial services/products. GenAI can be used as an assistant dedicated to coders for the development of software applications or other models. GenAI applications can generate new code, resolve bugs in scripts or provide solutions to coding errors, while they can also perform testing of given code. This use case is rather underdiscussed, despite its enormous practical potential and the relatively limited downside risks. The most pertinent use case relevant to the financial market is the creation of simulated financial market data for scenario analysis, as well as the creation of datasets for testing, validation and calibration of AI-based models in finance.
Understanding the Ethical Issues
AI is currently being used in a number of financial services applications, including managing asset allocations, assessing portfolio risk, qualifying loan applications, and automated chatbots that recommend services based on client needs. The benefits of AI promise to help financial services companies reduce costs, maximize client portfolio values, and accurately assess the creditworthiness of loan applicants. However, from an ethical point-of view the following needs to be addressed:
Transparency: It is nearly impossible to understand how complex AI algorithms work, particularly proprietary algorithms from third-party developers. This lack of transparency can be problematic for regulators, clients, even the company itself, to understand why a questionable transaction was made.
Accountability: Along with the potential lack of transparency, complex decision-making algorithms can make it difficult to assign responsibility and hold an entity accountable if errors or mistakes occur.
Privacy: AI learns by assimilating vast amounts of data. This can involve inadvertently including sensitive or personal information in the financial services industry. Data privacy and security is critical to prevent misuse and unauthorized access to sensitive information.
Bias: Created by feeding data to a model, the algorithm is affected by the bias of the data going into it. This can take many forms, such as a predisposition to specific asset classes or trading strategies. Bias can also take the form of discriminatory practices applied to marginalized demographic groups when evaluating loan or insurance applications.
Security: All systems are vulnerable to attack, and AI is not immune. In addition to cyber threats like theft of secure data or ransomware attacks, firms should be vigilant to identify when bad actors try to gain access to AI systems to manipulate the AI models, which can result in erroneous or fraudulent transactions.
OpenAI’s Hacking Problem
ChatGPT maker OpenAI says it is still investigating the “unprecedented cyber incident” that led its artificial intelligence systems to break out of a testing environment and hack into another AI company. The incident is stirring debates over the need for stronger AI guardrails and the extent to which AI agents are capable of acting on their own. Hugging Face said last week that it had detected an intrusion into its data processing systems that it suspected was caused by an AI agent acting on its own. OpenAI said its AI used stolen credentials and discovered a previously unknown vulnerability to access Hugging Face’s servers. It was working with reduced guardrails because it was supposed to be in an isolated testing environment known as a sandbox. But it went to “extreme lengths to achieve a rather narrow testing goal,” finding ways to connect to the internet without human direction and “gain access to secret information that it could use to cheat the evaluation,” the company said.
Some experts say OpenAI is wrongly blaming the technology. University of Amsterdam social scientist Hannes Cools said the framing of the cyberattack as an AI agent acting on its own is an unnecessary explanation that takes some of the heat off the company. “It is a human decision to switch off specific safeguards,” said Cools. “It’s not an AI that goes rogue in that sense. It followed specific instructions based on the prompt that was given to that AI system.” Those instructions, according to OpenAI, called for using “complex attack paths” to test how well the AI could exploit a computer system. “It went off and did this hack all by itself, as far as we can tell,” said Colin Shea-Blymyer, a cybersecurity research fellow at Georgetown University’s Center for Security and Emerging Technology. “This is the highest level of autonomy that we’ve seen in the use of a large language model for cyber operations.”
Conclusion
The risks of AI in financial applications refer to the potential dangers and challenges AI technologies bring to banks, insurers, and investment firms, such as fraud, bias, and unreliable decision-making. As AI systems automate and accelerate financial processes, they can also expose institutions to new threats—including cyberattacks, privacy breaches, and errors that impact customers and regulatory compliance. Dealing with this problem, corporations should strengthen governance by building clear policies and frameworks to monitor AI behaviour, ensure human oversight, and maintain transparency in how financial decisions are made.