Experts warn of security lapses in open-source AI deployments

Recent findings have revealed the unintentional exposure of sensitive data by open-source large language model (LLM) builder servers and vector databases. The revelations come as these models are increasingly integrated into business workflows, a process that has at times neglected critical security measures.

Legit security researcher Naphtali Deutsch was instrumental in unveiling these vulnerabilities. The primary concern stems from the fast-paced adoption of artificial intelligence (AI) technologies that often outstrip the implementation of necessary security protocols to safeguard the information they handle.

James Sherlow, Systems Engineering Director for EMEA at Cequence Security, elucidated the issue comprehensively. "LLMs bring significant benefits to companies aiming to automate processes. Open-source implementations provide a rapid path to enhancements, enabling optimisation of business processes and customer interactions," Sherlow commented. However, he emphasised that the sheer volume of data processed by these models, particularly Personally Identifiable Information (PII), necessitates a robust lifecycle protection scheme.

Sherlow pointed out that the popularity and functionality of open-source LLMs, often demonstrated by their high ratings and numerous stars, do not exempt them from thorough security testing. "Testing LLMs before they go into production is critical to determine if vulnerabilities can be exploited or data exfiltrated," he stated. He further argued that beyond initial testing, continuous monitoring of all activities involving LLM APIs is essential to ensure that exposure of sensitive data is limited to authenticated and authorised viewers only.

Highlighting more sophisticated threats, Sherlow noted, "Runtime monitoring should extend to vulnerability exploits and even more dangerous business logic abuse, where attackers aim to exfiltrate data by exploiting the business logic and flaws within it. LLMs will expose new flaws, making self-learning models necessary to identify these."

The importance of quick action in security measures was another critical point raised by Sherlow. He asserted that reliance on third-party tools for response is inadequate, as malicious actors can quickly exploit open-source LLMs. "Rapid inline native response is critical for any security platform," he warned, stressing that without immediate reaction capabilities, by the time a third-party tool responds, "the malicious actor will already have moved, probably using open-source LLMs themselves."

Endorsing this view, Katie Paxton-Fear, an API hacker and technical marketing manager at Traceable AI, added crucial insights. "As organisations look to leverage generative AI in their software and toolsets, it's important to recognise that while some parts of this technology are new, often they are powered by technology staples," Paxton-Fear noted. She observed that the AI models typically employ a web-based interface to interact with data, facilitated by APIs.

Paxton-Fear cautioned against becoming overly focused on newer types of vulnerabilities at the expense of fundamental security practices. "With the sudden explosion of AI, it can be tempting to focus on new vulnerabilities such as model inversion attacks. However, as this research shows, even the most advanced technology still needs to be protected with a password," she remarked.

Both experts underscore a common theme: while AI and LLMs offer substantial advantages in enhancing business functionality and customer experience, they concomitantly introduce new opportunities for cyber threats. Organisations adopting open-source LLMs or any generative AI technologies must not assume built-in security. Instead, these technologies should be viewed as a foundation for functionality that requires additional and ongoing security measures.

The consensus among security professionals is clear: robust security protocols, continuous monitoring, and immediate response mechanisms are essential in protecting sensitive data processed by rapidly evolving AI technologies. In an era where data breaches can have far-reaching consequences, neglecting these precautions is a risk businesses can ill afford to take.