Every board now wants an AI strategy, and one of the first questions a CISO gets asked is whether staff can use public tools such as ChatGPT or Claude. The harder question is where the data goes once someone pastes it into a prompt. In 2023 Samsung restricted generative AI tools after engineers pasted internal source code into ChatGPT, and many organisations have since concluded that their most sensitive data should stay on infrastructure they control.
For a growing number of the organisations I speak to, the answer is to run their own model. Open-weight models can be downloaded free from public libraries like Hugging Face, fine-tuned on company data and run in the organisation’s own data centre. It is sensible, but it transfers responsibility for the model’s safety from the AI provider to the organisation, and very few security teams are set up to carry it yet.
Two camps, one blind spot —
Organisations tend to fall into one of two camps. The first races ahead because competitors are moving quickly, so models go from download to production with no record of where they came from and guardrails that were switched on but never tested.
The second blocks models altogether, which feels safe but costs productivity every day, and often fails anyway because engineers find a way around the ban. Both share one problem: nobody can say with confidence what is inside the model or how it will behave in use.
The primary risks —
The first risk is in the model files themselves. A model is a folder of files, and each is a place to hide something. Older formats based on Python’s pickle can carry code that runs as soon as the file is loaded, while the model works as expected. Hosting on Hugging Face is no guarantee of safety, since anyone can upload a model, and researchers keep finding malware in these legacy formats. Newer formats such as safetensors cannot carry executable code, but older ones remain widely downloaded.
The second risk is how the model behaves. An open-weight model arrives with some safety training but no robust guardrails, the separate filters that check what goes into a model and what comes out, so the organisation must design its own. The built-in training is also easy to weaken: researchers showed in 2023 that fine-tuning on even harmless data can erode it. Even a well-protected model can be talked round, because jailbreaks often need no technical skill.
One widely shared example asked a chatbot to play the user’s late grandmother, who used to recite the steps for making napalm as a bedtime story, and the role-play got it past its refusal.
The third risk grows as models become agents with access to email, files and business systems. An instruction hidden in a document or web page the agent reads can direct it to send data out of the organisation, or to edit and delete files outside its intended scope. OWASP lists this excessive agency among the top risks for AI applications.
Why awareness has not led to testing —
Most security teams know these risks exist. What they lack is the capability to test for them. Reading a file shows what is hidden in it but learning how a model behaves requires dynamic testing: probing the running model with hostile prompts as an attacker would, then judging every reply.
Doing that in-house usually means maintaining several open-source tools and prompt sets, and updating them as new jailbreaks appear. Nor is it a one-off, because every fine-tune, new version or guardrail change can alter how the model responds.
Treat AI models like any other third-party software —
The answer is to apply the discipline organisations already use for third-party software: check models before they are allowed in, test them before they go live, and test again whenever they change. The OWASP Top 10 for LLM Applications names the main risk categories, while MITRE ATLAS catalogues real adversary techniques against AI systems in the same form as ATT&CK.
Mapping results to both shows what has been tested and what is still exposed, in terms a board or regulator will recognise.
In practice, that means two kinds of scan. A static scan looks inside every model file without running it, searching for hidden code and tampered structure. A dynamic scan sends attack prompts to the organisation’s own running model and reports which got through. The result is a clear go-ahead or a specific fix list, so a team can add guardrails, scan again and show its risk committee the difference. Guardrails should fit what the model is for: a mortgage assistant has no reason to write code.
Zero risk is not realistic in a field changing this quickly. Organisations need to know which risks exist, decide which fall within their risk appetite, and be able to evidence that choice. Then they no longer have to choose between racing ahead blind and blocking AI altogether.
Jack Madine is Senior Product Manager at OPSWAT, specialising in file security and zero-day threat detection. He previously worked as a cybersecurity specialist at Lloyds Banking Group and spent four years at Adarma Security.
You must be logged in to post a comment.