Security· 9 min read

AI Infrastructure Security: A 10-Point Checklist for Self-Hosted Models and Cloud AI

Small teams are moving fast on AI: a model server for internal chat, a Jupyter notebook for the data team, a Vertex AI or Azure OpenAI key for a new feature. Each one is new infrastructure, and most of it was set up for speed, not security. This guide shows where AI infrastructure gets exposed, two real examples, and a checklist you can work through this week.

Rutuja Mali
By · Founder & Managing Director, Sentrivox
Builds SentriOps to give small teams one clear view of their IT assets and security risk. LinkedIn

What counts as AI infrastructure

  • Model servers. Machines in the office or in the cloud that run models with tools like Ollama or vLLM, often on GPU virtual machines.
  • Data and ML tools. Jupyter notebooks, MLflow and the Ray dashboard that data teams use to build and run models.
  • Model and training data storage. Cloud storage buckets holding model files, datasets, documents for retrieval, and embeddings.
  • Cloud AI services and their keys. Vertex AI, Amazon Bedrock and Azure OpenAI, plus the API keys and service accounts your apps use to call them.

How AI infrastructure gets exposed

  • No login by default. Many AI tools were designed for one researcher on a trusted network. Jupyter, the Ray dashboard and Ollama can all end up reachable with no password if they are started on a public address.
  • Public storage. A bucket of training data or model files made public to share it quickly, and never locked down again.
  • Leaked API keys. Keys pasted into notebooks, scripts and shared repositories, where anyone with access can copy them and run up costs or read data.
  • Over-powered service accounts. The account an AI app uses has project-wide admin rights when it only needs to call one model.
  • Unpatched AI software. AI frameworks release often, and older versions carry known flaws that nobody tracks because the tool is not on the asset list.

Two real examples

NameAffectedWhat it allowedSeverity
ShadowRay, CVE-2023-48022Anyscale Ray 2.6.3 and 2.8.0A remote attacker could run code through the Ray job submission APICritical, 9.8 (CISA ADP)
Probllama, CVE-2024-37032Ollama before 0.1.34A path-handling flaw that researchers showed could lead to remote code execution on exposed serversHigh, 8.8 (CISA ADP)

ShadowRay. In March 2024, Oligo Security reported attackers using this weakness against Ray clusters left reachable from the internet, to steal data and credentials and to mine cryptocurrency on the victims' GPUs. Anyscale disputes the CVE: its position is that Ray is not meant to be used outside a strictly controlled network, and newer versions (2.52.0 and later) offer token authentication. Either way, the lesson is the same: a Ray cluster on the open internet is an open door.

Probllama. Wiz Research found that Ollama did not properly check model file paths, which could let an attacker who can reach the Ollama server write files outside the model folder. Ollama fixed it in version 0.1.34. The risk is highest when Ollama has been set to listen on the network rather than on the local machine only, which is its default.

Not on the KEV list
As of November 2026, neither CVE appears in the CISA Known Exploited Vulnerabilities catalog. That does not make them safe to ignore: ShadowRay was exploited in the wild regardless.

Cloud AI services need the same care

Using Vertex AI, Amazon Bedrock or Azure OpenAI removes the server patching, but not the access risks:

  • Least privilege. Give apps a narrow role, for example Vertex AI User on Google Cloud, permission to call bedrock:InvokeModel on specific models in AWS, or Cognitive Services OpenAI User on Azure. Avoid Owner or Editor.
  • Keys in a secrets manager. Store API keys in Secret Manager, AWS Secrets Manager or Azure Key Vault, and rotate them.
  • Logging on. Turn on audit or invocation logging so you can see who called which model, and how often.
  • MFA everywhere. Every human account that can reach these services needs multi-factor authentication.

The 10-point AI infrastructure checklist

#CheckWhy it matters
1List every AI server, notebook, dataset bucket and cloud AI service you runYou cannot secure what you have not listed
2Confirm no AI tool or dashboard is reachable from the internet without a loginThis is how ShadowRay victims were found
3Open AI ports (11434, 8888, 8265, 5000) only where needed, never to the internetShrinks the attack surface
4Update Ollama, Ray, Jupyter and other AI tools on a scheduleKnown flaws like CVE-2024-37032 have fixes
5Make every bucket holding models or training data private and encryptedStops data and model theft
6Give AI service accounts only the permissions they needLimits damage if a key leaks
7Move API keys out of code and notebooks into a secrets managerKeys in code get copied
8Turn on MFA for every cloud account that can reach AI servicesBlocks most account takeovers
9Turn on logging for AI service useLets you spot misuse and unexpected costs
10Repeat this review monthly and for every new AI projectAI estates change quickly

For risks inside the AI applications themselves, such as prompt injection and leaking sensitive data in answers, use the OWASP Top 10 for LLM Applications alongside this checklist.

How SentriOps helps

SentriOps by Sentrivox brings the infrastructure side of this checklist into one dashboard:

  • Cloud inventory and posture checks. SentriOps connects to AWS, Azure and Google Cloud with read-only access, lists your cloud resources, including AI servers and storage, and flags public storage buckets, unencrypted volumes, over-broad access rights, missing MFA and exposed ports. That covers checks 1, 3, 5, 6 and 8 for cloud resources.
  • Windows endpoints and servers. On Windows machines, the SentriOps agent reports installed software, versions and listening ports, so AI tools running on Windows show up in the inventory with their known vulnerabilities.
  • Vulnerabilities ranked by real risk. Findings are matched against the National Vulnerability Database and ranked with EPSS and the CISA KEV list, so the few that matter come first.
  • SentriOps AI (early access). Explains what a finding means and the recommended fix, in plain language, with the source it came from.

Your team makes the fix, and SentriOps shows the result so you can prove the gap is closed. One limit to know: the SentriOps agent runs on Windows, so model servers running Linux in your own office are not covered by the agent; cloud-hosted ones are covered through the cloud connection.

Frequently asked questions

What is AI infrastructure security?

Protecting everything that runs or feeds your AI: servers and GPUs that host models, tools like Jupyter and Ray, the storage that holds models and training data, and the cloud accounts and API keys that call AI services.

Is a self-hosted model safer than a cloud AI service?

It keeps data in your environment, but you take on the security work: patching the model server, keeping it off the open internet, and controlling who can reach it. Cloud AI services move that work to the provider, but you still own access rights, keys and logging.

Do AI tools really get attacked?

Yes. In 2024 researchers reported an active campaign, named ShadowRay, against Ray AI clusters that were reachable from the internet. AI tools are software, and exposed, unpatched software gets found.

Where should a small team start?

Make a list of every AI server, notebook and cloud AI service you run, then check that none of them is reachable from the internet without a login. Those two steps remove most of the risk.

Sources and further reading

Related reading