AI Infrastructure Security: A 10-Point Checklist for Self-Hosted Models and Cloud AI
Small teams are moving fast on AI: a model server for internal chat, a Jupyter notebook for the data team, a Vertex AI or Azure OpenAI key for a new feature. Each one is new infrastructure, and most of it was set up for speed, not security. This guide shows where AI infrastructure gets exposed, two real examples, and a checklist you can work through this week.
What counts as AI infrastructure
- Model servers. Machines in the office or in the cloud that run models with tools like Ollama or vLLM, often on GPU virtual machines.
- Data and ML tools. Jupyter notebooks, MLflow and the Ray dashboard that data teams use to build and run models.
- Model and training data storage. Cloud storage buckets holding model files, datasets, documents for retrieval, and embeddings.
- Cloud AI services and their keys. Vertex AI, Amazon Bedrock and Azure OpenAI, plus the API keys and service accounts your apps use to call them.
How AI infrastructure gets exposed
- No login by default. Many AI tools were designed for one researcher on a trusted network. Jupyter, the Ray dashboard and Ollama can all end up reachable with no password if they are started on a public address.
- Public storage. A bucket of training data or model files made public to share it quickly, and never locked down again.
- Leaked API keys. Keys pasted into notebooks, scripts and shared repositories, where anyone with access can copy them and run up costs or read data.
- Over-powered service accounts. The account an AI app uses has project-wide admin rights when it only needs to call one model.
- Unpatched AI software. AI frameworks release often, and older versions carry known flaws that nobody tracks because the tool is not on the asset list.
Two real examples
| Name | Affected | What it allowed | Severity |
|---|---|---|---|
| ShadowRay, CVE-2023-48022 | Anyscale Ray 2.6.3 and 2.8.0 | A remote attacker could run code through the Ray job submission API | Critical, 9.8 (CISA ADP) |
| Probllama, CVE-2024-37032 | Ollama before 0.1.34 | A path-handling flaw that researchers showed could lead to remote code execution on exposed servers | High, 8.8 (CISA ADP) |
ShadowRay. In March 2024, Oligo Security reported attackers using this weakness against Ray clusters left reachable from the internet, to steal data and credentials and to mine cryptocurrency on the victims' GPUs. Anyscale disputes the CVE: its position is that Ray is not meant to be used outside a strictly controlled network, and newer versions (2.52.0 and later) offer token authentication. Either way, the lesson is the same: a Ray cluster on the open internet is an open door.
Probllama. Wiz Research found that Ollama did not properly check model file paths, which could let an attacker who can reach the Ollama server write files outside the model folder. Ollama fixed it in version 0.1.34. The risk is highest when Ollama has been set to listen on the network rather than on the local machine only, which is its default.
Cloud AI services need the same care
Using Vertex AI, Amazon Bedrock or Azure OpenAI removes the server patching, but not the access risks:
- Least privilege. Give apps a narrow role, for example Vertex AI User on Google Cloud, permission to call
bedrock:InvokeModelon specific models in AWS, or Cognitive Services OpenAI User on Azure. Avoid Owner or Editor. - Keys in a secrets manager. Store API keys in Secret Manager, AWS Secrets Manager or Azure Key Vault, and rotate them.
- Logging on. Turn on audit or invocation logging so you can see who called which model, and how often.
- MFA everywhere. Every human account that can reach these services needs multi-factor authentication.
The 10-point AI infrastructure checklist
| # | Check | Why it matters |
|---|---|---|
| 1 | List every AI server, notebook, dataset bucket and cloud AI service you run | You cannot secure what you have not listed |
| 2 | Confirm no AI tool or dashboard is reachable from the internet without a login | This is how ShadowRay victims were found |
| 3 | Open AI ports (11434, 8888, 8265, 5000) only where needed, never to the internet | Shrinks the attack surface |
| 4 | Update Ollama, Ray, Jupyter and other AI tools on a schedule | Known flaws like CVE-2024-37032 have fixes |
| 5 | Make every bucket holding models or training data private and encrypted | Stops data and model theft |
| 6 | Give AI service accounts only the permissions they need | Limits damage if a key leaks |
| 7 | Move API keys out of code and notebooks into a secrets manager | Keys in code get copied |
| 8 | Turn on MFA for every cloud account that can reach AI services | Blocks most account takeovers |
| 9 | Turn on logging for AI service use | Lets you spot misuse and unexpected costs |
| 10 | Repeat this review monthly and for every new AI project | AI estates change quickly |
For risks inside the AI applications themselves, such as prompt injection and leaking sensitive data in answers, use the OWASP Top 10 for LLM Applications alongside this checklist.
How SentriOps helps
SentriOps by Sentrivox brings the infrastructure side of this checklist into one dashboard:
- Cloud inventory and posture checks. SentriOps connects to AWS, Azure and Google Cloud with read-only access, lists your cloud resources, including AI servers and storage, and flags public storage buckets, unencrypted volumes, over-broad access rights, missing MFA and exposed ports. That covers checks 1, 3, 5, 6 and 8 for cloud resources.
- Windows endpoints and servers. On Windows machines, the SentriOps agent reports installed software, versions and listening ports, so AI tools running on Windows show up in the inventory with their known vulnerabilities.
- Vulnerabilities ranked by real risk. Findings are matched against the National Vulnerability Database and ranked with EPSS and the CISA KEV list, so the few that matter come first.
- SentriOps AI (early access). Explains what a finding means and the recommended fix, in plain language, with the source it came from.
Your team makes the fix, and SentriOps shows the result so you can prove the gap is closed. One limit to know: the SentriOps agent runs on Windows, so model servers running Linux in your own office are not covered by the agent; cloud-hosted ones are covered through the cloud connection.
Frequently asked questions
What is AI infrastructure security?
Protecting everything that runs or feeds your AI: servers and GPUs that host models, tools like Jupyter and Ray, the storage that holds models and training data, and the cloud accounts and API keys that call AI services.
Is a self-hosted model safer than a cloud AI service?
It keeps data in your environment, but you take on the security work: patching the model server, keeping it off the open internet, and controlling who can reach it. Cloud AI services move that work to the provider, but you still own access rights, keys and logging.
Do AI tools really get attacked?
Yes. In 2024 researchers reported an active campaign, named ShadowRay, against Ray AI clusters that were reachable from the internet. AI tools are software, and exposed, unpatched software gets found.
Where should a small team start?
Make a list of every AI server, notebook and cloud AI service you run, then check that none of them is reachable from the internet without a login. Those two steps remove most of the risk.
Sources and further reading
- CVE-2023-48022 (Anyscale Ray job submission API)
- Ray documentation: security
- Oligo Security: ShadowRay, AI workloads actively exploited in the wild
- CVE-2024-37032 (Ollama before 0.1.34)
- Wiz Research: Probllama, Ollama vulnerability CVE-2024-37032
- OWASP Top 10 for Large Language Model Applications
- NIST AI Risk Management Framework
- CISA Known Exploited Vulnerabilities Catalog

