Boards are looking to improve productivity with large-scale language models and AI assistants. But the same features that make AI useful, such as viewing live websites, remembering user context, and connecting to business apps, also expand the cyber attack surface.

Tenable researchers published a series of vulnerabilities and attacks titled “HackedGPT” that demonstrated how indirect prompt injection and related techniques can enable data exfiltration and malware persistence. While some issues have been fixed, others reportedly remain exploitable at the time of Tenable’s disclosure, according to an advisory issued by the company.

Removing the inherent risks from operating AI assistants requires governance, controls, and operational methods that treat AI as a user or device, and the technology must be subject to rigorous auditing and monitoring.

Tenable research shows failures in AI assistants that can lead to security issues. Indirect prompt injection hides instructions in web content that the assistant reads while browsing, instructions that trigger data access that the user did not intend. Another vector involves using front-end queries to seed malicious instructions.

The business implications are clear, including the need for incident response, regulatory review, and steps taken to mitigate reputational damage.

There is already research showing that assistants can leak personal and sensitive information through injection techniques, and AI vendors and cybersecurity experts need to patch them every time a problem occurs.

This pattern is familiar to everyone in the technology industry: as functionality expands, so do failure modes. You can improve resiliency by treating your AI assistant as a live, internet-connected application rather than a productivity engine.

How to actually manage your AI assistant

1) Establishment of AI system registry

Inventory all models, assistants, or agents used in public clouds, on-premises, and Software-as-a-Service according to the NIST AI RMF Playbook. Record owner, purpose, functionality (browsing, API connector), and data domain accessed. Even without this AI asset list, “shadow agents” can continue to have the power to track no one. Shadow AI is a serious threat at a time when it was promoted by companies such as Microsoft, which encouraged users to bring home Copilot licenses to the workplace.

2) Separate human, service, and agent identities

Identity and access management unifies user accounts, service accounts, and automation devices. Assistants that access websites, call tools, and write data require separate identities and are subject to a least-privilege, zero-trust policy. Mapping the chain between agents (who asked them to do what, on what data, and when) is the minimum crumb trail to ensure some degree of accountability. It’s worth noting that while agent AI is susceptible to “creative” outputs and actions, unlike human staff, it is not constrained by disciplinary policies.

3) Restrict dangerous features depending on context

Opt in to AI assistant browsing and independent actions for each use case. For customer-facing assistants, set a short retention time unless there is a special or legal reason to do so. For internal engineering, use AI assistants, but only in isolated projects with strict logging. Apply data loss prevention to connector traffic if your assistant has access to file stores, messaging, or email. Previous plug-in and connector issues demonstrate how integration can increase risk.

4) Monitor like any internet-connected app

• Capture assistant actions and tool calls as structured logs.
• Anomaly alert: A sudden spike in browsing to unfamiliar domains. Attempts to summarize opaque blocks of code. Abnormal memory write burst. or connector access outside the policy boundary.
• Incorporate injection testing into pre-production checks.

5) Build human muscle

Train developers, cloud engineers, and analysts to recognize injection symptoms. Encourage users to report strange behavior (for example, the Assistant unexpectedly summarizing content from a site you haven’t opened). After a suspicious event, it’s normal to isolate your assistant, clear its memory, and rotate its credentials. The skills gap is real. Without upskilling, governance implementation will be delayed.

Decision points for IT and cloud leaders

Risk, cost visibility, and human factors

• Hidden costs: Assistants that browse or hold memory consume compute, storage, and output in ways that are not modeled by finance teams or teams monitoring Xaas usage per cycle. Registries and measurements reduce surprises.
• Governance gap: Audit and compliance frameworks built for human users do not automatically capture delegation between agents. Align controls according to OWASP LLM risks and NIST AI RMF categories.
• Security Risk: Research has shown that indirect prompt injection can be passed through media, text, or code formatting and is invisible to the user.
• Skills gap: Many teams have yet to integrate AI/ML and cybersecurity practices. Invest in training that covers threat modeling and injection testing for your assistant.
• Evolving attitude: We expect new defects and fixes to emerge over time. OpenAI’s fix to the zero-click path in late 2025 is a reminder that vendor attitudes change rapidly and require validation.

conclusion

The lesson for business owners is simple. Treat AI assistants as powerful network applications with their own lifecycles, prone to attacks and unpredictable actions. Put registries in place, isolate identities, restrict dangerous features by default, log everything that makes sense, and rehearse containment.

With these guardrails in place, agent AI is more likely to achieve tangible efficiencies and resiliency without quietly becoming the latest breach vector.

(Image source: “The Enemy Within Unleashed” by aha42 | tehaha is licensed under CC BY-NC 2.0.)

Want to learn more about AI and big data from industry leaders? Check out the AI ​​& Big Data Expo in Amsterdam, California, and London. This comprehensive event is part of TechEx and co-located with other major technology events. Click here for more information.

AI News is brought to you by TechForge Media. Learn about other upcoming enterprise technology events and webinars.

A new academic review suggests that AI benchmarks are flawed and could lead to companies making high-stakes decisions on “misleading” data.

Business leaders are committing eight to nine-figure budgets to generative AI programs. These procurement and development decisions often rely on public leaderboards and benchmarks to compare model features.

The large-scale study, “Measuring What Matters: Building Validity for Large-Scale Language Model Benchmarks,” analyzed 445 individual LLM benchmarks from major AI conferences. The team of 29 expert reviewers found that “almost every paper had weaknesses in at least one area”, undermining claims about the model’s performance.

For CTOs and chief data officers, this is at the heart of their AI governance and investment strategy. If benchmarks that claim to measure “safety” or “robustness” don’t actually capture those qualities, organizations may deploy models that expose them to serious financial and reputational risks.

“Configuration Validity” Issues

The researchers focused on a core scientific principle known as construct validity. Simply put, this is how well a test measures the abstract concept it purports to measure.

For example, “intelligence” cannot be measured directly, but tests are created that serve as measurable surrogates. The paper states that if a benchmark has low construct validity, “high scores may be irrelevant or even misleading.”

This problem is widespread in AI evaluation. The study found that key concepts were often “poorly defined or operationalized.” This can lead to “poorly supported scientific claims, misguided research, and policy influences that are not based on solid evidence.”

When vendors compete for enterprise contracts by emphasizing top benchmark scores, leaders are effectively trusting these scores to be reliable indicators of real-world business performance. This new research suggests that trust may be in the wrong place.

Where enterprise AI benchmarks are failing

The review identified flaws throughout the system, from how benchmarks were designed to how results were reported.

Ambiguous or controversial definitions: You can’t measure what you can’t define. The study found that even when a definition of a phenomenon is provided, 47.8% “disagree” with a concept that has “many possible definitions or no clear definition at all.”

This paper takes ‘non-maleficence’, a key goal in corporate safety coordination, as an example of a phenomenon that often lacks a clear and agreed-upon definition. If two vendors score differently on a “harmless” benchmark, it may simply reflect two different arbitrary definitions of the term rather than a true difference in the safety of their models.

Lack of statistical rigor: Perhaps most concerning for data-driven organizations, the review found that only 16% of 445 benchmarks used uncertainty estimates or statistical tests to compare model results.

Without statistical analysis, it is impossible to know whether Model A’s 2% lead over Model B is a true difference in ability or simply a coincidence. Corporate decisions are being made based on numbers that would not pass basic scientific or business intelligence review.

Data contamination and storage: Many benchmarks, especially those for inference (such as the widely used GSM8K), suffer when questions and answers appear in the model’s pre-training data.

When this happens, the model is not doing any inference to find the answer. It’s just memorization. A high score may indicate good memory rather than the advanced reasoning ability that companies actually need for complex tasks. The paper warns that this “undermines the validity of the results” and recommends building taint checks directly into benchmarks.

Non-representative dataset: The study found that 27% of benchmarks used “convenience sampling,” such as reusing data from existing benchmarks or human trials. This data often does not represent real-world phenomena.

For example, the authors point out that reusing “calculator-free exam” questions means using numbers chosen to facilitate basic arithmetic in the questions. While the model may score well on this test, this score “does not predict performance at larger numbers, where LLMs struggle.” This creates significant blind spots and hides known model weaknesses.

From public indicators to internal verification

For business leaders, this study is a strong warning that public AI benchmarks are not a substitute for internal and domain-specific assessments. A high score on a public leaderboard does not guarantee suitability for any particular business purpose.

Isabella Grandi, Director of Data Strategy and Governance at NTT Data UK&I, commented: “A single benchmark may not be the right way to capture the complexity of AI systems, and expecting to do so risks reducing progress to a numbers game rather than a measure of real-world responsibility. What is most important is consistent assessment against clear principles that ensure technology not only progresses but also serves people.”

“ISO/IEC Good methodology as defined in 42001:2023 reflects this balance through five core principles: accountability, fairness, transparency, security, and redress. Accountability establishes ownership and responsibility for the AI systems deployed, making them ethical and accountable. Security and privacy are non-negotiable, prevent abuse and strengthen public trust. Remedies and appeals provide an important mechanism for society to monitor and allow people to challenge and correct outcomes if necessary.

“True progress in AI will depend on collaboration that brings together the vision of governments, the curiosity of academia, and the practical drivers of industry. When partnerships are underpinned by open dialogue and common standards are established, they build the transparency needed to instill trust in AI systems for people. Responsible innovation will always depend on collaboration that strengthens oversight while maintaining ambition.”

The eight recommendations in this document provide a practical checklist for companies considering building their own internal AI benchmarks and assessments that align with a principles-based approach.

• Define the phenomenon. Before testing a model, an organization must first create a “precise and operational definition of the phenomenon being measured.” What does a “useful” response mean from a customer service perspective? What does “accurate” mean for financial reporting?

• Build a representative dataset. The most valuable benchmarks are those built from your own data. The paper advises developers to “build representative datasets appropriate to the task.” This means using task items that reflect real-world scenarios, formats, and challenges faced by employees and customers.

• Perform error analysis. Beat the final score. The report recommends the team “conduct qualitative and quantitative analysis of common failure modes.” Analyzing why a model fails is more useful than simply knowing the score. It might be acceptable if the failures were all about low-priority, obscure topics. If you fail in the most common, high-value use cases, that one score is meaningless.

• Justify validity: Finally, the team must “justify the relevance of benchmark phenomena to real-world applications.” All evaluations should be accompanied by a clear rationale that explains why this particular test is a valid proxy for business value.

The race to adopt generative AI is forcing organizations to move faster than their governance frameworks can keep up. This report shows that the very tools used to measure progress are often flawed. The only sure path forward is to stop relying on popular AI benchmarks and start “measuring what matters” to your company.

See also: OpenAI spreads $600 billion bet on cloud AI across AWS, Oracle, and Microsoft

Want to learn more about AI and big data from industry leaders? Check out the AI ​​& Big Data Expo in Amsterdam, California, and London. This comprehensive event is part of TechEx and co-located with other major technology events such as Cyber ​​Security Expo. Click here for more information.

AI News is brought to you by TechForge Media. Learn about other upcoming enterprise technology events and webinars.