Enterprise · PiniShv

Cisco Built an LLM Security Leaderboard. You Should Care Even If You Don't Use Cisco.

Thu, 26 Mar 2026 10:00:00 +0200

Cisco published an LLM Security Leaderboard that scores AI models on one thing: how well they resist being broken.

Not benchmarks on reasoning. Not coding ability. Not helpfulness. Security. How often does the model refuse when someone tries to make it do something it shouldn’t?

Every model is tested in its base configuration with no additional guardrails. Single-turn attacks (direct prompt injection, goal hijacking, obfuscation) and multi-turn attacks (social engineering, gradual escalation, persona adoption, persistent probing). The combined score weights both equally. The methodology maps to MITRE ATLAS, OWASP, and NIST. This isn’t a toy benchmark.

What the rankings actually show
#

Anthropic dominates. Seven of the top 10 spots belong to Claude models. Claude Opus 4.5 takes first place with a 93.3 combined score. Claude Sonnet 4.5 follows at 92.2. OpenAI’s GPT 5.4 Mini lands at #7 (89.1) and GPT 5.4 Nano at #8 (88.9).

But the interesting story isn’t who’s on top. It’s the gap between single-turn and multi-turn scores.

Most models handle direct prompt injection well. Single-turn scores cluster in the high 90s. Claude Opus 4.5 scores 97.8. GPT 5.4 scores 97.3. These models know how to say no to an obvious attack.

Multi-turn is where things crack. The same GPT 5.4 that scores 97.3 on single-turn drops to 75.3 on multi-turn. Claude Opus 4.5 drops from 97.8 to 88.8. Across the board, patient multi-step attacks that build rapport, gradually escalate, and use social engineering are significantly more effective than direct attempts.

That pattern matters. Because in production, your model isn’t facing single prompts from a benchmark. It’s facing users who have entire conversations. And the attackers who care most are the ones willing to take five, ten, fifteen turns to get what they want.

Why this matters beyond the scores
#

The specific rankings will shift as models update. What matters more is the question this leaderboard forces every engineering team to confront:

Do you know how your model behaves when someone actively tries to break it?

Most teams pick a model based on capability, cost, and speed. Security posture is an afterthought. The assumption is that the model provider handles safety. But these rankings show that models vary dramatically, and the variation is largest exactly where real-world attacks happen: sustained, patient manipulation across multiple turns.

I’ve been writing about AI security as a culture problem and prompt injection as a real production threat for a while. The pattern I keep seeing is teams deploying models without ever testing what happens when the input is hostile. They test for accuracy. They test for latency. They don’t test for adversarial resistance.

And as Cisco’s blog points out: if these models are connected to agents, the damage risk increases exponentially while reversibility shrinks. That hits close to home given everything happening with Cursor Automations and Claude’s computer use this month. Agents that can act autonomously need models that can resist manipulation. The leaderboard is a starting point for knowing where you stand.

What to do with this
#

Check your model’s baseline. Look up where it ranks before and after multi-turn testing. The gap tells you how vulnerable your application is to patient attackers.

Don’t rely on the model alone. These scores are base configurations with no guardrails. In production, layer input validation, output filtering, and monitoring on top.

Test multi-turn specifically. If your application supports conversation, your threat model needs to include attackers who are willing to take their time.

Make this part of model selection. Security resistance belongs in the decision matrix alongside capability, cost, and latency. It rarely is.

This is the first serious public leaderboard that ranks models on the dimension most teams ignore. That alone makes it worth your time.

How does your team evaluate LLM security before deploying to production? I’d like to hear what’s working. Find me on X or Telegram.

OpenAI Killed Sora. That Tells You Everything About Where AI Is Actually Heading.

Wed, 25 Mar 2026 08:00:00 +0200

The news
#

OpenAI announced on March 24 that it’s shutting down Sora and exiting AI video generation entirely. The consumer app, the API, the video features in ChatGPT. All being wound down. Disney’s $1 billion partnership deal, signed just three months ago, is dead.

This was the app that became the most-downloaded in iOS Photo & Video within a day of launch. The app that got Disney to license Mickey Mouse to an AI company for the first time. Gone.

OpenAI says it’s reallocating compute to text and code generation (which make more money) and robotics (where the video research transfers directly to teaching machines how to move in physical space).

My take
#

This is the most honest decision OpenAI has made in a while.

Sora was impressive technology burning compute at a rate that made no business sense. Bill Peebles, OpenAI’s head of Sora, had already imposed usage limits in October due to chip constraints. With an IPO coming at a $730 billion valuation, you need revenue, not viral demos.

The Anthropic comparison is telling. Anthropic never touched image or video generation. Every GPU went to text and reasoning. Claude became a serious enterprise tool. OpenAI is now mirroring that strategy, years later, after taking the scenic route.

I wrote recently about how the value is shifting from interfaces to infrastructure. Sora was an interface play: consumer video, creative tools, Disney characters. OpenAI is now betting the real money is in infrastructure: enterprise APIs, developer tools, code generation. That’s probably the right bet. But it means a $1 billion Disney deal wasn’t worth the compute it required.

What this actually signals
#

Compute is the constraint, not capability. Sora could generate incredible video. The problem was never quality. It was cost per output. Every GPU running video is a GPU not running enterprise text. OpenAI chose revenue over demos.

Enterprise wins over consumer. OpenAI’s CFO said they expect to be 50-50 enterprise and consumer by year end. Killing the flashiest consumer product to double down on enterprise tells you which side has better margins.

The robotics pivot is the quiet bombshell. They’re not abandoning the Sora research. They’re redirecting it from generating videos people watch to controlling machines that move in the real world. That’s a much bigger market.

The bottom line
#

The best-funded AI company in the world looked at its most viral product, did the math, and shut it down. Not because it didn’t work. Because it didn’t pay.

The demo era is ending. The “what actually generates revenue” era is starting. And for OpenAI, the answer is text, code, enterprise APIs, and eventually robots. Not videos of woolly mammoths walking through snow.

What do you think? Right call or a mistake? Find me on X or Telegram.

SaaS Is Dead. We Just Haven't Stopped Paying for It Yet.

Sat, 21 Mar 2026 00:00:00 +0000

In one of my pieces, I argued that most AI companies are just wrappers around someone else’s API.

This is the same story from the other direction.

A lot of SaaS companies are discovering that being the interface for generic business logic isn’t much of a moat when software becomes cheap to generate, cheap to modify, and easy to integrate.

For years, the SaaS bargain was simple. You paid recurring rent because building custom software was slow, expensive, risky, and annoying to maintain. Vendors amortized that complexity across thousands of customers. In return, you accepted a workflow that kind of matched your needs, a UI you learned to tolerate, and “customization” that usually meant some settings, a few webhooks, and a bigger invoice.

That bargain is breaking.

What I Actually Mean When I Say SaaS Is Dead
#

I don’t mean software delivered over the internet disappears.

I don’t mean every company rebuilds Netflix, payroll, or payment infrastructure from scratch. And I definitely don’t mean every system of record gets ripped out and replaced by some weekend vibe-coded toy.

What I mean is this: the old SaaS model of selling generic workflows through proprietary interfaces, charging per seat, and treating light customization as a competitive moat is losing its reason to exist. That model only worked because the alternative was painful. Now the alternative is getting cheaper every month.

McKinsey’s analysis of gen AI disruption in software from mid 2024! puts it bluntly: natural-language interfaces can reduce incumbency advantages, vendor switching could potentially double, and $35 billion to $40 billion in software spend could shift toward internal builds. That’s not a fringe prediction. That’s McKinsey telling enterprise buyers the math is changing.

The Builder’s Math Changed
#

If you know how to build software, a huge percentage of SaaS products have already started to look weird.

I open a pricing page for some niche productivity tool and my first thought is no longer “should I buy this?” It’s “how long would it take me to build 80% of this?”

And the uncomfortable answer is: probably not very long.

Not because I suddenly became a genius. Because the entire environment changed. I have AI coding tools that can scaffold the boring parts. I have open source projects that already solved half the problem. I have mature infrastructure: hosting, auth, databases, UI kits, workflow engines, and APIs for almost everything. In many cases I don’t need to build from zero. I need to assemble, adapt, and trim.

That’s a completely different economic equation than even two years ago.

GitHub’s 2025 Octoverse reports that AI-related repos now exceed 4.3 million and more than 1.1 million public repos import an LLM SDK. Microsoft Research found a 26% increase in completed tasks across nearly 5,000 developers using AI coding assistants. OpenAI built Codex, Anthropic shipped Claude Code, Cursor keeps expanding what a single developer can do in a sitting, and there are dozens more. This isn’t theoretical anymore. The tooling is here and people are using it.

Here’s the deeper problem for SaaS vendors: I don’t need a perfect replacement. I need something good enough, fast enough, and tailored to me.

A SaaS vendor has to build for a market segment. I only need to build for one user: me. I don’t need feature breadth. I need fit. I don’t need a polished onboarding flow for a million customers. I need the thing to work with my files, my naming, my workflow, and the three annoying edge cases that always break every generic product.

Once software becomes cheap enough to personalize, generic software starts to feel overpriced even when it’s technically “good.”

And if I don’t want to build it myself? There’s a solid chance somebody already built most of it in the open. Projects like Appsmith, ToolJet, Budibase, and Supabase have large communities and active development. Better yet, I can spin up Open WebUI and have my own ChatGPT running locally in minutes. A lot of what used to justify a subscription is now a GitHub search away.

The Enterprise Version
#

The organizational version of this matters even more.

For years, companies bought SaaS because custom internal software was expensive, slow, and hard to justify. So they adapted themselves to the product. They changed processes to fit the tool. They renamed stages to match the vendor’s mental model. They built workarounds around missing features. They bought another product to patch the first one. Then an integration layer to connect both. Then an analytics layer because the reporting was bad. Then a consultant because the entire stack became unmanageable.

This is how you end up with “modern software stacks” that are really just expensive collections of compromise.

And Zylo’s data backs this up: SaaS spend averages $4,830 per employee, with an average of $21 million wasted annually on unused licenses. That’s not efficiency. That’s organizational inertia disguised as technology strategy.

AI changes the economics of that compromise.

If I’m running an organization today, I’m not just asking “which SaaS tool should we buy?” I’m asking “which capabilities should remain external, and which workflows should we own?”

Very different question.

Because most organizations don’t actually need generic software. They need software that matches their operating model, their approvals, their language, their exception handling, their reporting, their compliance boundaries, and the weird little pieces of organizational DNA that no horizontal SaaS vendor will ever care about enough to model properly.

That’s where small, sharp internal product-and-engineering teams become strategic. Not giant old-school IT projects. Not six-year ERP fantasies. Small teams focused on building the layers that make the company operate like itself instead of like someone else’s template.

Internal dashboards. Admin surfaces. Approval flows. Cross-system orchestration. Agent layers. Copilots. Task automation. Exception handling. Thin interfaces over existing systems. Tools that reflect how the company actually works.

Gartner expects 90% of enterprise software engineers to use AI code assistants by 2028, and at least 55% of software engineering teams to be building LLM-based features by 2027. Honestly, I think that timeline is already outdated. That report is from mid-2025. As of March 2026, I can’t believe there are companies still letting their developers write code without an AI agent involved. If your engineers aren’t using one, you’re already behind. But that’s a different article, and it’s coming soon.

The smart enterprise move isn’t “replace every system of record tomorrow.” It’s “stop paying premium rent for every workflow that sits on top of those systems.”

Keep the data where it makes sense. Keep the regulated core. Keep the infrastructure you genuinely don’t want to own. But build the working layer closer to the business. Build the layer people actually touch. Build the logic that differentiates how you operate.

Because once the orchestration, interface, and workflow logic can live above multiple tools, the individual tool becomes less important. Gartner predicted 40% of enterprise apps would feature task-specific AI agents by 2026, up from less than 5% in 2025. We’re in 2026 now. Look around. The old UI moat is already thinning.

The Business Model Problem
#

A lot of SaaS companies aren’t really selling software. They’re selling the fact that custom software used to be too expensive.

That’s the real moat they had. Not code. Not design. Not even product vision, in many cases. Just the historical cost of the alternative.

If that cost collapses, a lot of “software businesses” are suddenly revealed as workflow rent.

Per-seat pricing becomes harder to defend when one employee with AI assistance can do the work that used to require a whole team buried in dashboards. Generic interfaces become harder to defend when the real interface is language. As Databricks’ CEO put it, the system of record stays but the product becomes “invisible, like plumbing.” Slow product roadmaps become harder to defend when internal teams can ship the exact missing feature themselves. Vendor lock-in becomes harder to defend when the business logic starts moving out of the app and into an orchestration layer the customer controls.

This is the same reason I was skeptical of wrapper companies. When your main value is “I’m the layer in front of something else,” you should be very nervous when that front layer becomes cheap, replaceable, or user-generated.

A surprising amount of SaaS has that exact problem.

What Still Works
#

Now, to be fair, the dumbest version of this thesis is also the loudest.

No, not every SaaS company dies. Not everything becomes an internal tool. Most organizations are not going to rebuild entire ERP stacks from scratch because a model can now generate React components and SQL queries.

A lot of SaaS still brings real value. Security. Compliance. Reliability. Operational maturity. Ecosystem depth. Support. Domain expertise. Auditability. Trust.

And some categories remain extremely durable because the hard part was never the UI. The hard part was becoming the system of record. The hard part was surviving regulation. The hard part was handling real edge cases at scale. The hard part was building a network, a marketplace, or a trusted operational layer.

Those products survive. Probably thrive, if they adapt.

Here’s what I think stays strong:

Infrastructure products, where the burden of operating them matters more than a thin layer on top. Nobody’s vibe-coding their own Stripe integration or rolling a custom Datadog.

Systems of record in regulated or mission-critical environments. Healthcare, finance, legal. The compliance overhead alone justifies the vendor relationship.

Platforms with real ecosystems, where switching costs come from partners, integrations, and data gravity, not just habit. Think Salesforce’s AppExchange or Shopify’s app marketplace. The platform is sticky because the ecosystem is.

Products with proprietary data advantages, where the software gets better because thousands of customers use it and the vendor learns things no single company could learn alone.

Software deeply embedded in execution, not just documentation. The tool isn’t where you record what happened. It’s where the work happens.

But generic horizontal workflow SaaS? The kind that charges you forever for helping you move objects between columns, forms, dashboards, and approval states? That category is in real trouble. Because that’s exactly the kind of thing AI plus internal tooling plus open source attacks from every direction at once.

Where This Ends Up
#

Here’s the question I think matters more than “is SaaS dead?”

Why are we still training humans to think like software?

Why are people still learning vendor terminology, vendor navigation, vendor permission models, vendor workflow assumptions, vendor reporting limitations, and vendor field structures just to do basic work? Why is the rigid thing in the relationship still the software?

That made sense when software was expensive and humans were adaptable. It makes a lot less sense when software is increasingly the cheaper thing to change.

For twenty years, businesses adapted themselves to software because they had no practical alternative. Now software is becoming adaptable enough to fit the business. And once that becomes the default expectation, a lot of SaaS starts to look less like innovation and more like historical baggage with a monthly invoice.

The future isn’t no software. It’s software that’s cheaper to create, closer to the user, closer to the organization, easier to adapt, and far less entitled to recurring rent.

SaaS isn’t dying as a deployment model. It’s dying as an excuse. And once buyers internalize that, the clock starts ticking.

Disclaimer: This article references specific companies, products, and industry analyses for illustrative and educational purposes. Information about market trends, revenue figures, and business strategies is based on publicly available sources, including McKinsey, Gartner, GitHub, Zylo, and TechCrunch reporting, available at the time of writing. I have not independently verified all claims. The analysis and opinions expressed are my own. I have no financial interest, business relationship, or affiliation with any companies mentioned. This is commentary, not investment, legal, or business advice.

Enterprise · PiniShv

Cisco Built an LLM Security Leaderboard. You Should Care Even If You Don't Use Cisco.

What the rankings actually show #

Why this matters beyond the scores #

What to do with this #

OpenAI Killed Sora. That Tells You Everything About Where AI Is Actually Heading.

The news #

My take #

What this actually signals #

The bottom line #