How to Stop Holding AI Agents Back
May 29, 2026 – 8:37 am
Image by: Steve A. Johnson via Unsplash
Developers of agentic AI have been making some big claims—from booking flights and monitoring competitors in real-time to managing entire procurement cycles without human intervention. While the technology for many of these tasks is already available, the infrastructure needed to support them at scale remains inadequate.
Gartner recently projected that over 40% of agentic AI projects will be canceled before the end of 2027 due to escalating costs, unclear business value, and insufficient risk controls. Despite expectations that autonomous agents would mark AI’s maturity, this shouldn’t surprise anyone who has witnessed their limitations in real-world scenarios.
The underlying problem isn’t necessarily the quality of the models but rather the challenges these agents face when interacting with the web.
Why the Web Resists Agents
A capable agent needs to access a website and interpret its responses, then translate that information into something usable, consistently and in real-time. This is easier said than done on today’s web, where platforms often:
- Depend on personalized results to shape user behavior.
- Employ sponsored placements for revenue.
- Use urgency cues to influence decisions.
These tactics make automated access difficult, hindering AI agents from performing tasks like comparing different platforms and selecting the best option for users.
Oxylabs’ upcoming Web Openness Index sheds light on this issue by evaluating:
- Practical Reachability: The global average score is 83.4 out of 100, indicating reasonable access to sites through standard automated HTTP requests.
- Anti-automation Friction: Sites implement measures like CAPTCHAs, rate limiting, fingerprinting, and bot detection (average score: 62.8), creating significant obstacles for agents.
- Structured Data Interoperability: Many websites return data in formats that machines struggle to interpret (average score: 60.3).
These substantial disparities highlight a fundamental disconnect—while sites generally respond to automated access requests, they also implement numerous restrictions and deliver data in machine-unfriendly formats, hindering agent effectiveness.