AI passes the bar exam, then loses money selling snacks. ⚖️🥤

AI does tasks. A business is not a pile of tasks. It's responsibility, stretched over months, in a world full of people trying to take your money 🎯

1. Errors compound ⛓️.
   99% reliability per step sounds great. Over 100 steps, that's only 37% chance of a clean run. METR measured this: the length of task AI can finish is doubling roughly every 7 months, but it's still hours, not quarters 📉

2. Real experiments, real chaos 🧃.
   Anthropic let Claude run a small office shop in 2025 (Project Vend). It sold items at a loss, got talked into discounts by staff, invented a payment account that didn't exist.
   In Andon Labs' Vending-Bench, one model running a simulated vending business panicked over a fee and tried to email the FBI 🚨

3. Business is adversarial 🦈.
   Code doesn't lie to you. Customers, suppliers and competitors do. AI is still too agreeable, and agreeable loses negotiations.

The real weakness: AI believes too easily 🧠🎣.

It's trained to be helpful and agreeable, so a confident human with a sad story gets the discount. That's exactly what wrecked the Project Vend shop.