The robots wrote the brief… badly
Sullivan & Cromwell, one of Wall Street’s fanciest law firms, had to apologize to a federal judge after a court filing came packed with inaccurate citations and other mistakes apparently generated by AI. So yes: even the folks who bill by the minute are now having to fact-check the machine.
Why this matters
This isn’t just a funny lawyer-on-LinkedIn moment. It’s another very public example of AI hallucinations causing real-world headaches in a high-stakes setting. If you’re an investor in legal tech, enterprise software, or any company pitching “AI copilots” to professional services, the message is pretty simple: trust is fragile, and one sloppy output can turn a sales pitch into a liability tour.
The bigger takeaway
The incident also puts a spotlight on governance. Companies may love AI for speed and efficiency, but when the output touches court filings, compliance, finance, or healthcare, the downside of a bad answer can be way bigger than the upside of saving a few hours.
- AI can boost productivity, sure.
- But humans still need to be the last line of defense.
- And in regulated industries, the cost of a hallucination can be reputational, legal, and very expensive.
Big picture: this is less “AI is broken” and more “AI without guardrails is a chaos machine.” Investors should keep an eye on who can prove their tools are reliable, auditable, and boring in all the right ways.