Navigating Superintelligence at Work: The Role of Leadership, Trust, and Organizational Readiness

At her dentist’s office not long ago, Sandy Carter found herself in a surprisingly futuristic conversation—not with a doctor, but with a dental hygienist who was explaining how AI was creating a digital twin of Carter’s teeth. The hygienist, eager to keep up with the technology that had entered her workplace, had enrolled in a community college course so she could answer questions from curious patients.For Carter, it was a perfect illustration of the moment we’re in. AI isn’t arriving; it’s already embedded in the everyday tasks of ordinary workplaces, from dental chairs to marketing departments to customer service queues. The question isn’t whether to engage with it, but how to lead people through it.Carter, chief business officer at Unstoppable Domains and author of AI First, Human Always: Embracing a New Mindset for the Era of Superintelligence, shared that conviction, and a great deal of hard-won practical wisdom, during a fireside chat at From Day One’s Silicon Valley conference. In conversation with Steve Koepp, editor in chief and co-founder of From Day One, Carter explored AI adoption, organizational change, and the role of leadership in the AI-era. A Long View on a Fast-Moving TechnologyCarter has been working with AI since 2013, well before the concept became popular and well-known. She was part of the IBM team that deployed AI for Jeopardy! and later helped produce what she calls the first AI-generated cookbook, a collaboration with the Culinary Institute that she describes as an early glimpse of generative AI. “It was kind of like the first taste of Gen AI coming way long ago,” she said.That long view shapes her perspective on the current moment, which she described in her book using a chapter titled “Exponential Baby.” Change is accelerating, she acknowledged, but she’s skeptical of the anxiety it produces. To put the pace of adoption in context, she cited a chart tracking AI usage across millions of people. What it shows surprised even her: roughly 80% of people haven’t used AI at all. About 15% have tried it, but only the free version. Just a small fraction (around 2.5%) have used paid tools that allow them to actually build with AI. And the share developing agents, the most sophisticated form of AI deployment, is barely 1%.Carter signed copies of her book AI First, Human Always for session attendees Her point wasn’t to minimize the urgency, but to dispel the panic. “You’re not behind,” she said. “Everybody doesn’t have the pink cup today.” She was referring to her daughter’s conviction that all her classmates owned a coveted limited-edition Stanley Cup, until Carter called around and discovered that nobody actually had one. “The same thing applies here.”The Trust GapThat doesn’t mean AI adoption is going smoothly. One of the most significant obstacles Carter identified is what she calls the trust gap: a disconnect between how executives perceive AI’s capabilities and how employees experience them on the ground.She pointed to forthcoming research from WalkMe, recently acquired by SAP, which found a 4x trust gap between executives and employees in their confidence around AI. Carter illustrated the problem with a story. She was invited to review an AI dashboard at a Fortune 50 company. The executives walked her through it, everything was green. After they left the room, she turned to the team leads. “I said, ‘Really surprised that your dashboard was all green. I’ve never seen an all green AI dashboard before.’” The team leads confirmed her suspicion. Workarounds had been built; manual processes had been quietly substituted; but the dashboard continued to reflect optimistic metrics. The contrast she offered is Mercedes-Benz, where senior leaders have developed their own agents and brought employees across the entire organization, from assistants to car painters, into rooms together to evaluate where AI works and where it doesn’t. “That’s the best practice that we should be looking at,” Carter said.Agents as TeammatesAt Unstoppable Domains, Carter has put her philosophy into practice. Rather than deploying AI as a tool or using it as cover for layoffs, her team has built a structure in which AI agents function as named teammates, reporting to human managers in an expanded org chart.Inspired by Alice in Wonderland, the team’s 12 agents (including the Red Queen, who handles campaign analysis, and the Mad Hatter, who serves as a brainstorming engine) were chosen collectively, not handed down from the executive suite. The agents report to people managers, and the team has grown its roster from 12 to 45. To incentivize collaboration, when an agent produces something valuable, the human team it supports receives a bonus.The most striking data point from this experiment involves Gen Z workers. Citing a recent survey, Carter noted that 47% of them said they would prefer an AI manager. “It doesn't speak well to the quality of bosses,” said Koepp. But Carter’s explanation was more nuanced. “Why do they want an agent as a manager? Not political. They’re fair. And they don’t care if I work from home.”She sees this as an early sign that agents will eventually take on managerial roles, and that HR needs to be ready for the people questions this raises: Who owns agents? Do they have performance plans? How do you coach managers who are managing both people and AI?The Customer Use CaseFor businesses still on the fence about AI investment, Carter offered a concrete example from her company’s customer service operation. Unstoppable Domains has 4.8 million customers, and its AI agent now handles 48% of all customer service inquiries, without any layoffs. But the story she found most compelling wasn’t the efficiency gain. It was that the company moved to number one in customer satisfaction in its category.The key was rethinking what customer service could do, not just automating what it already did. “Why does customer service just identify a problem?” she asked. Now, when an agent identifies an issue, it can also resolve it, logging the fix in GitHub for an engineer to approve. The agent also flags incoming new customers who run into trouble, prompting personalized outreach from the community support team. New customer acquisition has risen as a result.This is the potential Carter returns to repeatedly: not AI as a cost-cutting mechanism, but AI as a means of raising the ceiling on what’s possible. She cited Deloitte, McKinsey, and a BMW report finding 38% higher productivity when humans and AI work together. “AI plus humans yield stronger results,” she said.What AI-First Leaders Look LikeCarter outlined three qualities she believes define effective AI-first leaders. The first is authenticity: knowing what you understand and what you don’t, and being willing to say so.The second is the capacity to reimagine. The most successful companies she works with don’t start by asking how to automate what they already do. They ask: if we were a startup today, with access to AI, how would we build this function from scratch?The third quality is what she calls being “fearless,” or, in her framing, shifting from brainstorming to what she calls “playstorming.” Executives who want to lead with AI have to be willing to get their hands dirty and fail in front of their teams. “This is not a technology that you can just think about theoretically,” she said. She described vibe coding the AI agent for her own book across 17 different platforms herself, learning from the experience rather than delegating it.Carter closed with what she considers the most important strategic reframe for organizations navigating AI. Most companies approach transformation in the wrong order: they select a platform, then redesign processes, then figure out what to do about people. The companies that fare best flip the sequence entirely: starting with readiness at the human level, then process, then technology.And in that people-first model, she says, HR is central. “I’m going to argue that I think the most important person in the transformation is you guys,” she told the room of HR leaders. “You deal with the people. And I think people is really where it’s at.”Grace Turney is a St. Louis-based writer, artist, and former librarian. See more of her work at graceturney17.wixsite.com/mysite.(Photos by Josh Larson for From Day One)