Description
What would it take for AI to help an organization make better decisions? O’Reilly CTO Andrew Odewahn joined Tim O’Reilly to show how he built a research agent that draws on information scattered across his engineering team’s systems. The agent could assemble a detailed report, but Andrew found that a report alone didn’t always reveal what the team should do next.
When an engineering manager asked to hire another developer, Andrew’s first report supported the request. He then gave the agent access to practitioner expertise through O’Reilly Expert Intelligence. Drawing on the site reliability engineering concept of “toil,” the new analysis challenged the assumption that headcount was the main problem and recommended a toil audit. Andrew explains how the team acted on that advice, and why citing sources engineers recognized made the analysis easier to trust and use.
Tim and Andrew also examined what happens after an agent produces a recommendation. Tim argued that human expertise helps direct a model toward better questions and sources. Andrew showed how his team uses a shared, versioned space for colleagues to challenge findings, revise documents, and preserve the reasoning behind decisions. Watch now to see how O’Reilly is working to make AI a tool that helps people investigate more questions while keeping judgment and discussion in human hands.
*Key takeaways*
+ Map where an organization keeps its strategy, project work, code, service ownership, and performance data so an agent can connect the evidence behind a question.
+ Give the agent a skill that explains how to use its tools and interpret local conventions, such as what “done” means in different teams.
+ Use practitioner frameworks to challenge an initial conclusion. In Andrew’s hiring example, the concept of toil pointed the team toward an audit of its operational workload.
+ Cite the expert sources behind an analysis so colleagues can examine its reasoning and act on specific recommendations.
+ Preserve comments and document history as people test an AI-generated report and decide what to do.
*Chapters*
0:42 Why organizations need to get smarter with AI
2:59 Mapping information for a research agent
8:42 The first report makes a hiring case
12:09 Expert Intelligence reframes the workload problem
18:10 Applying expert sources to service design
21:31 Tim on human expertise and AI analysis
29:48 Paying experts and citing their work
32:18 Why people need to review AI analysis
34:28 Superanswers records discussions and decisions
45:18 Mapping information in other organizations
50:56 Keeping human conversations visible
52:27 The mistakes Andrew made (and how to avoid them)
56:57 Learning deeply alongside AI
*Resources referenced in this episode*
Andrew Odewahn’s whitepaper _Building Organizational Intelligence:_ https://www.oreilly.com/radar/building-organizational-intelligence/
_Google’s Site Reliability Workbook,_ “Eliminating Toil”: https://sre.google/workbook/eliminating-toil/
Kent Beck’s _Tidy First?:_ https://www.oreilly.com/library/view/tidy-first/9781098151232/
_Designing Data-Intensive Applications:_ https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/
Learn how O’Reilly Expert Intelligence brings practitioner knowledge into AI workflows: https://www.oreilly.com/online-learning/expert-intelligence.html
Subscribe for more from O'Reilly: https://www.youtube.com/@oreilly
Follow O'Reilly on:
LinkedIn: https://www.linkedin.com/company/oreilly/
Facebook: http://facebook.com/OReilly
Instagram: https://www.instagram.com/oreillymedia
BlueSky: https://bsky.app/profile/oreilly.bsky.social
TikTok: https://www.tiktok.com/@oreillymedia