From ChatGPT Projects to Four Agents With Different Hats (David Pengelley)

# Dave at AIO IRL [00:00:00] **Dave:** so AI agents. We've, we've, we've heard a lot of different things tonight, and I think I could probably give a, a whole 30-minute TED Talk on, um, lots of different things that Kelly said, that Sean said, that everyone said. [00:00:08] Uh, but I will keep it succinct so we've got time for Q&A and drinks afterwards. Um, but AI agents, what they are, when you need one, and why I ended up with 32 of them. I shouldn't have had 32. Spoiler. Uh, we'll get onto that Um, but where did I start? I started like most people with ChatGPT. I was like the ChatGPT whisperer and getting it to, to prompt and I would do... [00:00:32] I would say prompt, I would give it a prompt and I would say, "Now write the prompt." I'd like what-- call it promptception like in- inception, where I'd, uh, just brain dump and say, "Now write the prompt to solve that problem," and it would write the prompt that would solve that problem for me. And then I started using project folders in ChatGPT, where then I could create sort of these static systems. [00:00:51] And so then I got really lazy with my prompting because I kind of taught it the rules. I taught, taught it the guardrails, and then I could put in really a- average ordinary prompts and just ask it [00:01:00] questions. But I'd baked in these rules and this context, so it gave me pretty good answers all the time. [00:01:05] But like we were hearing pricing, all these different things, I was like, "Ah, ChatGPT, is that where I wanna stay? There's lots of other tools coming out." You get a bit of FOMO, you wanna see what's out there. No one else had projects. So I was like, "Oh, kinda stuck with ChatGPT 'cause I really like the projects. [00:01:21] I really like my lazy prompting. What are my options compared to that?" And everyone started then talking about agents. Agents became the next big thing. And we saw n8n, the tool that we just saw Sean talking about, popping up everywhere. And on YouTube everyone was saying, "AI agents for everything in n8, n8n." [00:01:38] Build yourself an agent. All you do is you give your AI some tools. So here's my AI agent, and you give it a, a model, you give it some memory, that's the database that Kelly was talking about, and then you give it a CRM, Outlook, give it a web browser, give it the weather, whatever it needs. Give it all the things. [00:01:55] The problem is when you set up an agent like that, it's really expensive really, really [00:02:00] quickly. Because every time it passes a message, the agent goes, "Oh, I need to check the CRM," and it comes back and then it goes, "Oh, I need to check the email and I need to go back." And it, it starts stacking up all the words for every time it goes back and forth. [00:02:13] So when you start off with an email that's this long, and then it goes and gets some CRM data, and then it goes and gets some other data, and then it gets some more data, you're paying those tokens. You hear about token cost. Your little request, all of a sudden now it's passing through this many tokens every time that agent back and forth between a different tool to do one job. [00:02:31] Which is why you saw Sean's workflow look a lot more like the one at the bottom here, where you have a whole bunch of manual steps, very deterministic steps that always do the same thing. Like you said, math, you don't want agents doing math. That's not how agents work. Use JavaScript, use things like that for doing math And have the agent do the summarization. [00:02:52] Have the agent do the thing that agents are really good at, and keep it to that, 'cause that's much cheaper to run 'cause you're not burning all these AI [00:03:00] tokens on all this stuff you don't need. You really filter it down. And you saw, like from a time-saving point of view, you saw Sean's demo, and you're like... [00:03:07] We're all going, "Gee, this feels like it's taking a while." Imagine if a human was doing that job, it would take so much longer. And if this agent is also passing all this data back and forth, back and fo- back and forth and having a bigger and bigger prompt, it also takes longer. So this is quicker and cheaper when you start doing this. [00:03:23] So I started playing with n8n and going, everyone who's saying you need agents for everything, they're lying. They're lying to you, and I don't know why. What you need is workflows and processes, and we've heard a lot about that tonight from Richard, from Sean, from Kelly around what is that, what is that one job you'd never wanna do ever again? [00:03:41] Can you get AI to solve that problem? And if you can, it's probably a workflow, not an agent So anyway, my, my journey, and I, I documented most of this on my Substack, so davidpinguelli.substack.com. I was writing articles about all of my journey as I went through all these different stages and generating artwork along the way. [00:03:58] Um, but I [00:04:00] discovered Google's Antigravity platform, which is similar to Claude Code. It's similar to AI- OpenAI Codex. But back in sort of December, I discovered Antigravity, where all of a sudden, rather than being in project folders up in the cloud, now I was working in actual file folders on my computer. [00:04:17] So now all that information, all those prompts, all that structure, like Kelly said, you'd need a database to store the information. Now it was all storing on my laptop, on my hard drive. I had it with me all the time, which means if ChatGPT tripled their pricing, I didn't care anymore. I'd switch providers. [00:04:31] And guess what? Google did that to me. Google changed all their pricing, so I left Antigravity and went to Codex. I went to Claude Code. I tried all the different things. But in this world, I started actually playing with agents. And you, Richard mentioned my, my DC superheroes, where I'd have, you know, Lex Luthor as my business guy, and I had Kryptonite as my risk person, uh, looking for weaknesses, giving the comparison around all the different options. [00:04:55] And so I started moving into that world. And while I was doing that and [00:05:00] building some of these workflows, like having my little war room to compare different agent points of view, OpenClaw came out. Now, did anyone in the room hear about OpenClaw? Has anyone heard about OpenClaw? Before that, did you hear about Claude Bot and Malt Bot or any of those things? [00:05:14] You probably didn't. They came and went very quickly. But OpenClaw took off and blew the world away. Everyone's saying, "I have my OpenClaw, and it built five businesses for me, and now I'm a crypto millionaire while I slept." I'm like, I don't know how they did that. That's amazing. I wish they could... I wish I could work out how to get a bot to do that. [00:05:31] But I resisted the FOMO. And like many of you, like, "Oh, I'm scared of missing out on all this stuff. What should I do?" It's like, you know what? What I'm doing with Antigravity at the time is working for me. I have systems and processes. I can interact with it. I can ask it questions. It's got all my context and knowledge. [00:05:47] It's doing what I need. I don't need to chase the next shiny object straight away just because everyone else is doing it. I can't see the business value in chasing the next shiny object, so I'm not going to. Um, but as I mentioned, [00:06:00] um- I was using Antigravity. I was building all kinds of things. I thought, "I can build anything. [00:06:08] OpenClaw looks good, but, ugh, everyone's complaining about the security issues, and it's got this problem and that problem." I was like, "I could, I could do better. I could build a better, better AI agent than anything because I've built a SaaS application. I've built Chart Reporter. Look at this amazing thing. [00:06:23] I can build anything 'cause I've got these vibe coding tools." I, I... So I started building Sergeant. I started thinking, "What if I had all my agents as employees? What if I built a full hierarchical org chart where I could have, like, a CEO agent and a CMO agent and a CFO agent, and I could compartmentalize all the requests, and they all have their own little assignments and break it all down?" [00:06:43] That seems like a really good way. Like, humans organize like that for a reason. Maybe we should organize our AIs like that. Turns out, um, that took a long time to build and was really hard. And in the meantime, I had real work to do. So I went to Cowork. We heard, uh, about [00:07:00] Claude Cowork. I was like, "Well, Cowork's already out there and working. [00:07:03] I can just jump in and use that while I build my thing. My thing's gonna be great for the future, but I'm gonna use Cowork 'cause it's there today." And these were my workflows, my... that I had, Kelly, in, uh, in Cowork. My sort of dawn briefing, where I was going through, checking my emails, cross-checking all different agent profiles and personas and assessing things. [00:07:19] And then I had hourly heartbeats and the end-of-day check. Crazy. Crazy stuff. But Claude Code also got really expensive. Kelly, Kelly mentioned the 5X Max plan was 150 bucks. I'm like, "I can't justify that. Like, this is all fun, but I'm not really feeling enough, like, intrinsic business value to be spending that sort of money to have daily email summaries. [00:07:39] They're really cute and stuff, but do I really need them?" So I thought about what was going on with Claude Code, with OpenClaw, with Hermes Agents, which you've probably never heard of either. Hermes Agents are great. That's what I'm doing now. And went, "I'm living in the future. I've got all these agents. [00:07:57] I've got all these personas. Everything's going [00:08:00] amazing. It's just expensive. What's the next step?" I started migrating all of that stuff into Hermes. And when Hermes... Uh, my hubris to build my own thing is gone. Like, these, these new applications, these teams of people working on this sort of stuff, I need to kill my thing. [00:08:16] That's not an AI slop picture. That's me actually burning stuff in my backyard. Um- In a world of AI, I thought I'm gonna be authentic and use real fire. Um, I killed my project because, uh, when you start compartmentalizing agents that much, A, it actually doesn't work. My idea, my, my glorious idea of what I could build, A, building it was really, really hard to get it right, and B, what I've learned through having so many agents and having so many levels of hierarchy, and even all my personas, my Lex and everything, when I started setting it up in Hermes Well, it got complicated. [00:08:49] I ended up going from having these sort of AI agent personas that could put on different hats and pretend to be different people to having, like, 32 actually [00:09:00] isolated, compartmentalized AI agents that didn't share memory, they didn't share logins, they didn't share anything. And so I spent my entire life just trying to fix login issues. [00:09:10] They couldn't connect to that server anymore. I couldn't get that information into there. They didn't remember that what was happening, or they'd start bleeding, and I'd just... It was negative productivity. I, I actually went really, really badly backwards. Um, I thought, "They've got these Kanban project boards. [00:09:26] This is awesome. I'm gonna have all these agents, and I'm gonna have, like, the CEO agents, and they're gonna be assigning jobs off and doing all this kind of stuff." The problem is, like, who's... If you've employed people, have you ever employed 32 people at the same time and tried to give them all jobs and get everyone running effectively? [00:09:43] It doesn't work. You can't do it. Even in an org of 32 people, you're probably gonna have some management. You're gonna have people to help you manage. And if the managers that you're hiring are just crazy, super intelligent toddlers like AI is, they're not actually much better at managing the other super intelligent toddlers. [00:09:58] They just create [00:10:00] more work for you. And so I ended up with this massive issue with cognitive load because I had these agents that were trying to instruct other agents. They were creating work, they were creating ideas, but they were only half creating them. And then they were running off and then coming back and saying, "We need your help now, we need your help now, we need your help now." [00:10:16] Um, and I just felt like I was babysitting all these agents who were asking me to make decisions about things I had no idea about. They're like, "Make a decision on this thing." I'm like, "What do you... What is that?" Because I was so disconnected now from the process. Talk about the agents need-needed context. I needed context. [00:10:33] I didn't know what my agents were doing anymore because there were too many of them moving too fast, until they didn't, and then they all stopped. And I was like, "I'm stressed out. I've got cognitive load. I don't have the context that I need for all this sort of stuff. I've gone backwards. When I had CoWork, I was paying money for it, but at least it worked. [00:10:52] Now I've got this crazy, fancy, complicated system, and it's broken." So over the weekend, I've actually simplified [00:11:00] right back down. Right back down. I've got... For my core different projects that I work on, um, I've got different environments now that I log into, and I've kind of reverted back to where I was before, where now I've got, like, four employees, and they just know they can wear different hats. [00:11:16] It's like Mario, where he puts on the squirrel suit and starts running around as a squirrel, right? It's still Mario, but now he's got squirrel powers. So I do that with my AIs now. They can put on their squirrel suits, they can get different squirrel powers, et cetera, as they need them But it's one agent with one set of logins, with one set of memory that understands when it passes a job between the different members what was happening when the last person with the last hat did it, versus losing all of that in the back and forth between my superior hierarchical agent model that I was trying to set up. [00:11:48] Um, so if you think maybe you're not doing enough, maybe that's okay. 'Cause maybe if you do too much too quickly, it's actually not beneficial either. Uh, so this is a [00:12:00] bit of a, a cautionary tale around agents. But ultimately, as we've heard a little bit from different speakers tonight, the difference between your ChatGPT and an agent is an agent is connecting to systems. [00:12:08] An agent is actually doing things for you. You can teach it stuff so it can actually think about stuff with reliable context. You don't need to tell it all the rules every single time because you trained it. You've given it the instructions. It knows what to do, so you can do those repeatable processes. [00:12:24] You can see the little comic strip there I've started for the podcast that we do every Wednesday, live midday tomorrow. Um, getting the transcripts and trying to make comics out of that. And so I spent yesterday training it, teaching it what I look like, what Richard looks like, what the other co-hosts look like, so that way when we generate these comics, we kinda generally will always look the same or we'll look recognizable. [00:12:44] 'Cause if we just said, "Here's four guys, one's blond, one's gray, one's this," we'd get such different images. Even trying to train it, it's not 100%. But we do... I do have an agent now who's a comic agent, who knows who the four of us are and knows what our podcast is, so I can give it a [00:13:00] transcript, and it will generate these comics for me automatically next time. [00:13:03] So for those repeatable processes, and that's a job where you do want an agent reading through transcripts, making decisions, thinking through things. Uh, and to Sean's point, sometimes the pictures come back and I'm like, "Why does Richard have a beard now and I don't?" Um, so you still want that human in the loop element as well. [00:13:21] Um, but what it's also enabled me to do is get my hand back on the steering wheel. 'Cause no longer do I have 30 agents creating jobs for me that I don't understand, but I'm back to that interactive mode. There's a little bit of scheduling where I can put in and say, "Okay, once a day go check the email, generate the summary, give me an idea of what's happening. [00:13:37] Maybe create a few jobs." So one of the things that I loved about what I had in CoWork that I'm rebuilding now is it would look at my calendar and go, "Oh, you're having a meeti- meeting with Kelly today." It will then go to LinkedIn, it will go to Google, research Kelly, and write me a full dossier on who I'm meeting with on that day. [00:13:53] That kind of stuff. Automate, do that automatically. Amazing. High-value activities. Um, but I've got my hand [00:14:00] back on the wheel. I'm feeling more productive. I'm in control. I've got four people to talk to instead of 30 people, uh, and it's better. So, um, simplicity is the feature sometimes. Uh, that is... That's all I had to say on that one. [00:14:17] So that's the, the end of the formal talks. Thank you all for your attention. [00:14:24]