# Transcript

**Anand**: [00:00] All right. It wants to flop. Oh, okay. So, this is just for this room. Okay. The online mic is different. Okay. And I'll assume that people online can hear me. So, one possibility that we could use for collaboration is simply the WhatsApp, right? So, we are on a WhatsApp group and, so, as we go along, if we need to do some collaboration, we can use this as well. But I will also have another mechanism where we'll be able to do stuff.

**Anand**: [00:49] I'm not really going to do much of the work. This is a workshop; you're going to be doing most of the work. Actually, you are not going to be doing most of the work; you're going to be delegating most of the work to AI. That is what we are here to learn. And what is left, what it cannot do, is broadly what we will be doing. **The premise here is that earlier humans were, in some shape or form, the consumers of data. Software became a big consumer of data. Now agents are becoming a big consumer of data.** We kind of know how to organize stuff for software, for humans. We are not sure we really know well how to organize it for agents. And because nobody probably has a good enough point of view, and because agents are moving so fast that if we have an answer today, it will anyway become out of date, this is a "teaching you to fish" rather than "giving you a fish" kind of a workshop.

**Anand**: [01:46] Meaning, what we'll do is **figure out how to figure out whether a particular way of organizing data for agents works well or not.** And to do that, we will first have AI do some research. There are two QR codes on the screen, and the one on the left will give you a prompt. The one on the right will be a location where you'll be able to share it later, once I've started my server, which is sitting on my machine. But for now, could everyone please scan the QR code on the left side and see if you are able to open that link?

**Anand**: [02:37] Okay, I'm not able to. Is anyone else able to open the QR code? You're able to? Okay, I don't know why I'm not able to. So, I will try something. I'll just, sorry, change the background to white and remove the background image. And please try your luck now, in case it was failing earlier for you. Oh, okay, it's back.

**Anand**: [03:14] If you get a screen that has that begins with a "raw.githubusercontent.com," that is what you want. To be on the safe side, I will also paste the URL in a few seconds on WhatsApp, and that will be an alternate source. I'm just putting it on WhatsApp. Now, this is a—since it's on WhatsApp, in case you're not able to get it from the QR code, just open it from WhatsApp.

**Anand**: [03:45] Now, let's open this link. **This is a prompt that we will gently go through. We're not going to go through the full details of the prompt, but broadly it's saying: Look, AI agents are the ones that are consuming the data. Now, what we want to do is figure out, by you doing the research, what are the ways we should organize it for agents rather than for humans?** So, search online, go through some good sources, give me the top 10 techniques, and prioritize—give me the better techniques on top—and finally rank it on a table. And for each of the top 10 techniques, write it in this particular format.

**Anand**: [04:31] Now, this is step one of what we'll be doing today, where you'll just blindly copy-paste this into—and I'll suggest which agents you might want to use—and let it do some of the research. Then, we will collect our collective research and see, amongst those, are there some strong patterns that are emerging and test out those techniques? I don't know what are the best ways of organizing data. I have my opinions; so do others. None of us know. **Let's find out, and if we have an experimental approach by which we can actually validate if that organization of data is working well or not, great, we have learned an approach.**

**Anand**: [05:14] So, what I'm going to do, and here's what I invite you to do, is to copy this and put it into your best agent. And what I mean by best is, firstly, certainly paid. If you have a paid ChatGPT account, great; paid Claude account, great. Gemini may not be so great; Perplexity may not be so great.

**Participant**: [05:32] I've not used up my credits for today.

**Anand**: [05:34] *Are vah* [Oh wow], congratulations! Here's an opportunity. So, I have set this to GPT-4.5 on ChatGPT and set it to "high" as the level of thinking. This is my configuration for ChatGPT. And I'm going to paste this entire prompt exactly as it is and submit. I'm also going to submit the same thing to Claude on the highest model. If I'm not mistaken, yeah, Fable is vanishing in a short while. So, before it vanishes, on Fable, I'm not going to put the effort at max; I'll at least put it at high and paste the same prompt. But Opus is probably fine.

**Anand**: [06:23] Now, we'll let it do all of its research. But we are going to discuss what it finds. And one of the things that we will do is submit the results of what we are seeing on a forms page, which I will enable in a short while. Give me a minute. Just ignore everything that I'm doing; this is all part of the setup that I was to have done an hour ago.

**Anand**: [07:31] Now, while it's running, here is a question that I will pose to all of us for discussion. I'm going to paste a second link where we can submit the results. **What is your guess? How should we organize data differently for agents versus humans?** Please visit the link that you can scan on the top right, or you can just click on the link on WhatsApp. Either will lead you to the same page, or "forms.s-anand.net"—it's all the same. And this is a form I would suggest you just keep on your browser somewhere because we will be filling out a whole bunch of things in here. So, the first thing that we want to take a shot at is your guess on how we should organize data differently for agents versus humans. One sentence, one paragraph, one book, however long you want, feel free. And yes, please?

**Participant**: [08:35] JSON, maybe? Because it has named keys and so, like the prompt decides, it will be hard to understand which column and which row it belongs to, like JSON might be a bit better.

**Anand**: [08:48] One possibility. But I'll invite everyone to just write down one or two points before we get into the discussion. That will be the starting point of your hypothesis. Once you submit, you will not be able to change the results. So, just spend two, three minutes. And **the point is not to use AI to get to the answer. The point is not to get to an answer; it is to set a prior.** "Gut feel: this is what I think," so that we can compare, and that will give us more learning, hopefully. And you can do this while your agent is running whatever it's doing—ChatGPT, Claude, etc.—the other prompt, that is.

**Anand**: [09:39] For those who are joining remotely, in case you have any issues with any of these, please just ping on WhatsApp. I'll be keeping an eye on that. Just—yes, it comes back. Yeah, for some reason it's going off. Once—it's not a major worry.

**Anand**: [10:02] Okay, we have a couple of responses on how we should organize data differently. I'll start the discussion once we have maybe about 10 responses. Like I said, you're welcome to write as much as you like, or hold off until the discussion also if you want. I'll just wait till we have about 10 responses and then we'll take some of those and start discussing.

**Anand**: [10:59] Okay. Please continue submitting. I am going to see what the results look like. Okay, bunch of stuff. But I'm just doing a quick look at what are the words that are popping up on top, and you'll see that on the screen in a short while. So, **context and metadata seem to be a couple of words that have popped up.** Not going through the actual text, which we will in a short while. But yeah, context, metadata.

**Anand**: [11:47] Let's see what others have said. Context about the domain. Okay, let me start at the bottom. **Data should be organized as graphs as opposed to humans who may not need graphs as much.** Possible. **JSON or XML might perform better than CSVs because they have named attributes for each value.** That used to be a strong—I mean, there were some results around this. And then people found that with newer models, they are able to understand CSVs at least as well, and the token compression seems to have some benefit. But does that hold true for all models? Unclear.

**Anand**: [12:28] We need to provide semantic meaning and domain knowledge. Yes. What I wonder, though, is—so for each of these, I'm not saying that this is right or this is wrong. For every one of these points, because they are valid, I'm also offering a counterpoint. My main idea is, look, let's test. And then we will know for sure. Until we test, we don't know. But why would we even need to test? I'm offering a few reasons. Like providing semantic meaning and domain knowledge. Yeah, but wouldn't we do that for humans also? **Markdown all the way; content must originate as markdown.** Would that be different for humans? And would humans particularly find an alternative more convenient? **More context, self-cataloged data, data catalog as an integral part of the data.** Again, very true. Why would humans find it different? **An "agents.md" explaining the context, query, etc.** And one of the themes that I'm seeing, therefore, is that we need to explain the data well. Completely agree. And I think that is an important part. We really should do that. It is probably not very different from what we would need to do for humans as well. No harm in still learning that as a technique, especially because we can use AI to create and curate the data in that particular form, and we'll probably be doing a bit of that as well.

**Anand**: [13:54] Let's see. Context about the domain and use case, structuring the metadata, how it was collected, syntactic format, etc. And yes, please keep the responses coming.

**Anand**: [14:08] Now, while this is happening, we have few results from the models. And what I will invite all of us to do is fill out a second question. And this question, which I will call the "research source," is basically: **Which agent and model did you use to get the answer?** You may have used multiple models. Please pick any one that you're happy with. In my case, for instance, let me fill this form in. I will choose the result that I got from ChatGPT.

**Anand**: [15:05] So, I'm filling out ChatGPT with GPT-4.5 "high." And that will be the first part of my answer. In your case, whichever one you have chosen. If you have multiple, pick what you feel would be the best. The next thing that I'll request that we fill out is the actual response. The results—what did the agent say? In—okay, what techniques or differences did your agent suggest? Which now becomes the third question. And I'm going to copy the answer and paste it here, though it's probably going to be easier if we don't copy-paste it directly but rather use the copy button to do the copy-paste. The copy button for ChatGPT is at the bottom, and for Claude, it's at the bottom. Depending on what agent you're using, it may vary. So, I will just copy it. And the reason I'm suggesting that we copy it from here is it preserves the markdown formatting. That's a little convenient. And I'm just going to paste that and submit. So now we have three answers that have emerged.

**Anand**: [16:30] Once we have a few more of these answers, let's do a collective lift-up and say, across agents, what are we seeing as the overall pattern? But before that, I'm also curious what are the agents that we are using as a group. Let's look at it. Claude Opus 4.8, Gemini—okay. I'm curious what model in Gemini we used. But okay. Codex with GPT-4.5 High Reasoning, Claude Min-Max—okay. Gemini 3.1 Pro, Copilot with Opus is interesting.

**Anand**: [17:15] Okay, that's a wide range. **One thing that I would suggest for the Gemini users in this group: some of what we are doing, Gemini doesn't do as good a job today as a Claude or ChatGPT.** For the—okay, I'm—yeah, Hermes with whatever agent can probably do the job. So, just see if you could—and in case you have a paid version of ChatGPT or Claude—explore that in addition to Gemini and whatever else.

**Anand**: [17:51] And the—okay, we have seven people who have copy-pasted and submitted. Is that right, or has it updated? Seven.

**Participant**: [18:00] Can we get the link?

**Anand**: [18:01] Oh yes, sorry. The link to the form is out here. You can use that as a QR code. It is also on the WhatsApp group. It is the most recent link on the WhatsApp group, whichever is convenient. And for anyone who may have also joined online late or room late, what we are doing is there's a link just above the last link on WhatsApp. We copy-pasted that and put it into the best paid agent we have access to and are pasting the answers out here. In other words, we're delegating our research.

**Anand**: [18:43] And—two, four, five. Okay. Let me get rid of all of the initial—actually there's a good way of doing this. Select answer from responses.tsv. Okay. No, okay, doesn't matter. I'm just going to take all the answers so far and ask ChatGPT to summarize what we have.

**Anand**: [20:53] Let's spend—actually, let me share a bit about what we're going to do next. **The first half of this workshop is really about saying: let's find what agents think are different for agents versus humans. And then we will, in the second half, run these as experiments.** Yes, please?

**Participant**: [21:13] So you asked us to ask Claude or ChatGPT or whatever. But essentially it is just regurgitating what people have written about on the internet. So, are we like basically looking at a high-level multiple sources to get that? Because I don't think there's any logic behind what these guys would tell us.

**Anand**: [21:33] Fair point. The question is, if an agent is simply going to search online, find what is there on the internet, and give it back to us, maybe that has some value, but is that really valuable? And fair point, it has some value in that it would save us time and research and exploration. Maybe it doesn't if we know better. My hypothesis is that, on average, there are more things that the agent knows about than we know about. Where we are an expert, it makes sense to just say, "This is my answer; now you tell me where I might have missed something," and all that. But I will take a decision. In other areas, we don't have a basis to judge the agent. In that case, we say, "Now, I know a little bit about how to prompt and all that. Maybe I don't even know that. You tell me, this is my overall objective," and see how it is helping.

**Anand**: [22:36] **I'm taking the non-expert perspective here for two reasons: I believe that over time agents will get smarter—they certainly seem to be continuing to get smarter—and the number of areas in which we will be able to have deep expertise is small.** There's nothing stopping us from having both approaches. And which is what the first part was also about—our prior, where we are saying, "This is what I believe is the agent's response." And it will be good to compare, therefore, the first part versus the second, and see what it is that humans missed that agents caught, and what it is that agents missed that humans caught, which of these seem more valid. And the best part is we will be able to generate some synthetic data, test it out against that, see if it is in fact valid, and therefore know who's doing better. Benchmarking our own skill. Any other—sorry, please feel free to ask a question at any point.

**Participant**: [23:32] Isn't it too wide? Because we are asking about everything about HTMLs, PDFs, everything.

**Anand**: [23:46] Good point. Is it too wide? One of the principles I follow is, because agents are getting smarter and smarter, if as of, let us say, June 2024 I have a point of view on what an agent cannot do, I don't know when it is going to be out of date. I have a sense that I'm keeping up with the agents only when I give prompts that agents fail on. So, as a rule of thumb, **I start with something tougher than I believe agents can solve, see if it really fails and totally messes up, and then I know, "Okay, fine, this really is not possible."** Every now and then—happens more than half the time—I find that something that wouldn't have been possible last month is now suddenly possible. Then I mark it saying, "Okay, now this is new," which is an opportunity I will miss unless I pose harder and harder problems. Harder problems are becoming harder to find. So, I go to agents for that, saying, "Can you give me a problem that you can't solve?" and then it gives me something that I don't understand, which becomes the other problem.

**Anand**: [24:47] So, the other thing that I'm trying to do is get myself out of the loop. My understanding is bounded. I could learn, but that is a slow process. I could learn, but that requires initiative and I have no enthusiasm for it—so many other things. Instead, if there is a way of getting to a better outcome without me having to learn, good, I don't mind. I'm just a manager, not an executor or a subject matter expert. I just want to get the stuff done. Not saying that this is the right perspective, this is the only perspective. I'm saying that in many areas, this is an effective perspective, and there'll be a few areas where you will ideally want to be the expert.

**Anand**: [25:27] So, now, it is saying here are a set of—okay, let me do this. No, I'm just going to put this on the screen. We will walk through a few of the things that it's saying and discuss those. The idea is not to say the focus should be only on testing what the agents are saying. We will test a few other things as well. But what it's saying is **"progressive just-in-time access" is something that we would want to organize differently.** And this apparently is coming through in 11 out of the 15 submissions that we have from across agents. Popularity is not necessarily the same as correctness, but it is probably correlated and worth paying some attention to.

**Participant**: [26:13] It might be correlated because a lot of people are using the same models.

**Anand**: [26:17] That is one possible reason. Another possible reason is the internet is filled with the same kinds of—so the source could also be polluted even across—not polluted, but have common elements. Both are fair points.

**Anand**: [26:30] **It's saying for humans, assemble all plausibly relevant documents, schemas, etc., in one portal before the work begins. Whereas for agents, provide a compact catalog of paths, URLs, metadata, query handles, and let the agent search and fetch selected content only when needed.** Which is an interesting premise. Why? Pre-loaded material consumes attention regardless of relevance, and **progressive disclosure keeps distractors away.** There's no way I would have guessed—I would not have put this on my list. Any thoughts?

**Amit**: [27:11] I think progressive disclosure, like—I mean, this is not very different for humans, but in agent's phrasing it is, I guess, the hypothesis here, the why you want that for agents to do that, is to manage context window and the token budget for the agent would get exhausted. So, you want it to be—I mean, it can be more, but yeah, you want it to be able to, like, go down different paths when required. So, linking out, like the rest is probably helpful.

**Anand**: [27:48] Got it. Just to repeat for the benefit of those remote, Amit was saying that progressive disclosure may not be a surprise, and it may help it fan out and avoid crowding the context window. So, understandable, but perhaps not non-obvious. Fair point.

**Participant**: [28:08] One counter to progressive disclosure: it works perfectly well for narrow queries, but it also inhibits the ability to connect the dots. Like humans would do. In the course of time, I need this research, but I have looked across other sources. That table has this, so now they're right. So the connecting the dots, I feel like the progressive disclosure may be a little bit of an inhibitor.

**Anand**: [28:30] Sorry, maybe a—?

**Participant**: [28:31] Maybe an inhibitor to connecting the dots effectively across different problems in the same domain. I've seen the same problem elsewhere, this table may not be reliable because that table also was not reliable in that problem and that—

**Anand**: [28:46] Got you. Whereas if you had it all in your head, then you'd be able to make those connections better. That is certainly a possibility. And of course, context window being a limitation, but still that doesn't mean that this is—fair point.

**Participant**: [28:58] Yeah, I think very similar on that. Context setting, the way that humans think is they're going to have a 50-foot view of what is actually the domain that they're engaging with, then cascade down to very specific sets of hypotheses that they want to investigate, which is not really a pattern that you can do with agents because agents will much rather go for specific facts to build first high-level views and then work back down. I think there is clearly that difference. Like when you do like a research, for instance, you're first trying to figure out how to create an opinion about the subject matter at the high space and then find the set of facts that you want to collaborate—this same rationalization dynamic that goes on in humans.

**Anand**: [29:43] Fair point. And to paraphrase very crudely, it is that once you have the stuff loaded in your head, you know what's important, which direction to go on, etc., which if you have to do step-by-step, it's likely to be slower, harder for an agent. Fair point. And we know why it's suggested.

---

**Anand**: [00:00] ... suggested as a technique because it doesn't yet have enough of a choice about the context window, whereas we probably have a better way of organizing memory. Go on.

**Participant**: [00:09] I was thinking about progressive disclosure and I had similar thoughts. But I was also thinking that maybe the effort, right? Like when we say "effort" high or low, maybe that's basically what expands the search space, you know? And therefore, like if you want to high-effort, it's probably going to look at more options to arrive probably at a better answer and then rank it to give you a result.

**Anand**: [00:30] Right. In other words, once it has the full context, it knows which to cherry-pick as the most important ones. Quite possibly.

**Anand**: [00:44] For each of these, it would be good to just test it out. Nothing quite like it, right? And that's what we'll do in a short while. But let's go on to what it's saying. It's saying that I will be—well, "what matters when" I think is obvious; I'll skip that. It's saying that on 24 large corpus tasks, **index-and-fetch agents achieve equal or higher success with substantially fewer tokens than agents given the full corpus upfront**, as a testable claim. Now, maybe it will, maybe it will not. That's exactly what we are trying to test. And it sounds like a reasonable claim. What we are not testing is whether agents, when they are able to handle the full context, do better. That is an impossibility because context windows are limited. So, keeping the context window limitation reality, it's saying in one case I'll give the full context, in the other case I will not give the full context, and we will test both of these out. And the evidence will basically be—let's see. Oh okay, this is the evidence for why it's reporting this. Fair enough.

**Anand**: [01:59] At this point, let us branch out. I was going to go through a few of these, but since we have had a discussion on this, we will run a small benchmarking test against this. How are we going to do that? Let me copy this and we'll do a little bit of editing to get just T1.

**Anand**: [02:27] Now, there is another prompt that I will share which will do the benchmarking. But what we're going to do is take this, give it to an agent with a bit of a prefix and say, "**Write a benchmark for this particular claim.**" My proposal is that we do that together, meaning we all run with the same benchmark so that we will have a comparable base and then see across different machines and—rather than one person burning lots of tokens, we have lots of people burning a manageable number of tokens and still collectively learn from that experiment.

**Anand**: [03:03] So, the prompt that I would suggest we use is out here. Wait, before you copy that, I am going to paste the link in the chat window. This is the "benchmarking prompt." And let me show you the thing. So, the one on the left is the benchmarking prompt. You can take it from WhatsApp if that's easier or take it from here. **Please copy that prompt and put it into, preferably, a coding agent.** And don't submit yet because the hypothesis is the next thing that you need to paste.

**Anand**: [03:52] What I mean by "preferably a coding agent" is Codex, Claude Code, Copilot CLI, Antigravity if you like, Pi, OpenCode, Tao—whatever your preference is. And **why a coding agent? Because it will be able to, a bit more than ChatGPT online or Claude online, create datasets, download tools, install stuff, and so on.** Not that you can't use ChatGPT or Claude.ai for this; you could, and if you don't have an alternative, that's perfectly fine. I would not advise using Gemini for this unless you have no alternative because its environment isn't as good.

**Anand**: [04:42] So this is the first part of the prompt. The second part of the prompt is the actual test, and I will put that on WhatsApp as well. You could just copy this entire thing and paste it. So, the entire result will be—in my case, I'm just going to copy this text and this text and put it into some coding agent somewhere.

**Participant**: [05:32] Is it T1?

**Anand**: [05:33] Sorry? T1, yes. Yes.

**Anand**: [05:48] Does anyone need help with this step, running the benchmark, either online or in the room?

**Participant**: [06:01] If the agent is reading search results, it will be a tool calling thing that we provide. It will be—OpenAI how will it do that thing?

**Anand**: [06:14] **Pata nahi (I don't know).** It's a good question. The question is: how is this going to actually do a comparison between these two methods? I don't know. Maybe it's going to totally mess it up as well. So let's take a look at the prompt that we are asking it to test.

**Anand**: [06:31] The benchmarking prompt says, "Look, I'm going to give you one hypothesis and your job is to test the hypothesis." The second part of the prompt says, "**Create a data set.**" The ground truth is basically you have to construct a realistic data set that is suited to test the hypothesis. The third step is: create two parts. Organize one folder as if for humans based on whatever the hypothesis said, and then Part B is organizing it for the agent (or vice-versa). You should randomize it. And then, you know, document what are all the tasks that it's supposed to create and benchmark on both data sets, probably through sub-agents. So you will spawn sub-agent one that tests out a human-oriented data set and a sub-agent two that tests out an agent-oriented data set. Finally, **grade it using some deterministic script.** It has to figure out what the result is using something that will consistently give a—come back. Don't use LLM as a judge. Fine. And give me the report as a results.md that says: this is what I did, here's what I found, and whatever else it can think of that says T1 is probably true, probably not true, etc.

**Anand**: [07:59] Now, it may well be that the agent is going to make a total mess of it. It's not really testing it correctly; it's not getting the results correctly. Fair enough, we will find out. The beauty of it is that we're probably wasting a few tokens—big deal. And in the process, we would have learned (a) a technique of verification, and more importantly, **how we verify its verification**, which is really the skill to learn.

**Anand**: [08:30] So, let it run. And what I will do is add a couple of questions to the form once I find it. So, we'll call this "T1 source." And share what agent you used to test the first hypothesis, T1. And the results that it gets you once it finishes. In my case, it's still running. I have no idea what it's doing. I refuse to look at it; I'm going to have an agent look at it and tell me what it's doing.

**Anand**: [09:56] I'm just going to go back to the form and submit. In my case, the choice of agent that I'm using to test T1 is Codex with GPT-5.5 "media." And once the results come through, I will paste it here.

**Anand**: [10:34] A small note about the prompts that were used to research and benchmark: these are **meta-prompted**, meaning I have only steered—

**Participant**: [10:46] One quick question. Would you advise we start a new session or continue with the previous one? What are the best practices for organizing [inaudible]?

**Anand**: [10:54] Please start a new session. But how are you able to see questions?

**Participant**: [10:57] Oh, ChatGPT.

**Anand**: [10:58] Oh, thank you. Yeah, new session please.

**Anand**: [11:14] Absolutely. And I think it's worth spending a little bit of time to go through those. I'm sure that we could do much better, but here is my process. I pasted the workshop content, which is exactly what you have seen on the Fifth Elephant page, and told it a few things that I believe, which is what constitutes "data." In other words, everything that I can think of. And how did I get this? This is meta-prompted also. I said in another window, "Look, this is what I think data is, but can you give me a broader list?" And it extended it.

**Anand**: [11:59] But I also said I specifically want you to **exclude harness artifacts.** I do not consider the skill.md, Claude.md, hooks, etc., as data. Why? I don't know. Gut feel. But it just doesn't feel like data; it feels more like the processing system. And then I said, "Give me some questions." Well, okay, yeah, so this is the first part of the meta-prompting: Am I missing some other kinds of data? Then, I want a "research prompt." Search—I mean, think about what is a good way that we can have different agents do the research to find out what techniques will work, what techniques will not, etc. And yeah, all of this I actually sat and typed. So I'm guessing this is where my steering comes from.

**Anand**: [12:53] And the third question that I had was: how do I go about benchmarking it? I'm going to share this entire conversation on the chat, but there are a couple of things that I will highlight as we go along once I find the right window. This is the right window. "Chat used to generate the prompts."

**Anand**: [13:31] One element that I'll flag off here is the last question, which is how did the benchmarking prompt come about? It's based on steering that, "Look, I want a prompt that will allow participants to paste, pick any one technique to test, and run it. They should just be able to copy-paste the prompt and the technique, and it should somehow be able to test it." And it generated a prompt.

**Anand**: [14:05] Now, what it does not know is the nature of techniques that might come up. And it strikes me, by looking at even the first one, that **maybe just a folder is not sufficient. It might need to create an API server in one case for progressive disclosure; in another case actually have a folder and give the details differently.**

**Anand**: [14:28] So, one of the things we could do now is, now that we have results from the research prompt, feed it and say, "How would we modify the benchmark prompt?" That's something that we will do, but after we get the results of the first discussion, that is whatever the benchmark has produced. Has anyone's benchmark completed? No? Okay good. We are using AI to the fullest, very happy about it.

**Participant**: [15:05] It's using older models for the benchmarking and it's saying that I'm using 4.1 "mini" to do the benchmarking and those things.

**Anand**: [15:13] Interesting, and that's a fair point. One of the—the fact that as you go through the thinking you're noticing this is a good point. I used to do this, which is—and I still do it—go through the thinking to see what I observe, etc. It's certainly helping me learn. I'm trying to figure out if there's a way by which we can **asset-ize** that, for lack of a better word. Meaning, without me having to do this, is there a way by which we can have an agent catch the kinds of things that I'm spotting as patterns?

**Anand**: [15:51] So far, one method that kind of seems useful is to have some place where you note down these things. "Oh, agents tend to use relatively older models; I found this," etc., almost like a "Things I Learned" catalog. Feed it to the agent every week, month, whatever and say, "Look, here are some things that I observed. Are there any skill.md files that I could create out of these that will help it catch the kinds of weird patterns that it's seeing?" Sometimes it groups them and then I ask it, "Which of these in my logs are you actually seeing at least half a dozen times? Which of these happened only once so there's no point?" So, a frequency filtering. And then have it run the skill and see if running that skill actually produces useful results. If it does, then keep it; otherwise why waste even that one or two sentences of context? We will try and do that today. Meaning, after we identify which techniques are actually better for agents and test it, we'll take a few of those and tell it, "**You create a skill that I can use later on so that the knowledge that we have learned in this session doesn't have to sit in our head.**" It can sit in an agent's skill that can be used repeatedly.

**Anand**: [17:11] Put another way, **your job here is not actually as a learner but also as probably the head of HR for your organization team**, and you are basically saying, "I got this reference material from some workshop there; you go read it." It's a role that we can adopt. It's not too hard a role to play. It takes some adjusting to, but it's a valid role.

**Participant**: [17:41] Do you miss the good old days?

**Anand**: [17:43] No. Tell me more. The question for those online is: "Do I miss the good old days?" Go on.

**Participant**: [17:50] I just kind of feel like this is the reality. Like what you said, where we're going to shift people more like as speech and like managers and supervisors and stuff. And so, especially like for a craftsperson who enjoys like doing the work on the IDE, that probably hits a little bit hard. But for the things that we do and a lot of people do enjoy the work that's there.

**Anand**: [18:18] That is a good point. And to repeat for those online, the comment was: **craftsmen would probably miss the agents taking over their craft.** And I completely agree. In which case, as a craftsman, I have one solution, which is: **don't use AI in my craft, but everywhere else.** Good enough. Now, that may in the long term mean that we might get left behind in our craft, but at some level we have to ask ourselves the question: "What am I doing it for? Money or joy?" If I'm doing it for the joy, it's like a sport. I run because I want to run, not because I have to get somewhere. If I have to get somewhere I will take a bus or a car or whatever; those are faster than me running. We run because we want to run. We code because we want to code. Code.

**Anand**: [19:08] Put another way, I think therefore we will have higher leverage in areas where we have less expertise, which is an interesting perspective. Now I can be a reasonably good contract law reviewer. Reasonably good and apply it to a rental agreement, an employment agreement, a takeover agreement, a sale deed of any kind—which is not bad. I can be a reasonably good financial advisor. In that, I can figure out where to put my money in better than the majority of the average bankers or investment advisors who probably don't have as much context or the ability to analyze my bank statement, my investment portfolio, the internet, etc., as it is changing.

**Anand**: [19:56] So, great. That means I can also be a better counselor. I often serve as a human-as-an-interface. Very often people come to me and ask me a question, I copy the question or record the question, put it into ChatGPT, give them the answer. Not a bad thing. I used to get irritated. This happened even during the days of Google. People would ask me a question, I would Google it, send them the first link, and I wondered why they were not doing it themselves. Then I asked ChatGPT about me getting irritated, and like a great counselor, it gently told me that, look, **people are valuing your judgment, that you will look at it and you have kind of okayed it. And if you have okayed it, maybe there is some amount of trust that is passing through, some amount of validation that is passing through that is helping them.** At which point it struck me, hold on, this is almost a free way of making friends. See, the value of friends, human relationships, trust, and all of that is anyway going up in the AI era. Now all I have to do is take their question, put it into ChatGPT, give them the answer, I've made a friend, they think I've helped them. Good! AI is helping me build AI-proof skills.

**Anand**: [21:11] So, as a craftsman, that's what I do. I protect my craft, use AI to help me in things that I'm not good at, use AI to help me in things that are going to become more important in the future.

**Participant**: [21:23] One question on some experiments. After multiple iterations of the same experiment and different variations, I think at some point it starts making up the answers. It doesn't actually run them.

**Anand**: [21:36] Broadly, that falls under the category of hallucinations. And **I treat models, agents, etc., like strange humans.** I don't mean stranger in the sense of an unknown person, but strange in the sense that I have no idea how they think. And therefore, an agent says something. If I don't really care what the outcome is, I will go with it. Example: last evening I was at Brigade Road. I said, "Look, you kind of know my food taste preference. Where should I eat and what should I eat?" It said, "Go to Street Stories and there is a channa-something or the other," which actually has nothing to do with channa; it's a dahi-puri with mango yogurt. It knows that I like new, novel kinds of dishes. This was a fantastic recommendation, but too sweet. It failed in a second-order sense. So it gave feedback, saying, "Look, you've given me a good novel dish, normally I would have picked it, but the execution was slightly flawed; they made it too sweet. So continue giving me this sort of a recommendation." **In this case, the cost of error is negligible.** What if I have a bad dish one meal? Zero. In that case, I don't even cross-check.

**Anand**: [22:56] If it is more important, like for instance when I asked Google—asked Google Maps, whatever—"How do I get here?" and it told me that I should come from the back gate, and the person at the back gate security tells me, "No, no, no, you have to go to the front gate," and that's 22 minutes away and I'm 10 minutes late to my own session making you wait—that has a higher cost. It's an example where I should have verified where there are real-world costs. If it's a one-way door, I would verify very carefully. But here's the thing: that is what I would do with a human giving me advice as well.

**Anand**: [23:42] So, depending on what the cost of it is: verify. One perspective. Another perspective is: **I treat agents as cheap or almost free.** I'm getting free advice. As long as it doesn't cognitively overload me, I don't mind using it as an additional input. In other words, if it's medical advice, I will go to the doctor. I need a neck to catch, say, "Doctor, you told me this last month." But I will also ask AI and pass that input to the doctor saying, "Look, AI said this." The doctor may get slightly irritated, so I have to manage that relationship, but otherwise informationally and medically, it is a good thing. **In short, I don't particularly treat the advice from AI as being any more or less valid than from a really smart human who's weird in ways I don't understand.**

**Anand**: [24:41] Okay, we will take a break from this, which means we can just generally talk about all kinds of things. Anyway, any agents are done yet? Wow, okay! So for those whose agents are done, could you please go back to the form, paste the results.md, and I will just take a—

**Participant**: [25:05] Once we just lost the livestream?

**Anand**: [25:07] Oh, okay, we lost the livestream. About machines needing a reboot.

**Participant**: [25:09] Machines need a reboot. I can't explain anything else; people changed in here.

**Anand**: [25:13] I can't reboot until this is done. So, I'm going to try it once more. But we're done. Sure. Cool.

**Anand**: [25:21] So, I'm going to take a look at some of the results that we have so far, conscious that those online can't do anything about it, but okay. Okay, we have four results that have come up and—okay. By the way, I am curious: so there are 10 of us who have tried out using Sonnet, Codex, KiloCode. Is the person who used KiloCode here? Okay, I am hearing of KiloCode for the first time, very, very curious. The rest, okay. Who's using Pi? Okay, you're using Pi. Nice. What's your experience with Pi?

**Participant**: [26:13] It's been like three months now. Yeah, I've used the Pi AI library for agents' SDK.

**Anand**: [26:23] Got you. So you used Pi for three months using agent's SDK, and why Pi here?

**Participant**: [26:30] It's my go-to agent.

**Anand**: [26:31] Your go-to agent. Interesting, okay, that is impressive. All right.

**Anand**: [26:38] So now, here are five responses so far and let me start with the earliest response, which says: "Conclusion: **difference appears from Level 2, limitations blah blah blah**," but I don't know if it means that it was a successful test or a failed test. It's giving all the code and all of that. Smoke checks, lots of stuff. Okay. Primary metric on accuracy, okay. **Okay, so I guess then the hypothesis is validated.** In other words, the very first submission that we have says **at an easy level of testing it doesn't seem to make a difference, but at a slightly higher level of testing, there seems to be some preference for progressive disclosure.**

**Anand**: [27:47] Okay, that is just one input though. Let's see what else we find. The previous one is giving a lot of code and limitations. "Conclusion: **no difference at any tested level within the budget.**" Okay. So maybe it doesn't make that much of a difference according to the second one. What else? Third one has a lot of code—I should find a better way of reproduction. "Conclusion: **no difference at any tested level.**" At least that's a second result that has emerged. A fourth test reveals **no difference at any tested level within the budget.** That is good to know. And the fifth one that we have here: **no difference at any tested level.**

**Anand**: [29:10] So, fair point. Maybe it is true, but we have far too little evidence to conclude that progressive disclosure, the way we have tested it, is working. Now, remember we have two problems and one advantage. **The advantage is that while what it said sounded plausible, and if I had just read it I would have believed it and started incorporating this as a prompt, we have now managed to test it.** The beauty of this testing is that I have put in very little effort into the process of creating the test or the benchmark. Step one: look, I want you to give me a prompt that will create a test. Take that prompt, then say: create a test. Create.

---

**Anand**: [00:00] ...take a test, run it, tell me the result. Now, what we should do is a review of this, and we will, but we will use an agent as a judge for this. We should also probably revise the prompt and improve it. Both of these are good things to do. But where I'm going with this is the general premise is: **agents are good at generation; therefore, humans need to be good at curation—that is, what we put in—and verification—which is what comes out.**

**Anand**: [00:31] Which is true. However, **the boundaries are expanding because we can also use agents to brainstorm on what to submit. We can use agents to help us verify**, which is what we are doing now. So, what really constitutes verification is probably a slightly different skill than what we might have thought of as verification before agents. And by maximally delegating, I'm finding that it's easier for me to understand what to learn because it's something that agents can't yet do, versus what I may as well let agents do and therefore get a better result out of it. One of the results that we will be getting because of this is a benchmark that, across a bunch of people, tells us what's working and what's not working. I think we have one more result—let's see—the sixth one, maybe. Okay, no, we have five responses. Let me see if mine is done. Okay, mine is done, and I will take `results.md`, paste it as well, and submit the answer. But mine, conclusion, interpretation... okay, **difference appears from Level 1, which is interesting.**

**Anand**: [02:11] So, we have two results which indicate that there is some improvement and four results that indicate no improvement. Okay, maybe it is nuanced; maybe some kinds of data work better, some kinds of data don't work, etc. Let's find out. Yes, please?

**Participant**: [02:29] [inaudible] I had used Claude 3.5 Sonnet. I am wondering if that actually affected the quality of the tests themselves?

**Anand**: [02:37] Quite possible. So, let us take all of this. I will take the responses and the question that... okay, let me not even bother and just upload the `responses.csv`. Okay, having uploaded it, let me also upload the form—actually, why am I doing it here? I really should upload it to Claude, where I was having that earlier conversation. So, this is where I was having the conversation on what prompts to use, etc. So, let me give the same agent this data and say... I'm going to dictate out here.

**Anand**: [03:42] **What I've uploaded is the final form that I'm using to collect data and the responses that people have submitted.** What they did was took one of the hypotheses which I'm pasting below and the benchmark prompt that you had given, with a few small variations, and I'm pasting that also below, to have their coding agents, which you can identify from the `responses.tsv`, to analyze and come up with a set of results. **I want you to do a few things. First, help me understand whether the benchmarking prompt itself is doing a good job for this particular hypothesis or it needs to be improved.** To help you with this, I'm also sharing the full list of the hypotheses that were generated. Therefore, using this, think like an expert and come up with a better benchmarking prompt that I can use. Second, evaluate the results that we got for this hypothesis. Did it actually evaluate properly? Meaning, did the benchmark prompt actually get it to do a good evaluation? Again, think like an expert and see what the failure points are, but also to see if this can be used for any kind of a valid conclusion and give me the main takeaway. **I'd like you to summarize this using my "meeting response style" at the end.**

**Anand**: [05:13] And now I have to give it a whole bunch of context. Let's do all of that. So, I dictate into ChatGPT because the dictation is much better and easier. I can just copy-paste out here. Now, I need to pass it the prompt that we used. So, these are the techniques... that was one thing that I promised to give it. No, not this one. Somewhere... ah, this is the test run. And then the other techniques list comes somewhere from the other window. Yeah, this is the list of tests, techniques. What else did I promise it? `responses.tsv` I've given, benchmark prompt, full list of hypotheses. Yes, I've... okay, techniques. **Think like an expert**, which is basically—I have a skill called "expert lens." And I have a skill called "meeting response style," which will help it answer in the right way. And the last thing that I need to do is see if I'm going to run out of Fable credits—borderline, but I think it's smart enough. So, let that run.

**Anand**: [06:56] Now, a couple of things that I'll flag off here. I am not very good at experiment design. I really don't know whether the way it ran it is right or not, nor am I good at rapidly evaluating evaluation code. Generative code I'm good at evaluating; evaluation code I'm not good at evaluating. It requires a certain mindset—a "red teaming" mindset. I'm an optimist. So, I will not even try and compete with this. Secondly, in the past, I used to find that the naivety of the models was high; that if I said "be critical about it," it may not be doing a good job. I found that a prompt saying "**think like an expert**," roughly, which is encoded in one "expert lens" prompt, was working. I have not in the last six months tested to see if it is effective or not. That's pending on me, but my theory is that it still doesn't do too much harm even if it doesn't benefit. Let's take a look—actually, it's worth taking a look at the expert lens prompt skill, whatever, because there are some skill designs that I found useful.

**Anand**: [08:24] Normally, if you said, "Think like a security expert," and give it some details, that works. Some other times I have to say, "Think like an expert financial advisor," and so on. Now, that means that I'll have to type out the prompt, or at least change the prompt every time. I'm lazy. So, this skill is something that it can automatically pick up, even without... by first saying, "Well, okay, no, this has been modified a lot." Well, the first version of it that I had basically said, "**Find out who's the right expert to answer this and then answer it that way,**" which was working fine. Then I did a few iterations, tested what's working, what's not working, what is the best practice on this, and one of the things that we found is that, see, **there are some places where expertise matters, some places where expertise does not matter, or matters in a different way.**

**Anand**: [09:13] There are **high-validity domains where things don't change much. In that case, the expert's gut feel is useful.** So, everything that you've read on the internet, use that. **There are some places where things change a lot. In that case, don't use your gut feel, but the process that experts follow is generally robust; so, follow the process instead.** That is useful advice. Then it suggested rather than "think like an expert"—now, think like an expert means what? Should I use jargon like the experts? That's what models tend to go to because it's the lazier option. Instead, the agents were suggesting that, based on research, if you specifically guide it on a few lenses—say, "Look at what an expert would notice, look at what mental model they would apply, etc."—but don't tell it to do all of these, it will go overboard. Whichever is relevant, tell it to do it. And that seems to be working for me based on a few experiments. I have not validated this, like I said, extensively in some time.

**Anand**: [10:12] But anyway, let it apply the "expert lens." What we should now have is two things: knowing what techniques will make data better for agents, it will rewrite the benchmarking prompt. Second, it will tell me the results of the benchmark that we just ran and see if it is valid or not. If it is valid, we will hold it and later on, after accumulating a few of these, tell it, "**Create a skill for me so that we don't have to remember it; the agent will remember it.**" If it is not, we drop it. Now, this is where the second... okay. It has identified one structural flaw and three smaller ones. And it identified what we had expected, which is **progressive disclosure means you shouldn't be giving it the folder, but boss, you're creating folder A and folder B, obviously it has access to both the folders. So, the experiment itself is invalidated, and therefore the prompt is wrong.**

**Anand**: [11:23] Also, the default primary metric apparently contradicts the technique card. This testable claim is equal or higher success with substantially fewer tokens. The prompt says, "Pick one primary metric of accuracy." So, six of the seven runs that we have failed to find an effect. Did anyone understand this?

**Amit**: [11:52] [inaudible] I think the hypothesis was basically that there should be two primary metrics, one being the accuracy and the other being the cost. But the prompt only takes one primary success metric for your experiment, which is only the accuracy portion. So the cost metric was never considered as a primary metric to declare success or failure.

**Anand**: [12:13] A fair point. Got it. The calibration pilot got lost, token accounting was inconsistent and prompt-blind, and okay, some other fifth one. One run had no API key, so a deterministic script played the sub-agents. It was disclosed, but it still looks like blah blah blah. And based on this, it's giving a revised prompt, which let's just run. I mean, if nothing else, it has learned from all of these techniques. Let's take a break in 10-15 minutes, that might be a good thing. 10-15 minutes is perfect. Yeah, yeah, okay.

**Anand**: [12:53] And we will in a few minutes try this while we break; we'll let our agents run during the break. The second thing: did it actually evaluate T1? Now, for this, I'm going to go straight to the end because I have a skill that says, "Look, just tell it to me in English." **The protocol worked; the construct didn't.** What does that mean? Bloody hell. All seven teams produced something, okay, but every team tested files plus grep versus files plus an index and not the real contrast. Okay, that is still kind of useful because what that tells me is: with or without an index, is there a difference? And in two out of the seven, it looks like there was a benefit. Maybe that's good enough. Agents already do just-in-time access by default, so giving it a shell, it's able to do what it needs to do. So, five valid runs found no accuracy difference even up to the highest level.

**Anand**: [13:55] What I mean by highest level is: in the benchmarking prompt, I had earlier told it, "Look, if you just run one test and you find that there is no difference, that doesn't mean that the agent-specific arrangement of data is not really better. Maybe there is some more complex data for which it will be better." So, what I want you to do is create three levels of tests—Level 1 simple, medium, complex—and to progressively test and wherever it has a benefit, at that point tell me "Ah, there is a benefit," or vice versa. So, which is what it's talking about in terms of up to 500k token corpora. The way I told it to create—the way it decided to create the level is by saying some of the test data sets will have only 50k, then 200k, 500k, whatever it was. One significant result says that the technique can hurt. That is... okay, and a P = 0.03—that is a reasonable one. It lost on 6 out of the 24 tasks, specifically because entity shards showed stale and conflicting values. **Lesson is: sharding needs freshness and identity metadata attached, or it creates new errors.** What I am translating that into is: boss, if you're creating some kind of an index, do it properly and carefully and test it a little bit. You can mess it up. Cost savings are real but not universal. Okay, that is helpful. When we created an index, it ingested 2.5 to 7.5 times less context, but in one case it was 3X. Okay, I might consider that a win. Not sure.

**Anand**: [15:44] So, three fixes before anyone reruns this: put the intervention in the sub-agent prompt, something, pre-register something, ask each team to add one line. Yeah, but look, all of this it should have done by itself. So let me just say: you gave me some advice on what to change when the team reruns this. Did you incorporate it into the benchmark prompt? If yes, just say so and be done; else, give me the revised prompt. I have no intention of doing work sitting in the middle of a—standing in the middle of a workshop. Let it decide.

**Anand**: [16:39] With this, we will have the revised prompt. But here's what I'm taking away. **The original prompt was not able to anticipate the technique, which said "progressive disclosure."** But the prompt, the benchmark prompt, still gave us a reasonably useful test: if you create an index for the data versus if you don't create an index for the data, does it make a difference? **The conclusion seems to be: it seems to not matter too much most of the time. In one case it actually hurt, in one case it helped, in five of the cases it didn't matter, but the token savings mostly seem to be real. So, quality: don't—you don't need to—cost of if the agents are running at a higher cost, then you might want to reduce it and in that case, creating an index seems to be helping.** Okay, that is a conclusion that I find useful. And because it's coming from seven of you, not just one of my runs, I'm feeling a little more confident. And if a team of 30 people were running it, I would feel even more confident. Now, the revised prompt...

**Amit**: [17:44] But this is like... we just manufactured the data, right? Like, I mean, the whole idea of domains being different, or even in your case you said low quality, high quality... and in your script, you gave different things, right? I mean, like most engineering, we will have a scale so as you go up data, some things will make sense, some won't. So both on like the amount of data, this thing will change, the domain it will change, right? The type of problem it will change. **Are we doing premature optimization by thinking about quality versus service versus cost?** Like, this is what we're doing. This is like an engineering tradeoff. But if I am doing this consistently over time and repeatedly for my work, then I'm starting to get into the engineering part of it, which is like, what gets me best quality for the lowest cost and the fastest speed? Right, like kind of that. I'm not sure... like, I'm a little skeptical of how much we can take away from this, because we ran it in a very generic sense, right? Like, general principle makes sense, like progressive disclosure, you know, do that. So, I don't know, it's a question, I guess, not really a...

**Anand**: [19:17] And to paraphrase, I'm hearing two things. One, progressive disclosure generally makes sense, but it varies on a case-to-case basis. In this case, we didn't put in any specific domain, so how transferable are the results? Wouldn't we need judgment to apply it? Absolutely. So, what I am confident of is three things. Number one: **earlier, I would take an input from a person or an LLM and say, "Ha, this sounds reasonable." Now I have found a way of at least testing it.** If I know what domain I want to apply it in, I could then in the benchmarking prompt say, "For this particular domain, create a data set," or give it my data set. So at least one part of my validation effort gets saved. And it's a reusable one. So you're right: if I want to figure out if progressive disclosure works in, for instance, document OCR for financial services where it's mostly credit applications that are coming in and I have a folder of 300 such, my prompt is going to be very different. It is going to be: "I have given you Folder A. Now I want you to figure out whether this particular technique when applied to Folder A will create a Folder B based on progressive disclosure and benchmark against it." Spot on.

**Anand**: [20:38] However, if this had across all seven cases said, "Every single one of these runs at Level 1 is getting you 30% to 300% higher quality," I'd say, "Boss, this seems moderately universal; I would use an exception route." I'd say, "By default add this skill," and then I would in my domain just say, "Do one cross-verification to make sure that all of the skills that I'm running are still applicable here." Watch it for the first three to six months and then once you have a track record of success, I won't bother too much. So, completely aligned. This does not give me enough confidence to put it into a skill blindfolded. But I'm happy to put it into a conditional skill which says, "**Before organizing the data, test out, see if this makes a difference, at least from a cost perspective, and if cost is important in this project, use it.**" It will probably be a project-specific skill; it'll earn a mention in my "Things I Learned" notes, not a global skill.

**Amit**: [21:45] Within progressive disclosure, also there are so many different ways of doing that. Isn't this very abstract around that? Like in some cases it might be creating summaries and then testing it, in some cases it is doing that grep kind of that thing, in some cases there are ways of doing that tree structure, there are ways of doing the index of document, sub-agent approach—all of these are various methods of doing progressive disclosure.

**Anand**: [22:09] Absolutely. And therefore, that is the second failure point. What Amit was saying is that one failure point is that the data set needs to be specific and you have to test it on that data set. What you're saying is the technique of progressive disclosure also needs to be specific and that we have to test. Quite possibly. I don't know this, though. What I mean by that is: for instance, in a certain domain, maybe it doesn't even matter how you organize it and any kind of organization or progressive disclosure helps. Not very likely, but possible. More likely, the natural way any agent today organizes it is likely to converge towards it because it sees the data and says, "Yeah, look, this kind of data, obviously this is how you organize," and you don't necessarily need to tell it. So again, I would benchmark. And the cost of that benchmarking is becoming low. By the way, if we discover generalizable rules, very good, that's almost a bonus.

**Anand**: [23:05] So, let's try that second benchmarking. What I will do is now share the complete prompt on WhatsApp that we can just reproduce, which is out here... technique T1 card, okay... this goes into WhatsApp somewhere... wherever that is. So, my invitation to you is: take this full prompt, just create a new folder and—or new session or whatever—and try this. I'm going to do the same.

**Anand**: [24:03] I made only one correction to the prompt, which is I told it to use a new folder for this just in case it reran on the same folder. So...

**Participant**: [24:12] You mentioned the prompt is on WhatsApp? I don't have it on WhatsApp.

**Anand**: [24:19] On WhatsApp, I have the full prompt... I don't have it on WhatsApp... oh, oh, oh... let me share the link to that text file. You are not on WhatsApp? Okay, who is... can you add yourself?

**Anand**: [24:57] So, I think your approach right now is like: write something, let it run, then let it analyze, improve, let it run, right? The other way—I guess a human approach, but also maybe what I've seen doing, or at least I do or... is like before we run anything, you keep doing this discussion of what is correct, what is not correct. Right? So, like whatever people call planning, or it's like critique. You can do meta-critique: take this plan, give it to something else to critique. That would be more efficient, in terms of like getting it to critique and saying, like, "This thing about cost versus... the metric is not correct, the cost versus quality," you'll get some of this to be picked up without running these things. Quite possibly. What do you suggest? Like, what is your experience in which one works better?

**Anand**: [25:59] To paraphrase, **if we plan, we get more accurate results up front, whereas if we didn't, then we might end up wasting both tokens and potentially almost certainly quality.** So, do we plan and do, or do and review? In some sense, an abstraction. **When the cost is low, I find doing first better. When the cost is high, I find planning first better.** The traditional approach for software development is: let's create a requirements document. Why? Because if you make a mistake, the cost of fixing it is so high. The new approach to software development is: tokens are far cheaper than humans, and therefore to create four or five solutions and then say, "Oh, I like this one, let's go in this direction," is now becoming cheaper. **We are not used to this era of abundance. Sorry, I should say we're not used to this *kind* of abundance, and it takes a while to get used to it.**

**Anand**: [26:58] Let me give you another example. The classic approach in a consulting project is: you have a problem, I will create a plan to figure out how to solve it, we'll get it approved, we'll discuss, make sure we're solving the right problem. Here's an alternate approach. I told one of my colleagues, Tanooj, "You are sitting inside a waste management company. You have access to an agent, Cortex, which is Snowflake's agent. Now Snowflake has the entire commercial data of this organization. How about if you just toss it all the company's documents and ask it to create a bunch of use cases that they might want solved?" He did. And the result was... this is 100% Cortex-generated, by the way, anonymized. This is what it generated. So he—no, sorry, control minus minus, yeah.

**Anand**: [27:57] He took this to their head of analytics, who looked at it and said, "Tanooj, we did a consulting exercise three months ago, four months ago, in November. And that consulting exercise was a long, expensive one. You have identified independently 80% of those use cases." Tanooj got a huge kick. "Okay, now let me put together a project plan to implement each one of these," which is when he filled out the timeline and effort columns and all of that, and this is what it looked like. He said, "We can do revenue forecasting; this is the methodology we will use, these are the fields we will use. Snowflake already has an ML forecast function; our approach will be to do this, blah blah blah." I said, "Yeah, stop. Stop. Don't write a project plan; get it approved. Solve it. The agent is anyway doing the work." He said, "Oh, but I have some reconciliation to do." I said, "Tell the agent to do the reconciliation, and you tell the agent to do this also."

**Anand**: [28:55] Three days later he came back. "Anand, I finished that reconciliation in just four hours. It would have taken me two weeks." "Okay, good. What did you do with the rest of the time?" "I solved it." "Okay, what did you do?" He said, "I finished the revenue forecasting." "Okay, what else?" "Then I created an email out of it which I sent to the people," which was a really powerful thing. Because then he said, "Why does a human have to come, read the analysis result? Ultimately they're going to take some action. I may as well just tell them what action to take." And he said, "Not just this, Anand, I finished it for all 15. Every single one of those use cases."

**Anand**: [29:33] So, for instance, there was a pricing anomaly. They run a service which costs about $175 to $179. There are 105,000 transactions in which each of those has rates as low as 16 cents. Either you have a massive revenue leak, which is probably not likely, or at the very least you have a massive data quality problem. These are being tagged wrong; these are something completely different...

---

**Anand**: [00:00] ...different. At the very least, we need to sit or... customer churn. His email said—and the subject of this email was delightful—"We are losing $117 million in revenue. Here is the fix." I can't think of an exec who will not open that email. And it talked through: "Look, here are some customers" (anonymized, but US Department of Energy, $1.4 million). For the last 128 days for a monthly service, they have not been ordering from you. Or Meridian Chemical, for the last 126 days they have not been ordering from you. Go reach out to them. Who are the account managers? Just talk to them.

**Anand**: [00:39] So, what he did was, in this particular case, because we are in an era of abundance, **he solved a problem no one asked for. This may be spam; it may be useful.** The open-source ecosystem is facing this problem in a very visible way. For instance, I had ChatGPT... I was at SUTD—sorry, wait... okay, we are in the break. Are your agents working with that new prompt? Okay, then we can talk. Can you please add yourself to the WhatsApp group? Add numbers...

[01:03] *[Silence/Background noise while participants add numbers and agents run]*

**Participant**: [01:40] I have one question on something like this. So, we did exactly the same thing—automated churn modeling—and then we went into the weeds of it as to what it was predicting. There was some number, "This much we are losing," etc. And the problem started... the structuring was all over the place, because most of the time a bunch of ML things were wrong, and then we fixed it. But by then the message had already gotten out. One message already gotten out that "This is the number, these people are likely to churn," right? And then by the time the right ML approach comes and says, "No, no, these are the merchants who are actually likely to churn," now there is a conflicting narrative and then there is an extra coordination task saying, "Is my number better than your number?"

**Anand**: [02:22] Absolutely. And that is one of the risks of getting it wrong. And therefore, we would probably want to go for a plan before you execute. So this is a call that Tanooj takes very carefully. He says, "**This churn email, I know the guy, I can send it to him. Revenue forecasting, I will hold until the head of analytics approves.**" For which there is a big meeting after the finance team has gone through this and verified, saying, "Ha, we will present it to the CFO." No, very fair point. **The generation is cheap and the socialization requires judgment, which is a very fair point.** Let me re-share—or actually, could I request anyone who's on the group to re-share the prompt? And I will maybe just finish one last comment and then we can break... no, actually not, I will come back and share you a very interesting Robert Polie story about what's happening on the bounty hunting on GitHub after the break. Let the agents run. We'll take a break.

**Amit**: [03:24] Okay, we will ask the Microsoft employees to please direct people towards the cafeteria. Kindly be the agent.

**Anand**: [03:31] Thank you. Folks online, we will be taking a break for 15 minutes. Yeah, 15 minutes is good. And the 5-minute buffer, but try not to use the buffer. Sure. So for the folks online, we will interrupt the stream briefly because we need to fix the glitching and various devices need to be rebooted once.

[03:52] *[Break concludes. Audio resumes as participants return.]*

**Anand**: [03:54] Okay, people online should be able to hear me, right? Yes. Cool. Okay, and the screen is also... it's shared. Very good. So for those of you whose agent has managed to complete something, please go to the same form. Right at the bottom, there will be one more question which says something along the lines of—well, basically the last question, whatever it is—you could paste your `revised_results.md` on that. In my case, I have a result, and it turned out to be a positive result in this case. Yeah. Question number six for those of you... okay, looks like three of us have submitted against the `results.md`. That is great.

**Anand**: [04:54] And this time I'll wait for at least half a dozen of these before we start aggregating. **My guess is that this time we will have a better, sharper result. It may not be positive, it may not be negative, but it will be higher statistical significance because the benchmark was aware of what it's going to test.** One thing that I will share, just continuing from the earlier discussion on the abundance mentality: on GitHub, there are a whole bunch of bounty hunting robots—agents. How does this work? What happened?

**Anand**: [05:33] So, I asked... I was at a session at SUTD two weeks ago where somebody was teaching Z3, which is a software that can prove software. It's a software theorem prover. How does it work? I don't know, but for instance, it was used to find out bugs in Python itself. One of the classic problems in a reasonably popular Java library for binary search was: it was calculating the midpoint in binary search as `(low + high) / 2`. That has an overflow error bug if `low + high` becomes higher than the highest integer value. And the better algorithm is `low/2 + high/2` or any other bit-shift mechanism. And Z3 is the kind of thing that can at least make sure that you don't have this kind of an error anywhere in your code, and for other kinds of errors and so on.

**Anand**: [06:30] So while it is not something that proves necessarily that the code is correct—or maybe it is—it certainly can prove that certain kinds of bugs do not exist in your code, overflow error being one among several hundreds of these. So I learned this. Now, "learning" is a very weak term because for me, learning is: I am the head of HR for my agents, and I copy that learning and pass it to my agents. "ChatGPT, look, there is this thing called Z3. Can you just run it and find any other famous—just look, see what famous bugs were verified by it." And it said, "Oh, lots of things were verified by formal verification methods." "Okay, can you find me some bugs?"

**Anand**: [07:18] And it did. And some of these were fairly popular libraries. And some of these were utter rubbish. Even I could see that some of the bugs—there's no point fixing these, these sorts of scenarios don't really exist. So I had it run one more round saying, "ChatGPT, give me something which, if I submitted as a PR, the author would really accept it." And it did. One of them was in Xarray, where it said, "Look, if you try and split a range into an equal number of 2, 3, 4, 5 regions etc., the Xarray's `linspace` function equivalent will do it, but if you put `n=1`, it raises an exception."

**Anand**: [08:03] Now I had already told it, "Don't give me general stuff, give me stuff that the PR owner will accept," so it added a line saying, "Look, `n=1`—that is, taking a space and splitting it into one—doesn't make sense, but there is a particular... there are two reasons why you would want to consider it. Number one: you are not controlling the `n`. The `n` is coming from an external source, and a category of one is a valid point. But the authors may not accept it." It went on to continue, "This is the same behavior as NumPy's `linspace`. NumPy's `linspace` provides this output correctly. Why shouldn't Xarray, which is intended to copy exactly the same interface, produce the same output?"

**Anand**: [08:44] So I said, "That looks reasonable"—no, I didn't say that looks reasonable, I glanced at it and it seemed to have a reason, so I said, "Okay, create a pull request for it." It created a pull request. The author below it said, "Yeah, the NumPy `linspace` consistency is a valid argument, though I don't really see this as having a major use," and accepted it. This is Part A of the story.

**Anand**: [09:07] And let me spend a few... Part B. A few minutes after—a few hours after I submitted my PR and had raised the issue, somebody else had submitted a PR. And I said, "Okay, fine, somebody else is doing it. Is their code better than my code?" I had done some amount of manual review of the code and I was reasonably proud of the submission. Their code was not as good as mine. I was very happy. Then I looked at their profile. **Amazing actress-level photo. Korean.** "Wow, okay, some Korean company." Clicked on the link, led me to an African in Ivory Coast, Abidjan. "Okay, African in Ivory Coast working in a Korean company with a photo of somebody who looks like an actress. What's happening?" **And then it turned out that this is a bot.**

**Anand**: [10:07] All right. And self-declared bot. A few hours later, another bot submits a PR for it. So I went to ChatGPT and said, "What the hell is happening?" It said, "**These are bounty bots.**" If you go search for—just FYI, I am waiting for people to submit a bunch more of your `revised_results.md` in case anyone's done or doing and you're submitting. I'm just giving it a few minutes so that I'll have... well, I have half a dozen, I can work with it, but just filling it with the story.

**Anand**: [10:33] Then it said, "**These are bounty hunting bots. GitHub issues apparently has bounties.**" I knew it, I forgot about it, now I learned it again. You can say, "Anyone who solves this issue gets $5, $50, $500," whatever. And these bots are actively finding easy-to-solve bounties, submitting it, and putting in one note. This particular bot said, "PayPal account," something or the other. And one of the people who responded to one of its issues said, "This is an open-source project, sorry, we don't have budget to pay for any of these. We didn't put in a bounty or any such thing."

**Anand**: [11:06] Sindre Sorhus, who's one of the top JavaScript library authors, reviewed the code for one of these and had a detailed—not very long, but three-four paragraph—explanation of why this solution is not exactly an apt solution and how he might attempt to do it differently. Three others closed this as "bot, I'm not going to look at it" and so on. But Sindre Sorhus's response was probably the most instructive to me. Because what that got me thinking was: **Why am I discriminating against agents?** Something has, with good or bad intent, submitted to my open-source repository. He is treating that as, "Okay, you have contributed through tokens, and I'm reviewing it. It is worth reviewing, or if at a glance it is not worth reviewing, fine. If I'm not able to, then that's okay, I can probably use agents to filter out." **But it just gave me a perspective that I need not discriminate against agents simply because of their origin. They are potentially productive members of the society as well through their token consumption.** It's a perspective. Absolutely not saying it's right, because there are several people who have said "We will not accept AI contributions" and several people who have said "AI contributions must go through a slightly different process" and so on.

**Anand**: [12:35] Now, why am I saying this? **The premise is that because generation is cheap, we need to focus on validation.** That presupposes that there is a queue of use cases; those are the ones we have to solve. Because generation is cheap, now new use cases become possible, like this bounty bot. There is also a Kaggle bot. Go solve Kaggle problems, try and win the prize money. One of my students submitted one; he couldn't win yet, but the next step is now to scan across. If he wins, and the person who wants the problem solved is getting a solution, buyer is happy, seller is happy. Or not. Maybe it is intended to teach students how to learn data science, or repositories are meant to teach students how to submit repositories, in which case it's a very different purpose, in which case it should be banned. There are many other reasons why you would want to ban it.

**Anand**: [13:30] **But the use cases are emerging. New use cases that are becoming possible simply because agents can not just solve problems but identify problems that can be solved and solve them also.** It's something to keep in mind, and I see this whole notion of "data for agents" belonging in that category. Agents can also consume data. That doesn't mean I shouldn't organize it for humans. Organize it for humans, create a separate folder and organize it for agents also by having the agent do it—by having it figure out how best to do it. That's the premise.

**Anand**: [14:04] Okay, so we are stuck at six. I'm going to assume that we have or will likely have only six of these for now. And let's talk to ChatGPT and say what we need to say once it loads. All right. We tried the new prompt, six of us, and have a bunch of results. These are at the bottom. Now, analyze this and tell me what should I conclude from this and what action would you suggest I take? For instance, should I just go ahead and blindly put this as a `skill.md` saying whenever we have data organized in a folder, we should probably reorganize it in some different way, or should it be more nuanced than this? Put another way: **I want you to think like an expert and guide me on the actions that will have maximum impact at relatively high frequency for me.** I'm not really sure what I'm asking for. [Laughs] Okay. This we will run on what chat... somewhere. Yeah. Let's get this data engineering right. Expert... and meeting. Now let's upload the responses and see what we find.

**Anand**: [15:58] Now, while that runs, let's take up the next hypothesis. And I will share on WhatsApp a revised prompt. Let's run the second hypothesis and see if that is giving us a positive or a negative result. Nothing stopping us from running half a dozen of these. So we will do one more. This was T1. Let's discuss T2 first. It says, "**Execute queries and transformations outside context.**" So it's saying for humans, export the data into spreadsheets, reports, application screens etc. and inspect directly. For agents, expose SQL code, DuckDB or tool APIs so that filtering, joining, looping etc. happen in an execution environment and return only the small final result. Which is roughly saying, okay, humans give them spreadsheets, databases maybe... yeah, humans give them spreadsheets that they can load fully; agents give them data processing tools. This is kind of obvious. Should we test or skip?

**Amit**: [17:09] If it is obvious, then the implication is that we should put it into a skill and say, look, whenever you have data, say in a spreadsheet, always—almost always—convert it into DuckDB code, something SQL processable etc. Any reasons why we would not do that?

**Participant**: [17:30] I mean, I'm just curious to see how it will try to validate humans versus agents between both of these.

**Anand**: [17:36] Which is an interesting experiment. Fair enough. Let's just hold on that thought. Maybe we can come back to it. But if there's something else that we are not sure of, where it's not so obvious, let's use our tokens to test that. **Schema-constrained interfaces.** For humans, accept prose, JSON conventions etc. For agents, enforce schema. That is, when you are running a tool, make sure that it always gives an output in a schema. Again, does anyone think otherwise? Would we ever say, "No, no, enforcing a schema's a bad idea"?

**Participant**: [18:14] If it's an intermediate agent—processing agent—yes, always enforce because you don't want it to break. If it's an output agent, I want it to give me the insight. I don't know if maybe a loose structure is fine? I wouldn't enforce a very rigid schema at the output.

**Anand**: [18:32] A good point. For the output agent, maybe we don't want to enforce a rigid schema. Intermediate agents where a machine is consuming it, it's possible. Fair point. Usually I find that the output is consumed by some frontend software at best, and which is therefore also a machine. The number of times that it goes directly to a human or needs to go directly to a human from an agent—we can always add a tool in between to do the conversion, is what I'm finding. But no, which is not to say that that's invalid. I think it's quite valid. Let's just see if there's anything else that's slightly more... okay. **Search and filter before listing.** For humans, provide a complete list. For agents, require or prefer server-side searches, predicates, date ranges. Okay, so only—don't give it a complete list in the first case—sorry, in the first case give a full list or paginated let them scroll. Okay. This overlaps with the first one.

**Anand**: [19:37] Let me... this is obviously doing manual work. So let's shorten that and say, "Some of these hypotheses are clearly uncontroversial. For example, we tested one and it wasn't clear that that was an obvious win. But the second and third, maybe even fourth, seem like they would obviously and almost always be true. Is that the case? Or put another way: **If you had to pick one of these that people may not agree with and yet is true, then what would those be? Give me the top three that it is likely to be true.**"

**Anand**: [20:25] So that helps us pick the hypothesis. We will test one or two of those. I should of course share this prompt—I mean, this chat—with you as well. I will be doing that in a few minutes once it completes this. And then I know we have only half an hour, but we'll use that half an hour to test one more hypothesis, synthesize and see if we can create a skill out of it.

**Anand**: [20:49] "Changing... okay. For changing text-heavy corpora, grep-style navigation can beat embeddings." That I'm not sure is necessarily true. Embedding similarity is a powerful one. "**Uniform tables can be serialized less machine-readably.**" For... ah, this is something we discussed earlier. Is JSON versus CSV going to make a difference? So it's saying **CSV will beat repeated-key JSON is the claim.** Earlier it used to be the opposite. Today this is true. This, I think, is worth testing. It is definitely a changing wisdom. You may disagree, please run all the other hypotheses as well, but let's give this a shot. JSON... um, which is the... okay, let's see. For the uniform tables thing, which hypothesis number was it? Can you give it to me again? And... oh, it was Number 2. Okay, fine. Then it's reframed it. Okay, fine. Let's take this and test it. So now I have the revised benchmark prompt and the hypothesis out here. Let me paste this on chat, in WhatsApp.

**Anand**: [23:01] Do give this a shot. I will also test it. This will be the last of the hypotheses that we are testing. And let me run it. I'm just going to make one small change to this—I'll call it T2 instead of H2, but you don't need to worry. And let it run.

**Anand**: [23:38] What I will be doing is sharing the anonymized results of—anonymized meaning taking out the email column—and sharing all of the results that we have so far, the prompts and the outputs. In case your agent is not able to complete the `skill.md` now, just take a shot at creating it later before we start the end of today.

**Anand**: [24:03] But here is how... okay, and it's testing out with a bunch of agents like GPT-4o and all that. Okay. What ideally we would want to do is, to get the answer to this workshop's objective, we would take potentially these hypotheses, possibly more on a recurring basis, and setting up an infrastructure roughly like this: which is, I will create a new hypothesis, take this benchmark which I might constantly improve over time, run it, get the result, put that `result.md` into a new folder. Maybe run it three or four times so that that folder will contain `result.md` version 1, version 2, version 3—in this case, you are my versions, you may create your own versions. And then have an agent say, "**I've run all of these experiments. If you think this particular hypothesis is strongly true, add it to my skill.**"

**Anand**: [25:00] **That way, it will automatically learn.** This leads us to one possibility of loop engineering, auto-research, what have you. I have largely been a hammer struggling for—looking for—a nail. Meaning, loop engineering looks cool. What can I use it for? And I have nowhere near any clue to the answer. This looks like a possibility: **"Go through my code and figure out how to improve it."** And how exactly to go through my code, I don't know. This is one slice that struck me as a possibility: come up with new ideas or hypotheses every day, or review the ways in which my code is executing by looking at the logs and come up with hypotheses, or review recent research papers and come up with hypotheses. I don't really care how you come up with ideas. Second: efficiently test and see the results. Review whether you did a good job of testing, if required improve your benchmarking prompt, and then put it into the `skill.md`. If I can have an agent set this up so that every morning I just click one button and it runs, or every morning I don't have to click one button and it runs, something is automatically improving. Periodically I'll go in and see if it makes sense, if not edit, but that `skill.md` should be reasonably easy to edit.

**Anand**: [26:30] Okay, that was very quick. And it's saying...

**Participant**: [26:33] Just wanted to mention, for any agentic systems, their logs are like a good system from which you can work, because you're kind of taking traces from that system. Whether it's a chatbot or database or customers or whatever, you can rank the responses as 10, 8, 3, 2, 1 and then you want that system to...

**Anand**: [26:54] Got you. The comment is: any agentic system and its logs therefore are potential for continuous improvement and therefore a loop-based system, which is an interesting one. Let me come back to that in a few seconds because I just need to update the form to allow for the T2 results. I'll call it T2/H2... and here... and let me... T1, T2... I have a feeling something went wrong. I patched some experiment, regenerated a deterministic score. No, okay, I have a feeling it just didn't do what I had asked it to do. So I'm just going to re-run it. Not sure why.

**Anand**: [28:31] But now, to the earlier comment: would any loop be—would any agentic system with its logs be a candidate for a loop? Yeah, makes sense. I have this agentic system or any system operating system running. I will every day ask it to look at the logs, see if you can improve the code, iterate and so on. I've never tried this. Has anyone tried this?

**Participant**: [28:57] I mean, I tried a version but it didn't really—I mean, it started giving me not-so-good answers as what I was expecting. Do tell us more, what was failing? Like, we use an open-source kind of setup, and we call it "braids" a lot because it goes into wrong directions. So we added on top of that—at the third day, look at the logs and see where—what things can be improved, what are the things we can improve and move on those lines. And started giving like feedbacks like, "Okay, removing credentials from the logs," and those kind of things, not exactly the kind of inputs we were looking at—what will make you more efficient as a tool. It was good at providing the sync in case of errors, like when errors were happening it could create pull requests automatically to fix few errors. But yeah...

**Anand**: [29:54] Which is a fair point. Have you been using that as an approach for agentic systems anywhere else? Work?

[30:00] *[Audio Ends]*

---

**Participant**: [00:00] ...the whole what you mentioned, which is **any agentic system can effectively be loop engineered**. Like the open source kind of thing, but at least on a weekly basis, like I have an agentic system which my SaaS product uses. So I just put in the... export the Langfuse traces, for example, and then feed it back to an agent and ask it to optimize these traces. And then, you know, get suggestions out of it like, "These are the things which can be defined better." And every day if I do it, it actually starts improving with the quality of the response.

**Anand**: [00:37] Got you, which is interesting because I'll share how I'm using it to scrape LinkedIn in a crudish sense. But... okay. So it's saying in either case the claim holds from level one and... for whatever reason it seems to be putting it... okay. Save the results in a new folder. Code and whatever else... new folder. This is a new experiment, right? Or is it the same as the last one? Because I thought I copy-pasted the right one and this is talking about JSON versus CSV.

**Anand**: [01:43] Oh, I see. So it's putting it in the same one—it's the same family. Okay. And it's created a new folder. This `results.md` is testing... okay. Conclusion: claim holds from level one and... still not 100% sure from here. Okay, let me ask it: "What exactly did you test and conclude at that `results.md`? Use simple language."

**Anand**: [03:47] In the meantime, for anyone who has been able to come up with a revised result for the second hypothesis, which is the last one on the chat, please let us know. We are testing whether CSV works better or JSON works better.

**Participant**: [04:06] One of the problems is, for example... it depends on whether there is a code execution sandbox. If you have something like that, then for example if you have code execution... I mean, I'm talking mostly in production systems like what we are doing... like how you can provide like Dev and all these like... you have to provide them in a sandbox in a safe manner. If you don't have that in some of these environments, so it depends on what you are able to also provide them.

**Anand**: [04:54] Very true. And therefore the question then becomes: if you have a constrained set of tools that you can provide, the generic advice may not work—part A. Part B: is there a way by which we can somehow expand the space of what can be provided? **What I'm finding is that these constraints often exist in enterprise IT systems.** And enterprise IT systems largely are going for confidence rather than technical validation. Their question is not, "Is this a safe environment and therefore should I approve it?" It is, "Is this an environment that my auditor will approve? Is this an environment that my insurer will approve?" And that is easier when there is an enterprise backing. That is easy when it comes from a cloud provider. So, **the easiest route that I have found is: here is Microsoft sandbox.** Or here is, if you're on Google, Google sandbox, or Amazon sandbox. And people say, "Ah, okay, fine." And then if it comes as part of an enterprise plan, it usually has a reasonable amount of legal guarantees, which they are comfortable with. So my highest ROI nudge is find the enterprise version of it.

**Participant**: [06:10] I mean, it's not matured fully yet because I think Amazon has a sandbox environment essentially using the Sidecar pattern for LLM. I don't know Microsoft's yet, but I think Google recently published an open-source version of like gVisor substrate—[substring? substrate?]—on Kubernetes to support sandboxing, etc., but some of these are like extremely new.

**Anand**: [06:39] Yes, I agree, but they are maturing rapidly, I think. I don't find much of a gap in terms of the execution environment, partly because the agents themselves are offering as part of... OpenAI says, "I'll give you a sandbox in which you can run stuff and you can decide how much control you have." So does Claude and Google as well. Google, in fact, yeah, has probably leapfrogged a bit there. Plus, Cloudflare is now saying, "We will offer these micro-VMs." Fly.io is offering a whole lot of these. So, where we want to test and experiment, sandboxes with granular controls are emerging. For secure execution, they are part of the cloud environment. **Because of the growth, I suspect that yes, there is a gap, but it's a shrinking gap.**

**Anand**: [07:37] Okay, we have a couple of results. I just realized that this isn't as much testing CSV versus JSON; this was more a "should I put the entire content into the prompt or should I let it access and pull on demand?" And it is overwhelmingly saying, "**Pull on demand. No question about it.**" Which, interestingly, is the opposite of what we had concluded at the beginning of this discussion.

**Participant**: [08:12] The main prompt we submitted still said that we need to test progressive just-in-time extraction.

**Anand**: [08:18] Oh! Okay, sorry. I just basically gave you the wrong prompt and I've been using that wrong prompt multiple times. Sorry. No. Then I will not, since we have only ten minutes, take this up. My request is: once I share the chats with you, you will be able to run these by yourselves. First chat that I need to share is where the prompts came from. That's going to go on WhatsApp. Prompt generation and the other was the set of hypotheses that we generated.

**Participant**: [09:56] Was the prompt incorrect? Should we pause?

**Anand**: [09:57] Yes, the prompt is incorrect. Please stop it. There is no point in... I'm going to delete that last... oh, I can't delete the last prompt. Yeah, please ignore that last prompt and stop what I was doing. I mixed the first prompt and the second prompt in a very bad way. But what we'll now do is ask it the question: "Based on this, if we had to... **based on what you have seen so far as evidence, what skill would you suggest we create about it?**" Think about this. You don't necessarily need to create a skill if it should not be applied uniformly. You would create a skill only if there is some data or evidence here that suggests that it would be broadly applicable, of course making sure that there are caveats and filters so that it doesn't get applied in the wrong place. My aim is to set up a skill that, **when I'm converting from a corpus that is typically meant for humans to a corpus that is meant for agents, you automatically apply this skill so that it follows the benchmarked best practices that we have.**

**Anand**: [11:22] Let's give that a shot. The good part is Claude understands how to create skills. Co-dex is not bad at creating skills either. You just say "create a skill," and it creates a skill. And let's end with that. And we'll take a look at the skill that it's created. In all likelihood, it will need improvement on two dimensions. One: is the skill well-worded? We can test it. All we have to do is tell it, "Here is a skill. With and without this skill, are you doing a good job?"

**Anand**: [12:00] The second thing is: is this truly applicable? Is it worth even the few tokens that it is going to be consuming? That is likely to come from the logs. "Go back through my last hundred chats, thousand chats, thirty projects, whatever. See where this skill would have applied. Maybe even do a benchmark. Take a sample of three places where you think it's borderline—not sure. Would it have really helped? Based on that, do you want to refine this skill?" and so on. And it's saying: "Okay, create the skill. **The evidence supports that there is a different shape for agents.**" And it gives me a reason. I'm not too worried about the reason. Let it write. Okay.

**Anand**: [12:43] **This potentially is a skill that I will be adding to my growing list of skills.** Initially, it'll probably... whatever it says, I will probably eventually convert it to just two or three lines. Saying: "I'm going to have a large skill that does a whole bunch of things; you are too verbose for me. Just condense it to the few lines that we need and run with it." Just a quick spot check: how many of you have or are using more than, let's say, a dozen skills with Claude Code at any point in time? More than three skills at any point in time? More than... well, one to two skills at any point in time? Okay, that's a majority. Zero skills? So one to two seems to be the majority. Curious: what are the bottlenecks to increasing the number of skills or lack of need? Why don't you have more skills?

**Participant**: [13:46] I mean, so at least in my workflow, there's very specific types of applications that you want an agent to do. I think the more you have a lot of skills, the effectiveness I've found tends to deteriorate a little bit. It's a lot better for you to instead have multiple agents, having each of them their own skills and doing very specific things, and then for there to be something as a supervisor that is a little bit more good at reasoning the output of all of it. Yeah, that's usually the way for at least like larger scale custom setups.

**Anand**: [14:26] Which is interesting. Do the same agent configurations port into Co-dex as well? Just curious.

**Participant**: [14:35] I haven't used Co-dex at all actually.

**Anand**: [14:38] Which I'm finding to be a fairly powerful enabler. Meaning, anyone who's just on the Claude ecosystem, I usually see as going further ahead on the agents route. I'm on the "I switch between agents" kind of a thing and therefore I'm behind on that curve. That's a useful inference: that **agents work better than skills at scale.** Any other reasons?

**Participant**: [15:04] I think Claude has been building skills by itself and registering them without me actually doing anything separately. And it registers the skills in the cloud, so it's accessible from my coding agent, and I noticed this when I actually built a skill in Karthik's workshop—thing—and it said, "Should I register it?" Okay. And then it was also available in my internet chat. And it had other skills which I had not [inaudible].

**Anand**: [15:35] Yeah, I see this "Save Skill," which is probably the button that would install it. And yeah, I mean, I wouldn't for this particular skill, but I can see. And what you're saying is therefore that's another thing that encourages in the Claude ecosystem.

**Participant**: [15:50] As the result, I before this had never built a skill until Karthik's workshop. I had not decided, "Okay, now I will build a skill and register it." Claude worked; I just did it.

**Anand**: [16:03] Fair enough. You were going to say?

**Participant**: [16:07] So like what Karthik said, a layer where it was basically created something like onto [a prompt?]. So for instance, like if there is a certain analysis that needs... there are certain series of steps, we'll just input that into the semantic layer—steps along with the dictionary—instead of encoding it in a skill. We use the prompt to express that. So then my supervisor will take that and ensure each agent uses that instead of having to... so in terms of overhead on tokens, now I am trying to make it a little more sparse and ensure that the information is cataloged better and skills are a little more generic except for my consumption skills which are [inaudible].

**Anand**: [16:47] Got you. So you're putting it always for the agent to read and then over time spreading it out where it is needed on demand? Which I think is a very reasonable evolution and arguably we go further down this evolution when we use it more across projects etc., which is a fair point. **I just try to leapfrog that where I can with benchmarks and evals.** If I can create one that says, "Okay, three months down the line you will have this problem and you might as well solve it that way," just trying to see if I can stay half a step ahead. But not everywhere. I don't think it's worth... this would take three, four hours to run—maybe only twenty minutes of my time or even an hour of my time—but it's still an hour that you have to question whether it's worth investing.

**Anand**: [17:41] But if I were to wrap up and summarize what I'm taking away from this workshop—and this workshop was really a learning exercise for me, I haven't done this before. The things that I'm learning are: one, among other things, that **some of my hypotheses on what would be better to provide to an agent vis-à-vis human, I can go out with far greater confidence now.** Provide only the summary and let it pick up the stuff; I can now say, "Done a test; even at the lowest levels—that is even with small context—it is able to beat it with fetch-on-demand." I can probably once I run this on JSON versus CSV say with confidence I can do X. I did not realize that that confidence was worth what I now feel it's worth. To be able to say, "I've done it. It works." And that means then that I can start sharing it with my team, who will then also be able to run it with that confidence. This is part A.

**Anand**: [18:52] Part B: **I did not realize that creating a benchmark is as easy as I found it to be.** I'm not saying this is necessarily easy, but it is certainly easier than I had worried about because I had three problems. I don't know what to benchmark—what is the whole process of creating a benchmark. I don't have data for a benchmark. I don't have a basis for evaluating the benchmark. Now, given all three problems, I had no place to start. **What this exercise did for me was say: give an agent your objective; it will do all three for you, and you have to be a judge and see if it makes sense.** Which, to some extent, I'm able to poke, prod, and where I'm not sure, I'm not sure. Okay, fine. But the creation of the benchmark is not the headache now. It is judging whether the benchmark is a good one or not that becomes a problem. That is one step forward, which makes me feel a whole lot more comfortable. Then I can delegate this to one of my team members and look at it from the outside and say, "Okay, in this area I'm an expert; let me do this." Or better yet, create a benchmark, give it to somebody who is an expert in that area and say, "Poke holes in this," so that they can, you know, instead of worrying about how do they go about creating a benchmark, they can just evaluate the benchmark.

**Anand**: [20:16] We have some experts in chemical structure evaluation from papers—PhDs in chemistry. They're saying, "We currently have an agentic system which is extracting it; it is not doing so well. We want to switch models, but manual testing is taking a lot of effort," etc. So now my plan is to go to them and say, "Look, we can create a benchmark. You have your logs, you have stuff, right? I'll just tell an agent to create a benchmark. It will create a benchmark, and then you tell me if the benchmark is right." So I've delegated two pieces of work: one to an agent, one to an expert. Fantastic. They are enabled. I am happy. That is the second takeaway.

**Anand**: [20:53] A third takeaway is that—and we should really look at this skill—it's saying: "Okay, here's how you turn a human-oriented corpus into an agent-oriented one, but only where measurement says it will help." Fair enough. So it is introducing that element here. It's a decision rule. Step one: **measure before touching anything.** So figure out whether it will really save stuff. Decide under these circumstances whether you will convert, whether you will not. Building a rule of thumb, which I will review closely and see if I agree with it or not. And build a catalog out of it and some sharding rule... not sure.

**Anand**: [21:39] But **the takeaway for me here is: because I can always put stuff into some asset—could be a sub-agent which I'm going to try next, or a skill which is always going to get applied, or even just some directory or one prompt which I can copy-paste and run—I don't have to be the person that carries this knowledge in my head.** I have a place where my notes can get auto-used. I have to yet figure out what is the best place to put it, but it is storable and usable like a real asset. And that asset is a potential flywheel. Look at the logs, update it, see where it's working well, where it's not working well because it's ultimately the same benchmark that is leading to it. These were my takeaways from the workshop.

**Anand**: [22:25] Since you have logged in on Google, what I will do is have some agents do a little more analysis, see what else it finds, plus run a few more tests and share what we find. My request to you is: run a few of these benchmarks, run a few other benchmarks, create verification datasets, translate that to skills. See how much you can get out of the way. **The fact that we don't know something might be an advantage.** We are no longer wedded to the craft that we are following. If something doesn't work, as long as it doesn't cause us any harm, what is the big deal? **Tokens are cheaper than our time.** Let's see what we find. Do give it a shot. Thank you. [Applause]