## Transcript

**Zainab**: [00:00] ... making all the arrangements. So Anand, if you can just let people know about the rules, whether they can take coffee inside, and you know, where they should be going in between for lunch and tea. And you can take over. So I'm not the only person talking.

**Host**: [00:19] Thank you so much. Hi, good afternoon everyone. It is nice to host this session. I'm just giving you some small instructions, like the Wi-Fi password I have already shared in the group. You can have access to the Wi-Fi. And for the washrooms, just opposite to this MLR, we have the washroom areas. And just right side outside, if you can just take a little turn, there's a coffee machine and cookies and coffee. So coffee is not allowed inside. You can—I think we have a break time, right?

**Zainab**: [00:56] We will have a break time and that will now change.

**Host**: [00:58] Yeah, during that time you can go grab the coffee and cookies there. And yeah, that's it. Apart from that, if you have anything, let me know. I'll be available here. You can reach out to me. Thank you so much for your time.

**Anand**: [01:13] Thank you. Thank you. We are good to go. I am good to go. Thank you.

**Zainab**: Anand, there might be a few people who don't have the dataset, so if you have something that they can work off from, please speak to them.

**Anand**: No problem at all. There should be plenty.

**Anand**: [01:30] Firstly, apologies everyone, I had to literally get lost. I missed my morning walk; I didn't miss my afternoon walk. But am I clearly audible to the people at the back? Okay. Folks remote, please just bear with us, we're sorting out some logistics. Am I still audible to you at the back? Not a problem?

**Anand**: [01:54] So this session is a very systematic exercise. I'll have to somehow get this to be closer to my mouth. Okay. Check, check. I'll keep talking. Okay, can I clip this to get it up to my face? Oh, okay. Nice. Right, hold on. It wants to flop.

**Anand**: [02:35] So this is just for this room. The online mic is different. And I'll assume that people online can hear me. So one possibility that we could use for collaboration is simply the WhatsApp group. So we are on a WhatsApp group and as we go along, if we need to do some collaboration, we can use this as well. But I will also have another mechanism where we'll be able to do stuff.

**Anand**: [03:12] I'm not really going to do much of the talking. This is a workshop; you're going to be doing most of the work. Actually, you are not going to be doing most of the work; you're going to be delegating most of the work to AI. That is what we are here to learn. And what is left, what it cannot do, is broadly what we will be doing.

**Anand**: [03:31] **The premise here is that earlier humans were in some shape or form the consumers of data. Software became a big consumer of data. Now agents are becoming a big consumer of data.** We kind of know how to organize stuff for software, for humans. We are not sure we really know well how to organize it for agents. And because nobody probably has a good enough point of view, and because agents are moving so fast, that if we have an answer today, it'll anyway become out of date.

**Anand**: [04:05] **This is a "teaching you to fish" rather than "giving you a fish" kind of a workshop. Meaning, what we'll do is figure out how to figure out whether a particular way of organizing data for agents works well or not.** And to do that, we will first have AI do some research. There are two QR codes on the screen, and the one on the left will give you a prompt. The one on the right will be a location where you'll be able to share it later once I've started my server, which is sitting on my machine. But for now, could everyone please scan the QR code on the left side and see if you are able to open that link.

**Anand**: [04:58] I'm not able to. Are you guys able to open the QR code? You were able to? Okay, I don't know why I'm not able to. So I will try something. Let me just, sorry, change the background to white and remove the background image. And please try your luck now in case it was failing earlier for you.

**Anand**: [05:32] Okay, looks better. If you get a screen that has that begins with raw.githubusercontent.com, that is what you want. To be on the safe side, I will also paste the URL in a few seconds on WhatsApp. And that will be an alternative source. I'm just putting it on WhatsApp. Now this is a—since it's on WhatsApp, in case you're not able to get it from the QR code, just open it from WhatsApp.

**Anand**: [06:08] Now let's open this link. This is a prompt that we will gently go through. We're not going to go through the full details of the prompt, but broadly it's saying: "**Look, AI agents are the ones that are consuming the data. Now what we want to do is figure out, by you doing the research, what are the ways we should organize it for agents rather than for humans? So search online, go through some good sources, give me the top 10 techniques, and prioritize, give me the better techniques on top, and finally rank it on a table. And for each of the top 10 techniques, write it in this particular format.**"

**Anand**: [06:55] Now this is step one of what we'll be doing today. Let's just blindly copy-paste this into—and I'll suggest which agent you might want to use—and let it do some of the research. Then we will collect our collective research and see amongst those are there some strong patterns that are emerging, and test out those techniques. **I don't know what are the best ways of organizing data. I have my opinions; so do others. None of us do. Let's find out. And if we have an experimental approach by which we can actually validate if that organization of data is working well or not, great, we have learned an approach.**

**Anand**: [07:31] So what I'm going to do, and here's what I invite you to do, is to copy this and put it into your best agent. And what I mean by best is, firstly, certainly paid. If you have a paid ChatGPT account, great; paid Claude account, great. Gemini may not be so great; Perplexity may not be so great.

**Unsure (Male)**: [07:52] I'm one of them. I'm not used up credits for today.

**Anand**: [07:54] Ah, wonderful! Congratulations. Here's an opportunity. So I have set this to GPT-4o on ChatGPT and set it to "High" as the level of thinking. This is my configuration for ChatGPT. And I'm going to paste this entire prompt exactly as it is and submit. I am also going to submit the same thing to Claude on the highest model. If I'm not mistaken, yeah, Fable is vanishing in a short while, so before it vanishes on Fable, I'm not going to put the effort at Max, I'll at least put it at High and paste the same prompt, but Opus is probably fine.

**Anand**: [08:42] Now we'll let it do all of its research. But we are going to discuss what it finds. And one of the things that we will do is submit the results of what we are seeing on a form's page which I will enable in a short while. Give me a minute, just ignore everything that I'm doing. This is all part of the setup that I was to have done an hour ago.

**Anand**: [09:50] Now while it's running, here is a question that I will pose to all of us for discussion. I'm going to paste a second link where we can submit the results. **What is your guess? How should we organize data differently for agents versus humans?** Please visit the link that you can scan on the top right, or you can just click on the link on WhatsApp. Either will lead you to the same page. Or forms.sh-anand.net. It's all the same. And this is a form I would suggest you just keep on your browser somewhere because we will be filling out a whole bunch of things in here.

**Anand**: [10:41] So the first thing that we want to take a shot at is your guess on how we should organize data differently for agents versus humans. One sentence, one paragraph, one book, however long you want, feel free.

**Unsure (Male)**: [10:57] JSON, [inaudible] like keys and values, the raw data is very hard to understand which column and which row [inaudible] for agents [inaudible].

**Anand**: [11:08] One possibility. But I'll invite everyone to just write down one or two points before we get into the discussion. That'll be the starting point of your hypothesis. Once you submit, you will not be able to change the results. So just spend two, three minutes. And **the point is not to use AI to get to the answer. The point is not to get to an answer; it is to set a prior. Gut feel: this is what I think. So that we can compare, and that will give us more learning hopefully.**

**Anand**: [11:47] And you can do this while your agent is running whatever it's doing—ChatGPT, Claude, etc.—the other prompt that is. For those who are joining remotely, in case you have any issues with any of these, please just ping on WhatsApp. I'll be keeping an eye on that.

**Anand**: [12:19] Okay, we have a couple of responses on how we should organize data differently. I'll start the discussion once we have maybe about 10 responses. Like I said, you're welcome to write as much as you like or hold off until the discussion also if you want. I'll just wait till we have about 10 responses and then we'll take some of those and start discussing.

**Anand**: [13:04] We have seven responses. We'll give it a minute. Nine responses. And 10. Okay. Please continue submitting. I am going to see what the results look like.

**Anand**: [13:38] Okay, a bunch of stuff. But I'm just doing a quick look at what are the words that are popping up on top, and you'll see that on the screen in a short while. So **"Context" and "Metadata" seem to be a couple of words that have popped up.** Not going through the actual text, which we will in a short while. But yeah, context, metadata... let's see what others have said.

**Anand**: [14:09] **"Context about the domain."** Okay, let's start from the bottom. **"Data should be organized as graphs,"** as opposed to humans who may not need graphs as much. Possible. **"JSON or XML might perform better than CSVs"** because they have named attributes for each value. That used to be a strong—I mean, there were some results around this, and then people found that with newer models, they are able to understand CSVs at least as well, and the token compression seems to have some benefit. But does that hold true for all models? Unclear.

**Anand**: [14:47] **"We need to provide semantic meaning and domain knowledge."** Yes. What I wonder, though, is—so for each of these, I'm not saying that this is right or this is wrong. For every one of these points, because they are valid, I'm also offering a counterpoint. My main idea is, let's test and then we will know for sure. Until we test, we don't know. But why would we even need to test? I'm offering a few reasons. Like providing semantic meaning and domain knowledge—yeah, but wouldn't we do that for humans also? **"Markdown all the way. Content must originate as markdown."** Would that be different for humans? And would humans particularly find an alternative more convenient?

**Anand**: [15:26] **"More context. Self-cataloged data. Data catalog as an integral part of the data."** Again, very true. Why would humans find it different? **"An agents.md explaining the context, query, etc."** And one of the themes that I'm seeing therefore is that we need to explain the data well. Completely agree. And I think that is an important part; we really should do that. It is probably not very different from what we would need to do for humans as well. No harm in still learning that as a technique, especially because we can use AI to create and curate the data in that particular form, and we'll probably be doing a bit of that as well.

**Anand**: [16:14] **"Context about the domain and use case. Structuring the metadata, how it was collected, syntactic format, etc."** And please keep the responses coming.

**Anand**: [16:26] Now while this is happening, we have few results from the models. And what I will invite all of us to do is fill out a second question. And this question, which I will call the "Research Source," is basically: **Which agent and model did you use to get the answer?** You may have used multiple models; please pick any one that you're happy with. In my case, for instance, let me fill this form in. I will choose the result that I got from ChatGPT. So I'm filling out ChatGPT with GPT-4o High. And that will be the first part of my answer. In your case, whichever one you have chosen. If you have multiple, pick what you feel would be the best.

**Anand**: [17:42] The next thing that I'll request that we fill out is the actual response. **The results—what did the agent say? What techniques or differences did your agent suggest?** Which now becomes the third question. And I'm going to copy the answer and paste it here, though it's probably going to be easier if we don't copy-paste it directly but rather use the "copy" button to do the copy-paste. The copy button for ChatGPT is at the bottom, and for Claude, it's at the bottom. Depending on what agent you're using, it may vary. So I will just copy—the reason I'm suggesting that we copy it from here is it preserves the markdown formatting. That's a little convenient. And I'm just going to paste that and submit. So now we have three answers that have emerged.

**Anand**: [18:48] Once we have a few more of these answers, let's do a collective lift-up and say: across agents, what are we seeing as the overall pattern? But before that, I'm also curious, what are the agents that we are using as a group? Let's look at it. Claude Opus 3.5, Gemini... okay, I'm curious what model in Gemini we used. But okay, Cortex with GPT-4o High reasoning, Claude Minimax, okay, Gemini 1.5 Pro, Copilot with Opus is interesting. Okay, that's a wide range.

**Anand**: [19:39] One thing that I would suggest for the Gemini users in this group: some of what we're doing, Gemini doesn't do as good a job today as Claude or ChatGPT. For the—okay, yeah, Hermes with whatever agent can probably do the job. So just see if you could—in case you have a paid version of ChatGPT or Claude, explore that in addition to Gemini and whatever else.

**Anand**: [20:11] Okay, we have seven people who have copy-pasted and submitted. Is that right?

**Unsure (Male)**: [20:20] [Inaudible] link?

**Anand**: [20:21] Oh yes, sorry! The link to the form is out here; you can use that as a QR code. It is also on the WhatsApp group; it is the most recent link on the WhatsApp group, whichever is convenient. And for anyone who may have also joined online late or the room late, what we are doing is there is a link just above the last link on WhatsApp. We copy-pasted that and put it into the best paid agent we have access to and are pasting the answers out here. In other words, we're delegating our research. And... yeah, okay, that is my response. Fine.

**Anand**: [21:05] We have nine responses so far. Okay, we have 10. So what I'm now going to do is take these 10 responses—more will emerge—but let's do a synthesis. On top of this, what are the broad patterns that will emerge? To do that, I am going to—I'm going to—I'm not going to share this prompt because this is not something that you need to run, but you'll be easily able to locate it. This is the collation prompt. And I'm going to run this on ChatGPT. In which I'm supposed to paste all the responses. Now let me get to the responses.

**Anand**: [22:12] [Muttering while working] Get rid of all of the initial set of... okay. Is there a good way of doing this? Set answer from responses.csv? No, okay, doesn't matter. I'm just going to take all the answers so far and ask ChatGPT to summarize what we have.

**Anand**: [23:14] Let me share a bit about what we're going to do next. See, the first half of this workshop is really about saying: let's find what agents think are different for agents versus humans. And then we will, in the second half, run these as experiments.

**Unsure (Male)**: [23:33] Yes, please. This is just to ask ChatGPT or what is good for LLMs. But this results to work on what you have written about on the internet [inaudible]. So are we like just looking at AI system output for suggestions that might be relevant for them or are like finding the truth or?

**Anand**: [23:54] Fair point. **The question is: if an agent is simply going to search online, find what is there on the internet, and give it back to us, maybe that has some value, but is that really valuable?** And fair point, it has some value in that it would save us time and research and exploration. Maybe it doesn't if we know better. My hypothesis is that on average, there are more things that the agent knows about than we know about.

**Anand**: [24:28] Where we are an expert, it makes sense to just say, "This is my answer; now you tell me where I might have missed something" and all that, but I will take a decision. In other areas, we don't have a basis to judge the agent. In that case, we say, "No, I know a little bit about how to prompt and all that, maybe I don't even know that. You tell me." This is my overall objective and see how it is helping me.

**Anand**: [24:53] I'm taking the non-expert perspective here for two reasons: **I believe that over time agents will get smarter—they certainly seem to be continuing to get smarter—and the number of areas in which we will be able to have deep expertise is small.** There's nothing stopping us from having both approaches, and which is what the first part was also about. Our "prior," where we're saying this is what I believe is the agent's response. And it's going to be good to compare, therefore, the first part versus the second, and see what it is that humans missed that agents caught, and what it is that agents missed that humans caught, which of these seem more valid. And the best part is we will be able to generate some synthetic data, test it out against that, see if it is in fact valid, and therefore know who's doing better. Benchmarking our own skill. Any other—sorry, please feel free to just ask a question at any point. Can I just check your display settings to see why it's glitching? Yes. Any other questions?

**Unsure (Male)**: [26:01] Is the context too wide because we are asking about everything about LLMs, PDFs and everything?

**Anand**: [26:08] Good point. Is it too wide? One of the principles I follow is: because agents are getting smarter and smarter, if as of, let us say, June 2024, I have a point of view on what an agent cannot do, I don't know when it is going to be updated. I have a sense that I'm keeping up with the agents only when I give prompts that agents fail on. So as a rule of thumb, **I start with something tougher than I believe agents can solve, see if it really fails and totally messes up, and then I know: okay, fine, this really is not possible. Every now and then—happens more than half the time—I find that something that wouldn't have been possible last month is now suddenly possible.** Then I mark it saying: okay, this is new. Which is an opportunity I will miss unless I pose harder and harder problems.

**Anand**: [26:56] Harder problems are becoming harder to find. So I go to agents for that, saying, "Can you give me a problem that you can't solve?" And then it gives me something that I don't understand, which becomes the other problem. So the other thing that I'm trying to do is get myself out of the loop. My understanding is bounded. I could learn, but that is a slow process. I could learn, but that requires initiative and I have no interest for it, so many other things. Instead, if there is a way of getting to a better outcome without me having to learn—good, I don't mind. I'm just a manager, not an executor or a subject matter expert. I just want to get the stuff done. Not saying that this is the right perspective, this is the only perspective. I'm saying that in many areas, this is an effective perspective, and there'll be a few areas where you will ideally want to be the expert.

**Anand**: [27:49] So, now it is saying: here are a set of... okay, let me do this. No, I'll—I'm just going to put this on the screen. We will walk through a few of the things that it's saying and discuss those. The idea is not to say the focus should be only on testing what the agents are saying. We will test a few other things as well.

**Anand**: [28:11] But what it's saying is: **"Progressive just-in-time access"** is something that we would want to organize differently. And this apparently is coming through in 11 out of the 15 submissions that we have from across the agents. Popularity is not necessarily the same as correctness, but it is probably correlated and worth paying some attention to.

**Unsure (Male)**: [28:35] [Inaudible] page 10?

**Anand**: [28:38] That is one possible reason. Another possible reason is the internet is filled with the same kinds of—so the source could also be polluted even across—polluted, yeah, have common elements. Both are fair points.

**Anand**: [28:51] It's saying: **"For humans, assemble all plausibly relevant documents, schemas, etc. in one portal before the work begins. Whereas for agents, provide a compact catalog of paths, URLs, metadata, query handles, and let the agent search and fetch selected content only when needed."** Which is an interesting premise. Why? "Preloaded material consumes attention regardless of relevance, and progressive disclosure keeps distractors away." There's no way I would have guessed and I would not have put this on my list. Any thoughts?

**Unsure (Male)**: [29:33] I think progressive disclosure, like... this is why we're different from humans, right? For humans, fetching something is a big desk [inaudible] so you want to minimize that for agents. For agents, the cost of search is so minimal. You don't manage context. Context is very low for humans. The context limit for an agent is very high, so why do you want that for agents to do that? Then [inaudible] manage context. And the token budget for the agent would be charged for it [inaudible]. Do you want that to happen?

**Anand**: [30:00] [Inaudible]

---

**Anand**: [00:10] Got it. Just to repeat for the benefit of those remote, Amit was saying that progressive disclosure may not be a surprise and it may help it fan out and avoid crowding the context window. So, understandable but perhaps not non-obvious. Fair point.

**Unsure (Male)**: [00:29] One thing on progressive disclosure is it works fine for humans and for LLMs because human memory is very short-term. But it also introduces an inhibitor to connecting the dots across multiple pieces. I have this, but I haven't come across this yet. ChatGPT has this property as well. So, connecting the dots effectively across different problems in the same context, maybe a—

**Anand**: [01:10] Oh I see. Progressive disclosure may be an inhibitor to connecting the dots effectively across different problems in the same field?

**Unsure (Male)**: [01:22] I've seen the same problem elsewhere, and you're not able to resolve it because that piece was also not available in that problem I saw.

**Anand**: [01:27] Got you. Whereas if you had it all in your head, then you'd be able to make those connections better. That is certainly a possibility. And of course, context window being a limitation. But still, that doesn't mean that—fair point.

**Unsure (Male)**: [02:08] Thinking based on that point, essentially what LLMs do is they want to have a 50,000-foot view of what is actually the domain that they are engaging with and then cascade down to very specific sets of hypotheses that they want to investigate. Which is not really a pattern that you can do with agents because agents still want to go for specific facts to build first that 50,000-foot view and then go back into specific details. Like when you do a search, for instance, you're first trying to figure out how to create an opinion about the subject matter in the first place and then find the set of facts that you want to corroborate. It's the same rationalization dimension going on.

**Anand**: [02:32] Fair point. And to paraphrase very crudely, it is that once you have the stuff loaded in your head, you know what's important, how, which direction to go on, etc. Which if you have to do step-by-step, it's likely to be slower or harder for an agent. Fair point. And we know why it's suggested as a technique because it doesn't yet have enough of a choice about the context window, whereas we probably have a better way of organizing memory. Go on.

**Unsure (Male)**: [03:01] I thought about progressive disclosure and I had same thoughts, but I was also thinking that maybe the effort—when we say effort, higher or lower—maybe that's basically what expands the search space. That's what I thought. Having an expert knowledge into what the higher effort is probably gives you look into more options to deliver probably a better answer and rank them accordingly.

**Anand**: [03:09] Right. In other words, once it has the full context, it knows which to cherry-pick as the most important ones. Quite possible.

**Anand**: [03:14] **For each of these, it would be good to just test it out. Nothing quite like it, right?** And that's what we'll do in a short while. But let's go on to what it's saying. It's saying that I will be—well, "what matters when" I think is obvious. I'll skip that. **It's saying that on 24 large corpus tasks, index-and-fetch agents achieve equal or higher success with substantially fewer tokens than agents given the full corpus up front, as a testable claim.** Now, maybe it will, maybe it will not. That's exactly what we are trying to test. And it sounds like a reasonable claim. What we are not testing is whether agents, when they are able to handle the full context, do better. That is an impossibility because context windows are limited. So keeping the context window limitation reality, it is saying in one case I will give the full context and in the other case I will not give the full context, and we will test both of these out. And the evidence will basically be—let's see. Oh okay, this is the evidence for why it's reporting this. Fair enough.

**Anand**: [04:32] At this point, let us branch out. I was going to go through a few of these, but since we have had a discussion on this, **we will run a small benchmarking test against this.** How are we going to do that? Let me copy this and we'll do a little bit of editing to get just T1. Now, there is another prompt that I will share which will do the benchmarking. But what we're going to do is take this, give it to an agent with a bit of a prefix and say, "**Write a benchmark for this particular claim.**" My proposal is that we do that together, meaning we all run the same benchmark so that we will have a comparable base and then see across different machines. Rather than one person burning lots of tokens, we have lots of people burning a manageable number of tokens and still collectively learn from that experiment.

**Anand**: [06:11] So, the prompt that I would suggest we use is out here. Wait, before you copy that, I am going to paste the link in the chat window. This is the benchmarking prompt. And let me show you the thing. So the one on the left is the benchmarking prompt. You can take it from WhatsApp if that's easier, or take it from here. **Please copy that prompt and put it into preferably a coding agent.** And don't submit yet, because the hypothesis is the next thing that you need to paste. What I mean by preferably a coding agent is Codex, Claude Code, Copilot CLI, Antigravity (if you like), Pi, OpenCode, Tau, whatever your preference is.

**Anand**: [07:42] And why a coding agent? Because it will be able to, a bit more than ChatGPT online or Claude online, create data sets, download tools, install stuff and so on. Not that you can't use ChatGPT or Claude.ai for this; you could. And if you don't have an alternative, that's perfectly fine. **I would not advise using Gemini for this unless you have no alternative, because its environment is not as good.** So this is the first part of the prompt. The second part of the prompt is the actual test. And I will put that on WhatsApp as well. You could just copy this entire thing and paste it. So the entire result will be—in my case, I'm just going to copy this text, this text, and put it into some coding agent somewhere. T1, yes.

**Unsure (Male)**: [08:35] So, index and search [inaudible] indicate this is a tool for using index and [inaudible] toolset [inaudible]?

**Anand**: [08:51] Good question. The question is: how is this going to actually do a comparison between these two methods? I don't know; maybe it's going to totally mess it up as well. So let's take a look at the prompt that we are asking it to test. The benchmarking prompt says, "**Look, I'm going to give you one hypothesis, and your job is to test the hypothesis.**" The second part of the prompt says, "**Create a data set.**" The ground truth is basically you have to construct a realistic data set that is suited to test the hypothesis. The third step is: create two paths. Organize one folder as if for humans based on whatever the hypothesis said. And then part B is: organize it for the agent. Or vice versa, you should randomize it. And then, you know, document what are all the tasks that it's supposed to create and benchmark on both data sets. Probably through sub-agents. So you will spawn sub-agent one that tests out a human-oriented data set and a sub-agent two that tests out an agent-oriented data set. Finally, grade it using some deterministic script. It has to figure out what the result is using something that will consistently give a comparison. **Don't use LLM as a judge.** Fine. And give me the report as a `results.md` that says: this is what I did, here's what I found, and whatever else it can think of that says T1 is probably true, probably not true, etc.

**Anand**: [10:21] Now, it may well be that the agent is going to make a total mess of it. It's not really testing it correctly, it's not getting the results correctly. Fair enough, we will find out. **The beauty of it is that we're probably wasting a few tokens—big deal—and in the process, we would have learned a technique of verification.** More importantly, how we verify its verification, which is really the skill to learn. So let it run. And what I will do is add a couple of questions to the form. Once I find it.

[Anand types for a while, muttering] Let's call this T1 source.

**Anand**: [12:57] So the two questions that we're adding are: just share what agent you used to test the first hypothesis, T1, and the results that it gets you once it finishes. In my case, it's still running. I have no idea what it's doing. I refuse to look at it. I'm going to have an agent look at it and tell me what it's doing. A small note about the prompts that were used to research and benchmark: these are meta-prompts, meaning I have only steered—

**Unsure (Male)**: [13:08] One online question: would you advise to start a new session or continue with the previous one? What are the best practices for organizing data?

**Anand**: [13:14] **Please start a new session.** But how were you able to see questions? Oh, from YouTube. Thank you. Yes, new sessions.

**Unsure (Male)**: [13:31] My question is: should we use both—the prompt says to paste LLM tools to get some responses because it's—?

**Anand**: [13:37] Absolutely. And I think it's worth spending a little bit of time to go through those. I'm sure that we could do much better, but here is my process. I pasted the workshop content, which is exactly what you have seen on the Fifth Elephant page, and told it a few things that I believe, which is: what constitutes data? In other words, everything that I can think of. And how did I get this? This is meta-prompted also. I said in another window, "**Look, this is what I think data is, but can you give me a broader list?**" and it extended it. But I also said, "**I specifically want you to exclude harness artifacts.**" I do not consider `skill.md`, `claude.md`, hooks, etc., as data. Why? I don't—gut feel, but it just doesn't feel like data, it feels more like the processing system.

**Anand**: [14:56] And then I said, "**Give me some questions.**" Well okay, yeah, so this is the first part of the meta-prompting. "**Am I missing some other kinds of data?**" Then I want a research prompt. Search—think about what is a good way that we can have different agents do the research to find out what techniques will work, what techniques will not, etc. And yeah, all of this I actually sat and typed, so I'm guessing this is where my steering comes from. And the third question that I had was: how do I go about benchmarking it? I'm going to share this entire conversation on the chat. But there are a couple of things that I will highlight as we go along. One element that I'll flag off here is the last question, which is: how did the benchmarking prompt come about? It's based on steering that, "**Look, I want a prompt that will allow participants to paste, pick any one technique to test, and run it.**" They should just be able to copy-paste the prompt and the technique, and it should somehow be able to test it. And it generated a prompt.

**Anand**: [16:31] Now, what it does not know is the nature of techniques that might come up. And it strikes me, by looking at even the first one, that maybe just a folder is not sufficient. It might need to create an API server in one case for progressive disclosure; in another case actually have a folder and give the details differently. So one of the things we could do now is, now that we have results from the research prompt, feed it and say, "**How would we modify the benchmark prompt?**" That's something that we will do, but after we get the results of the first discussion, that is whatever the benchmark has produced. Has anyone's benchmark completed? No? Okay, good. We are using AI to the fullest, very happy about it.

[Sound of Anand adjusting things] I'm going to change the cable to see if the glitch stops. Yes.

**Unsure (Male)**: [17:26] Is it using different models? It's saying that it's using GPT-4o Mini to do the benchmarking and [inaudible]?

**Anand**: [17:42] Interesting, and that's a fair point. One of the things—the fact that as you go through the thinking, you're noticing this is a good point. **I used to do this, which is—and I still do it—go through the thinking to see what I observe, etc. It's certainly helping me learn.** I'm trying to figure out if there's a way by which we can "assetize" that, for lack of a better word. Meaning, without me having to do this, is there a way by which we can have an agent catch the kinds of things that I'm spotting as patterns?

**Anand**: [18:25] So far, one method that kind of seems useful is to have some place where you note down these things. "**Oh, agents tend to use relatively older models. I found this, etc.**" Almost like a "things I learned" catalog. Feed it to the agent every week, month, whatever, and say, "**Look, here are some things that I observed. Are there any `skill.md` files that I could create out of these that will help it catch the kinds of weird patterns that it's seeing?**" Sometimes it groups them, and then I ask it, "**Which of these in my logs are you actually seeing at least half a dozen times? Which of these happened only once so there's no point?**" So, frequency filtering. And then have it run the skill and see if running that skill actually produces useful results. If it does, then keep it; otherwise, why waste even that one or two sentences of context? We will try and do that today.

**Anand**: [19:33] Meaning, after we identify which techniques are actually better for agents and test it, we'll take a few of those and tell it, "**You create a skill that I can use later on so that the knowledge that we have learned in this session doesn't have to sit in our head.**" It can sit in an agent's skill that can be used repeatedly. Put another way, **your job here is not actually as a learner, but also as probably the head of HR for your organization team.** And you are basically saying, "**I got this reference material from some workshop there; you go read it.**" It's a role that we can adopt. It's not too hard a role to—well, it takes some adjusting to, but it's a valid role.

**Unsure (Male)**: [20:07] Do you miss the good old days?

**Anand**: [20:11] Do I miss the good old days? No. Now tell me more. The question for those online is: do I miss the good old days? Go on.

**Unsure (Male)**: [20:12] I asked this because in the expert era, what you said—where people used to code each line and code each string and code each matter—everything was somehow in possession of the person. And so, for a person who enjoyed that, putting everything into dots... the person [inaudible] and I've seen that. So that brings some interesting [inaudible] to all the people who really enjoy the craft aspect.

**Anand**: [21:01] That is a good point, and to repeat for those online, the comment was: **craftsmen would probably miss the agents taking over their craft. And I completely agree. In which case, as a craftsman, I have one solution, which is: don't use AI in my craft. But everywhere else.** Good enough. Now, that may in the long term mean that we might get left behind in our craft, but at some level, we have to ask ourselves the question: what am I doing it for? Money or joy? If I'm doing it for the joy, it's like a sport.

**Anand**: [21:30] I run because I want to run, not because I have to get somewhere. If I have to get somewhere, I will take a bus or a car or whatever; those are faster than me running. We run because we want to run. We code because we want to code. Code. Put another way, **I think therefore we will have higher leverage in areas where we have less expertise.** Which is an interesting perspective. Now I can be a reasonably good contract law reviewer. Reasonably good—and apply it to a rental agreement, an employment agreement, a takeover agreement, a sale deed of any kind. Which is not bad. I can be a reasonably good financial advisor. In that I can figure out where to put my money in better than the majority of the average bankers or investment advisors who probably don't have as much context or the ability to analyze my bank statement, my investment portfolio, the internet, etc., as it is changing.

**Anand**: [22:24] So, great. That means I can also be a human as an interface. Very often, people come to me and ask me a question. I copy the question or record the question, put it into ChatGPT, give them the answer. Not a bad thing. I used to get irritated. This happened even during the days of Google. People would ask me a question, I would Google it, send them the first link, and I wondered why they were not doing it themselves. Then I asked ChatGPT, "**I'm getting irritated,**" and like great counselors, it gently told me that, "**Look, people are valuing your judgment, that you will look at it and you have kind of okayed it.**" And if you've okayed it, maybe there is some amount of trust that is passing through, some amount of validation that is passing through that is helping them. At which point it struck me: hold on, this is almost a free way of making friends. The value of friends, human relationships, trust, and all of that is anyway going up in the AI era. Now all I have to do is take their question, put it into ChatGPT, give them the answer. I've made a friend, they think I've helped them. **Good, AI is helping me build AI-proof skills.** So as a craftsman, that's what I do. I protect my craft, use AI to help me in stuff that I'm not good at, use AI to help me in stuff that is going to become more important in the future.

**Unsure (Male)**: [23:44] [Inaudible] same style and [inaudible] different valuations and [inaudible]?

**Anand**: [24:02] Broadly falls under the category of hallucinations. And I treat models, agents, etc., like strange humans. When—I don't mean stranger in the sense of an unknown person, but strange in the sense that I have no idea how they think. And therefore, an agent says something. If I don't really care what the outcome is, I will go with it. Example: last evening I was at Brigade Road. I said, "**Look, you kind of know my food taste preference. Where should I eat and what should I eat?**" It said, "**Go to Street Stories and there is a Channa... something or the other,**" which actually has nothing to do with channa; it's a Dahi Puri with mango yogurt. It knows that I like new, novel kinds of dishes. This was a fantastic recommendation, but too sweet. It failed in a second-order sense. So I gave it feedback saying, "**Look, you've given me a good novel dish, normally I would have picked it, but the execution was slightly flawed; they made it too sweet. So continue giving me this sort of a recommendation.**" In this case, the cost of error is negligible. What if I have a bad dish one meal? Zero. In that case, I don't even cross-check.

**Anand**: [25:57] If it is more important, like for instance when I ask Google Maps—whatever—"**How do I get here?**" and it told me that I should come from the back gate and the person at the back gate security tells me, "**No, no, no, you have to go to the front gate,**" and that's 22 minutes away and I'm 10 minutes late to my own session making you wait—that has a higher cost. Is an example where I should have verified, where there are real-world costs. If it is a one-way door, I would verify very carefully. But here's the thing: that is what I would do with a human giving me advice as well. So depending on how—what the cost of it is, verify. One perspective. Another perspective is: I treat agents as cheap or almost free. I'm getting free advice. As long as it doesn't cognitively overload me, I don't mind using it as an additional input. In other words, if it's medical advice, I will go to the doctor. I need a neck to catch, say, "**Doctor, you told me this last month.**" But I will also ask AI and pass that input to the doctor, saying, "**Look, AI said this.**" The doctor may get slightly irritated, so I have to manage that relationship, but otherwise informationally and medically it is a good thing. In short, I don't particularly treat the advice from AI as being any more or less valid than from a really smart human who's weird in ways I don't understand.

**Anand**: [27:01] We're going to take a break from this, which means we can just generally talk about all kinds of things. Anyway, any agents are done? Yes? Wow, okay! So for those whose agents are done, could you please go back to the form, paste the `results.md`, and I will just take a—[display glitches again] also just lost the cable. Oh okay. Both machines need a reboot. I can't reboot until this is done. So I'm going to try it. Reboot while we are down. Sure.

**Anand**: [27:45] Cool. So I'm going to take a look at some of the results that we have so far. Conscious that those online can't do anything about it, but... Okay, we have four results that have come up. Okay, by the way, I am curious: so there are 10 of us who have tried out using Sonnet, Codex, Kilocode. Is the person who used Kilocode here? Okay, I am hearing of Kilocode for the first time, very, very curious.

**Anand**: [28:29] The rest... who's using Pi? Okay, you're using Pi. Nice. What's your experience with Pi?

**Unsure (Female)**: [28:51] [Inaudible] used Pi AI library [inaudible]?

**Anand**: [28:58] Got you. So you used Pi for three months using their SDK, and why Pi here? Interesting, okay. That is... All right. So now here are five responses so far. And let me start with the earliest response which says: **conclusion, difference appears from level 2, limitations blah blah blah. But I don't know if it means that it was a successful test or a failed test.** It's giving all the code and all of that, smoke checks, lots of stuff. Okay. Primary metric on accuracy, okay. Okay, so I guess then the hypothesis is validated. In other words, the very first submission that we have says at an easy level of testing... [video cuts off] Spring-loaded [inaudible].

---

**Anand**: [00:00] ...level of testing it doesn't seem to make a difference, but at a slightly higher level of testing, there seems to be some preference for progressive disclosure. Okay, that is just one input though. Let's see what else we find. The previous one is giving a lot of code. Implementation conclusion: **no difference at any tested level within the budget.** Okay. So maybe it doesn't make that much of a difference according to the second one.

**Anand**: [00:51] What else? Third one has a lot of code. I should find a better way of reproduction. Conclusion: no difference at any tested level. At least that's a second result that has emerged. A fourth test reveals no difference at any tested level within the budget. That is good to know. And the fifth one that we have here: no difference at any tested level.

**Anand**: [01:34] So, fair point. Maybe it is true, but **we have far too little evidence to conclude that progressive disclosure, the way we have tested it, is working.** Now, remember we have two problems and one advantage. The advantage is that while what it said sounded plausible and if I had just read it I would have believed it and started incorporating this as a prompt, we have now managed to test it.

**Anand**: [02:08] **The beauty of this testing is that I have put in very little effort into the process of creating the test or the benchmark.** Step one: look, I want you to give me a prompt that will create a test. Take that prompt and say, "Create a test." Run it, tell me the result.

**Anand**: [02:26] Now, what we should do is a review of this, and we will, but we will use an agent as a judge for this. We should also probably revise the prompt and improve it. Both of these are good things to do. But where I'm going with this is, **the general premise is agents are good at generation, therefore humans need to be good at curation—that is what we put in—and verification—which is what comes out.** Which is true.

**Anand**: [02:53] However, the boundaries are expanding because **we can also use agents to brainstorm on what to submit. We can use agents to help us verify**, which is what we're doing now. So what really constitutes verification is probably a slightly different skill than what we might have thought of as verification before agents. And by maximally delegating, I'm finding that it's easier for me to understand what to learn, because it's something that agents can't yet do, versus what I may as well let agents do and therefore get a better result out of it. One of the results that we will be getting because of this is a benchmark that across a bunch of people tells us what's working, what's not working. I think we have one more result. Let's see. Sixth one maybe? Okay, no, we have five responses. Let me see if mine is done.

**Anand**: [03:55] Okay, mine is done and I will take `results.md`, paste it as well, and submit the answer. But mine... conclusion, interpretation: **difference appears from level 1**, which is interesting. So we have two results which indicate that there is some improvement and four results that indicate no improvement. Okay, maybe it is nuanced; maybe some kinds of data work better, some kinds of data don't work, etc. Let's find out. Yes, please?

**Unsure (Male)**: [04:51] [Inaudible] results, there is no much difference. I had used this with Sonnet 3.5. Wondering whether that actually affected the performance?

**Anand**: [04:59] Quite possible. So let us take all of this. I will take the responses and the question that... Okay, let me not even bother and just upload the `responses.csv` file. Workshop data for agents. Okay, having uploaded it, let me also upload the form. Actually, why am I doing it here? I really should upload it to Claude, where I was having that earlier conversation. This is where I was having the conversation on what prompts to use, etc. So let me give the same agent this data and say... I'm going to dictate out here.

**Anand**: [06:05] "**What I've uploaded is the final form that I'm using to collect data and the responses that people have submitted. What they did was took one of the hypotheses, which I'm pasting below, and the benchmark prompt that you had given with a few small variations—and I'm pasting that also below—to have their coding agents (which you can identify from the `responses.csv`) to analyze and come up with a set of results. I want you to do a few things.**"

**Anand**: [06:40] "**First, help me understand whether the benchmarking prompt itself is doing a good job for this particular hypothesis or it needs to be improved. To help you with this, I'm also sharing the full list of the hypotheses that were generated. Therefore, using this, think like an expert and come up with a better benchmarking prompt that I can use.**"

**Anand**: [07:07] "**Second, evaluate the results that we got for this hypothesis. Did it actually evaluate properly? Meaning, did the benchmark prompt actually get it to do a good evaluation? Again, think like an expert and see what the failure points are, but also to see if this can be used for any kind of a valid conclusion. And give me the main takeaway. I'd like you to summarize this using my meeting response style at the end.**" And now I have to give it a whole bunch of context. Let's do all of that.

**Anand**: [07:41] So, I dictate into ChatGPT because the dictation is much better and easier; I can just copy-paste out here. Now I need to pass it the prompt that we used. So these are the techniques. That is one thing that I promised to give it. Not this one. Somewhere... ah, these are the... no, okay. This is the test run. And then the other techniques list comes somewhere from the other window. Yeah, this is the list of tests. Techniques. What else did I promise it? `responses.csv` I've given, benchmarking prompt, full list of hypotheses. Yes, I've... okay, techniques.

**Anand**: [08:52] "**Think like an expert,**" which is basically I have a skill called "expert lens." And I have a skill called "meeting response style," which will help it answer in the right way. And the last thing that I need to do is see if I'm going to run out of Fable credits. Borderline, but I think it's smart enough. So let that run.

**Anand**: [09:20] **Couple of things that I'll flag off here: I'm not very good at experiment design.** I really don't know whether the way it ran it is right or not. **Nor am I good at rapidly evaluating evaluation code.** Generative code, I'm good at evaluating. Evaluation code, I'm not good at evaluating. It requires a certain mindset, a red-teaming mindset. I'm an optimist. So I will not even try and compete with this.

**Anand**: [09:55] Secondly, in the past I used to find that the naivety of the models was high, that if I said "be critical about it," it may not be doing a good job. I found that a prompt saying "**think like an expert**" roughly—which is encoded in one expert lens prompt—was working. I have not in the last six months tested to see if it is effective or not. That's pending on me, but my theory is that it still doesn't do too much harm even if it doesn't benefit.

**Anand**: [10:31] Let's take a look, actually, it's worth taking a look at the "expert lens" prompt. My skill, whatever. Because there are some skill designs that I found useful. Normally, if you said, "Think like a security expert," and give it some details, that works. Some other times I have to say, "Think like an expert financial advisor," and so on. Now, that means that I'll have to type out the prompt or at least change the prompt every time. I am lazy. So this skill is something that it can automatically pick up, even without...

**Anand**: [11:13] Well, okay, now this has been modified a lot. The first version of it that I had basically said, "Find out who's the right expert to answer this and then answer it that way," which was working fine. Then I did a few iterations, tested what is working, what is not working, what is the best practice on this. And one of the things that we found is that, see, there are some places where expertise matters, some places where expertise does not matter—or matters in a different way. **There are high-validity domains where things don't change much. In that case, the expert's gut feel is useful.** So everything that you've read on the internet, use that. **There are some places where things change a lot; in that case, don't use your gut feel but the process that experts follow is generally robust, so follow the process instead.** That is useful advice.

**Anand**: [12:00] Then it suggested rather than "think like an expert"—**"think like an expert" means what? Should I use jargon like the experts? That's what models tend to go to because it's the lazier option.** Instead, the agents were suggesting that based on research, if you specifically guide it on a few lenses—say, look at what an expert would notice, look at what mental model they would apply, etc.—but don't tell it to do all of these, it'll go overboard. Whichever is relevant, tell it to do it. And that seems to be working for me based on a few experiments. I have not validated this, like I said, extensively in some time.

**Anand**: [12:38] But anyway, let it apply the expert lens. What we should now have is two things: knowing what techniques will make data better for agents, it will rewrite the benchmarking prompt. Second, it will tell me the results of the benchmark that we just ran and see if it is valid or not. If it is valid, we will hold it and later on after accumulating a few of these, tell it, "Create a skill for me so that we don't have to remember it, the agent will remember it." If it is not, we drop it.

**Anand**: [13:13] Now, this is where the second... Okay. **It has identified one structural flaw and three smaller ones. And it identified what we had expected, which is progressive disclosure means you shouldn't be giving it the folder.** But if we are creating folder A and folder B, obviously it has access to both the folders. So the experiment itself is invalidated and therefore the prompt is wrong.

**Anand**: [13:48] Also, the default primary metric apparently contradicts the technique card. This testable claim is equal or higher success with substantially fewer tokens. The prompt says, "Pick one primary metric of accuracy." **So six of the seven runs that we had failed to find an effect.** Did anyone understand this?

**Unsure (Male)**: [14:14] Because we were comparing accuracy on both the runs, and the [inaudible] was accuracy and cost, basically. But the prompt only asked to take one primary metric as an accuracy, which is basically the same thing. So cost, even if there's a difference only in the cost, it's not being considered as a primary metric to determine the success of the test.

**Anand**: [14:35] A fair point. Got it. The calibration pilot got lost, token accounting was inconsistent and prompt-blind, and okay, some other fifth one: one run had no API key, so a deterministic script played the sub-agents. It was disclosed, but it still looks like a blah blah blah. And based on this, it's giving a revised prompt which... let's just run. I mean, if nothing else, it has learned from all of these techniques.

**Anand**: [15:10] If you want to take a break in 10 minutes, that might be a good thing. 10:15 is perfect. Yeah? Okay. And we will in a few minutes try this while we break; we'll let our agents run during the break.

**Anand**: [15:24] The second thing: did it actually evaluate the T-test? Now for this, I'm going to go straight to the end because I have a skill that says, "Look, just tell it to me in English." **The protocol worked, the construct didn't. What does that mean?** All seven teams produced something, okay, but every team tested files plus grep versus files plus an index, and not the real contrast. Okay, that is still kind of useful because what that tells me is with or without an index, is there a difference?

**Anand**: [16:04] And in two out of the seven, it looks like there was a benefit. Maybe that's good enough. **Agents already do just-in-time access by default, so giving it a shell, it's able to do what it needs to do.** So five valid runs found no accuracy difference even up to the highest level. What I mean by highest level is in the benchmarking prompt, I had earlier told it: look, if you just run one test and you find that there is no difference, that doesn't mean that the agent-specific arrangement of data is not really better. Maybe there is some more complex data for which it will be better. So what I want you to do is create three levels of tests: level one simple, medium, complex. And it will progressively test and wherever it has a benefit, at that point tell me "ah, there is a benefit," or vice versa.

**Anand**: [16:49] So, which is what it's talking about in terms of up to 500 token corpora. The way I told it to create—no, the way it decided to create the levels is by saying some of the test datasets will have only 50k, then 200k, 500k, whatever it was. One significant result says that the technique can hurt. That is... okay, and a p = 0.03, that is a reasonable one. It lost on 6 out of the 24 tasks specifically because **entity shards showed stale and conflicting values. Lesson is shard data needs freshness and identity metadata attached or it creates new errors.**

**Anand**: [17:36] What I am translating that into is: **boss, if you're creating some kind of an index, do it properly and carefully and test it a little bit. You can mess it up.** Cost savings are real but not universal. Okay, that is helpful. When we created an index, it ingested 2.5 to 7.5 times less context, but in one case it was 3x. Okay, I might consider that a win. Not sure.

**Anand**: [18:07] So three fixes before anyone reruns this: put the intervention in the sub-agent prompt something, pre-register something, ask each team to add one line. Yeah, but all of this it should have done by itself. So let me just say, "**You gave me some advice on what to change when the team reruns this. Did you incorporate it into the benchmark prompt? If yes, just say so and be done; else give me the revised prompt.**"

**Anand**: [18:52] I have no intention of doing work sitting in the middle of a—standing in the middle of a workshop. Let it decide. With this we will have the revised prompt. But here's what I'm taking away. **The original prompt was not able to anticipate the technique which said "progressive disclosure." But the benchmarking prompt still gave us a reasonably useful test.** If you create an index for the data versus if you don't create an index for the data, does it make a difference?

**Anand**: [19:24] **Conclusion seems to be: it seems to not matter too much most of the time; in one case it actually hurt, in one case it helped, in five of the cases it didn't matter. But the token savings mostly seem to be real.** So quality: don't, you don't need to; cost: of—if the agents are running at a higher cost, then you might want to reduce it, and in that case creating an index seems to be helping. Okay. That is a conclusion that I find useful. And because it's coming from seven of you, not just one of my runs, I'm feeling a little more confident. And if a team of 30 people were running it, I would feel even more confident. Now the revised prompt...

**Unsure (Male)**: [20:07] But this is like... we just manufactured the data, right? Like I mean, the whole idea of domains being different or even in your case it was set low context to high context. You know, in your script you had different things, right? I mean like in most engineering we will have a scale of jobs as we go up. Data something would make sense, some things wouldn't. So both on the amount of data this thing will change, the domain it will change, the type of problem it will change. Are we doing premature optimization by thinking about quality versus service versus cost? Like this is what we do, this is like an engineering trade-off. And if I am doing this consistently over time and repeatedly for my work, then I am starting to get into the engineering part of it, which is like, you know, what gets me best quality for the lowest cost and the fastest speed, right? Like kind of that. I'm not sure, like I'm a little skeptical of how much we can take away from this because we ran it in a very generic sense. Right? Like the general principle makes sense, like progressive disclosure, you know, do that. So I don't know, it's a question I guess. Anand will be a time to paraphrase.

**Anand**: [21:42] I'm hearing two things. One, **yeah, progressive disclosure generally makes sense but it varies on a case-to-case basis. In this case, we didn't put in any specific domain, so how transferable are the results? Wouldn't we need judgment to apply it?** Absolutely.

**Anand**: [21:58] So what I am confident of is three things. Number one, **earlier I would take an input from a person or an LLM and say, "ha, this sounds reasonable." Now I have found a way of at least testing it.** If I know what domain I want to apply it in, I could then in the benchmarking prompt say "for this particular domain create a dataset" or give it my dataset. So at least one part of my validation effort gets saved. And it's a reusable one.

**Anand**: [22:26] So you're right, if I want to figure out if progressive disclosure works in, for instance, document OCR for financial services where it's mostly credit applications that are coming in and I have a folder of 300 such, that prompt is going to be very different. It's going to be, "I have given you folder A. Now I want you to figure out whether this particular technique when applied to folder A will create a folder B based on progressive disclosure and benchmark against it." Spot on.

**Anand**: [23:02] However, **if this had across all seven cases said every single one of these runs at level one is getting you 30% to 300% higher quality, boss, this seems moderately universal.** I would use an exception route. I'd say by default add this skill and then I would in my domain just say "do one cross-verification to make sure that all of the skills that I'm running are still applicable here." Watch it for the first three to six months and then once you have a track record of success I wouldn't bother too much. So completely aligned. **This does not give me enough confidence to put it into a skill blindfolded.** But I'm happy to put it into a conditional skill which says, "Before organizing the data, test out, see if this makes a difference, at least from a cost perspective, and if cost is important in this project, use it." It'll probably be in a project-specific skill. It will learn a mention in my "things I learned" notes, not a global skill.

**Unsure (Male)**: [24:06] Also like within this progressive disclosure also there are so many different ways of doing that. This is very abstract around that. Like in some cases it might be creating summaries and then testing it; in some cases it is doing that grep and that thing; in some cases there are ways of doing that tree structure; there are ways of doing the index of documents. Sub-agent approach. All of these are various methods of giving progressive...

**Anand**: [24:33] Absolutely. And therefore that is the second failure point. What Amit was saying is that **one failure point is that the dataset needs to be specific and you have to test it on that dataset. What you're saying is the technique of progressive disclosure also needs to be specific.** And that we have to test. Quite possibly. I don't know this too. What I mean by that is for instance in a certain domain, maybe it doesn't even matter how you organize it and any kind of organization or progressive disclosure helps. Not very likely, possible.

**Anand**: [25:03] More likely, the natural way any agent today organizes it is likely to converge towards it because it sees the data and says, "Look, this kind of data, obviously this is how you organize," and you don't necessarily need to tell it. So again, I would benchmark. **And the cost of that benchmarking is becoming low.** By the way, if we discover generalizable rules, very good, that's almost a bonus.

**Anand**: [25:29] So let's try that second benchmarking. What I will do is now share the complete prompt on WhatsApp that we can just reproduce, which is out here. Technique 1 card, okay. This goes into WhatsApp somewhere, wherever that is. So my invitation to you is take this full prompt, just create a new folder—or new session or whatever—and try this. I'm going to do the same.

**Anand**: [26:25] And I made only one correction to the prompt, which is I told it to use a new folder for this test in case it ran on the same folder.

**Unsure (Male)**: [26:34] So could you also share the link [inaudible]? I don't have WhatsApp [inaudible].

**Anand**: [26:39] On WhatsApp I have the full prompt. I don't have it on WhatsApp... oh, oh, oh.

**Unsure (Male)**: [26:47] You can just create that `.md` file, your global portal [inaudible].

**Anand**: [26:53] Ha, but this is a new prompt. I have not created a link for this. It only exists on Claude, but could we add you to WhatsApp and that may be easier? I don't know how to...

**Unsure (Male)**: [27:07] No, I'm not on the group.

**Unsure (Male)**: [27:19] So, I think your approach primarily is write something with an LLM, then let an LLM analyze, improve, write that back, right? The other way, I guess a conventional approach would also be what I've seen doing, or at least I do, is like before we run anything, you keep doing this discussion of what is correct, what is not correct. Right? Like so, whatever we call as planning and [inaudible]. It's like critique, you do meta-critique, take this plan, give it to something else to critique. Right? That would be one dimension, I guess, in terms of like getting it to critique and say, "this thing about cost versus whether the metric is not correct, the cost versus accuracy," we could get some of this to be picked up without running these tests. Quite possibly. What do you suggest? Like what has been your experience in which one has better [inaudible]?

**Anand**: [28:21] To paraphrase: **if we plan, we get more accurate results up front, whereas if we didn't, then we might end up wasting both tokens and potentially—almost certainly—quality. So plan and do or do and review?** Some sense an abstraction.

**Anand**: [28:39] **When the cost is low, I find doing first better. When the cost is high, I find planning first better.** The traditional approach for software development is create a requirements document. Why? Because if you make a mistake, the cost of fixing it is so high. **The new approach to software development is code first, tokens are far cheaper than humans.** And therefore to create four or five solutions and then say, "oh I like this one, let's go in this direction," is now becoming cheaper.

**Anand**: [29:13] **We are not used to this era of abundance.** Sorry, I should say we're not used to this *kind* of abundance, and it takes a while to get used to it. Let me give you another example. The classic approach in a consulting project is you have a problem, I will create a plan to figure out how to solve it, we get it approved, we discuss, make sure we're solving the right problem.

**Anand**: [29:34] Here's an alternative approach. I told one of my colleagues, Thanoj: you are sitting inside a waste management company. You have access to an agent, Cortex, which is Snowflake's agent. Now Snowflake has the entire commercial data of this organization. How about if you just toss it all the company's documents and ask it to create a bunch of use cases that they might want solved? He did. And... [audio ends]

---

**Anand**: [00:01] The result was this. This is 100% Cortex-generated, by the way, anonymized. This is what it generated. So he took this to their head of analytics, who looked at it and said, "**Thanoj, we did a consulting exercise three months ago, four months ago in November, and that consulting exercise was a long, expensive one. You have identified independently 80% of those use cases.**"

**Anand**: [00:37] Thanoj got a huge kick. "Okay, now let me put together a project plan to implement each one of these," which is when he filled out the timeline and effort columns and all of that, and this is what it looked like. He said, "We can do revenue forecasting; this is the methodology we'll use; these are the fields we will use; Snowflake already has an ML forecast function; our approach will be to do this," blah, blah, blah. He said, "**I will write a project plan, get it approved, and then solve it. The agent is anyway doing the work.**"

**Anand**: [01:08] He said, "Oh, but I have some reconciliation to do." "**Tell the agent to do the reconciliation, and you tell the agent to do this also.**" Three days later he came back. "Anand, I finished that reconciliation in just four hours; it would have taken me two weeks." "Okay, good. What did you do the rest of the time?" "I solved it!" "Okay, what did you do?" "**I finished the revenue forecasting.**"

**Anand**: [01:29] "Okay, what else?" "**Then I created an email out of it which I sent to the people.**" Which was a really powerful thing, because then he said, "**Why does a human have to come, read the analysis result? Ultimately they're going to take some action. We may as well just tell them what action to take.**"

**Anand**: [01:52] And he said, "Not just this one, I finished it for all 50. Every single one of those use cases." So for instance, there was a pricing anomaly. They run a service which costs about $175 to $179. There are 105,000 transactions in which each of those has rates as low as 16 cents. Either you have a massive revenue leak, which is probably not likely, or at the very least you have a massive data quality problem. These are being tagged wrong, these are something completely different; at the very least we need to sit.

**Anand**: [02:24] Or customer churn. His email said—and the subject of this email was delightful—"**We are losing 117 million in revenue. Here is the fix.**" I can't think of an exec who will not open the email. And it talked through, "Look, here are some customers, anonymized, but US Department of Energy, 1.4 million. For the last 128 days for a monthly service they have not been ordering from you. Or Meridian Chemical, for the last 126 days they have not been ordering from me. Go reach out to them. Who are the account managers? Just talk to them."

**Anand**: [02:56] So what he did was, in this particular case, **because we are in an era of abundance, he solved a problem no one asked for.** This may be spam; it may be useful. The open-source ecosystem is facing this problem in a very visible way. For instance, I had ChatGPT... I was at SUPD...

**Anand**: [03:26] Wait, we are in the break. Are your agents working with that new prompt? Okay, then we can talk. Can you please add yourself to the group? I need to reshare the prompt after I add you. Ah, I—after I add him, I will reshare the prompt. Sorry about this. I'm not able to find you.

**Unsure (Male)**: [04:03] [inaudible] entity shards... So if the person was only [inaudible] I'll take you back into the weeds of what was discussed. If there were some numbers that the LLM is supposed to calculate from the folder, and then you split it. But you have to ensure that all of the context goes out. One which is already cautioned outside, this is—this is where accuracy versus the [inaudible]. And then by the time the final answer comes out, this is—these are the numbers that actually might be wrong because now there is a contradiction directly and then there is an extrapolation that's essentially my number better than your number.

**Anand**: [04:43] Absolutely. **And that is one of the risks of getting it wrong. And therefore, we would probably want to go for a plan before you execute.** So this is a call that Thanoj takes very carefully. He says, "**Anand, this churn email, I know the guy, I can send it to him. Revenue forecasting, I will hold until the head of analytics approves.**" For which there is a big meeting after the finance team has gone through this and verified, saying, "Ha, we will present it to the CFO."

**Anand**: [05:14] Now, very fair point. **The generation is cheap and the socialization requires judgment**, which is a very fair point. Let me reshare—or actually, could I request anyone who's on the group to reshare the prompt?

**Anand**: [05:33] I will maybe just finish one last comment and then we can break. No, actually not. I will come back and share a very interesting Robert Police story about what's happening on the bounty hunting on GitHub after the break. Let the agents run. We'll take a break.

**Unsure (Female)**: [05:46] Okay, we will ask the Microsoft employees to please guide you towards the cafeteria.

**Anand**: [05:52] Thank you. **Folks online, we will be taking a break for 15 minutes.** 15 minutes is good. We have a 5-minute buffer, but let's try not to use the buffer.

**Unsure (Male)**: [05:58] Yeah, sure. So for the folks online, we will interrupt the stream briefly because we need to fix the glitching and various devices need to be rebooted once, in the hope that that will fix it. [audio goes to incidental noise during break]

**Unsure (Male)**: [06:30] Amit, can you please [inaudible] pass the cable? So what happened was two requests. One, can you please wait for Cortex to finish? Okay. Two, start with the rest. I suspect it's actually physical. Okay. See that? Oh, yes, something is a loose contact. Okay. Sure. Do you have any [inaudible]? I'm going to move the keyboard up. I forgot to put my [inaudible] back. Just a moment.

**Unsure (Male)**: [06:56] HDMI? No. Wait, tell me if this—no, not HDMI. Wait, tell me if this—no, it's not HDMI. [inaudible]. So as long as this display supports USB-C, what I'm going to try is... I'm going to try it on my [inaudible]. I'm going to try this. Let me see if this does anything for cable reliability and then we'll reboot the Cortex. [inaudible]

**Unsure (Male)**: [07:41] I will also ask... Arvind was my last sort of like, you know, direct link to Karthik, so if you're going to [inaudible] a lot of mutual [inaudible]. Yeah, and I'm now discovering more and more directions. You just said that editor is also like [inaudible]. So yeah, so I think Karthik is basically known to a lot of people and like I said, you know, the nodes are reaching [inaudible].

**Unsure (Male)**: [08:06] This is weird. Okay. [inaudible] Yeah, like [inaudible] digital [inaudible]. Yeah, I can just imagine. [incidental chatter during the remainder of the break].

**Anand**: [28:56] For those—so for those of you whose agent has managed to complete something, please go to the same form. Right at the bottom there will be one more question, which says something along the lines of—well, basically the last question, whatever it is. You could paste your `revised_results.md` on that.

**Anand**: [29:24] In my case I have a result and it turns out to be a positive result in this case. Yeah. Question number six for those of you... Okay, looks like three of us have submitted against the `results.md`. That is great.

**Anand**: [29:46] And this time I'll wait for at least half a dozen of these before we start aggregating. **My guess is that this time we will have a better, sharper result.** It may not be positive, it may not be... [audio ends]

---

**Anand**: [00:00] ...it may not be negative, but **it will be higher statistical significance because the benchmark was aware of what it's going to test.** One thing that I will share just continuing from the earlier discussion on the abundance mentality—on GitHub there is a—there are a whole bunch of bounty hunting robots, agents. How does this work? What happened?

**Anand**: [00:26] So I asked... I was at a session at SUTD two weeks ago, where somebody was teaching Z3, which is a software that can prove software. It's a software theorem prover. How does it work? I don't know, but for instance, it was used to find out bugs in Python itself. One of the classic problems in a reasonably popular Java library for binary search was: it was calculating the midpoint in binary search as `low + high / 2`. That has an overflow error bug if `low + high` becomes higher than the highest integer value. And the better algorithm is `low/2 + high/2` or any other bit shift mechanism. And Z3 is the kind of thing that can at least make sure that you don't have this kind of an error anywhere in your code, and for other kinds of errors and so on.

**Anand**: [01:28] So while it is not something that proves necessarily that the code is correct—or maybe it is—it certainly can prove that certain kinds of bugs do not exist in your code, overflow error being one among several hundreds of these. I learned this. Now, learning is a very weak term because **for me learning is: I am the head of HR for my agents and I copy that learning and pass it to my agents.**

**Anand**: [01:51] "ChatGPT, look, there is this thing called Z3. Can you just run it and find any other famous... just look, see what famous bugs were verified by it." It said, "Oh, lots of things were verified there, formal verification methods." "Okay, can you find me some bugs?" And it did. And some of these were fairly popular libraries. And some of these were utter rubbish. I mean, I could see that some of the bugs are—there's no point fixing these; these sorts of scenarios don't really exist.

**Anand**: [02:23] So I had it run one more round, saying, "**ChatGPT, give me something which if I submitted as a PR [pull request], the author would really appreciate it.**" And it did. One of them was in Xarray, where it said, "Look, if you try and split a range into an equal number of two, three, four, five regions etc, the Xarray's `linspace` function equivalent will do it, but if you put `n=1` it raises an exception."

**Anand**: [02:54] Now I had already told it, "Don't give me general stuff, give me stuff that the PR owner will accept." So it added a line saying, "Look, `n=1`—that is, taking a space and splitting it into one—doesn't make sense, but there are two reasons why you would want to consider it. Number one, you are not controlling the `n`. The `n` is coming from an external source, and a category of one is a valid point. But the authors may not accept it."

**Anand**: [03:26] It went on to continue, "**This is the same behavior as NumPy's `linspace`. NumPy's `linspace` provides this output correctly. Why shouldn't Xarray, which is intended to copy exactly the same interface, produce the same output?**" So I said, "That looks reasonable." No, I didn't say, "That looks reasonable." I glanced at it and it seemed to have a reason, so I said, "Okay, create a pull request for it." It created a pull request. The author below it said, "Yeah, the NumPy `linspace` consistency is a valid argument, though I don't really see this as having a major use," and accepted it.

**Anand**: [04:02] This is part A of the story. And let me spend a few—Part B. A few minutes after, few hours after I submitted my PR and had raised the issue, somebody else had submitted a PR. And okay, fine, somebody else is doing it. Is their code better than my code? I had done some amount of manual review of the code and was reasonably proud of the submission. Their code was not as good as mine, as it happened. Then I looked at their profile. Amazing actress-level photo. Korean. Wow, okay, some Korean company. Clicked on the link; led me to an African in Ivory Coast, Abidjan. Okay, African in Ivory Coast working in a Korean company with a photo of somebody who looks like an actress—what's happening? And then it turned out that this is a bot. All right.

**Anand**: [05:02] And self-declared bot. Few hours later another bot submits a PR. So I went to ChatGPT and said, "What the hell is happening?" It said, "**These are bounty bots.**" If you go search for—just FYI, I am waiting for people to submit a bunch more of your `revised_results.md` in case anyone's done or doing and you're submitting. I'm just giving it a few minutes so that I'll have half a dozen I can work with it, but just filling it with a story.

**Anand**: [05:28] Then it said, "**These are bounty hunting bots.**" GitHub issues apparently has bounties. I knew it, I forgot about it, now I learned it again. You can say anyone who solves this issue gets $5, $15, $500, whatever. And these bots are actively finding easy-to-solve bounties, submitting it, and putting in one note. This particular bot said, "PayPal account something or the other."

**Anand**: [05:54] And one of the people who responded to one of its issues said, "This is an open-source project. Sorry, we don't have budget to pay for any of these; we didn't put in a bounty or any such thing." Sindre Sorhus, who's one of the top JavaScript library authors, reviewed the code for one of these and had a detailed—not very long, but three, four paragraph explanation of why this solution is not exactly an apt solution and how he might attempt to do it differently. Three others closed this as "bot, I'm not going to look at it" and so on.

**Anand**: [06:31] But Sindre Sorhus's response was probably the most instructive to me. Because what that got me thinking was, "**Why am I discriminating against agents?**" Something has, with good or bad intent, submitted to my open-source repository. He is treating that as, "Okay, you have contributed through tokens and I am reviewing it. It is worth reviewing, or if at a glance it is not worth reviewing, fine. If I'm not able to, then that's okay. I can probably use agents to filter out."

**Anand**: [07:01] But it just gave me a perspective that **I need not discriminate against agents simply because of their origin. They are potentially productive members of the society as well through their token consumption.** It's a perspective. Absolutely not saying it's right, because there are several people who have said "we will not accept AI contributions" and several people who have said "AI contributions must go through a slightly different process" and so on.

**Anand**: [07:32] Now, why am I saying this? **The premise is that because generation is cheap, we need to focus on validation.** That presupposes that there is a queue of use cases; those are the ones we have to solve. Because generation is cheap, now new use cases become possible, like this bounty bot. There is also a Kaggle bot. Go solve Kaggle problems, try and win the prize money. One of my students submitted one. He couldn't win yet, but the next step is now to scan across three wins. And the person who wants the problem solved is getting a solution. If seller is happy, buyer is happy.

**Anand**: [08:14] Or not! Maybe it is intended to teach students how to learn data science, or repositories are meant to teach students how to submit repositories, in which case it's a very different purpose, in which case it should be banned. There are many other reasons why you would want to ban it. But the use cases are emerging. **New use cases that are becoming possible simply because agents can not just solve problems, but identify problems that can be solved and solve them also.**

**Anand**: [08:44] It's something to keep in mind, and I see this—the whole notion of **data for agents**—belonging in that category. Agents can also consume data. That doesn't mean I shouldn't organize it for humans. Organize it for humans; create a separate folder and organize it for agents also, by having the agent do it, by having it figure out how best to do it. That's the premise.

**Anand**: [09:02] Okay, so we are stuck at six. I'm going to assume that we have or will likely have only six of these for now. And let's talk to ChatGPT and see what we need to say. Once it loads. All right. We tried the new prompt. Six of us. And have a bunch of results. These are at the bottom. Now analyze this and tell me: what should I conclude from this and what action would you suggest I take? For instance, should I just go ahead and blindly put this as a `skill.md` saying whenever we have data organized in a folder we should probably reorganize it in some different way? Or should it be more nuanced than this?

**Anand**: [09:51] Put another way, **I want you to think like an expert and guide me on the actions that will have maximum impact at relatively high frequency for me.** I'm not really sure what I'm asking for. This we will run on... what chat somewhere. Yeah. Skip this. Data engineering right. Expert. And meeting.

**Anand**: [10:38] Now let's upload the responses and see what we find. Now while that runs, let's take up the next hypothesis. And I will share on WhatsApp a revised prompt. Let's run the second hypothesis and see if that is giving us a positive or a negative result. Nothing stopping us from running half a dozen of these. So we will do one more. This was T1. Let's discuss T2 first.

**Anand**: [11:21] It says: "**Execute queries and transformations outside context.**" It's saying for humans, export the data into spreadsheets, reports, application screens, etc and inspect directly. For agents, expose SQL, code, DuckDB or tool APIs so that filtering, joining, looping etc happen in an execution environment and return only the small final result. Which is roughly saying, okay, humans: give them spreadsheets, databases maybe—not databases, yeah, humans: give them spreadsheets that they can load fully. Agents: give them data processing tools. This is kind of obvious. Should we test or skip?

**Anand**: [12:08] If it is obvious, then the implication is that we should put it into a skill and say, "**Look, whenever you have data, say in a spreadsheet, always, almost always convert it into DuckDB code, something SQL-processable, etc.**" Any reasons why we would not do that?

**Unsure (Male)**: [12:26] [inaudible] I'm just curious to see how it would try to validate human versus agents [inaudible].

**Anand**: [12:33] Which is an interesting experiment. Fair enough. Let's just hold on that thought. Maybe we can come back to it. But if there's something else that we're not sure of, that's not so obvious, let's use our tokens to test that.

**Anand**: [12:46] Schema-constrained interfaces. For humans, accept prose, JSON conventions, etc. For agents, enforce schema. That is, when you're running a tool, make sure that it always gives an output in a schema. Again, does anyone think otherwise? Would we ever say, "No, no, no, enforcing a schema is a bad idea"?

**Unsure (Male)**: [13:12] If you think from [inaudible] the intermediate processing agent, yes, you want it to be structured. If the output agent, I want it to give me the insight. I don't know if [inaudible] maybe a loose structure is fine. I wouldn't enforce a very rigid schema at the output level.

**Anand**: [13:30] Good point. For the output agent, maybe we don't want to enforce a rigid schema. Intermediate agents where a machine is consuming it—it's possible. Fair point. Usually I find that the output is consumed by some frontend software at best, and which is therefore also a machine. The number of times that it goes directly to a human or needs to go directly to a human from an agent... we can always add a tool in between to do the conversion, is what I'm finding. But no, which is not to say that that's invalid. I think it's quite valid.

**Anand**: [14:02] Let's just see if there's anything else that's slightly more... Search and filter before listing. For humans, provide a complete list. For agents, require or prefer server-side searches, predicates, date ranges. Okay, so only—don't give it a complete list. In the first case, give a full list or paginated; let them scroll. Okay. This overlaps with the first one. Let me...

**Anand**: [14:34] This is obviously doing manual work. So let's shorten that and say, "Some of these hypotheses are clearly uncontroversial. For example, we tested one and it wasn't clear that that was an obvious win. But the second and third, maybe even fourth, seem like they would obviously and almost always be true. Is that the case? Or put another way, if you had to pick one of these that people may not agree with, and yet is true, then what would those be? Give me the top three." It is likely to be...

**Anand**: [15:21] So that helps us pick the hypothesis. We will test one or two of those. I should of course share this chat with you as well. I will be doing that in a few minutes once it completes this. And then I know we have only half an hour, but we'll use that half an hour to test one more hypothesis, synthesize, and see if we can create a skill out of it.

**Anand**: [15:47] Changing... Okay. "**For changing text-heavy corpora, grep-style navigation can beat embeddings.**" That I'm not sure is true. Embedding similarity is a powerful one. Uniform tables can be serialized less machine-readably for humans... This is something we discussed earlier. **Is JSON versus CSV going to make a difference?** So it's saying CSV will beat repeated-key JSON is the claim. Earlier it used to be the opposite. Today this is true. This I think is worth testing. It is definitely a changing wisdom. You may disagree—please run all the other hypotheses as well—but let's give this a shot.

**Anand**: [16:38] So JSON... which is the... okay, let's see. For the uniform tables thing, which hypothesis number was it? Can you give it to me again? And oh, it was number two. Okay, fine. Then it's reframed it. Okay, fine. Let's take this and test it.

**Anand**: [17:31] So now I have the revised benchmark prompt and the hypothesis out here. Let me paste this on chat in WhatsApp. Do give this a shot. I will also test it. This will be the last of the hypotheses that we are testing. And... just going to make one small change to this; I'll call it T2 instead of H2, but you don't need to worry. And let it run.

**Anand**: [18:35] What I will be doing is sharing the anonymized results of—I'm just taking out the email column—and sharing all of the results that we have so far, the prompts and the outputs. In case your agent is not able to complete the `skill.md` now, just take a shot at creating it later. We'll start it off today.

**Anand**: [19:00] Here is how... okay. I mean it's testing out with a bunch of agents like GPT-4o and all that. Okay. **What ideally we would want to do is to get the answer to this workshop's objective. We would take potentially these hypotheses, possibly more, on a recurring basis and setting up an infrastructure roughly like this.** Which is: I will create a new hypothesis, take this benchmark which I might constantly improve over time, run it, get the result, put that `result.md` into a new folder. Maybe run it three or four times so that that folder will contain `result.md` version 1, version 2, version 3. In this case you are my versions. You may create your own versions.

**Anand**: [19:49] And then have an agent say, "**I've run all of these experiments. If you think this particular hypothesis is strongly true, add it to my skill.**" That way it will automatically learn. **This leads us to one possibility of loop engineering, auto-research, what have you.** I have largely been a hammer struggling for—looking for a nail. Meaning, loop engineering looks cool. What can I use it for? And I have nowhere near any clue to the answer.

**Anand**: [20:23] This looks like a possibility: go through my code and figure out how to improve it. How exactly to go through my code, I don't know. This is one slice that struck me as a possibility: come up with new ideas or hypotheses every day. Or review the ways in which my code is executing by looking at the logs and come up with hypotheses. Or review recent research papers and come up with hypotheses. I don't really care how you come up with ideas.

**Anand**: [20:50] Second: efficiently test and see the results. Review whether you did a good job of testing—if required, improve your benchmarking prompt—and then put it into the `skill.md`. If I can have an agent set this up so that every morning I just click one button and it runs, or every morning I don't have to click one button and it runs, something is automatically improving. Periodically I'll go in and see if it makes sense, if not edit, but that `skill.md` should be reasonably easy to edit. Okay, that was very quick. And it's saying...

**Unsure (Male)**: [21:30] I think human loop is a good... I think any agentic system and its logs therefore are potential for continuous improvement and therefore a loop-based system, which is a good system for continuous improvement because you're constantly taking traces from that system and even the chatbot that you build for support, customer support, you can rank the responses and identify where the improve [inaudible].

**Anand**: [21:51] Got you. **The point is any agentic system and its logs therefore are potential for continuous improvement and therefore a loop-based system**, which is an interesting one. Let me come back to that in a few seconds because I just need to update the form to allow for the T2 results. I'll call it T2/H2 here. And let me... I have a feeling something went wrong. I patched some experiment, regenerated a deterministic score...

**Anand**: [23:05] No, okay, I have a feeling it just didn't do what I had asked it to do. So I'm just going to rerun it. Okay. But now, to the earlier comment: would any loop be a—would any agentic system with its logs be a candidate for a loop? Yeah, makes sense. I have this agentic system or any system, operating system running—I will every day ask it to look at the logs, see if you can improve the code, iterate, and so on. I've never tried this. Has anyone tried this?

**Unsure (Male)**: [23:54] We have tried it but didn't really look at the performance as such but [inaudible] what was failing. Like, we use an OpenTok kind of setup and because it breaks a lot—because it goes into wrong directions quite a few times for me—and an automated job that every third day like, look at the logs and see where the—what things can be improved, what prompts we can improve and do those things. And it started giving like, feedback like, removing credentials from the logs and those kind of things. Not exactly kind of inputs we were looking at, like what will make you more efficient as a tool. So those kind of things. It was good at providing this in case of errors, like when errors were happening then it created pull requests automatically to fix those errors. But yeah.

**Anand**: [24:51] Which is a fair point. Have you been using that as an approach for agentic systems as well?

**Unsure (Male)**: [24:56] Not—the whole what you mentioned which is any agentic system can effectively be loop engineered. Yeah, I haven't tried with like, automation like an OpenTok kind of thing, but at least like, on a weekly basis, like, I have an intermediate system that summarizes the query and context and then like, it tells where it might perform better. And then your system instructions or the prompt gets improved. And every week I do it, which actually is a way to maintain the quality of responses. So I just put the—for the code based experiment for example—and then build like an internal documentation for the agents. And then your system instructions or [inaudible] gets changed. And every week I do it, which actually is a way to maintain the quality of results.

**Anand**: [25:34] Got you. Which is interesting. Because I'll share how I'm using it to scrape LinkedIn in a crudish sense, but... Okay, so it's saying in either case the claim holds from level one. And whatever the reason, it seems to be putting it... Okay, save the results in a new folder, folder or whatever else in a folder. This is a new experiment, right? Or is it the same as the last one?

**Anand**: [26:25] Because I thought I copy-pasted the right one and this is talking about JSON versus CSV. Oh, I see, it's putting it in the same one, it's the same family. Okay. And it's created a new folder. And okay, this `results.md` is testing... okay, conclusion: claim holds from level one and... I'm still not 100% sure from here. Okay, let me ask it: "What exactly did you test and conclude? Add that to `results.md` and use simple language."

**Anand**: [28:45] In the meantime, for anyone who has been able to come up with a revised result for the second hypothesis, which is the last one on the chat, please let us know. **We're testing whether CSV works better or JSON works better.**

**Unsure (Male)**: [29:03] I've run this on the—it depends on like if the agent also has a code execution sandbox. If it uses the code execution then it goes better because then it can just like... like CSV takes less tokens than JSON but because of code execution sandbox you can—I mean I am talking loosely here—but it can provide like suggestions and obviously like the response is also quite fast but you must have a sandbox. If you don't have that, then some of this might not apply. So it depends on what you also provide the agent.

**Anand**: [29:52] Very true. And therefore **the question then becomes: if you have a constrained set of tools that you can provide, generic advice may not work.** Are they...

---

**Anand**: [00:00] Part B: is there a way by which we can somehow expand the space of what can be provided? What I'm finding is that these constraints often exist in enterprise IT systems. And **enterprise IT systems largely are going for confidence rather than technical validity**. Their question is not "Is this a safe environment and therefore should I approve it?" It is "**Is this an environment that my auditor will approve? Is this an environment that my insurer will approve?**" And that part is easier when there is an enterprise backing. That is easy when it comes from big cloud providers. So the easiest route that I've found is: "Here is Microsoft Sandbox" or "Here is—if you are on Google—Google Sandbox, or Amazon Sandbox." They'll say, "Ah, okay, fine." And then if it comes as part of an enterprise plan, it usually has a reasonable amount of legal guarantees which they are comfortable with. So my highest ROI move is just find the enterprise version of it.

**Unsure (Male)**: [01:07] [inaudible] sandbox system... I don't think it's mature for everyday use. I think Amazon has like a sandbox... very early stages. I don't know about Microsoft and I don't think Google has officially launched something on the substrate—any substrate on Kubernetes supports sandboxes—that is a very costly [inaudible].

**Anand**: [01:32] Yes, I agree. But they're maturing rapidly, I think. I don't find much of a gap in terms of the execution environment, partly because the agents themselves are offering it as part of the tool. OpenAI says, "**I'll give you a sandbox in which you can run stuff and you can decide how much control you have.**" Anthropic has—I mean, Claude—and Google as well. Google, in fact, yeah, started as probably leapfrogging them there. Plus there are... Cloudflare is now saying we will offer these micro VMs maybe; Fly.io is offering a whole lot of these. So where we want to test and experiment, sandboxes with granular controls are emerging. For secure execution, they are part of the cloud environment. Because of the growth of agents, I suspect that yes, there is a gap, but it's a shrinking gap.

**Anand**: [02:33] Okay, we have a couple of results. I just realized that this isn't as much testing CSV versus JSON; this was more: "**Should I put the entire content into the prompt or should I let it access and pull on demand?**" And it is overwhelmingly saying pull on demand. No question about it. Which interestingly is the opposite of what we had concluded at the beginning of this discussion.

**Unsure (Male)**: [03:02] But look at the first line.

**Anand**: [03:04] Sorry, what?

**Unsure (Male)**: [03:05] The prompt you submitted still said that we need to test progressive just-in-time [inaudible].

**Anand**: [03:14] Oh! Okay, sorry. I just basically gave you the wrong prompt and I've been using that wrong prompt multiple times. Sorry. No, then I will not—since we have only 10 minutes—take this up. My request is once I share the chats with you, you will be able to run these by yourselves. First chat that I need to share is where the prompts came from. And that's going to go on WhatsApp. Prompt generation. And the other was the set of hypotheses that we generated.

**Anand**: [04:27] And the last thing that I will end with is simply converting this into a skill. Where is that? Okay.

**Unsure (Male)**: [04:50] What is that prompt?

**Anand**: [04:54] Yes, the prompt is incorrect. Please stop it. There's no point in... I'm going to delete that last—oh, I can't delete the last prompt. Yeah, please ignore that last prompt and stop what I was doing. I mixed the first prompt and the second prompt in a very bad way. But what we'll now do is ask it the question: "**See, if we—based on this, if you had to create a skill... Based on what you have seen so far as evidence, what skill would you suggest we create about it?**"

**Anand**: [05:32] Think about this; you don't necessarily need to create a skill if it should not be applied uniformly. You would create a skill only if there is some data or evidence here that suggests that it would be broadly applicable. Of course, making sure that there are caveats and filters so that it doesn't get applied in the wrong place. **My aim is to set up a skill that, when I'm converting from a corpus that is typically meant for humans to a corpus that is meant for agents, you automatically apply this skill so that it follows the benchmark best practices that we have.** Let's give that a shot. The good part is Claude understands how to create skills. Codex is not bad at creating skills either. You just say "create a skill," it creates a skill. And let's end with that. And we'll take a look at the skill that it's created.

**Anand**: [06:41] In all likelihood, it will need improvement on two dimensions. One: is the skill well-worded? We can test it. All we have to do is tell it, "Here is a skill. With and without the skill, are you doing a good job?" The second thing is: is this truly applicable? I mean, is it worth even the few tokens that it is going to be consuming? That is likely to come from the logs. "**Go back through my last 100 chats, 1000 chats, 30 projects, whatever. See where the skill would have applied.**" Maybe even do a benchmark. Take a sample of three places where you think it's borderline—not sure—would it have really helped? Based on that, do you want to refine the skill? And so on.

**Anand**: [07:23] And it's saying, "Okay, create the skill. The evidence supports that there is a difference for agents." And it gives me a reason—I'm not too worried about the reason. Let it write. Okay. **This potentially is a skill that I will be adding to my growing list of skills.** Initially, whatever it says, I will probably eventually convert it to just two or three lines. Instead of having a large skill that does a whole bunch of things—you are too verbose for me—just condense it to the few lines that we need and run with it.

**Anand**: [08:02] Just a quick spot check. How many of you have or are using more than let's say a dozen skills with Claude or Codex at any point in time? More than three skills at any point in time? One to two skills at any point in time? Okay, so majority. Zero skills? So one to two seems to be the majority. Curious, what are the bottlenecks to increasing the number of skills or lack of need? Why don't you have more skills?

**Unsure (Male)**: [08:42] I mean for my workflow there's very specific types of applications that you want the agent to do. I think if you have a lot of skills, the effectiveness I've found is it gets diluted a little bit. **It's a lot better to instead have multiple agents, having each their own skills and doing very specific things, then for there to be something as a supervisor that is uniform across these things in the output for all of it.** Yeah, essentially. So at least for like larger scale custom builds.

**Anand**: [09:23] Which is interesting. Do the same agent configurations go into Codex as well?

**Unsure (Male)**: [09:31] I haven't been using Codex at all. I've just been using Claude.

**Anand**: [09:35] Which I'm finding to be a fairly powerful enabler. Meaning anyone who is just on the Claude ecosystem, I usually see as going further ahead on the agents' route. I'm on the "I switch between agents" kind of a thing and therefore I'm behind on that curve. That's a useful inference: **agents work better than skills at scale.** Any other reasons?

**Unsure (Male)**: [10:01] I think Claude has been building skills by itself and registering them without me actually doing anything separately. And it registers the skills in the cloud, so it's accessible from the coding agent. I noticed this when I actually built a skill in Karthik's workshop thing. And it asked, "Should I register it?" And then it was also available in my internet chat. And it had other skills which I had not [inaudible].

**Anand**: [10:31] Yeah, I see this "save skill" which is probably the button that would install it. And yeah, I wouldn't in this—for this particular skill—but I can see. And what you're saying is therefore that's another thing that encourages it in the Claude ecosystem.

**Unsure (Male)**: [10:47] So as a result, I before this had never built a skill before. Until Karthik's workshop I had not decided, "Okay now I will build a skill and register it." But seeing how the Claude work [inaudible].

**Anand**: [11:00] Fair enough. Yes?

**Unsure (Male)**: [11:04] So the other thing is like, where should these be saved as well, I don't know. Like, long term. So for example, like you take some sort of analytical type workflow, where there's a series of steps. But then you just input that into an instruction along with the context, instead of including it in a skill, which is basically what you have across etc. So then I just have these specific scripts as an instruction before going to the next step, instead of [inaudible].

**Anand**: [11:30] Okay. So, basically you're saying a workflow is better than a skill?

**Unsure (Male)**: [11:31] Yeah, because they're over a long list of skills. Now I'm starting to build out some more stuff and ensure that information is captured better, and keep the list a bit more clean. Except for my standard script for [inaudible].

**Anand**: [11:42] Got you. So you're putting it all—always for an agent to read—and then over time spreading it out where it is needed on demand. Which I think is a very reasonable evolution. And arguably we go further down this evolution when we use it more across projects etc. So which is a fair point. **I'm just trying to leapfrog that where I can with benchmarks and evals. If I can create one that says, "Okay, three months down the line you will have this problem, and you may as well solve it that way," just trying to see if I can stay half a step ahead.** But not everywhere. I don't think it's worth—this would take three, four hours to run—maybe only 20 minutes of my time or even an hour of my time, but it's still an hour that you have to question whether it's worth investing.

**Anand**: [12:37] But if I were to wrap up and summarize what I'm taking away from this workshop—and this workshop was really a learning exercise for me; I haven't done this before. The things that I'm learning are: one, among other things, that some of my hypotheses on what would be better to provide to an agent vis-à-vis human, I can go out with far greater confidence now. **Provide only the summary and let it pick up the stuff. I can now say I've done a test; even at the lowest levels, that is even with small context, it is able to beat it with fetch-on-demand.** I can probably once I run this on JSON versus CSV, say with confidence I can do X. I did not realize that that confidence was worth what I now feel it's worth. I can say, "Yeah, look, I've done it. It works." And that means then that I can start sharing it with my team who will then also be able to run it with that confidence. This is part A.

**Anand**: [13:49] Part B: **I did not realize that creating a benchmark is as easy as I found it to be.** I'm not saying this is necessarily easy, but it is certainly easier than I had worried about because I had three problems: I don't know what to benchmark—I mean what is the whole process of creating a benchmark. I don't have data for a benchmark. I don't have a basis for evaluating the benchmark. Now given all three problems, I had no place to start. What this exercise did for me was say, "**Look, give an agent your objective, it will do all three for you, and you have to be a judge and see if it makes sense.**" Which to some extent I'm able to poke, prod, and where I'm not sure, I'm not sure—okay, fine. But the creation of the benchmark is not the headache now. It is judging whether the benchmark is a good one or not that becomes the problem. That is one step forward which makes me feel a whole lot more comfortable. Then I can delegate this to one of my team members and look at it from the outside and say, "Okay, in this area I'm an expert, let me do this." Or better yet, create a benchmark, give it to somebody who is an expert in that area and say, "Poke holes in this." So that they can, instead of worrying about how they go about creating a benchmark, they can just evaluate the benchmark.

**Anand**: [15:13] We have some experts in chemical structure evaluation from papers, PhDs in chemistry. They're saying, "We currently have an agentic system which is extracting it; it is not doing so well. We want to switch models but manual testing is taking a lot of effort," etc. So now my plan is to go to them and say, "Look, we can create a benchmark. You have your logs, you have stuff, right? I will just tell an agent to create a benchmark, it'll create a benchmark, then you tell me the benchmark is right." So **I've delegated two pieces of work: one to an agent, one to an expert.** Fantastic. They are enabled, I'm happy. That is the second takeaway.

**Anand**: [15:48] A third takeaway is that—and we should really look at this skill—it's saying, "Okay, here's how you turn a human-oriented corpus into an agent-oriented one, but only where measurement says it will help." Fair enough. So it is introducing that element here. It's a decision rule: Step 1, measure before touching anything. So figure out whether it will really save stuff, decide under these circumstances whether you will convert, whether you will not, building a rule of thumb which I will review closely and see if I agree with it or not. And build a catalog out of it and some sharding rule. Not sure. But the takeaway for me here is: **because I can always put stuff into some asset—could be a sub-agent which I'm going to try next, or a skill which is always going to get applied, or even just some directory or one prompt which I can copy-paste and run—I don't have to be the person that carries this knowledge in my head.** I have a place where my notes can get auto-used. I have to yet figure out what is the best place to put it, but it is storable and usable like a real asset. And that asset is a potential flywheel: look at the logs, update it, see where it's working well, where it's not working well because it's ultimately the same benchmark that is leading to it.

**Anand**: [17:20] These were my takeaways from the workshop. Since you have logged in on Google, what I will do is have some agents do a little more analysis, see what else it finds, plus run a few more tests and share what we find. **My request to you is: run a few of these benchmarks, run a few other benchmarks, create verification datasets, translate that to skills, see how much you can get out of the way.** The fact that we don't know something might be an advantage. We are no longer wedded to the craft that we are following. Something doesn't work; as long as it doesn't cause us any harm, what is the big deal? **Tokens are cheaper than our time.** Let's see what we find. Do give it a shot. Thank you.

[Applause] [18:09]