Okay, it's usually a bummer to have the last, second-last talk. Nilesh is of course, at the end after this presentation. But thanks to those of you who have stuck around and I really hope that I can sort of finish this in less than 15 minutes and then not come in the way of Nilesh and then the ending of this conference.

So, what I'm going to speak about, which the introducer mentioned, was building this culture of openness in peer review. And when I was thinking about this talk, it struck me that you know, it's about roles, rules, and returns. So let's see how far we can get in this journey.

What was the problem that we were trying to solve? The problem that we were primarily trying to solve was to say, how do you make this process of the peer review itself public? So if any of you have—how many of you have published in a peer-reviewed journal? Please raise your hands here. Okay, Jaidev is one. Okay, not, there's one more person there. All right, so the typical process in a peer-reviewed journal, if I understand correctly because I've published in a peer-reviewed journal, is that the comments that, first of all, you don't know who your reviewers are. Most of the times the reviewers are anonymous. And secondly, you get sort of a bunch of comments on your draft saying, "improve this, improve that, these claims are not right, substantiate with data," etc.

And so, by the time, and so the process itself is something that's hidden from the consumer, the end consumer. And it is something that the end consumer just gets to read the final article and whatever the findings of the article are. Now, in the case of Hasgeek, what we were trying to do was to say that because we want practitioners, people like yourselves who are actually in hands-on, day-to-day practice, how do we kind of open up this very process of giving feedback to the public? And I think there's, I mean, the learning that we've had particularly this year has been that there's the aspect which says that, oh, you know, at the end of this whole process of, you know, all the submissions, there is a conference, which is a showcase, and that's a consumption experience. But where is the participatory experience?

So, the problems that we were trying to solve is: how do you make this review public? How do you move this conversation that you have from closed spreadsheets to more public forums? So let me try to give you an example over here. Here is our bulky spreadsheet that we created for this conference we ran on SRE and systems and security last year. And you can see like this is like all of this thing, okay, this person has to comment, this person has to comment. None of these comments are necessarily showing up over here. Wrong, wrong funnel. Okay, so here you can see that, you know, here are people who've submitted their work based on the work that they've done. So they've said, okay, I would like to speak about, you know, how do you move databases from Kubernetes in Flipkart's DBaaS journey. You have some comments over here, but it's by and large extremely thin.

Similarly, what you find over here is, let's see the next one. Here's someone who's talking about how to move, I mean, from sidecars to serverless, Achal's experience of what they did at Tecton. Now, if you see, the feedback here is extremely transactional. There's no feedback. There's just a transactional conversation about slides. Similarly, here, what has happened is the person who had tried to champion the slides, the champion, the submitter here has put in some sort of comments about the slides, but there's no substantial conversation around here with respect to saying, okay, what is the problem you're trying to solve? Why is it interesting for the audience? etc., etc.

So essentially, what was happening for us until about the end of last year was that this process of saying, okay, we're going to organize a conference, but we're going to keep this review process hidden from the public. And not intentionally, but it's because, oh, we are trying to manage this. And you know, the final outcome will just be this, you know, sort of a talk that people will consume, just like you're consuming my talk over here. And so we said, you know, that's not the way to kind of build a participatory community. So how do we open up this review? And how do you turn this into a conversation where you do not come in with the assumption that, you know, the person has a well-formed idea, that you actually help them go through that journey, and that journey itself becomes public. And I'll show you a few examples as we go through this today.

And finally, the fact of, you know, if you look over here, there are about, you know, this conference last year received about 75 submissions, which is based on the work that people were doing. And, you know, typically the selection is, you know, maximum you've taken about like 20% of the submissions. Now, what do you do with the rest of the 80% submissions? Is it throwaway? Or rather, do you treat this as an asset and say that, hey, we're going to build this and, you know, we'll have conversations around this through the year, and that's how we will sort of, you know, build the community.

So these were the three problems we were trying to solve. Before I move further, I'd like to qualify that the context that I'm referring to over here is that what we have been trying to build at Hasgeek have been these sort of knowledge-based communities where people want to come together to discuss a particular problem space. So that might be platform engineering, it might be as niche as supply chain security, or it's about how do you use AI in software life cycle development. And there's a desire to learn and share from tacit knowledge, which is that this is not knowledge that is out there in the public. It is something that you have learned over the course of your own practice, which you think is insightful, and that insight is something that, you know, people want to learn more about. So that's by and large the context, and so it may not apply in this case, it may not apply to, you know, sort of a more, you know, a more generic community, but this is generally the context in which we are operating.

Now, to move quickly, the way in which this whole peer review process works is that there's a variety of actors. And what I will do in this presentation is just walk you through the role of the reviewer. And as it happens, Murphy's law. Okay, we're back. So what I will do is walk you through just one of the actors over here and happy to chat with you about the others if you have an interest afterwards. And so, among the actors that we have, we have the editor who's essentially the steward of the community. They champion the audiences. Then you have reviewers, who I'll explain sort of in the subsequent slides. And you have scouts who are basically people who kind of like, you know, get people from their network to submit ideas and who want to, and who basically, it could be a manager in an organization who wants to get, you know, their team members to speak, or it could just be, hey, I have a friend who's got an interesting idea, let me, you know, get them forward, etc. And then we have an actor called the auditors, who basically are the people who review the reviewers. And then you have an ops coordinator. But like I said, in this talk, we'll just speak about the reviewers as the role.

So what is the incentive for somebody to participate in reviewing? And please note that this is a completely voluntary activity in the sense that you're not necessarily paid for your time to sit and listen and give feedback and to write your notes about feedback, etc. So the incentives to review are, I think, you know, everybody sitting here would sort of agree that the industry in which we operate in technology is fairly siloed. It's very, it's, one operates in isolation, whether you are inside an organization or whether you're in a domain space, you generally do not know what's really happening to organizations outside unless you have friends in other companies who are sharing with you what's going on. So there is generally a need to know what's happening in a domain or in the industry, and so that becomes an incentive to review. Then there's an aspiration to say, hey, I'd like to contribute because if my contribution helps in like, you know, shaping someone else's work, in their career, in their influence, then, you know, it gives me that sort of a satisfaction that I was able to contribute. But this contribution is something that I'd like to do in a very limited, scoped role, which means that I need to know how much time I'm going to give, what is my role, and where is my boundary sort of like, you know, begin and end. And then, of course, there is always a need to say, hey, you know, I was a reviewer for someone, and you know, that goes into my personal branding later. So these are, this is typically the sort of, you know, hierarchy of needs that we have. And what we essentially look in most of our reviewers is where the, all right.

So then, how do you kind of align the incentives that people have with respect to saying that, hey, I'd like to know what's happening in the industry? So what we end up doing is create this sort of a funnel or a pool of what we call submissions where people come and sort of, you know, submit their ideas, the work that they are doing. And, you know, this is, this becomes an incentive to say, okay, you know, here I can pick and choose from like, you know, a pool of submissions that I'd like to review.

Now, what we also do over here is to induct the reviewers into their role and into rules. And this is extremely important because if you fail here, then you basically fail the entire process. And so who is a reviewer? Typically, it's a, it's a past speaker or past presenter who has received feedback and who knows how to give feedback. Or it's somebody who's an ecosystem contributor who's kind of been there, been around, and has a strong need to contribute back. It's also an individual who elevates and pushes the bar for discussions and for sharing knowledge. So if you remember the context that I shared over here, which is to say that we're trying to build communities of practice where people want to share tacit knowledge, the idea is that the reviewer is somebody who wants to kind of push the bar and say, hey, I'm not interested in just learning about like what tools you use, but I want to know like what were the outcomes over here? Where did you fail? Where did you succeed? So essentially, you know, kind of get to more honest discussions and just posturing. And then, you know, a reviewer is somebody who strives to create high-quality content and experience for the audience. And this sometimes even contradicts, you know, the editor's vision or roadmap, but that's a healthy tension to have.

And then there is, of course, the rules. What is it that you're reviewing and how do you actually go about reviewing? And here, you know, we sort of are very anal about following some of these rules, which is depth is far more important than breadth. You can always like, you know, sort of skirt over a lot of, you know, a broad area, but what matters is how deep do you go into this and how nuanced is your, is the experience and is the argument that you're making. Then details versus claims. So, if somebody says, hey, you know, I saw this XYZ outcomes, where are the metrics? What are you, you know, what is the proof that substantiates that, you know, this was a success in, you know, your sort of understanding of whatever success is? So essentially, you know, sort of push the boundary to making it more and more honest and not just posturing in the sense of like, of making claims that are not substantiated. And similarly, you know, the metrics and data points matter. So if you say that, you know, there is a success in the organization, then where are the data points to show what constitutes a success? What outcomes did your customers experience, etc.? And then among the other rules that we have is that, oh, I'm just going back and forth on my slide. Okay, and then among the other rules that are there is, what is relevant is that for the reviewer is to essentially help a submitter to kind of tease out the very ideas and the very sort of insights that would otherwise not be so obvious to others. So the idea is that if you are sort of, you know, in the consumption experience, you have something that you take away at the end of listening to a presentation or a session. And this is not just something that's shallow enough over here. So the rule is that you have to really help sharpen the takeaway. And if you were sitting in the audience, you know, what is it that you would take in as an insight? And then, of course, we have to manage conflicts of interest. Let's say a submitter is from the same organization as a reviewer, in which case then we call out the conflict of interest and replace the reviewer with someone else. So, the, these are sort of by and large the rules, and this is something that we follow very strongly now in our induction process with reviewers, in a way to just ensure that, you know, the points of failure are sort of fewer.

Does this mean that this process works foolproof? Well, you know, in some cases, you know, there's always challenges. And so I've mentioned failure here, but I mean, this could, you know, very well just be challenges. You have people who sort of are enthusiastic at the start but then who become uninvolved for various reasons. And, you know, then it becomes a communication overhead because what you're managing is 15-20 people at a time. If the onboarding is not right, then it leads to hiccups where if you do not specify the rules clearly, then you find that, you know, reviewers can contradict each other. And then I think the most important thing that we've seen is that, you know, while there is enthusiasm to say, "Hey, I'm going to do review," you know, when it actually comes to it and saying, "Hey, you know, you've got to post your comments in public." So for instance, you know, here is a submission that someone's made on building production-grade LLMs, blah, blah, blah with DSPy. And you can see that here, you know, the feedback is all out there, which is open for you also to see, and it's not just on my computer, which is, you know, sort of asking a lot of, you know, details and a lot of questions about saying, okay, you know, why are you doing this and, you know, why is it, you know, you know, how is it that it will help the audience, etc.

So going back to, you know, the points of failure is that many a times people are enthusiastic to say, "Yes, I'm going to participate to review," and then it's like cold feet saying, "Oh shit, my comments are going to be public. It's not going to be private." And then also the fact that, you know, you have to ask for details in public. You have to kind of give your feedback in public. So that, you know, if you didn't expect this transparency, then you kind of start out and totter in the beginning. And then you've also seen in our experience that, you know, when it comes to organizing a conference as large as IndiaFOSS, if you haven't sort of like, you know, quantified the number of reviewers you have, you find that a lesser number of people are doing more work, in which case then the overheads increase. And because, you know, the context in our case is sort of building these practice and, you know, knowledge-based communities, if the expertise is mismatched, then you have a subpar review for the submitter. And then, of course, you know, there's the gender dynamic that is always a challenge, which is that when you're building these communities of practice, particularly in the area of technologies, technology, you have few women reviewers. And so the gender dynamic sort of gets skewed more towards having male reviewers than having and fewer female reviewers.

So I'm going to try and conclude over here in the interest of time, which is that our overall learnings have been that it's really important to be able to onboard your reviewers both in terms of their role and the rules very clearly at the outset, and the expectations that everything you're going to kind of discuss is going to be in public. We've had reviewers who've, you know, sort of, is that the warning bell? Yeah, it is the warning bell. Okay. So we've had reviewers who've, you know, sort of said, "Oh, can I resort to ChatGPT to write out my comments?" And we're like, no, you speak in your own way. So if you've sort of read the Cluetrain Manifesto, what matters is your own authentic voice. So there is this sort of, you know, pushing happening saying that, hey, you know, use your own authentic voice. Role clarity is important, and you also need to scope and bound the role because your reviewer is coming in as a volunteer. If you're asking them to do too many things, then, you know, that's where collapsing happens, and it leads to overload, and then, you know, there can be points of hiccups and tottering and, you know, lack of success.

And finally, success is really proportional to the investment of all the actors. In this case, you know, the reviewers play a very important role in terms of not just the quality of content that comes forward over here, but also the quality of interactions that sort of leads to that, and then also the success of the presenters out there. And if, you know, the metrics and, you know, tracking these outcomes of success are not very clear, then the reviewers also do not know in terms of, you know, what are they really contributing towards. So the end learning that you know, we've had in this process is, it's not the good feelings that matter... because usually people say, "Oh, you know, as a community, it's very nice, we've all come together and, you know, it's a very nice feeling." But what matters is the outcomes. And it's important to kind of quantify and give metrics to the outcomes if you would like to have your reviewers and the review process to be invested. So yeah, with that, thank you very much, and I'm done.