Lessons from Social Code Analysis

Pythonic Grading that Rewards Sharing

PyConSG Education Summit · 11 Aug 2025 · SUTD, Singapore
Anand S · LLM Psychologist · Straive
Video · Blog · Slides · Transcript

I asked ~1,000 students to automate analysis

Project 2 Tools in Data Science, BS Data Science, IIT Madras

1. Write a Python script that uses an LLM to analyze, visualize, and narrate a story from a dataset.
2. Convince an LLM that your script and output are of high quality.

Running uv run autolysis.py DATA.csv must create a README.md with automated analysis as a story + data visualizations.

Evaluation & feedback were LLM-automated

Rubric 17/27: Code sends only carefully chosen analysis to the LLM?

• The code focuses on selective data analysis required to prepare a report but doesn't have a filtering mechanism to only present certain metrics to the AI.
• The code only processes relevant analyses by summarizing key findings (missing values, statistics, etc.) before sending it to the LLM, thereby avoiding unnecessary data.

Copying was allowed, but originality scores

8 bonus marks for code diversity.

You're welcome to copy code and learn from each other. But we encourage diversity too.

We evaluate code similarity. Unique responses get bonus marks.

If your response is similar to priors, you don't get these marks.

I evaluated similarity based on overlap

• Removed comments and docstrings. (These are easily changed to make the code look different.)
• Get all 5-word phrases in the program. (A “word” is a token from tokenize. A “phrase” is a 5-token tuple. Chosen by trial and error.)
• Calculate % overlap with previous submissions. (Jaccard Index via datasketch.MinHash.)

Evaluation code: similarity.py

Some copied exactly from each other

Rule 1: Copy late to pick best submissions

• The original was on 11 Dec afternoon. Scored 7 marks.
  • The first copy was on 12 Dec evening.
  • The second copy as on 14 Dec night.
  • Then 29 others streamed in, just before the deadline
• Another original was on 12 Dec late night. Scored 10 marks
  • The first copy on 14 Dec afternoon.
  • Several were within a few hours of the deadline.

Who you copy from matters! Copy late for more options.

Some made good changes to code

Original | Copy: used API keys from the environment.

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "pandas",
#   "seaborn",
#   "matplotlib",
#   ...
#   "scikit-learn",

# ]
# ///

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "pandas",
#   "seaborn",
#   "matplotlib",
#   ...
#   "scikit-learn",
#   "python-dotenv"
# ]
# ///

Rule 2: Change code only if you're sure

Original | Copy: hard-coded the API key.

import os
import sys

...
from sklearn.cluster import KMeans
from sklearn.impute import SimpleImputer

AIPROXY_TOKEN = os.getenv("AIPROXY_TOKEN")

import os
import sys

...
from sklearn.cluster import KMeans
from sklearn.impute import SimpleImputer

AIPROXY_TOKEN = os.getenv("eyJhbGciOi...")

Don’t change their code unless you KNOW what you’re doing.
Spend more time testing than changing.

Half of submissions were standalone

Code with <50% Jaccard similarity (like below) are standalone.

f"You are a data analyst.
Given the following dataset information,
provide an analysis plan and suggest
useful techniques:\n\n"
f"Columns: {list(df.columns)}\n"
f"Data Types: {df.dtypes.to_dict()}\n"

f"You are a data analyst.
Provide a detailed narrative based on
the following data analysis results
for the file '{file_path.name}':\n\n"
f"Column Names : {list(df.keys())}\n\n"
f"Summary Stats: {analysis['summary']}"

About 50% of the submissions were standalone!
They didn't copy despite encouragement.

Rule #3: Let others copy from you

Strategy	% of submissions	Average score
Standalone	50%	6.23
Be the first to copy	12%	6.75
Copy late	28%	6.84
Original – let others copy	11%	7.06

Students who let others copy from them got feedback and improved.
They scored the most.

Lessons for Students

1. Copy late to pick from the best submissions
2. Double check before you change code
3. Let others copy from you. Improve from their feedback

Lessons for Educators

• Analyze code similarity! You can discover:
  1. Which students are isolated? Who're the future teachers / TAs?
  2. When do they learn? Who do they learn from?
  3. What concepts have they understood well enough to tweak?
  4. Which code segments have the most diversity? Least?
• Scale helps! More students ⇒ more data, more experiments

Learn from Social Code Analysis

Pythonic Grading that Rewards Sharing

PyConSG Education Summit · 11 Aug 2025 · SUTD, Singapore
Anand S · LLM Psychologist · Straive
Blog · Slides · Transcript