Processing 2.5k Marksheets: Lessons in Thinking of Scale

I just find it funny how no matter what you do, someone always finds a way to do the unexpected.

I was recently working on a bigger project where I had to process marks from around 2.5k marksheets. Each student had two parts, so that’s around 1.2k students. The only way to know whose sheet it was: the student number in the filename. You can already imagine what kind of nightmare awaits.

Finding the naming patterns

As expected, files were saved in a couple of different ways, with one or two typos here and there. This wasn’t too bad. A few people did the marking, so the naming standard, even if different, was consistent within each batch. Once I had each batch’s pattern, I could process a good chunk of the marksheets.

Then it got more interesting

Everyone had a rubric to use, but people also had their own ideas 😂

• Some marksheets had a changed structure. Maybe someone preferred their own layout.
• Some had sheets deleted, so I couldn’t tell which sheet to pick.
• Some had sheets renamed, so I couldn’t pick by name.
• Some files were corrupt.

What I took from it

For one, you can’t always account for everyone functioning the same way. I notice I sometimes do things my own way too, and I’m sure the person processing it isn’t always happy 😂

Another: no matter what tests you do, it’s interesting to see what happens with 1000+ items. This was my new record, with 2.5k files and the extra challenge. Because it was so large, I still checked every error spotted to make sure the process worked smoothly. It was a good confidence booster.

Think of scale

With huge numbers, you begin to feel the difference in time. As I worked through the errors, waiting for the processing became annoying. It’s a lesson you only learn when time actually starts to matter.

My tool processed every sheet, then created the result. The ideal way is to process each sheet and capture its result straight away. Then, when a sheet causes an error for whatever reason, you don’t redo everything. You build on the previous results.

So when they tell you to think of scale, it really matters in the big picture.

How has your experience been with working with huge data? What tricks do you use?