Autograding in pluto notebooks

[mkschleg]

We began discussing this on the Julia slack channel. But @fonsp thought it would be a good idea to move to here!

Original Question:

Has anyone setup any kind of auto-grading system for Pluto notebooks? We are thinking about using them for University of Alberta’s ML undergraduate course. One thing we’ve though of is just having them export to html or something, but having some type of auto-grader would be nice.

Others replied. But I'm not comfortable copy and pasting their responses to a public forum from a semi-private forum.

[fonsp]

I worked on this for the computational thinking course, implemented inside pluto notebooks. The code is outdated, and will not run with latest Pluto, but you can still read it to get an idea of what I was going for:

https://github.com/fonsp/disorganised-mess/tree/main/autograding

The idea is to run all notebooks and run test statements inside them that either return true/false (correct/incorrect) or a value (extract student.name). Results are stored in a DataFrame, and you can view student notebooks in a little window inside the grader notebook to review them.

[fonsp]

We never persued it much, because we only had ~15 notebooks to grade, so it was more efficient to do manually. We also wanted to grade code style, so manual review was needed.

[fonsp]

If anyone is interested in picking this up, I would be happy to call and go over the current code together!

[mkschleg]

I will take a look and start from this! Thank you so much for your help! I would like to pick this up and see if I can't make some type of standard after I further develop things for the course I'm working on. But I have to manage research, teaching, and thesis work, which means much less time working on open source projects than I would like unfortunately.

[mkschleg]

Maybe we can sync when you are done with your vacation!

[fonsp]

For completeness, I will also mention the idea of live checking student code. We used this for almost every exercise in https://computationalthinking.mit.edu/Spring21/homework/, and it gives a fantastic experience for students.

The idea is that every exercise is structured like:

md"""
# Exercise 1.1
👉 Implement `hello` that returns two times the argument `x`.
"""

function hello(x::Number)
	
	return missing
end

hello(4)

let
	result = hello(7)
	if !(result isa Number)
		md"Make sure that you return a number!"
	elseif result == 14
		md"Well done!"
	else
		md"Keep working on it!"
	end
end

and the last cell would be folded. Because of reactivity, the last two cells re-run whenever the student updates their function, giving them immediate feedback. I discuss this in more length here: https://www.youtube.com/watch?v=QBkXmTen6P4 and the upcoming JuliaCon 2021 talk about https://computationalthinking.mit.edu/

[DhruvaSambrani]

Given that a pluto notebook is basically a julia file, isn't running tests good enough to solve the problem? Say the requirement is to define a function say_hi(name) to return "hello $(name)", then includeing the submission file and running @assert say_hi("John Doe") == "hello John Doe" as a test will tell you if the solution is correct. Making a whole Test Suite like this will allow you to see which tests failed and grade accordingly. Or, simply catching the errors and decreasing the score accordingly also will work. Obviously this is unable to grade code style (which is subjective anyway) and give partial marks(this can be partially solved by checking if some tests failed, and common mistakes by adding extra assertions).

[fonsp]

The problem is that a Pluto notebook will run if one of the cells errors, or contains incomplete syntax, but the include will fail. This would unexpectedly give 0 points to students that make an error.

[DhruvaSambrani]

Right. Didn't think of that.

Is it possible to programmatically add a cell to a running Pluto notebook? If so, the test conditions could be added as an added cell(s) and the same process be followed. Cells unaffected by syntax errors will anyway run because of Pluto's awesome cell dependency set up.

[DhruvaSambrani]

Oh I just noticed that that's what your previous solution was

[mkschleg]

For those wondering here. I've taken what @fonsp has done and updated it a bit. I've put up A1_grader here https://github.com/mkschleg/PlutoGrader, if anyone wants to use it as a template. It is self contained because ofPluto v0.15.x. I can send a version of the assignment if you want, I just want to keep it off git because it will be for a course that will be taught this fall.

[ctessum]

Here's what I've come up with so far: https://github.com/ctessum/Grader.jl

It currently integrates with https://www.prairielearn.org/ but with a different output format could work with any other system.