Noteable - When Submissions go wrong

Tips on writing an assignment

Obviously we can't teach you how to make an assignment - as tutors, you already know how to create an assignment - however we can offer some suggestions to make creation of assignments in noteable more fulfilling.

The first task is to have a plan. Just like any other test / assignment / homework task - the 5 Ps: "Poor Planning Prevents Proper Performance"

A quick overview of how the assignment system works

To help use the assignment grading system successfuly, it's useful to know how it works... this chart may help: Graphic showing the flow of Assignment files, from creation to receiving feedback.

  1. You create a complete & working assignment (with solutions for all tests, etc)
  2. You make a version for students (which takes out said solutions and other markup)
  3. The release notebook is a normal notebook. Whilst it has some metadata in some cells to know which ones need magic applied, the students have no way to find out what that magic is
  4. You distribute that release version to students, who
  5. ... work on it, then submit it back for marking
  6. ... (The release notebook is a normal notebook. It does not need nbgrader to be worked on. The fetch and submit processes do, but the students can work on the notebook somewhere else if they wish)
  7. You then grade the submitted work (which puts back in hidden tests etc), possibly adding feedback - in which case
  8. ... you release feedback for students to collect

Cell Types

Notebooks basically have 2 cell types: code and markdown. They do what they say on the tin. All cells in a notebook can be edited (though markdown cells are generally shown as rendered unless the cell is clicked on to edit it)

Assignment cell types

nbgrader (the tool that manages assignments) additionally sub-divides cells:

cell typesub-options
markdown Manually graded task, Manually graded answer, and Read-only
codeAutograded answer, Autograded tests, Manually graded answer, and Read-only
  • Autograded answer do not have points. This is where the student enters code - either as a function or to set variables, which are then used by...
  • Autograded tests cells have points. If the cell runs with no error, then the student gets the points.
  • Manually graded task cells have points. These cells are not editable by students. They are generally used to give instructions for students to then produce new content below.
  • Manually graded answer cells have points. The marker has to manually view the contents of the cell, and choose the number of points to award (up to the number given)
  • Read-only cells are just that: they cannot be edited.

What's in an assignment

An assignment should include everything a student needs to be able to complete the work.

That definitely requires a notebood document (the .ipynb file), and may (or may not) include data files.

The constituent parts of an assignment.

Generally speaking, assignments include some form of grading, to test a student has understood what has been taught. HOW that teaching happens is a different question - that could be the primary purpose of the assignment (and the grading just a minor part at the end), or it could be a complete test to complement other teaching elsewhere. Your Mileage, as they say, May Vary

You have two choices, when it comes to grading: auto and/or manual.

Obviously the former is going to cut down your workload, however definitely limits what you can test. The latter allows you to mark graphs & images, as well as essay-like answers.

Auto-grading

Auto-grading generally requires two components:

  1. Some code the students writes (`autograde answers`), and
  2. Some tests that exercise that code & validate the outputs (`autograde tests`)

For example,

def squares(n):
  """Compute the squares of numbers from 1 to n, such that the
  ith element of the returned list equals i^2.
  """
   ### your code here

is a valid piece of work to ask a student to fill in.

likewise,

"""Check that squares returns the correct output for several inputs"""
assert squares(1) == [1]
assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

is a valid set of tests for that function.

Manual grading

Manual grading also uses two components:

  1. A task, which tells students what to do (`manual grade tasks`), and
  2. Free-form answers (`manual grade tests`)

For example,

Calculating the circumference of a circle

The Ramanujan formula for determening the cirfumference of an elipse is:

C ~~ pi(3a + 3b - sqrt[(a+3b)(b+3a)])

In the cell below, write some python code to print the circumference when a=2 & b=3

is a valid description of work to ask a student to create... abd below it would appear an empty cell (for this example, a code cell).

Creating the source document

As mentioned above, you want to create a source document which will run to completion, with no errors, and thus needs to have sample solutions in it. You may also want to add some hidden tests... just to stop students creating monkey solutions (ones that pass the tests, rather than answer the question.)

Obviously, we don't want solutions and hiddent tests shown to students... so this is how you create them:

Autograde Answers

These are the cells where students fill in code, code which is automatically tested elsewhere. Unlike normal cells, these cells cannot be deleted (accidentally or otherwise.)

def squares(n):
  """Compute the squares of numbers from 1 to n, such that the
  ith element of the returned list equals i^2.
  """
  ### BEGIN SOLUTION
  data = []
  for i in range(n):
    i += 1
    data.append(i*i)
  return data
  ### END SOLUTION

The "### BEGIN SOLUTION" and "### END SOLUTION" are the magic that identify the code to remove when creating the student version.

The student version of the above will be:

def squares(n):
  """Compute the squares of numbers from 1 to n, such that the
  ith element of the returned list equals i^2.
  """
  ### your code here
  raise NotImplementedError()

Autograde Tests

Autograde tests are code tests that run as part of the notebook... and if they pass, the student gets awarded the marks defined for that cell. Unlike normal cells, these cells can be neither deleted or edited (accidentally or otherwise.) As mentioned above, you can include hidden tests:

"""Check that squares returns the correct output for several inputs"""
assert squares(1) == [1]
assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
### BEGIN HIDDEN TESTS
assert squares(2) == [1, 4]
assert squares(11) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121]
### END HIDDEN TESTS

The "### BEGIN HIDDEN TESTS" and "### END HIDDEN TESTS" are the magic that identify the code to remove when creating the student version.... but the tests are re-inserted before autograding.

The student version of the above will be:

"""Check that squares returns the correct output for several inputs"""
assert squares(1) == [1]
assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Pro Tip: Do not create a cell which is all hidden tests - that creates an enmpty, uneditable, undeleteable cell in the student notebook.

Autograde tests have an interesting feature: if the output of the cell is an number, then the student is awarded that number of points (up to the point-award defined for the cell.) For example:

"""Check that squares returns the correct output"""
assert squares(1) == [1]
assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
len(squares(3))

.... will have an output of 3... so award 3 points for the test (or the maximum value defined, if it's lower) if all the assert's pass as well.

Manual Grade Tasks

These are Markdown cells where students are told what to do, where they write a free-form answer elsewhere. Unlike normal cells, these cells can be neither deleted or edited (accidentally or otherwise.)

A handy feature of Manual Grade Task cells is you can embed a "Mark Scheme" in the source, which is removed for the student notebook... and re-inserted for grading:

Calculating the circumference of a circle

The Ramanujan formula for determening the cirfumference of an elipse is:

C ~~ pi(3a + 3b - sqrt[(a+3b)(b+3a)])

In the cell below, write some python code to print the circumference when a=2 & b=3

=== BEGIN MARK SCHEME ===

from math import sqrt
a = 2
b = 3
part_b = sqrt((a+(3*b)*(b+(3*a))))
c= 3.1415 * ((3*a) + (3*b) - part_b)
print(f"Circumference = {c}")
Circumference = 18.502072911118184 for pi=3.1415

=== END MARK SCHEME ===

The "=== BEGIN MARK SCHEME ===" and "=== END MARK SCHEME ===" are the magic that identify the text to remove when creating the student version.... and re-inserted before grading.

The student version of the above will be:

Calculating the circumference of a circle

The Ramanujan formula for determening the cirfumference of an elipse is:

C ~~ pi(3a + 3b - sqrt[(a+3b)(b+3a)])

In the cell below, write some python code to print the circumference when a=2 & b=3

Manual Grade Answers

These can be either Markdown or Code cells where students enter a free-form answer. Unlike normal cells, these cells cannot be deleted (accidentally or otherwise.)

Helpful hints

Text formatting

When it comes to presentation (of text comment), then you have quite a few options available to you to get creative.

The first & formost is that your text is created using markdown - so all the goodness of headings, lists, and tables are available

Notebooks have the bootstrap Style Sheets built in, so colour and so forth become available:

<div class="alert alert-success">
## Instructions to students

Do not change the name of this exam file.

Before starting ....

<code>
### your code here
raise NotImplementedError()
<code>

Will become:

Instructions to students

Do not change the name of this exam file.

Before starting ....

<code>
### your code here
raise NotImplementedError()
<code>

What to include in the assignment

It's always an interesting juggle as to what to include in the assignment:

  • You are required to include at least 1 .ipynb file
  • You can include more that one notebook. however more than one can make [manual] marking a challenge
  • If you include data, remember that you're going to get a copy of that data back.... from every single student. That can take a lot of disk space! Consider making the data available over the net - either as a git download, or use something like pandas to load remote data (ie: df = pandas.read_excel("http://www.example.com/data/something/spreadsheet.xlsx"))