Noteable - When Submissions go wrong

Tips on writing an assignment

This guide to writing assignments is intended to give you an overview of how to effectively use the assignment system in Noteable. While we assume you're familiar with creating assignments, we’ve compiled some tips to enhance your experience.

As with any task, a good plan is essential. Remember the 5 Ps: "Poor Planning Prevents Proper Performance." Here’s a step-by-step guide to help you get started:

A quick overview of how the assignment system works

To help use the assignment grading system successfully, it's useful to know how it works... this chart may help: Graphic showing the flow of Assignment files, from creation to receiving feedback.

Create a Complete Assignment: Start with a fully functional assignment, including solutions and all necessary components
Prepare the Student Version: Remove solutions and specific markups to create a version suitable for student use
Distribute to Students: Share the release notebook with students, who will then complete and submit it for grading
Grade Submissions: Grade the student submissions, adding any feedback as needed
Release Feedback: Once grading is complete, release feedback for students to review.

Cell Types

Notebooks in Noteable consist of two primary cell types:

Code Cells: For executing code
Markdown Cells: For text content, including instructions and explanations

All cells in a notebook can be edited (though markdown cells are generally shown as rendered unless the cell is clicked on to edit it)

Assignment-Specific Cell Types

In addition to standard cells, the nbgrader tool introduces specialized cells:

cell type	sub-options	Description
markdown	Manually graded task, Manually graded answer, and Read-only	Used for instructions, explanations, or any non-code content.
code	Manually graded task, Autograded answer, Autograded tests, Manually graded answer, and Read-only	Contains code to be executed or tested, either by students or automatically.

Autograded answer do not have points. This is where the student enters code - either as a function or to set variables, which are then used by...
Autograded tests cells have points. If the cell runs with no error, then the student gets the points.
Manually graded task cells have points. These cells are not editable by students. They are generally used to give instructions for students to then produce new content below.
Manually graded answer cells have points. The marker has to manually view the contents of the cell, and choose the number of points to award (up to the number given)
Read-only cells are just that: they cannot be edited.

What's in an assignment

An assignment should contain all the necessary materials for students to complete their tasks, including:

A notebook document (the .ipynb file): The primary document containing instructions, tasks, and cells for code execution.
Data Files: If applicable, include any required data files. Remember, large datasets may be better shared via a network link.

The constituent parts of an assignment.

There are two grading options in nbgrader: Auto-Grading and Manual Grading.

The former is going to cut down your workload, however definitely limits what you can test. The latter allows you to mark graphs & images, as well as essay-like answers.

Auto-grading

Auto-grading generally requires two components:

Some code the students writes (`autograde answers`), and
Some tests that exercise that code & validate the outputs (`autograde tests`)

For example,

def squares(n):

      """Compute the squares of numbers from 1 to n, such that the

      ith element of the returned list equals i^2.

      """

      ### your code here

is a valid piece of work to ask a student to fill in.

likewise,

"""Check that squares returns the correct output for several inputs"""

    assert squares(1) == [1]

    assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

is a valid set of tests for that function.

Manual grading

Manual grading also uses two components:

A task, which tells students what to do (`manual grade tasks`), and
Free-form answers (`manual grade tests`)

For example,

Calculating the circumference of a circle

    The Ramanujan formula for determining the circumference of an ellipse is:

    C ~~ pi(3a + 3b - sqrt[(a+3b)(b+3a)])

    In the cell below, write some python code to print the circumference when a=2 & b=3

is a valid description of work to ask a student to create... abd below it would appear an empty cell (for this example, a code cell).

Creating the source document

To prepare an assignment, create a source document that includes:

Sample Solutions: Ensure the document runs without errors.
Hidden Tests: These are removed in the student version but reinserted during grading.

Autograde Answers

These are the cells where students fill in code, code which is automatically tested elsewhere. Unlike normal cells, these cells cannot be deleted (accidentally or otherwise.)

def squares(n):

      """Compute the squares of numbers from 1 to n, such that the

      ith element of the returned list equals i^2.

      """

      ### BEGIN SOLUTION

      data = []

      for i in range(n):

        i += 1

        data.append(i*i)

      return data

      ### END SOLUTION

The "### BEGIN SOLUTION" and "### END SOLUTION" are the magic that identify the code to remove when creating the student version.

The student version of the above will be:

def squares(n):

      """Compute the squares of numbers from 1 to n, such that the

      ith element of the returned list equals i^2.

      """

      ### your code here

      raise NotImplementedError()

Autograde Tests

Autograde tests are code tests that run as part of the notebook... and if they pass, the student gets awarded the marks defined for that cell. Unlike normal cells, these cells can be neither deleted or edited (accidentally or otherwise.) As mentioned above, you can include hidden tests:

"""Check that squares returns the correct output for several inputs"""

    assert squares(1) == [1]

    assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

    ### BEGIN HIDDEN TESTS

    assert squares(2) == [1, 4]

    assert squares(11) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121]

    ### END HIDDEN TESTS

The "### BEGIN HIDDEN TESTS" and "### END HIDDEN TESTS" are the magic that identify the code to remove when creating the student version.... but the tests are re-inserted before autograding.

The student version of the above will be:

"""Check that squares returns the correct output for several inputs"""

    assert squares(1) == [1]

    assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Pro Tip: Do not create a cell which is all hidden tests - that creates an empty, uneditable, undeletable cell in the student notebook.

Autograde tests have an interesting feature: if the output of the cell is an number, then the student is awarded that number of points (up to the point-award defined for the cell.) For example:

"""Check that squares returns the correct output"""
assert squares(1) == [1]
assert squares(10) == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
len(squares(3))

.... will have an output of 3... so award 3 points for the test (or the maximum value defined, if it's lower) if all the assert's pass as well.

Manual Grade Tasks

These are Markdown cells where students are told what to do, where they write a free-form answer elsewhere. Unlike normal cells, these cells can be neither deleted or edited (accidentally or otherwise.)

A handy feature of Manual Grade Task cells is you can embed a "Mark Scheme" in the source, which is removed for the student notebook... and re-inserted for grading:

Calculating the circumference of a circle

    The Ramanujan formula for determining the circumference of an ellipse is:

    C ~~ pi(3a + 3b - sqrt[(a+3b)(b+3a)])

    In the cell below, write some python code to print the circumference when a=2 & b=3

    === BEGIN MARK SCHEME ===

    from math import sqrt

    a = 2

    b = 3

    part_b = sqrt((a+(3*b)*(b+(3*a))))

    c= 3.1415 * ((3*a) + (3*b) - part_b)

    print(f"Circumference = {c}")

    Circumference = 18.502072911118184 for pi=3.1415

    === END MARK SCHEME ===

The "=== BEGIN MARK SCHEME ===" and "=== END MARK SCHEME ===" are the magic that identify the text to remove when creating the student version.... and re-inserted before grading.

The student version of the above will be:

Calculating the circumference of a circle

    The Ramanujan formula for determining the circumference of an ellipse is:

    C ~~ pi(3a + 3b - sqrt[(a+3b)(b+3a)])

    In the cell below, write some python code to print the circumference when a=2 & b=3

Manual Grade Answers

These can be either Markdown or Code cells where students enter a free-form answer. Unlike normal cells, these cells cannot be deleted (accidentally or otherwise.)

Helpful hints

Text formatting

When it comes to presentation (of text comment), then you have quite a few options available to you to get creative.

The first & foremost is that your text is created using markdown - so all the goodness of headings, lists, and tables are available

Notebooks have the bootstrap Style Sheets built in, so colour and so forth become available:

<div class="alert alert-success">

    ## Instructions to students

    Do not change the name of this exam file.

    Before starting ....

    <code>

    ### your code here

    raise NotImplementedError()

    <code>

Will become:

Instructions to students

Do not change the name of this exam file.

Before starting ....

<code>
### your code here
raise NotImplementedError()
<code>

What to include in the assignment

It's always an interesting juggle as to what to include in the assignment:

You are required to include at least 1 .ipynb file
You can include more that one notebook. however more than one can make [manual] marking a challenge
If you include data, remember that you're going to get a copy of that data back.... from every single student. That can take a lot of disk space! Consider making the data available over the net - either as a git download, or use something like pandas to load remote data (ie: df = pandas.read_excel("http://www.example.com/data/something/spreadsheet.xlsx"))