Noteable - nbgrader Guide

The Formgrader menu is the instructor's hub for all assignment related activity on Noteable.

When logged into Noteable as an instructor, click on the Formgrader tab to be taken to the Formgrader menu:

On the left hand side of the screen you will notice 3 tabs:

• Manage Assignments - The main Formgrader menu, showing assignment status and providing access to assignment tools.
• Manual Grading - Used to manage manual marking and feedback.
• Manage Students - Display information about students and allows the instructor to add additional information.

In brief, the core functionality of the main Formgrader menu is as follows:

• Name - Displays the name of assignment folders, and links to instructor versions of assignments ("source" folder).
• Due Date - Displays the due date for assignments.
• Status - Display the status of assignments (eg: "draft", "released").
• Edit - Allows the instructor to edit metadata for an assignment by clicking the pencil icon.
• Generate - Generates the student version of an assignment, stored in the release folder. Special syntax (see below) and cell toolbars will not be visible in this version.
• Preview - Links to student version of an assignment.
• Release - Makes the student version of an assignment available to students - students can then fetch, complete and submit assignment from Noteable.
• Collect - Loads all student submissions for an assignment.
• # Submissions - Displays the number of collected student submissions to date.
• Generate Feedback - Once marking is complete, clicking the cap icon generates feedback files for students.
• Release Feedback - Makes feedback files for an assignment available to students.

Multiple instructors on the same course can access Formgrader for assignment related activity on Noteable. Find more detail on the multiple marker documentation page.

More detail is available for each step in the Create Assignment section below. Generally speaking, the workflow is as follows:

Click onto the Formgrader tab on Noteable. From the Formgrader menu:

1. Create assignment - give a name and optionally due date.
2. Click into assignment folder and add content (question notebooks, data required).
3. Add cell toolbars to notebooks (View>Cell Toolbar>Create Assignment).
4. Specify cell type for relevant cells. Assign marks and IDs to cells.
5. Write tests for automarked code cell (similar to JUnit tests), make final edits to the assignment notebook(s).
6. Generate the student version from Formgrader and preview, Optionally edit assignment files and repeat.
7. Release the assignment to students from Formgrader.
8. Students fetch, complete, submit the assignment (multiple submissions allowed, most recent collection used).
9. Collect student submissions.
10. Automark submissions.
11. Manually mark and add feedback to submissions.
12. Generate and release feedback to students.

Here is a summary of cell types available within cell toolbars:

Autograded answer cells (code cells only):

• Use: cell in which student can answer a question.
• Marking: Marks assigned autograded test cells.
• Syntax: This type of cell may include solutions which are hidden from the student by use of special syntax (see below).

Autograded test cells (code cells only):

• Use: pass/fail tests (usually assert statements) that test student code.
• Marking: Marks are automatically assigned on an all or nothing basis, depending on whether all tests in the cell pass.
• Syntax: This type of cell may include some tests that are hidden from students (see below).

Manually graded task cells (code and markdown cells):

• Use: Tasks that may require an unknown number of cells or do not require any cells.
• Marking: Grade for task is input by marker at manual grading stage.
• Syntax: This type of cell may include a mark scheme, hidden from students (see below).

Manually graded answer cells (code and markdown cells):

• Use: Space for student answer. Usually used to follow tasks requiring a non-code response.
• Marking: Grade for task is input by marker at manual grading stage.
• Syntax: This cell type does not have special syntax.

Read-only cells (code and markdown cells):

• Use: Any cells that should be locked from student edits.
• Marking: This cell does not correspond to a task/response.
• Syntax: This cell type does not have special syntax.

Automarking syntax is included in the teacher version of the assignment (found in the "source" folder). Once cell toolbars have been added, the following syntax is available to hide information from students and assist marking:

Manually graded task cells:

=== BEGIN MARK SCHEME ===

Text between the markers is invisible in student version, but visible to instructor in grading and feedback stages.

=== END MARK SCHEME ===

Autograder answer cells (code cells only):

### BEGIN SOLUTION

Specify solution inline. In student version, text between markers will be replaced with a code stub that raises notimplemented error.

### END SOLUTION

Autograder test cells (code cells only):

### BEGIN HIDDEN TESTS

Code between these markers will be hidden in the student version, leaving blank space. Hidden tests are marked by nbgrader during the autograding stage to assign marks to autograded code questions.

### END HIDDEN TESTS

(Note: These code-markers work because they are python syntax. For this reason, the grader system generally does not work well with other languages)

There is no seperate "student notebook" that needs to be maintained by the instructor. The appearance of the student version is controlled by the syntax (above) within the teacher version. This means that there is no need to edit any notebook other than the instructor version in the "source" folder.

Once all steps in the assignment workflow are complete, the following folders and files will be present in the instructors home directory:

• gradebook.db - database file containing information on student submissions and grades. Never edit or delete this file.
• source - instructor folder containing the instructor version of the assignment. Files in this folder can be edited.
• release - student release version, created in the generate step from Formgrader.
• collected - student submissions, organised by student_id then assignment_id.
• autograded - student submissions after autograding has taken place.
• feedback - contains html feedback files for each student.

There are several examples and templates to get you started in Noteable.

If you or your students are new to using notebooks, there is a quick start guide to interacting with Jupyter notebooks.

For transferring assignments to notebook format, please refer to the assignment skeleton which you can upload from the git repository and copy/paste questions.

To view an example assignment in notebook format, please refer to the assignment example.

If you are new to writing notebooks, you may be unfamiliar with the markdown used. Noteable provides a cheat sheet for this purpose.

Additionally, Noteable provides a range of quick start tutorials which cover the following:

• Using the bokeh library for interactive graphs
• How to create bar charts in notebooks
• How to create scatter graphs in notebooks
• How to create interactive elements within notebooks (such as slider bars)

In addition to the the Creating assignments notebook, you can find some addiional tips on writing an assignments.

Find our guidance on auto-grading coding assignments with Noteable and troubleshooting advice in this webpage.

Once clicked onto the formgrader tab, Click “Add new assignment…” at the bottom of the table:

A pop-up will appear allowing you to input the name of the assignment, and optionally the due date and timezone. This information is later referred to as the “metadata” of the assignment. The name you give here will be the name of the assignment folder on noteable. This folder will be released to students, and stores all the assignment files. When naming assignments, avoid space and “/” characters as this can confuse the system later on.

Clicking Save will add the assignment to Formgrader, and create the teacher version of the assignment (stored in the "source" folder).

The assignment will appear in the Formgrader table, and the status will be set to "draft":

Clicking the assignment name from this page will take you to the "source" assignment folder:

This is the teacher version of the assignment - the only version you need to edit. In this version, any special syntax that you add for solutions/tests/mark schemes will be visible. Later, in the Generate/preview step, a student view version will be created, but the teacher version is the only one you will need to edit.

You can add any files necessary for the assignment to this folder, such as data and the notebook worksheets themselves.

Note: If you are moving files from another folder in Noteable, you will not be able to move notebooks that are currently running, you will need to shut them down first.

You can also write assignments directly in the "source" folder by clicking New>Python 3, which will open an empty notebook you can edit:

If you go back to the files tab on Noteable you will notice that in addition to the "source" folder, a gradebook.db file has been created.

Do not edit or delete gradebook.db. This stores important information about student grades.

For notebook files that will be marked, cell toolbars must be added to structure information about:

• Whether the cell is automarked.
• Any mark schemes used.
• Any tests used.
• Points awarded for each question.
• Whether students are allowed to edit a particular cell.

To add cell toolbars, open an assignment notebook and click View>Cell toolbar>Create Assignment on the notebook toolbar. This indicates that the notebook should be marked.

There will now be an empty toolbar above each cell:

The cell toolbars are only visible in the instructor version of the assignment, students will not be able to see them in the released version.

You can still edit the notebook as before, including adding new cells.

Clicking the dropdown menu on a cell toolbar will present a list of available cell types. The list of options differs slightly between markdown and code cells.

Markdown cells may be marked: "-", manually graded answer, manually graded task, or read-only. For example:

Code cells have a few more options: "-", manually graded answer, manually graded task, autograded answer, autograder tests or read-only. For example:

Once you are happy that all cell toolbars have been assigned appropriately, move on to the preview and release stage. The next few sections provide information on how to appropriately assign cell types and information.

In cell toolbars, you will notice that some cells ask the number of points awarded. In automarked cells, this is all or nothing depending on whether all tests pass or not. For manual grading, this is simply useful information when manually grading.

Note that the point value of a cell is not visible in the student version. If you want this information to be available to students, you must include it explicitly within the cell.

Cells with “ID” can also be helpful to markers. Do not use the same ID for multiple cells. You may wish to use ID to help you identify tasks and associated tests: “q4” and “q4_tests” for example. Again, this information is not available to students, but will be visible to the marker.

Autograded answer cells can be edited by students. This option only applies to code cells; markdown cells cannot be autograded. No points are associated with the answer cell, instead marks are assigned based on the autograder test cells that follow.

Special syntax is available for autograded answer cells. To include an example solution to the question within the autograded answer cell, enclose solution in the following syntax:

### BEGIN SOLUTION

Solution code

### END SOLUTION

This will be replaced by a code stub in the student release.

Autograded test cells cannot be edited by students, as indicated by the lock symbol in the left of the cell toolbar. This option only applies to code cells. The contents of this kind of cell are tests similar to JUnit tests, that pass or throw an error. Points are associated with this kind of cell, assigned based on whether tests pass or fail.

Special syntax is available for autograded test cells. By default, tests will be visible to students unless enclosed in the following syntax:

### BEGIN HIDDEN TESTS

Test code

### END HIDDEN TESTS

Since hidden tests will be replaced by an empty cell in the student version, it is generally wise to include at least one visible test to avoid confusion.

Points are assigned on an all or nothing basis depending on whether all tests pass or not. To achieve partial grading, split tests between two autograder test cells.

Here is an example set of autograded answer and test cells:

In this example, the student version will look like this:

Students will be awarded 1 point if the first test is passed, and 6 points if all tests in the second cell (including hidden tests) are passed. Where a test is failed, no points will be awarded for that test cell.

For further information on writing good tests for these cells, refer to the nbgrader documentation regarding this.

Manually graded task cells cannot be edited by students, as indicated by the lock symbol in the left of the cell toolbar. This option can be used with either markdown or code cells. These cells are associated with points (partial grading possible). This kind of cell is not automarked, a human marker will assign points manually.

This option is used for cells that describe a task that students must complete. There is not a specific cell in which students need to complete the task. Therefore, this kind of cell is appropriate for long answer questions requiring multiple cells, or cells describing tasks with a response that applies to the whole notebook.

For this kind of task, students may add any cells they require to complete the task.

There is special syntax to hide mark schemes within these cells:

=== BEGIN MARK SCHEME ===

Mark scheme here.

=== END MARK SCHEME ===

This can be useful at the marking stage, as the mark scheme is available inline with the task.

Here is an example use:

The student version of the cell above will look like this:

Manually graded answer cells can be edited by students. This option is available for both code and markdown cells. These cells are associated with points (partial grading possible). This kind of cell is not automarked, a human marker will assign points manually.

This option is used in student answer cells that cannot be automarked. In particular, this cell is appropriate when the student response appears in a specific cell, and is especially useful for written responses.

Here is an example, viewed from the teacher version of an assignment. Note that the solution is included in the cell (no special syntax required):

Here, the cell containing the corresponding question has been marked Read-only to prevent students from editing instructions.

In the student version, the body of the answer cell will be replaced by a stub, so the solution can be included in the teacher version.

Here is the student version corresponding to the above example:

Read-only cells cannot be edited by students. Both Markdown and code cells can be marked as read only. These cells are not graded.

This markdown cell gives some general information to the student therefore shouldn’t be edited or deleted by students:

Similarly, the contents of this code cell is intended for students to use in later cells, so shouldn’t be edited by students:

Ensure that the final version of the notebook is saved by clicking the disc icon in the menu bar:

Return to Formgrader and select the cap icon in the Generate column, which will produce a student version of the notebook, stored in the "release" folder.

A log output will appear on the screen to confirm that the student version has been generated successfully.

The Preview and Release icons will become available in Formgrader:

Click on the looking glass icon in the Preview column to be taken to the student version folder ("release" folder).

Clicking the name of the assignment notebook from this page will open the student version of the assignment.

You can now check that the student version looks as expected. In particular, it may be valuable to check the following:

• Code stubs are expected.
• Solutions are hidden as expected.
• Test cells are visible/hidden as expected.
• Student version allows/disallows editing of cells as appropriate.
• Cells containing hidden tests in the teacher version also contain visible information in order to reduce likelihood of student confusion.

You can adjust the teacher ("source") version to your taste, repeating the generate and preview steps as many times as you wish.

Once you’re happy with the appearance of the student version, be sure to save the notebook, generate and preview one last time to check the result is as desired, and then return to the Formgrader menu.

From the Formgrader menu, press the cloud icon in the Release column to make the assignment available to students:

A log output pop-up will confirm that this has been done successfully.

The status of the assignment will change to "released":

Your students will now be able to fetch, complete and submit the assignment.

Press the collect button from Formgrader to retrieve all submissions:

You may collect the assignment as many times as you wish.

The number of submissions collected at any point is shown in the # Submissions column.

Clicking the submissions entry will take you to the Manage Submissions menu:

Names and IDs of students that have submitted the assignment will be displayed.

The assignments, in this case just one submission, have now been collected but not yet marked (notice the Status column which reads “needs autograding”).

Once student submissions have been collected, the assignment can be marked.

From the main Formgrader menu, press the lightning bolt in the Autograde column in order to autograde the submission. Any autograded cells will be assigned marks.

A log-output pop up will confirm that the autograding step has been successful, and the status of the submission will change to "needs manual grading". The score of the automarked cells is displayed in the Score column:

Since some elements of this notebook are manually graded, it is necessary to click the Manual Grading tab on the left hand side of the Formgrader menu screen:

The Manual Grading menu contains a list of assignments:

Select the desired assignment to see the list of submissions:

Here, information is displayed on the grade breakdown, as well as a tick icon in the Needs Manual Grade? column.

Click on the submission entry to view the individual piece of submitted work. From each individual submission notebook, manual marks and feedback can be added using the toolbar above each cell:

Note: Information that is hidden in the student version (for example mark schemes) will be visible to the marker at this stage.

Notice that autograded cells already have points assigned to them, and that graded cells are automatically awarded 0 if the student does not provide a response:

Once you have finished marking the student's work, press next in the top right hand corner of the screen to move onto the next submission.

Pressing next on the final submission, will return you to the Manual Grading menu on Formgrader:

Return to the Manage Assignments tab on the left hand side, and click the number in the # Submissions column to show the list of submitted work.

Feedback can now be generated by clicking on the speech bubble icon in the Generate Feedback column. A log output pop up will appear to confirm that this has been successful.

Going back to the Noteable launch page, you will see that there is now a feedback folder. Clicking on this folder will take you to a list of folders named by student ID:

Clicking on an individual student's folder will show all available feedback for that student. In particular, an HTML file containing the grades and feedback for the named assignment:

As the instructor, you are able to view this file by clicking on it.

Once you are happy with the feedback document, return to the Formgrader menu and press the envelope in the Release Feedback column to make the feedback available to students.

An instructor may wish to edit an assignment or add data files after it has been released and fetched by students on a course. Please note that students cannot re-fetch assignments if they already have the assignment.

To re-fetch a file a student needs to delete (or rename) the assignment directory (it will be <Course Code>/<Assignment Name>.

Note that students who have already started the assignment will lose the work they have done if they need to re-fetch the updated assignment.