Documentation > Hub for instructors
Multi-marker options for Instructors and Tutors
Why Multiple Markers?
Consider this in a physical, paper-based, world:
With a small class (say 1 teacher and 30 students) assignments are fine: the teacher hands out assignments; collects them; marks them; and returns them with grades (and possibly comments.
With a much larger class (say 300 undergaduates), then asking a single teacher to mark all 300 submissions is a big ask - if we assume each paper can me marked & recorded in just 60 seconds, that's a massive 5 hours. In the physical, paper-based, world - marking is often distributed across multiple teaching assistants, so a lecturer and 9 assistants can mark the same 300 papers in just 30 minutes... or spend 2.5 hours marking, and devote 5 minutes to each submission.
In the Noteable service, prior to Multiple Markers, there was no way to divide up the electronic task: the system was firmly rooted in the concept of "1 class, 1 instructor"
You could "game" the idea:
- the 9 assistants and 1 lecturer each release their own version of the assignment,
- the student cohort is divided into 10 groups (one group for each "instructor")
- students see all 10 assignments, but only work on the one for their group
- each "instructor" marks their group (feedback as needed)
- each "instructor" copies their grades back to a central place, for collation.
With Multiple Markers, we have created a shared environment for all "instructors" - so there is only 1 assignment, and all marking happens in the one place.
You still have 10 markers, and you still have 300 submissions... however all the grades are in one place.Multiple Markers is an "Opt In" system.
- With the normal assignments system, the instructor uses a local database, and stores all assignment files in their home directory
- With the Multiple Marker system, the assignments system uses a centralised database, and a piece of shared file-storage.
An Instructor, using the "multiple marker" feature, should essentially notice no difference when using the Notebook web inferface
The "multiple marker" feature has absolutely no impact on the student's experience in the service
Advantages of the Multiple Marker system
With the Multiple Marker system, all instructors for the course share the same database, and the same filestore
This means that:
- all student submissions are kept together
- all instructors [ie, markers] have the same access to collected stuident submissions
- any instructor can mark any submission
- the same submission could me marked my multiple instructors
Features
- Multiple Markers for instructors on a given course in a Virtual Learning Environment
- None of the files used in the complete assignment cycles count towards the instructors file store quota
- All instructors have access to all submissions
- All Grades [given by different markers] are all held in the same place
Considerations
- The Multiple Marker system in not interchangeable with the current assignments system: for any given course, you have to use one or the other.
- Managing who manually grades, and provides feedback for, which student is firmly for the instructors to agree themselves, the system makes no assumptions of provisions.
- You will NOT be able to collect data from the old gradebook.db database
- The file will remain present, but none of the nbgrader db ... commands will talk to it whilst you are on a course that's "opted in"
- If you access noteable on a course that's not "opted in", the nbgrader db ... commands will revert to the gradebook.db database
Additional Features
There are two other features of note:
- The Command Line Interface now works for all steps
- We have changed the directory structure [having received feedback]
Instructor side
| step | old path | new path |
|---|---|---|
| source | ~/source/<assignment code> | ~/instructor_courses/<course code>/source/<assignment code> |
| assignment_generated | ~/release/<assignment code> | ~/instructor_courses/<course code>/release/<assignment code> |
| collected | ~/<course code>/collected/<student>/<assignment code> | ~/instructor_courses/<course code>/submitted/<assignment code>/<student> |
| autograded | ~/autograded/<student>/<assignment code> | ~/instructor_courses/<course code>/autograded/<assignment code>/<student> |
| feedback_generated | ~/feedback/<student>/<assignment code> | ~/instructor_courses/<course code>/feedback/<assignment code>/<student> |
Student side (unchanged)
| step | path |
|---|---|
| assignment_fetched | ~/<course code>/<assignment code> |
| feedback_fetched | ~/<course code>/<assignment code>/<feedback> |
Opting In
By default, we will not force anyone to use the new system - there has to be a concious request to do so.
Opting in can be done for the whole customer, or for specific courses. Note that we need the course code (MATH101_2020_SEM1), rather than course name ("Maths 101 - An Introcustion to Bayesian Statistics")