Write-off contests: critiquing and judging

Instruction page index

Contest Coordinators
Enable permission to submit critiques
Assigning critics to contests
Generating the critique assignments
Monitoring incoming critiques
Troubleshooting the critique process
Transitioning to judging mode
Monitoring the judging process
Compiling, recording, and delivering results

Critics
Navigating to the critique queue
Viewing entries
Submitting a critique
Additional assignments

Judges
Navigating to the awards page
Viewing critiqued entries
Submitting awards

The Write-offs judging overview

Once contests go live, the contest coordinator can begin assigning critics to contests.

Entry deadline passes – all entries received.

Coordinator assigns entries to critics for review and rating.  Each entry is initially assigned two critiques.

Critics begin logging in and submitting critiques.

For any entries that receive split ratings (a thumbs-up and a thumbs-down), the system will assign a third critique as a tie-breaker.

Once all the critiques are completed, a judge reviews only critiques that have received two thumbs-up ratings, and chooses which entries receive awards.  Note the distinction we make between “critiquing” and “judging”.

Contest Coordinators

The contest coordinator recruits and assigns people to critique the entries submitted for online contests.  To perform any of these tasks, the coordinator must have an “admin” account set up on the system.

Enable permission permission to submit critiques

Most critics will already have an account in the Write-offs system if they are a JEA member.  For anyone volunteering to critique, who is not already in the system as an adviser, we can set up a “critic” account for them.  We are no longer allowing the practice of creating duplicate accounts for advisers to use for critiquing, due to the amount of confusion that created.

In that case, to assign them as a critic, follow this sequence.

  • Search Users
  • Search on the name of a person to assign as a critic.
  • Click the edit symbol next to their name in the search results.
  • Check the box near the top of the User Information form that says, “Allow User to Critique”.
  • Click [Submit] at the bottom of the page.

Now this person’s name will show up in the list of critics you can assign from the contest information page.

Assigning critics to contests

As the system is currently programmed, critics are assigned to a contest, not to individual entries.  Once they are are assigned to a contest, the system assigns a random selection of entries to each critic.  Each entry starts with two assigned critiques.  If an entry receives a unanimous rating (two “thumbs up” or two “thumbs down” ratings), no further critiques are assigned for that entry.  For entries that receive a split rating (one “thumbs up” and one “thumbs down”), a third critique is assigned for that entry.

Coordinators are responsible for ensuring that the critics are informed about this process, and that they are prepared for two or more rounds of critique assignments.

Once people have volunteered to rate entries in the contest, the coordinator can begin assigning them to contests.  This part can be done in advance, as soon as the contest goes live.

Navigate to the contest

Once logged in, click on View Contests.

Find the contest in the list, and click on the magnifying glass icon (the “View Contest” control) at the right of the contest.

critique-assignments-01-600px

Add people to the critics list for the contest

Near the top of the contest information page, click on the [Assign a critic] button from the task bar.

critique-assignments-02-600px

A list of all people with critiquing permission will be displayed.  If any critics have already been assigned to this contest, they will appear in the list with a checkmark next to their name.  The critics shown with checkmarks are only assigned to the contest currently in view.  To assign critics to more than one contest, the coordinator must navigate to each contest and add them to each one.

Choose at least three people from the list.  There is no need to click a submit or save button; your selections will be saved as soon as you click on them.

critique-assignments-03-600px

Click [Assign a Critic] a second time to collapse/remove the list from the screen.

Everything up to this point can be done anytime while the contest is running.

Generating the critique assignments

After the entry submission deadline passes, and all the entries have been received, begin assigning entries to critics by clicking on the link near the top of the contest information page labeled “Assign entrants to be critiqued”.

critique-assignments-04-600px

As the system is currently programmed, explicit critique assignments cannot be made (i.e., assign a specific critic to critique a specific entry).  All entries in a contest are randomly assigned and equally divided between however many critics are selected to critique entries in this contest.  Initially, two critiques are assigned to each entry.

Once this command is executed, a message appears to confirm the operation was successful.

critique-assignments-05-600px

To view the list of critiques that have been assigned, click the link just below the assign entrants link labeled “View Critique Queue”.

critique-assignments-06-600px

You should now see a list of critique assignments.  It shows which critics are critiquing which entries.

critique-assignments-07-600px

At this point, you may now notify your critics they can begin reviewing entries and submitting critiques.

Monitoring incoming critiques

As critiques start coming in, your critique list will be updated to show which assignments have been completed, and indicates the ratings they received from the critics.

critique-assignments-08-600px

If an entry receives unanimous thumbs-up or thumbs-down ratings, no further critiques will be assigned for that entry.

Here, we see that one entry received one thumbs-up rating, and one thumbs-down rating.  In this case, a third critique must be assigned to break the tie.  The system handles this automatically.  In this example, “Test StudentOne” received a thumbs-up and a thumbs-down, and now a third critic has been assigned a critique for that entry.

critique-assignments-10-600px

Troubleshooting the critique process

There are occasions when it may be necessary for a coordinator to intervene, in order to keep the rating process going.

When tie-breaker critiques are assigned, the critic receiving the new assignment may not be aware of the new assignments.  In the Critics manual, they are instructed to check back once a day until all the critiques are completed.  It’s a good idea to make sure they are aware this may happen, and ask them to check back for new assignments, until they see the contest pass into the judging phase, or until you notify them that the critiques are all complete.

In the event a critic is unable to submit a critique, or doesn’t show up, that can be problematic, since a critique assignment cannot be explicitly removed or reassigned, and judging cannot commence until all the ratings are received.  Fortunately, there are a couple of possible actions that can resolve this.

  • One way is to remove the critic from the contest and, if that leaves fewer than three critics for that contest, add another one.  Then, rerun the “Assign entries to be critiqued” command to “reshuffle” any remaining (un-submitted) critique assignments.
  • In the event there are insufficient critics to cover the assignments, the coordinator does have the ability to complete the critiques and submit them.
  • If all else fails, call or email the JEA web developer (webmaster@jea.org) for assistance.

Occasionally, a critique may need corrections or have to be redone.  This cannot be done by the critic or the contest coordinator, since the system does not allow additional edits to a critique once it’s been submitted.  If this happens, the JEA web developer can reset that critique, which will allow the critic to redo that critique.

Transitioning to judging mode

Once all the assigned critiques have been received, the system will automatically set the contest into judging mode.  Once the contest reaches this milestone, a notice will appear at the top of the contest information page, and a new link appears for the judges to assign awards.

critique-assignments-11-02-600px

Another indication the contest has passed into judging is on contest list page.

judging-01-600px

At this point, you can dismiss your critics, and notify your judges that they can begin.

Monitoring the judging process

 

Compiling, recording, and delivering results

 


Critics

If you have volunteered to critique contest entries, your contest coordinator will notify you when critiques have been assigned.  You can then log into your account in the Write-offs system to view your assignments and submit your critiques.

Navigating to the critique queue

Once logged in, navigate to your list of assigned critiques.  Start by clicking “View Critique Queue” from the main menu.

submitting-critiques-01-600px

A list of contests containing entries you’ve been assigned to review appears next.  If a critic is assigned to more than one contest, this list will include all of them.  Click the View icon (the magnifying glass) to go to that contest.

submitting-critiques-02-600px

Here, the contest information is displayed, including the contest rules, and other contest details.  Here you can see the number of entries you’ve been assigned to critique.  To see the list of entries you are assigned to, and to submit your critiques, click on “View Critique Queue”.

submitting-critiques-03-600px

Viewing entries

Here, you will see a list of entries you can submit critiques for.  To view the entry, click the View icon next to each entry.

submitting-critiques-04-600px

In the next screen, you will see a thumbnail of the entry and a form to fill out your critique.

submitting-critiques-05-600px

Clicking on the thumbnail image of the entry will display the entry full-sized.

submitting-critiques-06-600px

Submitting a critique

Choose a rating for the entry, type your comments, and click [Submit].

Note: Please do not copy/paste your comments from word processing or other programs that produce styled text.  Instead, type your comments into the comment box, and if you are using other programs to keep records of your comments, copy/paste from the comment box into whatever program you’re using.

Once you hit [Submit], you will not be able to go back and make additional edits, so please be sure everything is correct before submitting the critique.

In the event that a critique must be corrected or redone, please have the contest coordinator contact the JEA web developer.

submitting-critiques-07-600px

Once you submit your critique, you are returned to your list of assigned critiques.  The system displays a green checkmark next to critiques that have been successfully submitted.

submitting-critiques-08-600px

Repeat the process for all critique assignments on your list.

Additional critique assignments

After you have completed all the critiques assigned to you, it is possible, in the event that some entries receive split ratings, you may be assigned additional critiques to complete as tie-breakers.  It’s a good idea to check back about once a day until you receive word from the contest coordinator that all the critiques are completed, or you see that the contest has passed into judging mode, as indicated in the contest list.

judging-01-600px


Judges

Once the critiques for a contest are all completed, the Write-offs system will automatically change the contest status to “Judging”.  In Write-offs contests, this is the process of reviewing entries that received two thumbs-up ratings, and determining which among those should receive an award.

Navigating to the awards page

judging-01-600px

critique-assignments-11-03-600px

Viewing critiqued entries

assigning-awards-01-600px

assigning-awards-02-600px

assigning-awards-03-600px

Submitting awards

assigning-awards-04-600px

When viewing the awards queue, the assigned award will appear next to each entry in the queue.

assigning-awards-05-600px

Note that all entries must receive an award entry – even if it’s “No award” – before the results will be released to the contestants, where they can see them when they log into their accounts.

Password Reset
Please enter your email address and press [Return] or [Enter]. You will receive a new password via e-mail.

If you don't receive it within a few minutes, please call (785-532-5532) or email us (staff@jea.org), and we'll be happy to assist you.