Rubric Assessment:Grading a Design Rubric
An assessment of whether you can tell a working evaluation rubric from a broken one. You are given a finished presentation deck, the brief that produced it, and a rubric somebody wrote to grade that deck, together with the ten rules they were told to follow. You review the rubric and say what is wrong with it, up to five findings, each ruled on by two other people who read the same packet. The ten rules are published up front, and so are the defects we already found; restating either scores nothing. What is measured is whether the rubric would actually work. No design software needed, about 45 minutes.
This board is not published yet.
Every finding has to be ruled on before a cohort can be scored, and a board goes up once that is done rather than while it is under way.
What the order means.
What earns a place, what decides the sequence, and what a board does and does not put in public.
Submitting. A row appears as soon as somebody sits the exam, and carries what is known about their attempt at that point. A result is what has to be earned: at least one finding that cleared all three checks, meaning specified well enough to be checked, not already on the risk list the exam published, and confirmed by two people who sat the same exam. You also have to return the reviews the exam asked of you, because the checks that settled your own findings ran on other participants' time.
Rarity. Every confirmed finding is weighted by how few other people found the same thing, and a participant's weights are summed. Precision breaks ties within equal rarity, then confirmed count, then who submitted first. Precision alone cannot decide it: it stops at 100 per cent, and a cohort of competent people all pile against that ceiling. Submitting more findings does not move you up, which is the point: an exam that rewarded volume would be measuring stamina.
They describe one attempt, not a person. Confirmed is how many of the findings submitted came back confirmed; before an attempt has been checked the same column reads how many cleared the first, mechanical check, and precision reads pending. Reviews is how many other people's findings that author went on to rule on. Precision counts only findings a checker settled, and breaks ties. Rarity is the column the order is made on: it has no ceiling, so it is printed as a figure rather than a percentage, and it reads n/a until the findings have been grouped.
Because that person has public visibility switched off in their profile settings. The row stays, because removing it would print rank three as rank two and nobody reading the board could see that happen. The result is published; the name is not.
Either they did not sit this exam, or they sat it and placed below the rows this board publishes. It carries a top twenty-five; the cohort count beside it says how many people sat, and the difference between the two figures is the rest of them. Nobody below the cap is scored or ranked any differently, and each of them can read their own position and their own row. An attempt that was opened but never submitted counts in the sitting total and reaches no board at all.
The finding is settled by nobody and counts for nobody. It is kept out of the tally in both directions, so a genuinely hard question that two people cannot resolve costs its author nothing. The same holds when both checkers say they cannot tell from what is in front of them.
Rows appear as people submit. Results appear later, when the human checks on a cohort are closed and the cohort is scored. Those are deliberately separate: a precision computed while findings are still unruled is not a smaller version of the real number, it is a different number that will move, so no result is published until every finding has been ruled on. That is also why the board prints no numbered order until a cohort is large enough to carry one. A ranking over a handful of confirmed findings is noise wearing an order.
No. What you write goes to two other participants with your name removed before they read it, and it stays there. There is no public archive of anybody's findings, and there is no plan for one.
