CAJ 3

This is the third of three articles on the use of computer technology in the judging of figure skating competitions. Part 1 discusses a rigorous approach to investigating the potential role of computer technology in the process of judging competitions, and describes the development of a computer model of the judging process as it currently exists. Part 2 describes many of the specific details of this computer model for the various figure skating events. Part 3 describes a software implementation of the model and includes a downloadable version of the annotation software developed with the model.

Overview

A brief description of the software developed in this research project was given in Part 2 of this series. In this article a more complete description of the software will be given, together with a description of how to use the software to annotate a program. All though at times it may seem otherwise, the main purpose of this article is to describe how to use the software. It is not to to teach how to judge or to advocate any particular standard for evaluating programs, other than how to handle the most extreme failures of an element.

The point model and software developed in this research effort are termed Computer Assisted Judging^TM, or CAJ^TM for short. CAJ is not a new method of evaluating or judging competition. It is a computer program that models the current judging process in accordance with the current rules and standards of skating; i.e., it is the name used here to refer to the current judging process coded into a computer program.

The CAJ software began its life in 1998 as a program to allow a user to annotate a program as a judge would in marking an event. The user can enter the element identifications and a quality assessment for each element. For short programs and other programs with deductions for required elements the deduction values can also be entered. In its original form, the CAJ software was simply a tool to replace the hand written notes judges make during an event. In its current state it can also used as a scoring tool or to do statistical analysis of program content to investigate the mathematical properties of a point model, and the mathematical properties of combining the assessments from a panel of judges.

Readers who are interested in participating in our research project should practice with the program before attempting to assess an actual performance. At first using the program may seem awkward, but with practice one will find that information can be quickly entered into the program without distracting attention from watching a performance excessively - and certainly no more than trying to take detailed notes on paper - once one gets accustomed to where everything is located.

This article will be revised from time to time in response to questions readers raise about how to use the software and/or suggestions on how to have the software more completely and accurately model the current judging process.

The annotation-only version of the CAJ software can be downloaded from the following link.

In this version of the software, all features other than the annotation features have been disabled and/or hidden from the user. The CAJ software has options to display, summarize and compare previous assessments after a skater has completed a performance, and even to change them. They have all been turned off to force the user to assess a performance on an absolute basis without reference to any previous assessments. Our current interest is research into the ability of humans to make assessments without the assistance such information would provide - which would otherwise turn an absolute assessment into a relative assessment. For the same reason, the annotation version does not provide the calculated cumulative mark for the elements performed or a placement for each performance. Again, this is to ensure independent assessments, unbiased by any comparison to any other performance or knowledge of where the previous skaters were placed. Also turned off is an error checker that looks for potential entry errors in the assessments or missing required information, and queries the user at the end of a performance to verify the questionable entries.

The complete CAJ software also includes several analysis tools for evaluating and comparing point models and to investigate the statistical properties of point models. These too have been disabled as they are of no interest to the general user.

The CAJ.exe file is a self extracting ZIP file approximately 3.5 MB in size. To install the software proceed as follows:

The software forms were designed using screen settings of 1024 X 768 and Large Fonts. If the forms display oddly, change your screen settings to match. The program is best used with a flat panel display of 15 in. or larger, or a CRT monitor of 17 in. or larger. The software is best used with a touch screen display, but a mouse will do.

Revised software will be posted, as needed, up until the start of Worlds. We will also attempt to post actual start orders for Worlds to use with the software, when the start orders become available.

General Approach

The goal in designing the user interface was to allow the user complete flexibility to describe a skating performance and assess the quality in accordance with the current rules. It was also designed to support the approach CAJ takes to apply a point model.

In applying the point model CAJ does much more than just sum up a list of tricks and point values. In writing the CAJ software an attempt was made to capture in the software the language of skating that describes a performance, and the ability to determine the base value of a complex element from its component parts. Simple movements and basic tricks are the words of the language (referred to below as keywords). These movements and basic tricks are combined together to describe more complex elements and form the sentences of the language. Finally, the elements are combined together like sentences in a paragraph to completely describe a program.

In the case of programs with required elements, the use of both quality assessments and required deductions proves both a challenge and a nuisance to avoid double booking errors. Were the goal the reinvention of skating, elimination or at least a reduction in the number of deductions would be very appealing - especially in dance where the number of different possible deductions is overwhelming. But reinvention was not the goal, and so both assessments and deductions are entered for required elements as is the case today.

If a required element is done without deductions, then use of the quality factor alone is straight forward. Even though an element has no deductions it still may be rated with negative quality, however. For example, a jump might have the required number of rotations and takeoff and land on a clean edge, but be executed so slowly, with such little height and with such awkward control or landing edge that a user would feel a negative quality assessment was appropriate. A required element without deductions, however, should never be marked "Failed."

If a required element is done so badly it cannot be recognized or virtually nothing is accomplished, then the element would be noted as "Failed" and the corresponding deduction would be 0.3-0.4. The CAJ software recognizes that the failed designation and the major deductions are duplicate ways of specifying the element is a disaster and does not double penalize the skater.

For the gray area in between the previous two cases, the assessment applies to the quality of that part of the element not compromised by the minor error that resulted in the deduction, and the deduction applies to the specific minor error alone. Specific examples will be provided below.

Another special case is a required element that is cleanly executed but doesn't meet requirements. For example, if a double Axel is required and a beautiful single Axel is executed, then the element would be identified as a single Axel and receive a high quality assessment, but would also receive the required deduction for having insufficient rotations.

Getting Started

The setup program will add an entry to your startup menu. When you execute the program an introductory screen will appear immediately followed by a prompt to open a text file containing the start order for the event. This is a simple ASCII file of skater names that can be created in Notepad or a similar program. A generic file is included in the application directory by the setup program to serve as an example.

Opening Form

After loading the start order, the user selects the event to be assessed, and one of the main assessment screens appears. The following screen shots show examples of some of the nine main assessment forms.

Ladies Short Program Assessment Form

Ladies Free Skate Assessment Form

Free Dance Assessment Form

[Note: (R) on the assessment form for free dance designates a required element in the free dance.]

When you begin execution, the program also creates an ASCII text file in which is written a record of every input you make to the program. It is currently formatted in a way to facilitate reading it back into an analysis tool, and thus is not the most attractive format for reading by the user, but hopefully is clear enough. The program also creates a binary file which is a more compact tabulation of the assessments made while the program was executing. The text file is written to in real time. The binary file is written when the program is closed. The files are automatically named according to the date and time when execution began, and have extensions of .txt and .caj. Upon execution of the program, a window called CONTENT is also created. This form contains the text entries that are written to the ASCII text file, and can be examined while the program is running

At the top of a main assessment screen is a pull down list that contains the starting order you selected at the start of execution. You cannot change start orders or events in mid-stream. If you use the wrong start order or select the wrong event you must exit the program and start over.

In general, CAJ does not care about the order in which you enter information. You can jump around freely and enter/change/inspect your assessments at any time during a performance. Be careful, however, when you select skaters from the pull-down list. You should not go back to try and change assessments once you move on to the next skater. The software will not allow you to make the changes, and every time you select a skater from the pull-down list the assessments forms are reset to their initial conditions. In this version of the software, if you go back to a previous skater your assessments already written in the ASCII assessment file will not be affected, but the entries that end up in the binary file will be zeroed out. If you have just started assessing a program and notice you have the wrong skater selected, you probably have time to select the correct name and reenter the data for one or two elements. If you are well into it, however, the best course of action is to just keep going and make a notation of the fact.

To the left of the start order list is a button to go back one entry in the list, to the right a button to go forwards one entry. Farther to the right are two text boxes for the first and second marks for the skater. The first of these will always be zero, because at this time we do not want the user to know where they have placed the skaters. The text box for the second mark will show the mark assigned by the user for that mark.

The large text box occupying the largest region of the form is for diagnostic text that has been disabled in this version of the software. In a scoring application, it is where a real-time video display of the performance or replay of the elements would be displayed.

Below the diagnostic text box are a row of push button controls and below that a place to enter the two auxiliary technical merit marks. Entry of the auxiliary technical merit marks should be obvious. These marks are specified on a 6.0 scale and can be entered/changed at any time.

Extra content refers to all brief spins, other than the major spin elements, with less than the minimum number of rotations required to be considered one of the four spin elements; all jumps of less than one rotation not part of a major elements; and all connecting moves not part of a step sequence.

The button labeled "Enter Mark 2" calls up a window to enter the second mark. There are two or three sub-marks on this form, depending on the event.

Example Presentation Mark Form

All sub-marks must be entered or the "Back to Mark 1" will generate a warning to enter them all. Pressing "Cancel" gets you back to the main assessment form without changing the second marks. Second marks can be entered/changed/inspected at any time and the total second mark is listed at the bottom and on the main assessment form.

The button labeled "Misc. Deductions" calls up a window to enter miscellaneous deductions. This window is event specific.

Free Skate Miscellaneous Deductions Form

Check a box to set a deduction. Click it again to uncheck it. The user can enter/change/inspect these deductions at any time during a performance. The window can be opened multiple times during a performance, and each time it is opened the current state of the miscellaneous deductions is displayed. For free skating the Well Balanced Program deductions are enabled, while for short programs the Extra Element deductions are enabled instead. For now, the user has to enter the Well Balanced Program deductions, though the program has the internal ability to figure that out for itself at the end of a routine.

For dance events there is also a button to enter deductions for falls. These are the deductions for falls that occur outside a required elements, and are program specific (CD, OD or FD). Deductions for falls during required elements are entered with the other required element deductions. Entering falls is cumulative. That is, every time a fall occurs an entry is made and the deduction is added to the previous deductions for falls. The user can only zero out the deduction for all falls if an error is made. The user also has the option of applying the deduction to only the first marks, or to both marks.

Form for Falls in Free Dance

To get out of this form without taking a deduction, just make sure none of the push buttons are depressed and click either of the bottom two buttons.

Technical Content

On the left side of the main assessment forms there are several columns of controls where assessments for the technical elements are entered.

The left most column is a list of generic labels for the different types of elements in each type of program. Click on the label to call up a window where the identification of the element is entered. The label is color coded green after you enter an identification.

The next column is for the entry of assessments. Click on the text box to call up a window where assessments are entered. After entering the assessment these boxes are color coded red for "Failed", yellow for negative quality, green for average quality, and blue for positive quality.

For elements with required deductions, the next column is for the entry of required element deductions. Click on the text box to call up a window where the deductions are entered. After entering deductions these deduction boxes are color coded red. At the bottom of the column is a text box showing the total deductions for all required elements.

The next column of buttons, labeled "Replay" is illustrative only to show where a replay buttons might go for each element. These buttons are disabled.

The identification, assessment and deduction controls can be used in any order and corrections can be made at any time and in any order. It is preferable, however, that elements be entered sequentially in the correct row; i.e., the first solo jump attempted should go into J1, the second into J2, etc. Because all data entries are time coded, one can sort this out after the fact if done otherwise, but doing it right the first time makes life a lot simpler for the analyst.

[Note: In coordinating the entries from multiple judges the replay technician, or other official, would update the skater's name for each judge and cue each judge to enter assessments in the correct locations to insure entries from all judges are synchronized.]

The nomenclature used for elements is not standardized. It is not claimed that the descriptions used here are the best, or could not be improved upon; and a graphical representation might be preferable instead. In any event, in the interest of expediency a nomenclature that makes sense to me was used (since this program was initially written for just the author's use). Perfection is not claimed and you are welcome to suggest improvements/changes.

To apply the point model correctly the software requires that certain practices be followed in identifying and assessing the elements, especially so far as failed elements are concerned.

Element Identification

The following is an example of a "one-click" identification window. For some elements only one button is needed to specify the element. When the user clicks a button the identification window unloads and the user is automatically returned to the main assessment form.

Example Single-Click Element Identification Form

For complex elements several keywords are required to describe the element. The controls on each element identification form are used to describe the element in detail. To return to the main assessment form depress "Enter." "Cancel" gets you back without changing the identification on the main assessment form. "Clear" clears out an entire description on the identification form. "Undo" backs up one keyword of an element description.

Example Multi-Click Element Identification Form

With only one exception, all of the buttons on the element identification forms refer to descriptions of the elements which go into determining the base value of the elements and are not error identifications. For example, Travelling Entry on the spin element identification form, refers to the traveling entry that is used in some spins (e.g., the camel spin) and does not refer to the travelling (centering) error in a poorly performed spin. The travelling error would be taken into consideration on the assessment form for the spin. The one exception to this is the button labeled .fail. on the jump combination form. The assessment boxes have only one failed designation for the element as a whole. On a jump combination/sequence, the .fail. button is provided to designate which individual jump of a combination/sequence is failed (has a major error). For example, on a triple Lutz - double toe combination one could have 3L2T, 3L.fail.2T, 3L2T.fail. or 3L.fail.2T.fail. depending on what the skater did.

The jump combination and spin combination forms also include some shortcut buttons for the most common examples of those elements. For example, in the ladies short program just about every lady attempts a triple Lutz - double toe loop combination, so it makes sense to have a single button to enter that combination and save the user some time entering the identifications.

For jump combinations and sequences enter each segment of the element in the order performed. Enter failed individual jumps as indicated above. The failed designation is entered immediately after the jump identification.

For spin combinations enter the positions and transitions (changes of foot) in order performed, plus the rotation count, and any added entry difficulty if appropriate. Only the positions and transitions have to be entered in the order performed.

For step sequences enter the type, pattern and assess the base difficulty. Add any notations for added content that enhances difficulty. For spiral sequences enter the edges in the spiral position. All these entries can be entered in any order.

For lifts enter the positions used in the order performed. Enter the number of rotations of the man and any added difficulty for the take off, air position, and set down. The lift positions must be entered in order performed. The other items can be added in any order.

Pair spins and side-by-side spins are identified using the same process as spins in singles.

For death spirals enter the edge, and any notation for added difficulty of the entry. For other pivot moves use the "Other" buttons.

Dance lifts do not have a formalized taxonomy as do pair lifts. Until such time as that comes into being, the user enters the difficulty of the lift on a scale ranging from 1.00 to 6.75 in steps of 0.25 using the buttons provided (Category 1A = 1.00 to Category 6D = 6.75). The number of rotations to the lift and added difficulty for the entry and exit can also be noted. These entries can be made in any order.

Dance step sequences are identified using the same process as all other step sequences.

Elements are identified by what was accomplished or actually attempted, no matter how badly; not by what was intended (i.e., planned). Badly attempted elements are then marked "Failed" or given negative quality assessments (and deductions for required elements) in accordance with the user's judgement.

For elements consisting of a major error at the start of the element and nothing more (e.g., popping a jump, falling out of a spin, a lift that doesn't go up), or which end prematurely with a major error, the identification entered is the description of the element up to the point of the major error which ends the element. For an element with a major error in the midst of the element, the entire element is identified, ignoring the major error for the purpose of the identification. For an element with minor errors, the entire element attempted is identified, ignoring the errors for the purpose of identification. The errors, of course, are not ignored for the purpose of assessments and deductions.

In free skating, if the skater has a jump combination planned but only the first jump is executed/attempted, regardless of the reason, the jump is identified as the appropriate solo jump in one of the solo jump controls (J1 through J8). In the short program, however, the identity of the one jump executed/attempted is entered in the control for the required jump combination.

A jump is identified as a single through quad based on the total number of rotations completed both on the ice (cheated) and in the air from throwing the jump to checkout. If a total of one half rotation or more is completed on the ice (either at the takeoff or landing or both), the jump is to be assessed "Failed."

If the jump attempted does not have the correct total number of rotations completed (both on the ice and in the air), the user must decides if the jump is an over-rotated example of one jump or an under-rotated example another. The user must also decide whether to assess the jump failed or not.

For side-by-side spins and jumps in pair events, if the element identification would be different for each team member according to the above guidelines, enter the identification for the more difficult/complete description. For example, in side-by-side triple toe loops if one skater attempts a triple and the other a double, the identification would be triple toe loop. If one team member falls out of the entry to a spin and the other completes the spin, the identification is the spin completed by the one team member.

Assessments

All the assessment windows consist of a series of quality factor buttons labeled "F" for Failed and "-3" through "+3". Enter the assessment and the form will unload and return the user to the main assessment window.

Below the quality factor buttons are some guidelines for how to assess the quality of each element. Each element has several factors that affect the quality. A given element may be high in one factor and low in another. The user must decide what the overall quality of the element is taking into account all the factors involved. It is not claimed that these guidelines are perfect or even complete, and like all other aspects of this program you are invited to make suggestions for corrections and improvements.

All pair and dance elements are to be assessed for quality as a team/couple. In pair events, side-by-side spin and jump elements with different difficulty/errors/quality for each team member, the user must decide if the element is failed or not as a team, and what quality should be assigned as a team. Some consideration was given to allowing different assessments for each team member (and also each factor affecting the overall quality) but it was decided this would add too much complexity to forms and the evaluation process.

Example Assessment Form

In order that the point model be correctly applied, CAJ requires the user mark elements failed for certain specific cases. These cases are listed in the following table. In addition to failing an element for a major error, elements may also be considered failed due to multiple minor errors, at the users discretion.

Deductions for Required Elements

Each type of required element has a corresponding deduction form that is element and event specific. There are two ways of entering elements on these forms.

Example Required Element Deduction Form

Along the top of each form is a row of buttons specifying the value for the deduction. If the user knows the value of the deduction, press the appropriate button and the form will unload and return to the main assessment form.

Below the deduction buttons are descriptions of the deductible errors for each element. Click on all the errors that apply and then press "Enter". The form will unload and return the appropriate total deduction for the element. In this second approach to entering the deductions, the user does not have to remember the value of each deduction and can focus solely on describing the error. The drawback is that several clicks are needed to enter the deduction whereas the first method enters the deduction with just a single click.

In terms of applying a point model (and documenting a program), the second approach is preferable since it provides more information to diagnose and analyze the program. For now, users can take which ever approach they prefer, and mix and match at will.

Required elements are assessed using the same criteria as any element, and the guidelines provided above for assessments apply to the required elements also. To assist the user in deciding how to combine assessments with deductions, note the following ways the point model is applied.

Some Examples for Required Elements With Errors, But Not Considered Failed

Abbreviations

Jumps	T	Toe Loop
	TW	Toe Walley
	S	Salchow
	R	Loop
	F	Flip
	L	Lutz
	A	Axel
	W	Walley
	OFA	One Foot Axel
	IA	Inside Axel
	TLL	Toeless Lutz
	pA	Popped Axel (less than 1 1/2 rotations)
Spins	COF	Change of Foot
Edges	RFI	Right Forwards Inside
	RFO	Right Forwards Outside
	LFI	Left Forwards Inside
	LFO	Left Forwards Outside
	RBI	Right Backwards Inside
	RBO	Right Backwards Outside
	LBI	Left Backwards Inside
	LBO	Left Backwards Outside
Pair Spins	U/U	Upright/Upright Position
	S/S	Sit/Sit Position
	C/C	Camel/Camel Position
	C/U	Camel/Upright Position
	C/S	Camel/Sit Position
	S/U	Sit/Upright Position
Death Spirals	FIDS	Forward Inside Death Spiral
	BIDS	Backwards Inside Death Spiral
	BODS	Backwards Outside Death Spiral
	FODS	Forward Outside Death Spiral

ELEMENT TYPE	ERROR	REQUIRED ASSESSMENT
Solo Jumps	Major error.	Failed
Jump Combinations/Sequences with 2 Jumps	Major error in both jumps or first jump only. Major error only on second jump.	Failed -3 Quality or less (*)
Jump Combinations/Sequences with 3 Jumps	Major error on 2 or 3 of the jumps. Major error on first jump only. Major error on second jump only. Major error on third jump only.	Failed -3 Quality or less -2 Quality or less -1 Quality or less
Solo Spins	Major error resulting in a spin with 3 rotations or less. Major error resulting in a spin with at least 4 rotations but less than the minimum number required.	Failed -2 Quality or less
Spin Combinations	Major error resulting in a spin with 4 rotations or less. Major error resulting in a spin with at least 5 rotations but less than the minimum number required.	Failed -2 Quality or less
Step/Spiral Sequences	Major error that result in half or more of the sequence being omitted.	Failed
Pair Lifts	Lift does not go all the way up. No catch or collapse on a twist lift.	Failed
Throws	Major error.	Failed
Pair Spins	Major error resulting in a spin with 3 rotations or less. Major error resulting in at least 4 rotations but less than the minimum number required.	Failed -2 Quality or less
Side-by-Side Spins	Same criteria as singles.	Failed if both skaters individually would be considered failed under singles criteria. -3 Quality or less if only one skater individually would be considered failed under singles criteria.
Side-by-Side Spin Combinations
Side-by-Side Jumps
Side-by-Side Jump Combinations
Death Spirals	No pivot position. Fall on entry. More than half the element not supported by the lady's blade alone.	Failed
Step/Spiral Sequences	Major error that results in half or more of the sequence being omitted by both skaters. Major error that results in 3/4 or more of the sequence being omitted by one skater only.	Failed
Dance Lifts	Does not take off (none of the lift is accomplished).	Failed
Dance Spins	Major error resulting in a spin with 3 rotations or less. Major error resulting in a spin with at least 4 rotations but less than the minimum number required.	Failed -2 Quality or less
Twizzles	Major error that results in half or more of the element being omitted by both skaters. Major error that results in 3/4 or more of the element being omitted by one skater only.	Failed
Dance Step Sequences	Major error that results in half or more of the sequence being omitted by both skaters. Major error that results in 3/4 or more of the sequence being omitted by one skater only.	Failed

Jumps Throws Side-by-Side Jumps	For minor errors assess the quality of what was actually accomplished, aside from the deduction. For a solo jump out of foot work that does not have preceding steps that meet requirements, assess the quality of the jump alone and take the deduction for the missing footwork. For a jump with too few rotations but otherwise clean, identify the jump actually executed and assess its quality, independent of the deduction.
Jump Combination	For a successful first jump and failed second jump assess the quality of the first jump. For minor errors assess the quality of what just was accomplished, independent of the errors.
Spins Pair Spins Side-by-side Spins Dance Spins	Assess the quality of the part of the spin that was successful, independent of the deductions. For a fall exiting the spin or ending the spin prematurely, assess the quality of the spin up until the fall.
Step Sequences Spiral Sequences Twizzles Dance Sequences	Assess the quality of that part of the sequence completed successfully, independent of the deductions. If the pattern is short or incorrect, assess the quality of the steps performed.
Lifts Dance Lifts	For a major error on the setdown or which ends the lift prematurely, assess the quality of the lift up to the error. For minor errors assess the quality, independent of the deductions. A twist with a major error is assessed "Failed" unless it is a clean single in which case the quality of the single is assessed.
Death Spirals	For a major error on the exit or which ends the element prematurely, assess the quality of the element up to the point of the error.

The Application of Technology to the Judging Process

Part 3 - Software Tool