Programming in visual arts organizations can come from a variety of sources, from in-house curated exhibitions to proposals from outside organizations and calls for entry. To manage the documentation generated by third parties, a simple online form that supports image and document attachments will work well most of the time. However, calls for entry often require the materials to be evaluated by a jury, and preparing to present those materials to jurors often becomes a time-consuming task for managers. This challenge is where regular online forms fall short and where YouJudgeIt, an adjudication solution by Westaf, provides a good solution.

Conventional online form services offer two interfaces: one where administrators build applications, set deadlines and access the submission information, the other where the users complete the forms and submit them for review. YouJudgeIt offers a third option, a module for jurors to access and score applications online. This optional feature is completely integrated into the system and only requires administrators to set it up once per application.

With a simple and intuitive set up process, YouJudgeIt allows users to create a call for entries, start receiving submissions, and assign jurors access to the proposals through the following steps:

1. Specifying call settings: Create name and description of the call for entries, determine start and end dates, and set file types and quantity that applicants can upload. Attachments can’t be tied to a specific field (eg. Bio) and appear by default at the end of each form, but applicants are able to give a title and provide comments for each of their documents. This step is also where the organization can set an application or work sample fee (YouJudgeIt collects a 3.5% commission for every fee charged).

2. Create application form: YouJudgeIt allows for four different application question types: short answer, long answer, one choice, or multiple choice. It’s also possible to add help text, pre-populated text, and to indicate a field as required. However, it’s not possible to set character count limits or to specify a field type (eg. text, number, email address). Questions can be rearranged with a simple drag and drop function, but the options in the questions themselves have to be typed in the exact format needed. There are no form templates available, an important limitation for novice form creators.

3. Determine the scoring schema: Determine a scoring range and if the application will be scored “by application” (work samples are scored as a whole) or “by work sample” (each sample submitted receives its own scorecard). The organization can also choose to make applications anonymous to the jury by hiding each applicant’s name.

4. Build a scorecard: Set questions for jurors and assign a weight for each of them.

5. Add jurors to the call: Input first and last name, email address, and password for each juror. As soon as an administrator continues to the next step (and before publishing the call) an email is triggered and the login information and password are sent to the jury.

6. Review and activate call: Preview the forms as the applicants and jurors will see them; make any necessary changes before publishing.

7. Manage submissions: An administrator can login to manage calls, applicant information, and jurors. This includes modifying, opening/closing a call, exporting results or support material, and viewing jury panelist’s scores and comments.

Apart from the jury module, YouJudgeIt is also notable for its pay-as you-go pricing scheme. Branded as a solution for organizations on a budget, Westaf developed a “token” system where each feature has a cost in tokens. Tokens can be purchased for $1.50 plus a transaction fee, which varies with volume. For example, when purchasing 10 tokens the cost is $15 and the fee is $0.83 (or 5.52%); when purchasing 100 tokens the cost is $150 and the fee is $5.55 (or 3.69%).

Every account receives 500MB of storage space and 115 tokens free upon registration. This amount is enough to create a basic call for entries with five questions that allow for image attachments (100 tokens), and to start receiving applications (1 token each). Every application that is received after an account runs out of tokens is automatically saved and placed in a “Possible Application” section, which administrators can access after purchasing the necessary tokens. It is important to note that although the standard form allows for five questions only, fields for personal information, such as name and mailing address, are already included and do not count towards this limit.

Other functionalities that can be added to the application forms include: unlimited questions (a one-time 500 token fee), an automatic email response for each submission (30 tokens), additional storage space (4GB for 35 tokens per year), video (350 tokens), and document upload (60 tokens). This pay-as-you-go system is equal parts useful and confusing. On one side, it provides flexibility for the user to adapt YouJudgeIt to their organization’s budget. But the token system is also not as transparent as just setting a price, in dollars, for every feature.

To cut through the confusion, I created the following sample budgets that estimate the cost of using YouJudgeIt:

Note that the above examples do not budget for video and audio attachment capabilities, or for additional storage space. These exclusions are mainly because it is often easier for applicants to upload their materials to services such as YouTube or Vimeo, and then share the corresponding link and password (when applicable) in the form. The sample budgets also assume that 200 applications will be received. If an organization has difficulty estimating the number of applicants the call will generate, they can leave that figure out of the budget and purchase the exact number of tokens needed to unlock the submissions from the “Possible Applications” status once the call has ended.

An initial cost of $1250 (or $6.25 per application[i]) seems high, especially since free form services exist, such as Google Forms, as well as reasonably priced alternatives like Adobe FormsCentral. However, if staff labor hours are taken into consideration when evaluating these budgets, it is possible that an investment in YouJudgeIt could be more cost efficient than having the staff prepare and mail all the application materials to the jurors each time the organization issues a call for entries. In some cases, a system like YouJudgeIt might even eliminate the need for a portion of the travel expenses associated with evaluation panels.

Overall, YouJudgeIt is a useful and well-designed tool that effectively meets specific needs of visual arts organizations. However, the discrepancy between the cost of using it and being a true “solution for sponsors of competitions on a budget”[ii] is worth considering. The tradeoffs between effectiveness and cost will be specific for each organization, yet it can be noted that in general this solution is recommended for organizations that manage a large volume of applications in their calls for entries, as it becomes cheaper with scale. In most cases YouJudgeIt will likely make the adjudication process more efficient, but not necessarily cheaper.

What is your experience managing online calls for entries? Please leave a comment below or contact the editor; we would love to hear your thoughts.

[i] Assuming 200 applications

[ii] https://youjudgeit.org/