The UMFG Adjudication Process in the Faculty of Science
posted 27 January 2012

This page describes the adjudication process in some detail and is primarily intended for staff in administrative roles related to the handling of UMGFs.  This information might also be of some interest to applicants.

1.    Background

Prior to the 2007 competition, the UMGF was managed entirely by the Faculty of Graduate Studies and adjudicated by a multi-disciplinary committee of researchers selected from across the University. There was general dissatisfaction with various aspects of the process that often revolved around the review of applications by adjudicators who were significantly out of area and the perceived need to have a common and rigid scoring scheme.

As a result, for the 2007 competition, the Faculty of Graduate Studies decided to allocate the UMGFs to individual faculties and ask each faculty to do its own, internal adjudication. Decentralization resulted in a system where responsibility for the UMGFs is, in some sense, shared among the various faculties and Graduate Studies. This change addressed some of the issues but also introduced new challenges.

Subsequently, the Faculty of Graduate Studies decided to further devolve handling of UMGFs (and MGSs now) to the “unit” level (using a formula based on graduate student enrolment and time in program).

Handling these awards at the department level had the potential to introduce a number of issues (mostly related to small departments and the limited number of UMGF awards available) so the constituent departments of the Faculty of Science agreed to continue with faculty-level adjudication of the fellowships based on individual merit. This system enables us to benefit all departments while avoiding disadvantaging any single department and still allocate awards to the most deserving applicants. (E.g. Handling situations such as “Computer Science has 4 awards allocated but only 2 truly excellent applicants while Statistics has 1 award allocated but 3 excellent applicants.”)

2.    The Process

Awards are made through a faculty-based adjudication committee with one representative from each department (normally the chair of the departmental awards committee or the chair of the departmental graduate studies committee). The reviews are thus, necessarily, non-expert but past experience has been positive with what seems to be general agreement among committee members on the award decisions. The Associate Dean Research chairs the adjudication committee.

Even within a single faculty, there are significant differences in what is expected of excellent applicants in terms of GPA, publications, experience, etc. In an attempt to address this problem, each departmental representative provides a statement of “departmental norms” for excellent MSc and PhD applicants. The departmental norms are distributed in advance of adjudication and are used by the reviewers in assessing the quality of applicants. This allows committee members to take departmental differences into account. Departmental representatives are asked to review the departmental norms annually.

Applications are collected and ranked by each department and the top ranked applications are submitted on paper to the Dean’s Office. Applications are scanned and made available electronically to the adjudication committee members via a secure website. Departments compute each applicant's GPA for their last two full years of study.

Care should be taken in calculating the GPA as FGS recalculates it and sometimes rejects applicants due to inadequate GPAs. A calculation error is very hard on the student and wastes the effort of people on the adjudication committee. If, for any reason, the last two full years cannot be used to calculate the GPA, please make explicit how the GPA was calculated.

Once applications are received in the Dean's Office, the chair will assign three readers for each application from departments other than the student’s department. This process should normally preclude conflicts of interest, but should any arise, please notify the chair immediately so alternate reviews can be arranged. A period of time will be given for reviewers to download and review the applications. Each applicant is to be scored on a 10-point scale. To help promote consistency in scoring, a document describing what characteristics might correspond to particular scores is distributed to those adjudicating the applications. There is some degree of flexibility provided by the document but reviewers are asked to try to adhere to it as much as possible while considering the departmental norms.

Scores are returned to the chair who will calculate average scores for each applicant and distribute a spreadsheet showing the combined scores and reviewer variance. This document will be reviewed and refined at the adjudication meetings. The top applications (those that will either receive an award or who may do so) are discussed to minimize variance where possible. In these discussions, the committee member from the student's department will act as a resource person to help clarify any issues that may arise. Once matters of variance and review accuracy are addressed, any ties will be broken. Additional reviewers may be assigned as needed to complete this part of the adjudication. A revised spreadsheet in refined-score order will ultimately be used to select the applications to forward to FGS for receipt of awards based on the quota for UMGFs assigned to the Faculty of Science.

Before finalizing the list of award recipients, a review of applications on either side of the cut-off point for funding is normally done to ensure fairness and accuracy.

There is an expectation from FGS that the distribution of MSc to PhD awards will be approximately in the ratio 2:1. This distribution may also need to be considered in making final decisions about which applications can be recommended to FGS.

The number of applications reviewed in a given year is determined by the number of UMGFs available to the faculty, and the need for an alternate list of applicants who may receive awards after tri-council awards are announced (since UMGFs must be declined by tri-council winners). For 2012, we will ask each department to submit their top 10 ranked applicants though we will likely only need to review the top 7. Assuming each department has at least 7 applications to submit, this will result in 21 applications being assigned to each adjudication committee member to review (3 reviews per application being required for each of 7 departments).

3.    Administrative Matters

Please note that for reviews in the Faculty of Science, GPA calculations can be made on the basis of departmentally printed student histories. FGS will require original transcripts (or departmentally certified true copies of transcripts or student histories) so students whose applications are recommended will need to have transcripts by March 31st. It may be easiest to simply ask students whose applications are forward to Science ensure that they have transcripts available (although only 24 of those will actually be initially forwarded to FGS).

As in previous years, the Manitoba Graduate Scholarships (MGSs) will be awarded as "top-ups" to the UMGF applications forwarded to FGS so no additional competition will be required.

Unsuccessful applicants are notified by the  Faculty of Science; successful applicants are notified by FGS.

4.    Timelines

Mid January Updated PDF form and other materials are available to students, referees and departments.
Mid February Complete draft applications should be available for use by the selected referees. Refinement of the application may continue until submission to the Faculty of Science (start of March).
Late February Departmental rankings need to be completed (departmental deadline dates determined on a per-department basis).
Start of March Applications and statement of updated department norms are submitted to the Dean's office.
Early March Applications and review assignments are available to adjudication committee online and reviewing will begin.
Mid/Late March Adjudication meetings will take place. Exact dates and times will be schedule through the Dean's office.
End of March Submission of recommendations to the Faculty of Graduate Studies.

5.   
Changes To The Form and Process (2012)

Fairly significant changes to the form and some of the requirements (e.g. two letters of reference are now needed instead of one) have been made for the 2012 competition. Please draw this fact to the attention of those involved.


RESEARCH STORIES

picture 1

Jörg Stetefeld
Canada Research Chair in Structural Biology,
Department of Chemistry, and
Department of Microbiology
with student Gregory Cox
Understanding Snake Venom


Picture 2

Gary Anderson
Department of Biological Sciences
Saving Sturgeon


Picture 5

Scott MacEwan
Ph.D. student in sub-atomic particle physics


Picture 7

(l-r) Werner Ens and Kenneth Standing
Department of Physics and Astronomy
Innovation Takes Flight