UMGF Adjudication Guidlines in the Faculty of Science and Tips for Improving Applications
posted 30 January 2012

This document provides guidance beyond the instructions provided with the application form. It is intended to be read by all those involved in UMGF applications: the applicants themselves, those writing letters of reference, and representatives of departments involved in processing the UMGF applications from students in their departments.

Background

Prior to the 2007 competition, the UMGF was managed entirely by the Faculty of Graduate Studies and adjudicated by a multi-disciplinary committee of researchers selected from across the University. There was general dissatisfaction with various aspects of the process that often revolved around the review of applications by adjudicators who were significantly out of area and the perceived need to have a common and rigid scoring scheme.

As a result, for the 2007 competition, the Faculty of Graduate Studies decided to allocate the UMGFs to individual faculties and ask each faculty to do its own, internal adjudication. Decentralization resulted in a system where responsibility for the UMGFs is, in some sense, shared among the various faculties and Graduate Studies. This change addressed some of the issues but also introduced new challenges.

Subsequently, the Faculty of Graduate Studies decided to further devolve handling of UMGFs (and MGSs now) to the “unit” level (using a formula based on graduate student enrolment and time in program).

Handling these awards at the department level had the potential to introduce a number of issues (mostly related to small departments and the limited number of UMGF awards available) so the constituent departments of the Faculty of Science agreed to continue with faculty-level adjudication of the fellowships based on individual merit. This system enables us to benefit all departments while avoiding disadvantaging any single department and still allocate awards to the most deserving applicants. (E.g. Handling situations such as “Computer Science has 4 awards allocated but only 2 truly excellent applicants while Statistics has 1 award allocated but 3 excellent applicants.”)

The Current Process

Each year, the Faculty of Science is given an allotment of UMGFs. This allotment assumes that there will be roughly a 2/3:1/3 split between MSc and PhD awardees. This is important since PhD awardees receive more money and may hold the award for a longer period than MSc awardees. The number of fellowships is based on a “share” allotted to units (e.g. the Faculty of Science) less the number of ongoing recipients in that unit (Science) from competitions in previous years. Based on this number, the Dean’s Office requests copies of the top ‘N’ (N determined based on the number of available awards) applications from each department. Normally, only a subset of the top N is actively adjudicated initially. Other applications may be adjudicated to facilitate the handling of anomalous situations (e.g. an original applicant is found to be ineligible, as additional awards become available, etc.).

The applicants are ranked (as described in the next section) by a Science-local evaluation committee, and then the Associate Dean, Research communicates a list of “to be awarded” along with their award GPAs, to the Faculty of Graduate studies. A number of “alternate” candidates is maintained by the Faculty of Science in case awards are declined or more awards become available.

The Faculty of Graduate Studies confirms the selections, verifies eligibility and availability of funding and informs successful applicants of their awards. The Associate Dean, Research informs unsuccessful applicants that reviewed by the evaluation committee. The departments notify unsuccessful candidates whose applications were not forwarded for review to the Faculty of Science.

Evaluation

The Faculty’s UMGF evaluation committee is composed of one representative from each department (normally but not necessarily the grad chair in the department) and is chaired by the Associate Dean, Research. Three reviewers are selected for each application (from committee members from departments other than that of the applicant).

Each reviewer assigns the application a score out of 10 to create an initial ranking. The resulting scores are then discussed in committee to ensure that the scoring is consistent, that all information has been taken into account and that the relative ranking of applicants is appropriate. During this process, the committee member from the applicant’s department acts on behalf of the applicant to provide clarification as needed. Ultimately, the final ranking of applications is decided by committee consensus.
Scoring is based on each student's performance and potential as measured by research experience, departmental ranking and scholarly achievement (including GPA).

Indicators of performance in these areas include the quality and originality of the research statement, research accomplishments, awards held, publications, conference presentations, and letters of reference.

To address the issue of differences among departments, the committee evaluates an applicant’s performance in a discipline-specific way, making use of information regarding “departmental norms” describing expectations for excellent students (either MSc or PhD) in each department. The committee also takes special care in the assessment of GPA's for foreign students considering the quality of the institution and, when possible, the quality of the individual department.

The Most Common Problems

The Faculty of Science UMGF evaluation committee has identified a number of common application problems. Such problems not only impact individual students negatively but also make it more difficult to fairly rank the applicants as a group. The most significant problems are described in this section. Possible mechanisms to address these problems are discussed in the next section.

The “big picture” problems fall into three areas: inconsistencies, lack of detail, and missing information.

There are two commonly occurring specific forms of inconsistency. The first is inconsistency by referees between their scoring of applicants on specific characteristics (e.g. “Top 10% in terms of Academic Ability”) and the subsequent description of their abilities and experience. The second is inconsistency in quality of the various sections of the application, likely resulting from more attentive proofreading by the proposed advisor of some sections than others.

Lack of detail is most commonly manifested in information provided by the head (or department awards committee) in Part III and, sometimes, in information provided by the referee in Part II. Occasionally, the applicants themselves also provide research descriptions that are “too thin”, particularly in discussing how they intend to approach their research problem.

Missing information is disappointingly prevalent. Small things such as the monetary value of awards and whether they were national or local may seem insignificant but can be important when differentiating between very closely ranked applicants. Also, what would appear to be obvious information in the research description is sometimes omitted. For example, saying what research is planned but nothing about why it is interesting to the target community is all too common.

Recommendations

The specific recommendations presented in this section are organized to reflect who is writing the various sections in which the problems occur.

For the Applicant (Student)

The first thing all applicants should do is to read all the directions including those embedded in the application form itself and make sure that all requested information is provided.

Second, it is important to remember that the adjudication will be done by committee members who are not in the applicant’s own department. This means that the entire application must be written so it will be understood by a non-expert audience but is still scientifically significant. (This can be challenging!) In particular, it is very important to explain the significance and potential impact of the proposed research since this may not be obvious to non-expert reviewers. (Significance need not be some application of the proposed work. Instead, for example, it might be a description of how it will advance the field.) Also, the relevance of previous experience (especially industrial experience) to the proposed research needs to be explained.

Third, the entire application is to be written by the applicant including the research proposal. While it is entirely appropriate for the applicant to discuss the proposal with his/her supervisor, it is inappropriate to include material that is written by anyone else. Including such material is normally obvious to the committee and must be avoided.

Finally, applicants should be careful to choose appropriate referees who know them well and are able to write in detail about them, their experience and abilities.

Some more specific suggestions include the following:

  • Describe the value of your prior experience and explain how this experience has helped prepare you for the proposed research. A particular case where this can be an issue is with co-op experience where the work is not described and might or might not have been research relevant.
  • If any of the awards or scholarships you have received is particularly selective, be sure to bring this to the attention of the committee.
  • When itemizing your publications and other contributions, please structure the list as requested in the form and briefly explain the significance of each publication and describe the venue it appears in. Papers submitted, poster contributions as well as quality, unrefereed material can be included but probably merit some brief, additional description.
  • Discuss the significance of any “unusual” research contributions you may have made (e.g. participation in competitions, contributions made while working in industrial jobs, and the like).
  • Be sure to talk about your proposed research not research from a previous degree. Also be sure to clearly layout the proposed research plan and briefly discuss the significance of the anticipated results. The committee needs to be able to judge how well prepared you are to do the proposed research and your research statement/proposal is a big part of this assessment.

For the Referees

The overall credibility of a referee’s report hinges on consistency, accuracy and the provision of evidence. Most important of all is that the comments and scores provided by the referee must be mutually consistent. It is probably good practice to re-visit the scores after the free-form part of reference has been written. Further, thought may need to be put into the selection of the scores themselves.

Wherever possible, the cohort of “similar students” should be as specific as possible. Thus, for example, a cohort of “new MSc students” is much more useful than one of “graduate students” generally. It is also advantageous to convey information about the cohort selected to the evaluation committee. Very small cohorts should be avoided if possible. This is sometimes problematic with referees who are new academics. Please remember, however, that there is no reason why a cohort has to consist of only University of Manitoba students.

Comparison to students at a similar stage that a referee may have known at other institutions is perfectly reasonable. Again, the cohort should be described to the committee. Further, ranking someone in the top 2% in a cohort of size 10 does not make sense and casts doubt on the validity of other scores as a result. Finally, simply saying that a student is “excellent” has little value since at this level all students are, to some extent, excellent. It is far more useful to precisely explain why/how the student is excellent and compare him/her (anonymously) to other students at similar levels.

It is also very important for a referee’s report to be as detailed as possible. The bulk of referees’ reports provide specific information on the qualities, abilities and potential of an applicant. A report that does not provide specifics, suggests that either the referee does not care whether or not the student receives the award or that the referee does not know the applicant well. In most cases, the referee should know the applicant well.

In all cases, it is useful to describe the context in which the referee knows the applicant. If there is limited knowledge of the applicant then the referee should explain why s/he is acting as a referee and why the referee’s report should be considered to be accurate. A referee who does not know the applicant likely should not have agreed to serve as a referee in the first place. If one has agreed to serve, the committee must be made aware of the extenuating circumstances that lead to this decision.

For the Head or Awards Committee

The importance of other factors besides grade point average is often critical. Among these is the departmental ranking and comments from the Head or designate (normally an awards committee of some sort). To ensure that students are not disadvantaged in the competition, it is very important for the information provided by the Head to be as detailed and informative as possible. In the past, the information provided has often been little more than a capsule summary of comments made by the referees. While it is sometimes difficult to provide a lot of additional information (especially for new students), any such information is very valuable as it comes from a separate source.

The Head’s recommendation can also provide information that the referee’s letter cannot. In particular, it is very useful to explain why the Head or awards committee ranked the applicants in the order they did. It is not uncommon for the UMGF evaluation committee to independently rank applicants from a department in a slightly different order, and this divergence can be a cause for concern. Statements such as “This applicant was ranked ahead of applicant X because…” are very valuable. Sometimes the head can also provide information about the value of a candidate’s experience, etc. that a referee may not be able to provide. This may happen when a referee is external or new to the University of Manitoba.

Finally, it is extremely important for the Head or awards committee to do a good job of describing the “departmental norms” provided to the Faculty of Science UMGF adjudication committee prior to the adjudication. There is significant variance in expectations for students in different departments in terms of publications and experience, etc. There are also real differences within departments for MSc and PhD programs. Expectations for “typical” excellent MSc and PhD students need to be clearly articulated so that the evaluation committee can fairly compare and ultimately rank the applicants. As in the recommendations to referees, the heads/awards committee members are strongly encouraged to provide evidence supporting their comments (e.g. “why a student is brilliant) and to cover as many criteria for evaluation as are practically possible.

Other Issues Affecting Adjudication

There are also some issues that affect how readily the UMGF evaluation committee can perform the adjudication. These typically correspond to certain “special cases”. Addressing these issues might be done by the applicant, referee and/or the head.

  • Comments about the quality of various publications and appropriateness of venues.
  • For recruitment cases, an argument for why the student is good and why his/her previous institution(s) are good.
  • Distinguishing between new MSc and in-progress MSc, and new PhD and in-progress PhD as it relates to departmental norms.
  • Addressing any abnormalities in grades, publication record, etc.
  • Ensuring that (particularly) external referees provide sufficient detail in their reports.
Pic 1

Steve Whyward, Biological Sciences
Battling Mosquitoes and the West Nile Virus


Pic 2

A fluorsecence image of non-biological material:  science meets art in microscopy imaging. Photo courtesy of Erwin Huebner, Biological Sciences


Pic 3

A color 3-D projection of the DNA contained in nuclei of the female germ tissue in an ovary of a marine worm Ophryotrocha labronica. The high colorful peak shows the DNA amount in a nurse cell as compared with the low blue flatter one of the oocyte.  Photo courtesy of Erwin Huebner, Biological Sciences.