How Pilots Work

When a new learning technology enters a formal pilot phase at UBC, it is released and supported in specific courses for a trial run, typically lasting one to two terms. After this trial run completes, the LT Hub gathers feedback from applicable users of the technology to summarize the pilot in an official report. Details of this evaluation process and outcome are provided below.

Evaluation Process

A typical pilot includes a limited number of courses and three major groups of users: instructors, teaching assistants, and students. For each course, the evaluation team solicits feedback to hear how these groups perceived the technology itself and its usefulness for teaching and learning in their specific context.

Most often, feedback is gathered in the following ways:

  • Instructors:
    • Provide context and intended use case through written description or a half-hour interview before or soon after the start of the pilot
    • Participate in an hour-long reflective interview approaching or at the end of the pilot
  • TAs:
    • Participate in a half-hour reflective interview or 45-minute focus group session approaching the end of the pilot
  • Students:
    • Participate in a class-wide survey near the end of the course
    • Optionally participate in an 45-minute focus group session sometime during the course

Expectations of instructors piloting a tool

When an instructor’s course or courses are included in a formal pilot, the instructor should be prepared to assist the evaluation team in five main ways:

  1. Provide details of their context(s) and use case(s) for the technology
  2. Distribute the evaluation student survey (online or paper-based) in each course at an appropriate time
  3. Actively encourage student participation in the survey and (if included) related focus group
  4. Connect any TAs for the course(s) with the evaluation team to enable coordination of TA feedback
  5. Set aside an hour of their own time for a reflective interview near the conclusion of the pilot

Final Outcome

All information LT Hub gathers for the pilot is summarized in an official evaluation report. This report is shared with learning technology governance groups and forms one piece of a larger assessment they undertake in making a decision about campus-wide support for the piloted technology.

The report does not make any recommendation regarding the adoption of the technology at UBC nor make any claims regarding the technology’s overall effectiveness. The report’s scope is limited to summarizing how people perceived using the technology during the pilot period—focusing on strengths, weaknesses, and best ways to implement.

Once the groups review the report and decide on an outcome for the piloted technology, the decision is made available to the public through the LT Hub website (the report may be available upon request).

Secondary use of data

Instructional teams can use raw data collected from evaluation student surveys and focus groups in their specific course(s) for their own purposes. However, LT Hub will not normally seek BREB approval as part of this evaluation process, so anyone wishing to apply the data to wider research goals may need to consider submitting an application, ideally prior to data collection. Feel free to contact Ido Roll at CTLT for additional information and advice.