Tag: peer evaluations

Evolution of a Group Research Project
1-1-1, Assessment, Best Practices, Research, TLT

Faculty Guest Post: Evolution of a Group Research Project

Jessica Smith

Today’s Faculty Guest Post is from Chris Mothorpe, Assistant Professor of Economics.  Chris attended TLT’s 2015 Faculty Technology Institute.  In this post, he reflects on the process of revising and improving a group research project in two of his courses: Urban Economics and Economics of Geography and Transportation.  This is an excerpt from Chris’ own blog.  To read the entire post, please visit: https://sites.google.com/site/chrismothorpe/home/group-research-projects

I am writing this blog post based on my experience conducting research projects in my upper level economic classes over the past three semesters. This post will not discuss the research project in its entirety; instead, it will provide a general overview of the project and then focus on specific challenges I have faced each semester and different strategies I have employed (or I am planning on employing to overcome them).  There are two main challenges I will discuss: 1) group formation; and 2) peer evaluations.

Project Overview

I decided to require a group research project after reading several magazine and newspaper articles discussing what companies are looking for in college graduates.  Atop many of the surveys were not the hard-technical skills taught in the classrooms, but many soft-skills developed in the non-academic, extracurricular setting.  These soft-skills include: 1) leadership; 2) ability to work in a team; 3) written communication skills; 4) problem solving skills; 5) work ethic; 6) verbal communication skills; 7) initiative; 8) interpersonal skills; 9) creativity; and 10) organizational ability.  Conducting a group-based research project provides students the opportunity to practice many of these skills — practice they would otherwise not receive if the class is taught in a more traditional manner.   A second motivating factor is to allow the students the opportunity to apply economic models to real world problems.

I decided to require a group research project after reading several magazine and newspaper articles discussing what companies are looking for in college graduates.  Atop many of the surveys were not the hard-technical skills taught in the classrooms, but many soft-skills developed in the non-academic, extracurricular setting.  These soft-skills include: 1) leadership; 2) ability to work in a team; 3) written communication skills; 4) problem solving skills; 5) work ethic; 6) verbal communication skills; 7) initiative; 8) interpersonal skills; 9) creativity; and 10) organizational ability.  Conducting a group-based research project provides students the opportunity to practice many of these skills — practice they would otherwise not receive if the class is taught in a more traditional manner.   A second motivating factor is to allow the students the opportunity to apply economic models to real world problems.

The stated objectives for the research project are:
  1. Analyze a contemporary economic issue or social issue using economic theory and models
  2. Demonstrate versatile and competent written, oral and digital communication skills
  3. Evaluate communication situations and audiences to make choices about the most effective ways to deliver messages
  4. Appraise written communication skills through self and peer evaluations
  5. Manage diverse teams successfully

The project is set up as a paper submission to the (fictional) Charleston Journal of Economics, which I reside over as Editor.  At the beginning of the semester, I pass out the Fall/Spring 20XX Charleston Journal of Economics (CJE) Request for Papers (RFP), which contains the objectives of the journal, the strategic areas, scoring criteria, formatting requirements, and examples of correctly formatted submissions. Throughout the semester, groups are required to submit portions of their project to the Editor and receive feedback (in the form of a letter from the editor). I have required the research project in the Spring of 2015, the Fall of 2015 and the Spring of 2016.  These three iterations have proven valuable as I continually update the project to improve on its effectiveness and efficiency in delivery.

Group Formation

In the first iteration (Spring 2015) of the research project, I allowed each student to write his/her own paper and choose any topic as long as it was related somehow Urban Economics.  While allowing each student the opportunity to write their own research paper provides the best learning opportunity for the student (since he/she receives individualized feedback), it is much harder (time consuming) on me. I realized that there were three main consequences to allowing students to complete their own project:
  1. Grading fatigue
  2. Increase time until work is returned to students
  3. Grading research projects detracts from other activities such as research

In the second iteration (Fall 2015), I switched from individual research projects to group based projects.  I allowed the groups to form endogenously — students selected their own groups.  Each research group was required to have 3-4 individuals.  The main problem that arose from students selecting their own groups is that the groups were not interdisciplinary in nature.  For example, Group A consisted of three Transportation and Logistics Majors.  One of the comments Group A received on one of their drafts was that their paper lacks a sufficient economic model.  The feedback I received from Group A was that there is not a economic major (or minor) in the group, and as a result no one is familiar with economic models.

In the second iteration, I also began restricting the topic selection by requiring each group’s research question to at least fall within one of the strategic areas of the Charleston Journal of Economics.  The strategic areas are:
  1. Transportation Infrastructure
  2. The Port of Charleston Expansion
  3. Coastal Community Resilience and the Impacts of Sea Level Rise/Climate Change
  4. The Long Savannah Development

In the third iteration (Spring 2016), I attempted to correct for the lack of interdisciplinary majors within a research group by assigning research groups.  To aid in the assignment of research groups, each student completed an Oaks quiz that asked the following questions:

  1. List the strategic areas in order of greater interest to least interest
  2. For your top ranked strategic area, list keywords of interest
  3. For your second ranked strategic area, list keywords of interest
  4. List your major(s)
  5. List your minor(s)
  6. List individuals you would like to work with

Students submitted their responses via an Oaks quiz and then I used their responses to assign groups.  Matches were made based on strategic areas and keywords; however, not all students receive their top ranked strategic area (most did) as I also sought to ensure that each group contained at least one each major or minor.  This mechanism worked well in solving the interdisciplinary problem previously encountered; however, the new problem that arose was that group members wanted a greater say about who was in their group as the “Free-Riding” problem arose in several groups.  The Free-Riding problem occurs when not all members contribute equally to the project, yet all group members receive the same grade.  Of the 8 research groups in the Spring of 2016, at least 4 registered complaints about one of their group members not contributing.

The Free-Rider Problem

I am planning on implementing two strategies to attempt to mitigate the Free-Riding Problem.  First, I plan on introducing a mechanism that will allow students to reveal information about themselves (e.g. work ethic) to other members in the class.  This mechanism is a series of group-based homework problem sets in the first few weeks of class and before the assignment of groups.  Groups will be randomly assigned.  The random assignment of groups will ensure that students are meeting and learning about other members of the class.  After the problem sets, students will again be asked to complete an Oaks quiz, but on their quiz there will be additional questions aimed at revealing their preferences for who they do and do not want to work with.

 

The second strategy is to have students submit peer evaluations of their group members when assignments are due.  A portion of the peer evaluation is a Grade Multiplier.  Each member of the group assigns every other member of the group a multiplier, which gives each group member control over every other group member’s grade.  The purpose of the multiplier is to provide incentive to group members to work hard towards the completion of the project.  In the Spring of 2016, I required the students to submit Peer Evaluations at the end of the semester; however, this did not provide strong incentives to students since at the time of submissions final class grades were almost known.  It was recommended to me, by a student, to conduct the peer evaluations more frequently.

 

Peer Evaluations are a useful tool that provide students with information on their performance over the course of the research project.  Since the goal of the project is to aid students in developing soft skills, the peer evaluations are particularly effective, since they address each student individually.  Herein lies the main problem since each time I require a peer evaluation I cannot write 20-40 individual letters commenting on their performance.  The remainder of this blog post discusses the tools I have developed to create individualized letters based on peer reviews in an (semi) automatic fashion.  Creating letters in this manner allows me to provide individualized feedback to students while at the same time not spending hours drafting letters.

 

The letter-creation process requires the following programs/files:
  1. The Form Letter – Microsoft Word Template
  2. Oaks Quiz and Excel File of Modified Data
  3. Microsoft Word Template File
  4. Microsoft Excel Template File
  5. Microsoft Excel Addin ExcelToWord

The procedure behind the automated process is to have students complete their peer evaluations through an Oaks quiz, text-mine their responses, and populate a form letter with student responses.  Note that this process relies on student responses on the peer evaluation but does leave open the possibility of directly editing the individualized letters.

[TLT Note: On his own blog, Chris provides instructions for using OAKS, Microsoft Word, and Microsoft Excel to facilitate the peer evaluations described above.  He also provides templates and examples. To access this information, please visit  https://sites.google.com/site/chrismothorpe/home/group-research-projects]

In this blog, I have discussed the research project that I conduct in my upper level economics classes, two of the challenges that have arisen, and various strategies I have or will employ to overcome the challenges.  To overcome group formation problems, I am employing an Oaks quiz and group based homework assigned in order to allow students the opportunity to reveal information about themselves to other students in the class as well as myself.  To overcome the “Free-Riding” problem, I am planning on employing a series of peer evaluations, which gives all members in the group some control over the grades of the other group members.

One key to conducting peer evaluations is returning individualized feedback to the student based on their performance.  I have also discussed a set of tools which will enable me to create individualized letters in a timely manner.  Providing timely and individualized feedback also enhances the learning outcomes of the research project since the project is geared towards student practice of their “soft” skills.  Receiving individualized feedback allows students to learn from their experience and develop a stronger set of skills that they can employ in the future.