Assessment of learning outcomes in higher education has become a significant driver in undergraduate education nationally but “rigorous, thorough, and holistic reviews of doctoral programs are uncommon” (Golde et al., 2006, p. 58). While graduate programs subject to accreditation have consistently adhered to agency assessment mandates, doctoral programs across the board are still struggling with competing perspectives on the efficacy of assessment methodologies. The overarching perspectives fall into two areas of concern: (1) the applicability of current trends in assessment to doctoral programs in light of the unique and sacrosanct student–advisor relationship (Gross, 2002); and (2) quantitative data being viewed as the most reliable means of assessing a doctoral program, that is, graduation/attrition rate and average time to degree as opposed to qualitative and student self-generated data. Assessment challenges are even more acute as graduate programs move to multidisciplinary training that encompasses a variety of disciplines and their different academic cultures, vocabularies, values, and traditions.
Team-based multidisciplinary research and graduate education is now becoming the rule rather than the exception in developing practical solutions to contemporary problems in environmental science (National Research Council, 2004; Lawrence and Després, 2004). The complex relationships between human systems and natural systems are widely recognized and frequently serve as the basis of natural resource management policy and planning (Liu et al., 2007). For example, many emerging environmental challenges occur at spatial and temporal scales that demand a multidisciplinary approach to understand causes, consequences, and opportunities for mitigation. Climate change, water shortages, habitat fragmentation and loss, and the spread of invasive species are but a few examples of contemporary environmental issues that seriously impact ecological and social systems and require multidisciplinary approaches to develop practical solutions. For example, habitat loss can result from natural causes such as fire, wind, disease, and flooding. But in many parts of the world, habitat loss is principally driven by land use change due to human activities (August et al., 2002). The drivers of land use change are diverse and include political factors, economic opportunities, and changing social values (Agarwal et al., 2002). While respecting the subtle differences between and among the concepts of trans-, cross-, inter-, and multidisciplinary (Jacob, 2008), for the purposes of this article we use the word “multidisciplinary” when referring to research beyond the disciplinary base of an individual. We have chosen this term because multidisciplinary research typically represents the first foray out of single discipline science by a doctoral student. Multidisciplinary inquiry can eventually mature into a truly integrative interdisciplinary research approach.
Although multidisciplinary integrative research is frequently the basis for transformative changes in how complex problems are addressed, there remain barriers to multidisciplinary research in an academic setting (Lin, 2008; Younglove-Webb et al., 1999; Benda et al., 2002). There is a growing literature on how training in multidisciplinary research is best organized. Considerations include the desired behavior of students and scientists involved in such research and methods to optimize program effectiveness (Morse et al., 2007; Rhoten and Parker, 2004). It is not surprising that some of the nation's foremost research universities are reshaping themselves to better engage in multidisciplinary inquiry (Lawlor, 2008); however, there is not commensurate growth in assessment tools to judge the efficacy of multidisciplinary doctoral programs (Maki and Borkowski, 2006). The traditional pedagogy for training graduate students is based on a discipline–centric relationship between student and advisor within a focused community of peers consisting of fellow students, faculty, postdocs, and research technicians, all of whom are working in areas of the same discipline. There is no question that this system works. Academic institutions worldwide have produced many generations of accomplished scientists who have advanced discovery in disciplinary science. In fact, it is the practice of this discipline-based method that has led to the evolution of multidisciplinary scientific inquiry as essential to the development of research questions, methods, and syntheses that address this complexity.
There is a growing recognition among university administrators and faculty, as well as graduate students and other potential employers, that graduate training needs to expand to embrace multidisciplinary approaches. Granting agencies such as the National Science Foundation (NSF) have developed transformative funding programs to spawn innovation in training multidisciplinary scientists. As LaPidus (1998, p. 102) notes, “doctoral study is an educational experience designed to prepare students for a variety of roles and responsibilities… This means more than simply adding on components; it requires examination of the basic purpose and goals of doctoral education.” The flagship program in the NSF for the expansion of multidisciplinary doctoral education is the Integrative Graduate Education and Research Traineeship (IGERT) program (Brown and Giordan, 2008). The rigorous assessment of program effectiveness is an important requirement by most of the agencies funding multidisciplinary research training. Working in multidisciplinary settings requires skills that are not part of the traditional single discipline-based pedagogy. Such a skillset includes: communication with peers from other disciplines—including an understanding of distinct disciplinary lexicons; conflict resolution; ethics and social justice; teamwork and leadership/followership; and the capacity to recognize and integrate different disciplinary elements of a problem (Broussard et al., 2007). Thus, multidisciplinary doctoral programs must be served by a variety of assessment methodologies that can be applied and understood across disciplines (Jacob, 2008).
Multidisciplinary graduate education and research are challenging the academy to develop assessment tools that speak to increasing complexity. Golde et al. (2006, p. 54) acknowledge that “sustained self examination leading to action is unusual and difficult” and they advocate a holistic approach to doctoral program assessment entailing “regular self-examination of the doctoral program by faculty and students.” Doctoral programs have established a traditional suite of assessment instruments to evaluate individual student development and these are typically administered by the advisor, examining committees of graduate faculty, or academic departments and programs (Borkowski, 2006). Benchmark assessment events include qualifying exams, comprehensive exams, and critical defense of dissertation research. Furthermore, doctoral programs have relied on a traditional suite of system-wide metrics to evaluate program efficacy (Golde and Dore, 2001). The most common quantitative measures include retention/attrition rates and time to graduation. However, these traditional metrics do not speak to the heart of outcomes assessment of doctoral programs, that is, the ongoing improvement of the program as it relates to the evolution of the student from “senior learner” to “junior colleague” to “disciplinary steward” (Walker et al., 2008). Further, the emphasis on working in multidisciplinary settings with multiple perspectives and lenses of disciplinary bias makes rapid assessment and resulting adaptation of curricula and research all the more imperative.
The University of Rhode Island's Coastal Institute leads an NSF IGERT-funded initiative to teach and engage doctoral students in multidisciplinary research in coastal ecosystem science and management (herein referred to as the Coastal Institute IGERT Project, CIIP). We developed an assessment tool to evaluate progress in multidisciplinary learning based on the concept of T competency (Reis, 2001). This tool involves collecting a combination of both qualitative and quantitative data through a simple exercise with each student. The objective of this article is to present the T tool and demonstrate how the data it generates are used by both students and faculty to evaluate student progress in our program.
T Competency to the T Tool
The term “T competency” was originally coined by Professor Richard Zare of Stanford University (Reis, 2001) in his reflections about multidisciplinary work, where “…disciplinary depth is the vertical bar of the ‘T’ and cross-disciplinary proficiency is the horizontal bar.” We have found the T to be a powerful metaphor for discussing and evaluating multidisciplinary vs. single discipline-based research and learning. In this article, we present a formal definition of the T metaphor as it has evolved from T competency, offer a tool to quantitatively measure an individual's T, and demonstrate how the T tool can be used to assess student progress in multidisciplinary graduate education.
As noted by Zare, the vertical element of the “T” represents disciplinary endeavors; the horizontal element of the “T” represents multidisciplinary engagement (Fig. 1). A highly multidisciplinary scientist will have a relatively broad horizontal in his or her T. A disciplinary scientist that does not work outside his/her field would have a deep T with a narrow horizontal component. In practice, adding dimension to one axis (vertical or horizontal) requires reduction of the other. This reflects practical limitations; time invested in working on disciplinary pursuits is unavailable to allocate to multidisciplinary projects and the converse. In addition, one's T is dynamic and its proportions would be expected to shift over time.
We do not use the T tool to evaluate skill level or depth of knowledge, which are assessed through other direct means such as comprehensive examinations, class grades, peer-reviewed publications, conference presentations, and so forth. Certainly these evaluative tools and activities are reflected in the building of an individual T; however, the purpose of the T tool is to capture the depth and breadth of one's intended focus and the dynamic representation of one's achievements over time. We recognize that students and faculty wrestle with the distinction between intention (as an internally driven goal) and achievement (as assessed both internally and externally). In the CIIP, we use a suite of tools to assess student progress and the effectiveness of the program and its faculty in meeting CIIP's overall learning goals: rubrics for learning outcomes assessment within each course and for the program as a whole; ongoing faculty and peer evaluation of student writing; presentation and leadership training; surveys that gather both quantitative and qualitative data about student performance in a course (both self-assessment and faculty assessment) as well as student evaluations of course effectiveness. The T tool complements these other assessment tools.
Materials and Methods
We have developed an exercise that requires students entering our program to create Ts representing their perceived current and aspirational future balance between disciplinary and multidisciplinary endeavors. The exercise involves asking them to allocate 20 blocks on the vertical and horizontal axes of the T that represent their perception of the shape of their current T. We also ask that they concurrently build an additional T that reflects their aspirations 5 to 10 years after attaining their doctorate. Each of the 20 blocks represents a unit of perceived interest, knowledge, and capability in disciplinary or multidisciplinary research. A T with 10 blocks vertically and 10 blocks horizontally represents an even balance of disciplinary and multidisciplinary interests. Tall, narrow Ts represent highly disciplinary interests and short, broad Ts represent highly multidisciplinary interests. Every student in the CIIP revisits his/her T annually. This process yields a suite of simple, quantitative metrics (Table 1) that allow us to capture current and aspirational allocation of time and emphasis in disciplinary and multidisciplinary focus among our students. In addition, the T tool provides qualitative data as students engage in self-reflection and articulate the rationale behind their T allocations. Aspect ratio (i.e., breadth divided by depth) simplifies the metaphor of the T into a single metric (Table 1). When students build a T that increases in aspect ratio (broadens) when compared with previous Ts, this demonstrates their perception of an increased ability to perform multidisciplinary research. Conversely, students whose T aspect ratios decrease demonstrate their perception that they are focusing on enhancing depth in their scientific discipline. Neither outcome is viewed as a value judgment; rather, it serves to provide students and faculty with a clear sense of individual goals and an ability to measure progress in meeting those goals. The T metaphor supplies students with a vocabulary and nomenclature to identify current learning profiles and develop career goals as well as the opportunity for regular self-reflection. The importance of both self-reflection and reporting and discussion of same is reinforced by Wulff and Nerad (2006): “Because doctoral education is a socialization process that can lead to either persistence or attrition, it is particularly important that we understand how students are experiencing the process as they proceed through it.” This point is further reinforced by Aanerud et al. (2006) when they say that it is “…the Ph.D. recipients who are in the best position to judge the relevance and value of their doctoral education in relation to their careers and lives.” We have elected to focus on the T tool as one methodology to enhance a more rigorous process of self-reflection. Furthermore, by gathering more data on how students are experiencing the program we can more readily design improvements as the program evolves.
|H||number of blocks in the horizontal dimension of the T|
|V||number of blocks in the vertical dimension of the T|
|HV||aspect ratio of the T. Measured by dividing H by V. HV > 1 trends toward multidisciplinarity; HV < 1 trends toward a single disciplinary focus|
|ΔHV||change in the future aspect ratio relative to the present (ΔHV = HVfuture/HVpresent). ΔHV > 1 indicates a change toward increasing multidisciplinarity. ΔHV < 1 indicates a change toward greater disciplinarity.|
We do not yet have a full complement of longitudinal data on student Ts to report on use of this tool as a long-term assessment instrument. However, to examine the temporally dynamic nature of the T, we asked URI faculty (n = 20) to construct T profiles from the 20 blocks to represent their engagement in multidisciplinary science: (1) in the present; (2) when they were graduate students; and (3) 5 to 10 years post-Ph.D. Half of the faculty surveyed is directly involved in the CIIP. The other half of the faculty is from similar disciplines (natural sciences, oceanography, policy, and environmental economics) but not directly involved in the CIIP pedagogy. Faculty disciplines were selected to match CIIP student disciplines.
We performed simple statistical analyses on the results that we obtained from the 20-block allocation exercise that was administered to 23 students in our program and 20 faculty. We present these results to show how the T tool can be used to quantitatively assess how students perceive their level of knowledge and interest in multidisciplinary learning. A more robust analysis will be done in the future when our sample size of students is larger. Central tendency and variation are reported as means and one standard error, respectively. Differences in mean values between groups were tested using two-sample or matched pairs t-tests, equality of variances between groups was evaluated using the F-max test, and determination if aspect ratios (HV—horizontal breadth divided by vertical depth, Table 1) significantly departed from 1 were done using one-sample t-tests (Zar, 1999). All statistical analyses were done using SAS 9.1 software (SAS Institute, Cary, NC).
Results and Discussion
Graduate student Ts for the present and future show a significant emphasis on multidisciplinarity (Table 2). The average T for students early in their doctoral training had twice the emphasis on multidisciplinary interests than disciplinary (mean HV = 2.0). Students aspire to have future Ts that tend to be more disciplinary than their present Ts, but the difference between present and future Ts is not significant (t = 1.0, n = 23, P = 0.29). However, there is significantly greater variation in student perceptions of their current Ts than in their future Ts (F = 6.5, n = 23, P < 0.001). Our assessment of future vs. present Ts is based on the mean and variance of the difference between each student's present and future HV. It is not based on simply subtracting the mean of the future HV from the mean of the current HV.
|Group||HV now||HV future||ΔHV|
|All Students (n = 23)||2.0 (0.5)||1.5 (0.2)||1.0 (0.1)|
|Probability HV = 1||0.04||0.02||0.73|
|Female (n = 16)||2.1 (0.7)||1.4 (0.2)||1.0 (0.1)|
|Male (n = 7)||1.8 (0.4)||1.5 (0.3)||1.1 (0.3)|
|Probability genders are the same||0.71||0.80||0.84|
|Probability variances are the same||0.04||0.88||0.58|
|Natural vs. social scientists|
|Natural (n = 16)||1.4 (0.2)||1.4 (0.2)||1.2 (0.2)|
|Social (n = 7)||3.6 (1.4)||1.7 (0.5)||0.6 (0.1)|
|Probability disciplines are same||0.16||0.42||0.02|
|Probability variances are the same||<0.0001||0.06||0.02|
The comparison of T profiles between male and female students proved to be similar (Table 2). Women tended to be more variable in describing their present Ts than men. Students from the social sciences (e.g., planning, policy, and environmental economics) showed broader present Ts compared with students from the natural sciences (e.g., oceanography, ecology, natural resources science, and fisheries). Students from the natural sciences showed significantly less variation than their counterparts from social sciences in describing their present Ts. There was no difference in the average shape of post-graduate Ts between natural and social science students. Natural science students aspire to increase emphasis on multidisciplinary research (mean ΔHV = 1.2), whereas social science students aspire to develop greater disciplinary depth (mean ΔHV = 0.6).
Faculty Ts showed a number of interesting patterns. The faculty delivering our IGERT program and the major professors of IGERT Ph.D. students had current Ts that were nearly identical (mean current HV IGERT faculty = 0.9, SE = 0.1, n = 7; mean current HV major profs = 0.9, SE = 0.04, n = 6), whereas faculty not affiliated with the IGERT program had Ts that were significantly more disciplinary (mean current HV = 0.6, SE = 0.1, n = 7; t = 2.3, df = 12, P < 0.05). Clearly, faculty engaged in the IGERT pedagogy have a measureable bias in supporting multidisciplinary research. Reflecting back upon their time as graduate students, all faculty as a group considered themselves highly disciplinary when they were in their Ph.D. programs (mean HV = 0.7, SE = 0.2) and this differed significantly (t = 2.4, df = 41, P = 0.02) from our current students who show a strong emphasis in multidisciplinary interests (mean HV = 2.0, SE = 0.5). These same faculty viewed themselves as significantly less multidisciplinary in the period 5 to 10 years after receiving their Ph.D. degrees than later in their careers (t = 2.7, n = 13, P < 0.05; Fig. 2). Many faculty in our sample noted that this pattern corresponded to the time in which tenure and promotion decisions are made. Given the difficulties inherent in assessing productivity in multidisciplinary research (Lawrence and Després, 2004), it is not surprising that faculty tended to invest significantly more time in developing their disciplinary strength at the tenure and promotion-seeking phase of their career. However, this could also reflect greater acceptance by the scientific community of multidisciplinary research now as compared with past decades when faculty were students or in the early stages of their careers.
Our intent in developing and using the T tool is to help us and our students state learning goals, track progress and program effectiveness, as well as to respond to student feedback through curricular modification, that is to say “closing the assessment loop” (Maki and Borkowski, 2006). Although our sample sizes are small and limit our ability to draw strong inferences from the quantitative results we have obtained so far, the data do suggest a number of interesting emerging patterns.
Our CIIP students are very committed to multidisciplinary research. This is not surprising since they sought out and were accepted into a highly competitive NSF IGERT program that emphasizes multidisciplinary scholarship. They do, however, recognize that disciplinary strength is a necessary prerequisite to participating in multidisciplinary research for most scientists.
Different disciplines show different patterns of present and future interests in multidisciplinarity. Natural science students and their social science counterparts had different T profiles as students and how they aspire to change their T profiles (ΔHV) from present to post-Ph.D. Social science students hope to enhance their disciplinary depth whereas natural science students hope to increase their multidisciplinary breadth.
Faculty perceptions of their Ts when they were graduate students are much more disciplinary than current students. Faculty Ts were extremely disciplinary at the time in their career when tenure and promotion decisions are made. Faculty become more multidisciplinary later in their careers.
The T metaphor provides a simple nomenclature that allows us to discuss the balance between disciplinary and multidisciplinary education and research.
While the traditional student–advisor relationship is designed to develop disciplinary depth (the vertical of the T), the CIIP curriculum is designed to add breadth to the horizontal component of a student's T, a goal common to all NSF IGERT programs. Students completing the CIIP curriculum continue to work on the vertical component of their Ts with their advisor with an awareness of multidisciplinarity that serves to inform their research and broaden their views both conceptually (e.g., problem solving) and practically (e.g., career opportunities). Rather than challenging the traditional student–advisor paradigm, the CIIP has endeavored to create an intellectual community comprised of students and faculty—including student advisors—from a variety of disciplines. This design speaks to the findings of Walker et al. (2008), who argue that creating such a community is essential for the development of new ideas and intellectual risk taking, a key component of multidisciplinary research. The management of such a diverse learning environment can be daunting and could contribute to attrition rates, a serious concern of graduate education (McAlpine and Norton, 2006). By regularly engaging in the T exercise, students are encouraged to consider more deeply their goals and the specific steps necessary to achieve those goals, thereby providing some clarity as they navigate the world of multidisciplinary research. From the faculty's perspective, this insight improves our ability to deliver high quality multidisciplinary training through the identification of students’ self-perceived academic needs. While the applicability of the T tool will vary in specifics from program to program, our experience with the CIIP is that the T tool adds significant value to the assessment of our doctoral training.
Multidisciplinary doctoral programs attract students whose aspirational Ts are likely to be broader than their disciplinary-centric peers. By consistently gathering data (e.g., T dimensions, Likert surveys) on how a multidisciplinary doctoral program is helping students to develop skills characteristic of a successful multidisciplinary researcher (broad T), a program can learn where it is succeeding and where further adjustment is needed. In the CIIP, students have provided myriad curricular innovations that have strengthened the overall program. For example, the first iteration of our fall seminar attempted to cover five topics in a multidisciplinary framework. The students were empowered to design the second iteration of this seminar and elected to reduce the number of topics and increase the breadth and depth of each topic. In addition they have requested supplementary workshops to build additional skills, ranging from work/life balance to logical argumentation to grant writing. In this way students are active participants in the growth of a successful multidisciplinary doctoral program, furthering their development from “senior learner” to “junior colleague” (Walker et al., 2008).
The expansion of multidisciplinary doctoral programs is challenging academic institutions to devise new ways to collect and interpret useful assessment data from program participants, allowing for rapid and effective program adjustments. Student self-assessment has traditionally been viewed as subjective and therefore less valuable than other methods of assessment, especially those that generate quantitative metrics that can be tracked over time. However, it has become clear that student self-assessment is a crucial component of the assessment process, benefiting both student and program (Maki and Borkowski, 2006). Our T tool is especially useful for us because it permits the student to exercise reflective self-assessment, yet generates a quantitative result that can be tracked and compared with other students and categories of students.
We use a range of methodologies, both direct and indirect, to assess the efficacy of our multidisciplinary doctoral program. The T exercise described here is an assessment tool that not only generates quantitative data, allowing for longitudinal tracking of students’ perceptions of their progress and goals, but also yields valuable qualitative data arising from the act of the students’ deliberation of their T relating to the program and their individual academic choices and experiences. While the intention of this article is to focus on the value of the quantitative data, qualitative data are nevertheless exceedingly valuable to us in that they provide students and faculty with (1) nomenclature to describe and discuss the balance between disciplinary and multidisciplinary work and (2) an additional tool to evaluate the efficacy of program elements in serving the program's articulated goals. Revisiting the T tool on a regular basis provides grounding to students in the midst of a vast array of influences and opportunities and “closes the loop” by providing the program with regular feedback and encouraging students to contribute to the metamorphosis of the program based on that feedback. This type of direct engagement in self-reflection and program improvement will continue to serve them beyond the completion of their doctorate regardless of their chosen career path.