Research at ISU

Investigating the effectiveness of IADE (Intelligent Academic Discourse Evaluator)

The proposed study will attempt to bring together the fields of automated evaluation, academic writing pedagogy, and CALL, considering important issues that need to be addressed in these areas. Specifically, it intends to account for the scarcity of research on the formative potential of Automated Essay Scoring for non-native speakers of English, the problem of lack of plenteous practice and individualized instruction in academic writing, and the insufficiency of empirical analyses of potency of CALL programs and materials. The purpose of this study is evaluate the effectiveness the ICALL program, named IADE (Intelligent Academic Discourse Evaluator), by investigating its capability to enhance the formative assessment aspect of international graduate students’ academic writing instruction. To achieve this goal, the study will be guided by an explicit CALL evaluation conceptual framework (Chapelle, 2001), and evidence will be gathered about all the CALL qualities outlined in this framework, namely, language learning potential, learner fit, meaning focus, authenticity, impact, and practicality. A mixed-methods approach with a concurrent transformative strategy (Creswell, 2003) will be employed, which will allow for the integration of quantitative and qualitative data both during the analysis and the interpretation phase.

Learning to write academically with technology (Spring and Summer 2008)

Pedagogues have long been struggling to find effective ways of helping non-native speakers (NNS) develop their academic writing skills. Various pedagogical approaches have been attempted to achieve this goal (Cargill et al., 2001; Levis & Muller-Levis, 2003). Since the writing instruction for NNS students should “help initiate writers into their field-specific research communities” (Frodesen, 1995, p. 333) combining language and discourse with the skill of writing within professional norms (Kushner, 1997), issues of genre-specificity and disciplinarity have largely been addressed by corpus implementations (Cortes, 2006; Flowerdew, 2005; Lee & Swales, 2006, Swales, 2004). These, however, are generally limited to the use of concordancing software. Notwithstanding the voluminous literature highlighting the advantages of concordancers, new research is needed to explore the impact of innovative technologies on the development of NNS academic writing skills.
This purpose of this classroom-based study is to investigate the potential of computer-based applications implemented in a corpus- and genre-based academic writing course that combines a top-down approach to genre analysis and a bottom-up approach to corpus analysis. The overall students’ task was to examine organizational and linguistic patterns characteristic of their particular discipline and apply them when writing a research article. Two groups of NNS graduate students were subject to different types of technological treatment – using a concordancing program (WordSearch) and an annotation tool (Callisto), which aimed at enhancing students’ noticing of linguistic and discourse conventions in the discipline-specific input. Data from pre- and post-treatment questionnaires, semi-structured interviews, observations, analytical retrospective reflections, and student texts provided insightful evidence on the positive impact of the application that allows students to work with annotated corpora. This study not only greatly contributes to an improvement of corpus-based teaching methodologies; it also provides a framework for language learning and technology research of evaluative nature.

An analysis of academic discourse structure: Research article introductions (Fall, 2007)

Over the past few decades, written academic discourse has witnessed a considerable interest on the part of the Applied Linguistics community. Much research focused on research articles (RA) (Hyland, 2000), characterizing the macro-organization of this genre into abstracts (Samraj, 2005), results (Brett, 1994; Williams, 1999), discussions (Holmes, 1997; Hopkins & Dudley-Evans, 1988), and discussions and conclusions (Yang & Allison, 2003). Due to Swales’ seminal work on the move/step structure of research article introductions (Swales, 1981, 1990) these sections have been investigated the most (Ozturk, 2007; Samraj, 2002; Swales & Najjar, 1987). However, the studies conducted in this vein cannot inform the design of an ICALL academic discourse evaluation program. First, their analyses of variation within or across disciplines are based on a small collection of samples, which limits the degree of possible generalizations. Second, the semantic complexity of the discourse is overlooked; typically, the move or step that appears to be more salient is selected, and that affects the accuracy of results and the reliability of interpretations. Third, there is no conclusive evidence on which lexico-grammatical features signal the presence of a certain move/step. To address these limitations and to provide solid evidence for ICALL design, NLP-based approaches need to be employed.
This study is the beginning of a bigger research project, which aims at designing and evaluating the effectiveness of an ICALL program that will provide RA discourse related formative feedback. At this initial stage, the focus falls on identifying lexical means that indicate rhetorical shifts in RA introductions. To achieve this goal, a corpus of 400 RA introductions (1,322,089 words) representative of 20 academic disciplines was created and annotated in terms of Swales’ (1990, 2004) framework for moves and steps. Annotations were done at sentence level, each sentence being assigned a move and a step within that move as specified in the mark-up scheme. The scheme allows for multiple layers of annotation for cases when the same sentence contains more than one move or more than one step. This makes it possible to capture all the semantic shades rendered by a given sentence. Lexical means likely to signal a discourse move/step were found by (1) extracting unigrams, bigrams, and trigrams from the annotated corpus; (2) removing most frequent uni-, bi-, and trigram items (“stop words”); (3) calculating odds ratios for all the moves and steps; and (4) extracting the n-grams with highest odds ratio for each move/step. The results indicate that this method is capable of discriminating well between the moves. The steps that carry a more distinct rhetorical purpose appear to discriminate best. N-grams found in steps that comprise more general information, seem to be very discipline-specific.

From needs to instruction: Academic writing for international graduate students (Spring, 2007)

Co-researcher - Karina Silva, ALT PhD student

In the field of English for Academic Purposes (EAP), interest in identifying the academic writing needs of non-native speaker (NNS) graduate students has become the focus of an increasing number of research and pedagogical investigations (Casanave & Hubbard, 1992; Cooleey and Lewkowicz, 1995, 1997; Swales, 2004). Two major themes have emerged: a research trend that examines NNS students’ specific writing difficulties (Bitchener & Basturkmen, 2006; Braine, 2002; Hinkel, 2004; San Miguel & Nelson, 2007), and a pedagogical trend that considers effective ESL/EFL academic writing pedagogy (Cargill et. al, 2001; Cortes, 2006; Levis and Muller-Levis, 2003; Lim, 2006). While these two trends should be complementary, they often end up on parallel tracks and investigate student needs from separate perspectives, leading to results and recommendations that are frequently limited in scope. Only by using a more complex methodological approach, in which the writing difficulties and needs of NNSs are evaluated by all of the most important stakeholders involved, can these limitations be successfully addressed.
This study investigates the writing needs of NNS graduate students from multiple stakeholder perspectives: NNS students, EAP writing instructors, and professors in the students’ disciplines. Our results demonstrate that all stakeholders emphasize similar needs; what appears to be different is the degree to which they assess the importance of each category of needs. The implications of these findings are viewed in terms of practical considerations for writing courses that are struggling to bridge the gap between language instruction and disciplinary writing norms. Insights gained from the three sources simultaneously serve as guideposts for effective pedagogies grounded in research and research that is responsive to classroom practice.

Integrating learner corpus-based data-driven pedagogy in academic writing classrooms (Fall 2006)

The value of corpora in language teaching is evidenced by a significant body of research on native-speaker corpora (NSC) and learner corpora (LC). However, compared to NSC, research advocating the use of LC to inform pedagogical practice is underrepresented (Granger, 2004). While there is some progress in creating LC-based reference materials such as learner dictionaries or CALL programs (Wible et al., 2001; Hegelheimer, 2006), LC-based classroom implementations are still scarce (Nesselhauf, 2004). Moreover, practitioners rarely derive insights from previous research findings and suggested pedagogical implications when incorporating LC in their classrooms.
This study followed a methodologically complex way of establishing a relationship between LC research and pedagogy. Specifically, the study included (1) collecting a ‘working’ learner corpus of NNS academic writing (98,450 words), (2) analyzing the phenomena of underuse of linking adverbials (LAs) in it, (3) comparing learner use of LAs with LC research findings and recommendations as well as with NS corpus-based reference materials, and (4) designing and implementing data-driven pedagogical interventions. The purpose was to draw conclusions about the effectiveness of needs-based and research-informed LC activities in academic writing classrooms. It was achieved by adopting a concurrent triangulation strategy, converging both quantitative and qualitative data. Frequency counts and pre/post test results were integrated to measure the relationship between learners’ use of LAs and the pedagogical treatment they were subjected to. Additionally, participant questionnaires were examined to gain qualitative insights. The results indicate that data-driven activities that incorporate authentic learner output lead to an increase in the frequency and diversity of LA use, as well as to an increase in learners’ awareness of the LA phenomena.

Language learning potential of CALL tasks and the effect of annotation modes on incidental vocabulary acquisition (Spring 2006)

Interactionist perspectives adhering to SLA theories provide a compelling framework for an integrated design of tutorial-type CALL. Gass (1997) asserts that language acquisition is successful if learners receive, comprehend, and notice input. Extensive CALL research has investigated annotations (De Ridder, 2002; Martinez-Lage, 1997), claiming that they can enhance language learning by promoting salience and noticing (Doughty, 1991; Jourdenais et al., 1995; Shook, 1994), modified interaction (Chapelle, 2001), and therefore input intake. While empirical findings show that some modes of input modification appear to be more advantageous than others (Al-Seghayer, 2001), the cognitive demand of annotations as a possible factor influencing language learning has not yet been investigated. In addition, most studies have as a focal point the measurement of learning outcomes, overlooking the evaluation of specifically designed CALL materials through “observable data that provide evidence of CALL qualities”. (Chapelle, 2001, p.66-67).
This study attempts to account for these shortcomings. It focuses on one aspect of Chapelle’s (2001) framework by investigating the language learning potential of annotation-based CALL materials targeting vocabulary acquisition, created for low-intermediate L2 learners. Here, traditional glosses, in which the definitions and pictorial representations of words correspond with their specific contextual meaning, are compared with a cognitively more challenging type of input enhancement - glosses linked to online dictionary and concordancer entries. It is hypothesized and demonstrated that salient hyperlinks are conducive to incidental vocabulary acquisition in both types of treatment and that a more cognitively demanding annotation mode may hold more language learning potential. Overall, this study is an example of how theory and research can inform CALL materials design and guide the evaluation of their effectiveness by combining perspectives such as:  interactionist views in SLA (Gass, 1997; Long, 1996), interface design (Plass, 1998), instructional design (Chapelle, 1998), learner training (Hubbard, 2004), and task characteristics (Gutierrez, 2003).

Conferencing applications and their potential use for language learning (Spring 2006)

with Lily Compton, Maja Grgurovic, Anne O’Bryan, and Dessie Bekrieva-Grannis
The conferencing applications Yahoo Messenger, Skype, ICQ, Marratech, and Paltalk were reviewed according to a matrix based on existing guidelines for software evaluation (Bradin, 1999; Healey, 1998; Hubbard, 1992). Communication features in CALL research were also discussed and used to provide sugesstions for pedagogical implementations of the features of each application reviewed. Potential problems and solutions were brought into light as well.