Assessing Multimodal Texts Part 2

What we typically understand to be “literacy” has changed in recent years, due to an increase in the presence of digital technology and online practices. However, how teachers approach literacy, and particularly what is measured in literacy assessments, has not caught up with these changes (Cope, Kalantzis, McCarthey, Vojak, & Kline, 2011). Effective assessment in today’s world needs to be in spaces where students can demonstrate their knowledge and understanding using a variety of modes.

With regards to writing in particular, Cope et al. (2011) identify six principles for effective assessment of writing. They claim that meaningful assessment should:

• be situated in a knowledge-making practice
• draw explicitly on social cognition
• measure metacognition
• address multimodal texts
• be “for learning”, not just “of learning”
• be ubiquitous

Additionally, Cope et al. (2011) propose six technology-mediated processes for the assessment of writing:

• Natural Language Analytics
• Corpus Comparison
• In-text Network-Mediated Feedback
• Rubric-Based Review and Rating
• Semantic Web Processing
• Survey Psychometrics

Each of these is outlined in the table below. I personally find this chart a little overwhelming, and not very teacher-friendly. I would like to see a more simplified version of this chart, and written in a more coherent and concise manner. Also, I would like to see how this could be put into practice, by seeing how it could be used in the classroom.

Cope, B., Kalantzis, M., McCarthey, S., Vojak, V., & Kline, S. (2011). Technology-mediated writing assessments: principles and processes. Computers and Composition, 28(2), 79-96.                               

 The New Assessment: A Matrix of Principles and Practices for Writing Assessment

Principles and Practices	Situated in Knowledge-Making Practice	Social Cognition	Metacognition	Multimodal Texts	For/Of Learning	Ubiquitous
Natural Language Analytics	Responds to a text’s specific features with just enough information and just in time.	Environment becomes more socially and contextually intelligent as annotations are collected.	Reviewer annotations and queries to writers prompt metacognitive thinking about the writing and its contents.	Reading tags, captions, labels, descriptions of image, video and audio.	Assists the writer and provides data on their learning progress.	From a writer’s point of view, substantially automated, instant responses regarding textual specifics.
Corpus Comparison	On-the-fly comparison of same-discipline, same-level texts.	Environment becomes more accurate as more texts are collected and aligned according to discipline, subject contents and learning level.	Writer provided generalizations from corpus comparison, and the opportunity to rewrite addressing these generalizations.	Reading text ancillary to multimedia objects.	Provides the writer with peer comparison of writing quality in relation to cohorts, including an opportunity to rewrite and reapply comparison.	Automated response regarding overall text in relation to standards and equivalent levels and content areas.
In-text Network-mediated Feedback	Immediate feedback on written work in a knowledge producing community of practice.	Synchronous or asynchronous person-to-person conversations around textual specifics.	Parallel conversation speaks metacognitively about the text form and contents.	Dialogical feedback on non-textual multimedia contents.	Specific feedback and quantification of plus/minus evaluations.	Participants need not be proximate for the around-text dialogue to happen.
Rubric-Based Review and Rating	Establishing an overall frame of reference for the knowledge work.	Review and rating by self, peers, and invited critical friends, thereby creating a social culture of constructive evaluation.	An explicitly defined frame of abstract outcomes criteria in relation to a performance scale.	Review and rating of purely multimedia objects, as well as written and multimodal texts.	Pre- during- and post-task access to rubric, along with the option to rework to address reviewer comments in relation to rubric.	Asynchronous, web-accessible review and rating.
Semantic Web Processing	A subject-specific schema for mapping a knowledge domain.	Collaborative construction of concept maps.	Conscious markup of structure and semantics.	Tagging of images, videos, etc., using concept schemas.	Machine and person feedback on the application of concept map to a task.	Asynchronous web-accessible semantic mapping and markup.
Survey Psychometrics	Task-embedded quizzes, surveys, item-based tests.	Surveys can measure stance, attitudes, and perspective.	Surveys can address underlying knowledge and understandings.	Addresses knowledge acquired from multimodal work.	Before, during and after surveys to track what students already know and what knowledge they acquire.	Can be taken anywhere, anytime, for example when a task is completed.

 

 

 

 

Assessing Multimodal Texts Part 1

One question that will undoubtedly arise during the “Blogging in the Classroom” workshop is regarding assessment and evaluation. The next three posts will address this issue. Teaching with multimodal texts undoubtedly adds a certain degree of difficulty to this very important task. Therefore, it is essential for teachers to develop suitable assessments that can measure students’ literacy performance in relation to multimodal texts.

In a study aimed at addressing literacy instruction and assessment in today’s digital world, Hung, Chiu & Yeh (2012) developed a formative assessment tool, specifically a design rubric, to assess students’ design of multimodal texts. This research was particularly aimed at investigating the impact of the design rubric on English language learners’ multimodal text production (i.e., presentation slides) by providing students with feedback of their learning performance and constructive feedback for improving their learning performance (Hung, Chiu, & Yeh, 2012).

The design rubric proposed in this study uses five sets of questions (shown below) based on the established design elements (i.e., linguistic, visual, gestural, auditory and spatial) and uses a 5-point Likert scale, where 5 indicates excellent cohesion and 1 indicates poor cohesion. While this design rubric uses the cohesion of the design elements as the core criterion, I believe teachers could use the questions as a guide to create a more appropriate and simplified design rubric for the elementary school context, and their individual instructional purposes.

The authors claim that the design rubric below can be applied to various forms of multimodal texts (e.g., web pages, picture books, electronic portfolios).

Linguistic design

• Was the linguistic content comprehensible without major grammatical errors?
• Was the linguistic content structured in a logical and organized manner?
• How did the linguistic design represented in the multimodal text enable or limit the
author’s communication of meaning?

Visual design

• Did the author adopt a visual theme?
• Did the author carefully design the use of color and typology to reflect the selected
visual theme?
• If chosen to use, did the author make meaningful use of available visual elements,
such as graphics, to construct meaning in a cohesive manner?
• How did the visual design represented in the multimodal text enable or limit the
author’s communication of meaning?

Gestural design

• Did the author make use of any animated elements or special effects to design
dynamic sequencing of the content?
• If chosen to use, was the animation used purposefully and meaningfully to
complement or supplement the other design modes for meaning construction in a
cohesive manner?
• How did the gestural design represented in the multimodal text enable or limit the
author’s communication of meaning?

Auditory design

• Did the author make use of any auditory elements, such as music, sound effect or
narration?
• If chosen to use, were the auditory elements used purposefully and meaningfully to
complement or supplement the other design modes for meaning construction in a
cohesive manner?
• How did the auditory design represented in the multimodal text enable or limit the
author’s communication of meaning?

Spatial design

• Did the author adopt a specific layout to structure design elements?
• If chosen to use, did the author make use of text alignment and margins as design
elements to complement or supplement the other design modes for meaning
construction in a cohesive manner?
• How did the spatial design represented in the multimodal text enable or limit the
author’s communication of meaning?

Hung, H.,  Chiu, Y. J., & Yeh, H. (2012). British Journal of Educational Technology, 44(3), 400-409.

Thoughts on Assessment and Standards-based Tests

As a grade 3 teacher, the article by Asselin, Early, and Filipenko (2005) struck a chord with me. In Ontario, students in this grade are required to complete a series of tests to measure student achievement in reading, writing and mathematics in relation to Ontario Curriculum expectations. Having administered these tests several times throughout my career, I have first-hand experience with these paper-and-pencil print-based assessments. A great amount of time is spent throughout the year preparing students for these tests and ensuring they have the required knowledge and skills to achieve success.

In the assessment policy Growing Success: Assessment, Evaluation, and Reporting in Ontario Schools (2010), the Ontario government claims that assessments should be “carefully planned to relate to the curriculum expectations and learning goals and, as much as possible, to the interests, learning styles and preferences, needs, and experiences of all students” (p. 6). Additionally, they should be “ongoing, varied in nature, and administered over a period of time to provide multiple opportunities for students to demonstrate the full range of their learning” (p. 6). However, while this may be happening in classrooms, this certainly contradicts what is happening in province-wide testing.  Furthermore, as the notion of literacy and what it means to be literate is evolving, these traditional print-based assessments do not match what is happening in many classrooms across the province. As outlined in a previous post, the Peel District School Board has embraced the use of technology in the classrooms and is committed to providing its students with the 21^st century skills that will allow them to become active digital citizens. Collaborative inquiry, creativity and innovation, critical thinking and problem solving, and communication skills are at the forefront. Asselin et al. (2005) claim that as the curriculum changes to reflect these notions of literacy and 21^st century skills, so, too, should assessments. However, the question that arises for me is, how is this possible? How does one assess collaborative inquiry and creativity, and in particular, how is this possible in large scale, province-wide testing?

My next post will explore the use of multimodal texts (i.e., blogs) in the classrooms and assessment of these texts.

Asselin, M., Early, M.,  & Filipenko, M. (2005). Accountability, assessment, and the literacies of information and communication technologies. Canadian Journal of Education, 28(4), 802-826.

Ministry of Education. (2010). Growing Success: Assessment, Evaluation, and Reporting in Ontario Schools. Retrieved from http://www.edu.gov.on.ca/eng/policyfunding/growsuccess.pdf.

Getting to Know the PDSB

In preparation for our workshop, it is important that Joelle and I review the Peel District School Board’s (PDSB) “Vision for 21st Century Teaching and Learning” to ensure we are meeting the Board’s goals.

Vision for 21st Century Teaching and Learning (1)

In recognizing that students are already leaders in technology, the Peel District School Board (PDSB) has taken the proverbial (digital) bull by the horns, and recently implemented a Bring Your Own Device (BYOD) policy. Watch below as Tony Pontes, the PDSB’s Director of Education, explains the importance of their new policy.

They are ensuring parents are aware of the new policy as well.

Click to access FINAL%20BYOD%20brochure%20for%20parents.pdf

Why Blog?

Below is an excerpt from a paper I wrote last term in “Multilingualism and Multiliteracies: Teaching Language in a Globalized World”.

A weblog (“blog”) is an online personal journal on which the author (“blogger”) writes his or her reflections, opinions and/or comments regarding a particular topic. Blogs can include various modes of communication, such as images, sounds, links to other websites, etc., and provides the opportunity for others to read and comment on the content. Blogs allow individuals the opportunity to create a virtual community where one can post their thoughts for social, aesthetic and informational purposes (Alvermann, 2008). Not only are individuals creating their own content, but they are also “editing and remixing multimodal content they find online to share with others, using new tools to show and tell, and rewriting their social identities” (Alvermann, 2008, p. 10).

            In the classroom, a blog can be a relevant and worthwhile tool, particularly for second-language (L2) learners. According to Snodden (2010), not only can such practices enhance students’ engagement and increase their interest in learning, but can also serve as first-language models to other students. Trajtemberg and Yiakoumetti (2011) reiterate this notion by stating that blogs “may promote scaffolding in the fact that less knowledgeable learners can learn from more advanced learners simply by being exposed to their work” (Trajtemberg & Yiakoumetti, 2011, p. 438). They go on to offer many advantages to using blogs with L2 students, such as:

• providing a virtual community where students can actively engage in each other’s writing activities
• promoting self-expression, self-reflection, and self-evaluation
• promoting a sense of language progress
• promoting reflection on the part of the bloggers who have both their own work and their peers’ work at their disposal
• enhancing motivation for interaction and self-expression
• motivating learners to use language for real communicative purposes and to write in ways they have previously not experienced
• providing an “electronic portfolio” where both teacher and student can follow progress

The next post will provide some important information regarding the Peel District School Board.