FirstPass is being developed by Bolton College to support students and teachers with the formative assessment of open-ended questions. A couple of years ago, colleagues at Bolton College embarked on a tentative journey to discover if a computer could be trained to support teachers with the formative assessment of open-ended questions; and if real-time feedback improved the quality of student work when responding to such questions. We have discovered that if we make use of natural language classification, natural language understanding and other tools a computer can indeed be taught to analyse and assess responses to an open-ended question. It is also possible to offer real-time textual and graphical feedback to students as they respond to the questions set by their teachers.

The emergence of this new formative assessment tool enables teachers to make use of a richer medium for assessing their students' work. Traditionally, online formative assessment activities are undertaken using closed questioning techniques such as yes/no questions, multiple-choice questions or drag-and-drop activities. Whilst valuable, this is a rather narrow way to undertake formative assessment. Our solution enables teachers to pose open-ended questions which can be automatically analysed and assessed by a computer. The ability to offer real-time feedback means that students can review and amend their answers before submitting them to their teachers for final review.

FirstPass - discover more with open-ended questions

The following video shows how FirstPass makes use of natural language classification models to support the formative assessment of open-ended questions.

The subject knowledge of teachers plays an important role when training the classifiers that underpin the use of FirstPass. They train the classification models that underpin the open-ended questions that they want to present to their students. Teachers may also welcome the fact that the accuracy of the classification models improve as more students engage with each open-ended question and as the volume of training data rises. One of the defining traits for services like FirstPass is their desire to improve through a participatory or crowdsourcing model. Model accuracy improves as additional training data is supplied to the respective classifiers that underpin the FirstPass platform. These services start to perform with a high degree of accuracy when they have access to a very large pool of users who generate an even larger volume of training data to nurture the service.

A participatory model may also have implications about the ownership models for future AIED services. If every student, teacher, school, college or university curates subject topic classifiers and if they all offer training data to FirstPass through their participation and use of the platform who has ownership of the platform? Who has ownership of the classification models that underpin its use? And who owns the training data that informs the classification models?

The following slides were used during a recent dissemination event that was held in March 2021 about FirstPass. The slides provide an overview of FirstPass and the road-map for the service.

How does FirstPass currently support teachers and students?
FirstPass has the following functionality to support the formative assessment process of open-ended questions. The list is not presented in any particular order of importance.

• Teachers can pose open-ended questions to their students.
• Teachers can set up classifiers on a given subject topic and train the respective classifier with labelled sentences. Training can be done via the direct input and labelling of individual sentences or teachers can copy and paste a body of text from an external source before labelling individual sentences.
• Each classifier has a test page were teachers are able to test how accurately FirstPass assigns predicted labels to seen and unseen sentences. Teachers can re-assign labels to sentences as and when required.
• FirstPass offers real-time textual and graphical feedback to students and teachers as they compose their answers on the screen. Zero lag is exhibited by FirstPass during the feedback process.
• Teachers are able to create open-ended questions using one or more trained classifiers. This enables teachers to present questions with different components. For instance, the teacher may ask: 'Why do people set up as sole traders and can you think of any disadvantages of setting up as a sole trader?' This offers teachers the opportunity to re-use trained classifiers for numerous questions and contexts.
• Students have the opportunity to review, amend and improve on their answers before submitting them to their teacher for final commentary.
• As students respond to a given question their answers can be used to provide additional labelled training data to respective classifiers. As the volume of training data increases FirstPass will become more and more accurate at assigning the correct label(s) to each sentence.

How will FirstPass develop?
The road-map for the FirstPass service details how the platform will improve upon and refine the use of cloud computing to mediate the formative assessment of open-ended questions. Here is a short description for each potential area of development.

Assigning multiple labels to sentences
Language has many complexions which allows sentences to have one or more meanings. As a natural language classification service like FirstPass matures it will be tasked with assigning multiple labels to individual sentences when required. For example, the following sentence 'I found project management a particularly difficult skill to learn' could be assigned any of the following labels: learning, project management or ease of learning.

Annotated and audio feedback
There are numerous EdTech assessment tools that enable teachers to provide annotated feedback to their students. The same holds true for audio feedback which has become commonplace. FirstPass will follow suit. In addition, FirstPass will utilise natural language generation to deliver textual and contextualised feedback to each student; and do so at scale.

Training from prior knowledge sources
Over a period of time teachers naturally accumulate a large document library for all the subject topics that they teach across all their courses. These libraries typically include documents, web pages or video content. As FirstPass develops it will be natural for the development team at Bolton College to tap into these knowledge bases to inform the classification models that constitute the FirstPass service. With a relatively small training data set FirstPass will scrape the text from these subject topic libraries and assign predicted labels for each of the sentences that constitute the document library. The teacher will then approve or re-assign the predicted labels that have been suggested by FirstPass. As the volume of training data grows for each classifier the more accurate FirstPass will become in predicting the most appropriate label to assign to any given sentence.

Theme extraction and context analysis
If natural language classification services like FirstPass are to support the formative assessment of open-ended questions in a reliable manner they will need to address theme extraction and context analysis. A theme is a central topic or idea that is explored in a body of text; and context gives meaning or a place to these words or ideas. Teachers are well placed to support their students to develop their writing skills; and they can provide advice and guidance about exploring and developing ideas or themes in their writing; and teachers do so intuitively. Services like FirstPass will need to address these challenges if they are to have a place in supporting teachers and students with the formative assessment of open-ended questions.

Comparative judgment
One of the use cases for FirstPass could be its ability to support teachers with comparative judgment. FirstPass could carry out successive rounds of comparative judgments and do so instantaneously; ranking students as each round took place and presenting the results back to the teacher for final review. It will be interesting to examine if FirstPass could perform on par with multiple teachers who carry out comparative judgments on a written task that was given to a large group of students. As part of the training process teachers will also have the opportunity to improve the accuracy of the classification models that underly the FirstPass service by undertaking comparative judgments on student work.

Other use cases
The ability to classify text means that FirstPass can be used to support multiple use cases or contexts. For example, schools, colleges and universities regularly ask their students for feedback about campus services or about the courses or modules that they are undertaking. The ability to label and classify text means that free-form text responses from students can be labelled and classified on behalf of campus administration or course teams; thereby shortening and lessening the difficulty in processing and summarising student feedback. Natural language classification services may also support educational institutions with complaint handling or online student enquries about courses. You may ask if natural language classification services can support formative assessment why can't they be used to support summative assessment? At the present moment in time I would err on the side of caution. These services need to demonstrate that they can be used reliably to support the formative assessment process; research needs to be undertaken about their impact on formative assessment and we all know that lessons are always being learnt at this early stage of any product's development cycle.

Summary
Colleagues at Bolton College have demonstrated that natural language classification services like FirstPass can be used to support students and teachers with the formative assessment of open-ended questions; and to do so at scale in a reliable, efficient, affordable and sustainable manner. The project team looks forward to the further development of FirstPass and refining how FirstPass operates when carrying out discrete tasks and activities to support the formative assessment process of open-ended questions. We especially look forward to seeing a growing number of teachers and students using FirstPass at Bolton College and elsewhere over the course of the next academic year.