Going Below the Surface of AI - The Future of Assessments

Rachel Cossar: And welcome to another episode in conversations in the future of work. I am your host, Rachel Cossar. And today, we’re actually gonna take a deep dive into the world of education and educational assessments and and how technology and AI are coming into that space and pushing it forward or in some ways holding it back. And I’m very excited to welcome our guest, Tim Burnett, on the show. So, Tim, welcome. Hi. Welcome. Nice to meet you. Yeah. It’s great great to have you. And, yeah, share a little bit about your background and, you know, what you’re you’re up to these days. Okay. So, yeah, Tim Burnett.

Tim Burnett: I’m, a member of the assessment community related to the kind of main function of what I do, and that means that I get quite heavily involved in, anything really exams and assessment related, particularly the use of technology, whether it’s using traditional forms of, assessment but using technology to power those or or making the kind of big switch in the transition to, on screen computing, exams and online exams. My background really is, I I was a design technology student at university. I did education as part of that, and I did some teaching, but then I got quite heavily involved in, web design and development. This was back in kind of 2000, and from there my path is career’s taken me through various different kind of education related institutions or organisations, but I found myself working for an organisation which was focused in assessment. And assessment’s I suppose it’s the underdog, really, of the the education space. It’s a bit of the kind of this was the dirty topic that nobody really likes to talk about or admit is needed. Everyone focuses on the the nice shiny learning, side of things. At, yeah, that’s where it’s kind of taken me in various different roles. I’ve set up a thing called the test community network, which is a place to bring together as many assessment specialists, from around the world different because there’s lots of diverse routes and channels that people go down with assessment. I’m also a board member for the e assessment association. I do a lot with an organization called the Association of Test Publishers. So, yeah, few different things there. Awesome. Great. Thank you.

Rachel Cossar: And because the first question is gonna actually take a deeper dive into, you know, some of that assessment work, can you just give a few examples of of, like, what you’re talking about specifically with assessments? Like, is this something that’s global? Like, would would people in North America also experience these assessments, or is it like, at what level do people experience these assessments or or or not at all?

Tim Burnett: Like Well, people take people involved in assessment at every stage, really, of their their life. I think, certainly, as you get older, you may start to become slightly more disconnected from, from assessment cycles. But, you know, in schools, in teaching, in primary and secondary, k through 12 as as it’s called, you you will be taking both formative and summative best assessments. Formative is what your teacher is using as part of the the learning journey, so using assessment for learning as part of that process to to monitor and check on, your your progress as a student through that learning experience. And then you’ve got the summative cycles, whether it’s your school admin examinations, your college entrance examinations, university examinations. There’s lots of different kind of tiers. And then it goes on beyond that when you’re into adult education. And I think this is something we’re gonna see more of because the days of you going into one career and staying in one career path for the rest of your life is hugely changing, in particular as, you know, we’re gonna talk about AI and how that is changing, the world of work at the moment. But, you know, if you want to secure that new job and you need to, whether it’s, a license to practice in something or or, do a particular role or job, then you’re going to have to potentially take an assessment as part of that cycle and as part of the career development. And, you know, like I say, formative, it can be formative where you gain some feedback along it, or it can be a summative process where you’re awarded a certificate or something at the end of it and that’s it, there is no cycle. What’s interesting is that there is more of a call now for more feedback, regardless of where you are in that journey, to make, I suppose, more formative assessment than summative assessment, coming up. Okay. Fascinating.

Rachel Cossar: It’s so funny because my 1 year old is in daycare, and he got his, I think, first tech tech assessment evaluation at, like, the 6 month mark, and it was like, is he taking his own socks off? And I was like, yes. He is. So that’s so funny. He’s already started on the assessment journey. So it’s fascinating because, obviously, these assessments are so pervasive and so critical to the way people move through education and academia, and AI and technology come on the scene. And, like, how how are they like, what is, you know, the the disparity or, like, the gap that you experience between maybe where assessments are currently and where AI or tech maybe wants them to be?

Tim Burnett: Well, so assessment has traditionally been quite a conservative field, it’s not seen that much progression and change. There have been, you know, big innovations in assessment, you know, online proctoring was something that was around for a while and then the pandemic suddenly threw it into the limelight, and that with it brought some AI technologies in as passed there, which interestingly there was more of a pushback on that during that cycle and that process. It has been used for large scale marking exercises where you’ve seen, you know, human marking has been replaced by AI in some places, where if you’re able to train the AI model based on the responses that are given, then you can rely more on the AI to to actually undertake those, those kind of experiences. But I think this is one of the challenges. Learning, you know, I mentioned kind of learning is always the kind of shiny thing, you know, where there’s a lot of information goes in in learning, and learning’s constantly evolving and changing as we, I suppose, understand more about the psychology of people and how work and what works best for others, and the idea of personalised learning is something that’s come up a lot more as part of the assessment mode. I’d say assessment because it’s it has to make the right judgment call because it has to be valid, fair, and reliable. It’s been something which is has always been a little bit guarded against too much fast paced innovation, and, you know, people always often, why why aren’t we taking more exams on computers? And, you know, there’s lots of other variables that are kind of thrown into the mix, whether it’s access and diversity of of access to this computer in the first place, there’s the question types that are are in there, how the assessment is written, whether it’s something that’s prepared beforehand, is it something that’s taken on demand, or is it something that’s taken as part of a kind of an end of a series, so, you know, June exams or something like that? And that all these things kind of influence the style and approach that you might take, on assessment. And I think at the moment, the the big I suppose the big issue at the moment is that we’ve we’ve gone through a cycle where, there’s obviously, people talk about rote learning and the idea of knowledge based assessment. It’s actually quite a low level on the Bloom’s taxonomy piece being able to regurgitate facts, etcetera. And, you know, AI’s come along and, in a way, replaced that need for that knowledge. And as we move forward into a future where we’re gonna be more augmented by the use of technology and AI, it’s really thrown, you know, the whole education system, particularly, I’d say, universities. I’d say it’s probably one that’s, you know, the idea of generating an essay for a submission. The fact that you can get AI to create something, you know, as as almost as PhD level as we’re seeing with, you know, things like the o one and o three releases. Now it’s it’s really thrown the whole space into turmoil, and you’ve got this we’re we’re kind of stuck at the middle of this at the moment, so whether or not we move down the route of further locking down examinations, assessment, and almost some people have said let’s go back to paper based exams, in in big kind of school hall size settings, or whether, you know, we we try and just embrace this technology and and move with them and move forward, and it’s it’s it is dividing the industry, I think, at the moment. Yeah.

Rachel Cossar: So that’s that’s fascinating because, I mean, this wrote this idea of wrote learning did you ever, read oh, what’s his name? It’s, like, Richard Feynman. Is it Richard Feynman? Did you ever read his book? No. I don’t think so. No. So he is he he was a physicist who was, involved in the Manhattan Project and the, and all that crazy stuff, but he went to Brazil and sat in on a lecture at in one of their, like, central, you know, national universities, and was shocked that people were just regurgitating, like, facts based on textbook. Like, people were able to answer the question when it was asked a certain way, and they all answered it the same way, which was what had been written in their textbook. And he realized they actually didn’t know they hadn’t they didn’t know anything. Yep. Not processing. Yeah. Right? And and I thought that was fascinating because that was a that was a while ago. That was in the 19, forties. Right? And so this issue of because when I look at some of the things that AI does and the way most people interact with chat GPT and and things like that is, these underlying skills of critical thinking and being able to process something or take a step back from a piece of information and understand where it came from and whether or not you agree with it. Like, those types of things, I think, like, the things that you learn in a liberal, right, a liberal education, like political science or something. Right? Like, those things I think are so important, and I feel like they really come front and center again. And and to your point, it is even more so less about, like, remembering these facts and actually developing the ability to think and question and, you know, develop, like, debate, you know? But see assessment is about measurement. Yeah. Yeah.

Tim Burnett: That’s one of the reasons why it’s it’s easier to measure facts and people’s ability to regurgitate facts and things like that. It’s harder to, you know, measure metacognitive skills and, you know, critical thinking, those kind of things. And, you know, so kind of one of the the purpose of the assessment is always be a big driver as to how you approach it and the style that you take and the assessment questions and the, you know, what is it that you’re doing the assessment for, and you need to have, you know, to be a fair and valid assessment, you need to have good purpose for assessment, because if you’re just doing assessment for assessment’s sake, then you’re gonna make mistakes, I think. You’re not gonna have a perfect learning cycle. So, yeah, I think we we’re changing I mean, the big thing is we’re we’re changing, aren’t we, as a society in terms of we will be. I think not everyone’s working up to this fact just yet, but how we access and process knowledge and work with it, you know, is gonna have a a dramatic shift on, you know, the day to day working lives of everyone, and therefore the skills that we need in education. And I think my my worry is we might, you know, I’ve got a 3 or 7 year old. I just I don’t necessarily think the education system’s working up and caught up to the idea that things are gonna be significantly different in 10 to 15 years. Right.

Rachel Cossar: And on that note, so let’s say you could, like, wave a magic wand, and make change quickly and safely and whatever. Like, what what do you think would be the best way to leverage AI to enhance, you know, the current educational assessment experience?

Tim Burnett: Well, I think everyone goes with the personalized thing, and that’s that is the the big thing because we’re all different. I mean, we’re all we’re all on a different path and different journey at the moment. We’ve been measured, as if we’re all the same people and as if, you know, schools or factories aren’t churning out the same polished, products, and that’s not the case. So, you know, ideally, you wanna be in a situation where everyone has a personal learning experience, has a personal assessment experience as part of that, because assessment is still, you know, very valid in that process of making a call, but how you are assessed could vary from person to person, whether it’s someone doing a presentation, whether it’s someone doing an essay, whether it’s someone doing a practical, activity, you know, the kind of performance based or simulation. So, yeah, that’s the, I suppose, the dream really is to, to personalise the whole approach to the system and setup. You know, how we get there and whether we get there is a kinda different thing, but there’s some I think there’s some great initiatives, underway at the moment. I just I know that it’s gonna be quite hard for, the education system as a whole, which is is quite a big it’s like a supertanker. The turning circle of a of the education system is very slow and unwieldy. Right. And we will see I think we’ll see parts of society and community really kind of excel and lean into this, and we will see, unfortunately, you know, some, being left behind, I suppose, as well as part of it. But, yeah, I think moving towards a personalised kind of experience, as well as, you know, leveraging AI for kind of augmented or even collaborative space, going into an assessment where you’re you’re you’re you’re working with an agent throughout the kind of learning experience. When you take the examination, you’re alongside the same agent who’s, you know, been judging you along that process as well, and maybe there is a bit more of a summative assessment task, but they know what to expect from you, and they’re able to process all the information and make those kind of fair standardised kind of, judgment that calls across people so we know where we are, but not necessarily use that as a leverage for the institutions that are involved. And this is the other issue. It’s the the whole funding cycle, you know, results, funding connections there just it’s just wrong, really. Right. Yeah.

Rachel Cossar: That’s really interesting because we, at Virtual Sapiens, you know, we have an AI communication tool. So we have professionals who are engaging with our AI in a conversational way and then getting feedback from the AI and kind of interacting with the AI as a coach. And it it’s great because the individuals get access to their own personal personalized coach. The interesting thing is that the the learning mechanism is, I don’t know if you ever see this too, but sometimes we get actually, oftentimes, we’ll get people who, confuse AI with magic. Yeah. Yeah. And, you know, people will be like, okay. Like, they’ll be like, great. So, like, within my accounts, like, my AI will be, like, remembering the conversations we had and then, like, getting smarter as I interact with it and all this. And we we often have to explain that this just it’s actually not how it works and that there’s, like, the global LLM that we’re referencing. And, yes, that LLM is certainly, you know, be constantly being updated in Fed information, not necessarily yours because of the way ours is set up, but, but that the updates you will get are not actually necessarily from, like, within your private tiny little account. You know? It’s just funny to see the miss mismatch already between expectation and reality. Yeah. Yeah.

Tim Burnett: Well, it that’s sort of every release, we see that, though, don’t we? We kind of we all go, wow, and for the first five minutes of it, and then, yeah, within half an hour, we’re we’re expecting the next release or something like that. But Yeah. There there’s that side of it, and then there’s the other side that I think there’s a whole part of society that just has not experienced any of this so far. I was talking to my mom the other day about it, and, you know, she’s disabled, and there’s so many benefits that AI could bring to her in a kind of the way she lives and works and does things, but it’s it’s it’s partially not there. The only kind of experience so far is Alexa, you know, which is, as we all know, kind of old tech now, really, isn’t it, in the cycle? But, you know, it could do so many more things for it. You know, the the operator model that’s just been released, you know, the idea that you could, you know, get it to do all your shopping for, you take a picture of the fridge, and it it knows what you what you normally eat, what you’re gonna plan, help plan the meal out for you in a conversation, and then put that order together for for for the week. You know, those kind of things are just Yeah. Incredible, aren’t they, really? And supply that model to education and, you know, wow. Boom time. Totally. Totally.

Rachel Cossar: How do you see AI, being either maybe not detrimental, but, like, ill used, you know, in this in this constant kind of attempt to update things.

Tim Burnett: Well, I think, at the moment, you know, we’ve got a problem where people are, you know, could be delivered traditional forms of assessment, and they’re using AI to to basically cheat their way through the process. Right. And, you know, that’s that’s not really helping anyone as devaluing the education process, and it it makes you really start to question, especially when you’re paying a lot of money for a university education or something like that, it really starts to question the the value of that, experience. So, I think there’s a there’s a lack of, I suppose, education, a lack of appreciation and understanding as to how best to use these technologies, but they are still very kinda new and shiny, really, in the in the grand scheme of things. I think there is a potential for a long term, impact where, yeah, we don’t feel as though we need to acquire as much knowledge to do something because, or, you know, that background underlying, kind of experience because, yeah, we’re relying and leaning into AI to to do those things for us. You know, if you, if something’s coming around, you know, chasing after you and doing all the work for you, then you you’re gonna get a little bit lazy maybe in there. But Right. I I kind of hold on to the fact that, yes, that might happen a little bit, but, I kind of I want to hold on to the positives that this technology is going to bring, as opposed to the, you know, too many of the negatives in there. I think, you know, some of the amazing things, the fact that like coding and stuff like that are able to give someone who has that great idea, who’s never had the coding ability to create their app or dream, you know, I’ve been playing a lot with Repla, and you know, I’ve done some coding in the past, not a huge help in terms of web work and things like that, but the fact that you can now just chat away and build some some incredible AI based tools, is that, that’s the positive stuff I want to hold on to. But yeah, it’s like everything, you know, machines come in, and if you think about the, how the industrial revolution changed, you know, and it it maybe, you know, it took away the ability for us to stitch cloth, you know, by having this machinery in place, but, you know, there are still I think it it fuels the specialism in other areas, you know, people everybody’s, you know, people still knit and things like that. It still happens, doesn’t it? And people are very creative with that stuff, so I think it’s just gonna just shift, some of the the weight that hopefully people will still hold on to some of the core skills. Yeah. I mean, it’s it’s so interesting how things, like, change and evolve. Right?

Rachel Cossar: Because, the the point about the knitting and the sewing. Right? Like, what was it? Like, any of these, like, recent tech like, big, big leaps forward in technology or just advancement of machinery. Right? Like, there’s when, like, when the Internet came online at all. Right? Like, I think there was a lot of fear around some of those some of that in general and or like or like a with the when the car, like, the motor vehicle, you know, came online, there were probably lots of people who were, like, steadfast horse, like, you know? Yeah.

Tim Burnett: I read a book recently, the Victorian Internet, and I I about the kind of, you know, wireless and telegraph service and how that Yes. Change and impact. And it’s it’s fascinating to see. I I think some of these things are, you know, there’s there’s several different comparisons you can make in history. You know, people have often talked about the, you know, the space race, you know, in terms of the funding that’s needed to for AI innovation. You know, the the the damage that could be caused by AI and compare it with nuclear bombs and the fact that, you know, the nuclear, you know, bombs were very hard to create and develop and, you know, was a government based thing, but we’re seeing this has been a lot more of a, I suppose, a democratization of this fact, and we’re seeing, you know, whether you believe it or not, but rumors this week, you know, about, you know, deep seek and things like that, about, you know, how they’ve been developed and the the potential. And the fact that you could put these models locally and, you know, the Jetson Nano and, you know, this this kind of cheap technology that, you know, I’ve ordered one, and it’s not arrived yet, but, you know, I could have an AI computer on my desk for 250 quid. You know, it’s just Right. It’s incredible, where it’s it’s kinda gonna go. Totally. But yes. It’s a fascinating time, isn’t it? No.

Rachel Cossar: It really is. And, the deflationary and, pressure, I guess, of technology is what’s so fascinating. And I do think that the deep sea stuff has been really fascinating, you know, for the US to, like, because who I mean, of course, there’s still so much that’s up to, like, understand with that. But Yeah. There’s a lot more that’s gonna come out about this, I think. Right?

Tim Burnett: Like, so but, you know, you’re kind of, like, interesting.

Rachel Cossar: Like so who knows? But, yeah. Fascinating. Last question. So I we’ve touched a little bit on this, but I’d like to kinda tweak the direction of this and ask, you know, globally across the education space, are there leader like, you mentioned that that some at least in some sectors, like, on the learning side. Right? Things are shinier and move a little faster. So even just within the assessment side, like, are there countries who are leading the way? Like, and how does change typically happen? It’s interesting that.

Tim Burnett: So in terms of the assessment space, there’s there’s a there’s there is quite a difference between, I suppose across the pond, between kind of Europe and America, for instance, or North America, and how exams and assessment has evolved over time. In America, there’s been kind of multiple choice based evolution, and it’s been a lot more where people have made that step into it, because multiple choice is slightly easier to do it, there has been more of a technology kind of innovation in that space. In Europe, it’s been more kind of open book, open question, wrong response, a bit of multiple choice may be used in a kind of foundation level, kind of space. And so it kind of it it means that how people are innovating with these kind of technologies is, each to quite different really. The potential for AI to to to mark and, process, open response and essay response based answers or even performance based answers, you know, is very different from how it can be involved in the evolution of of multiple choice type examinations. It’s probably gonna be more there for generating items, because it doesn’t need, you know, marking of multiple choices is quite, is more of an automation process than an AI process, because it’s just kind of right or wrongs. So I think in terms of the education space and in in virtual, we’re definitely gonna see kind of the western countries, innovating in certain areas, but we are seeing some significant work in Korea and places like that, South Korea. You know, they’ve got moving to almost like a digital book based process, I think, which is I think it’s got a degree of AI in there, it’s got, AI, kind of personalised learning, books in there. Singapore’s always been a big space and an innovator in this. Other countries like Scandinavia and have always been big, kind of innovators in best practice in education. You know, people always look to New Zealand and Finland and places like that as being, you know, exemplars of of of really fine education, setups. At the moment, I wouldn’t really put my money on any space really kind of excelling with this. I think we’d yeah. It’s not gonna be a regional based thing. I just think we’re gonna see groups and communities within it, which have got different drivers. You know, if you’re in a state based education system, you’re you’re you’re held back by the traditional, yeah, limits of access of technology, limits of access of bandwidth in schools, those kind of things are all still gonna be, to play for there. Independent schools, they’re gonna have different drivers. Private education’s gonna have different drivers. Right. I think we are gonna see more of, kind of personalised tutoring style tools and personalised tools to assist students in there because that’s not that’s always been a kind of premium. Right. You know, opt in, you know, mom and dad paying for your private math tuition for the week. Yeah. Yeah. Those kind of things. And we’re gonna see a lot more of those kind of tools coming out, which are gonna really gonna aid and guide, people through the education, space. So, yeah, it’s it’s evolving. It’s changing. I think assessment, though, will will still be a little bit behind the rest of the world in terms of adoption just because it’s so critical. The, you know, the creation of assessment takes a good few years. If you’re creating an examination, there’s so much research, you’ve gotta get the subject matter experts involved, you’ve gotta make sure it’s, you know, fair, you’ve gotta ask generate the questions. You know, it it’s such a big cycle that, you know, there’s people working on assessments now that, you know, will be will be in place in 3 years’ time or 5 years’ time, which, you know, won’t have any kind of flavor of AI, in among them. I think one of the big areas, though, we’ll see a big surge in is the analytics and the ability to look at whatever whatever it is going in within the education space to be able to process that and and turn that into assistive, information for for for teachers. I think there’s there’s always that kind of worry that within the education that, you know, teachers are gonna be replaced. Do you need still need teachers? But they, yeah, they have such an important role. I’m a teacher, at a college for 3 hours a week, and it’s it’s less about the the the knowledge of the subject and and the the it’s it’s how you mentor and nurture, and those are the kind of soft core skills that, you know, are are important as part of the process. So Right. I think, you know, education is gonna continue to, evolve. It’s gonna evolve with hopefully a lot more personalization, and it and it’s gonna get I think it’s gonna be very exciting. It’s gonna be one of the areas that’s hopefully is gonna benefit the most from, this innovation. And people kinda talk about, I think Dan famous Dan Fitzpatrick talks about infinite learning and those kind of things. Yeah. There’s there’s so many opportunities for, you know, education to just blossom. Yeah. Well, I love that. I love that note of positivity.

Rachel Cossar: I, you know, I I definitely agree. I think we’re just at such a a wild point, turning point, or, you know, evolutionary point in our our world, and it’s amazing to get to, you know, witness it all and participate in whatever way we can. Right? Yeah. I think that’s the best thing. It’s it’s been Yeah. It’s been part of it, isn’t it? Yeah. This is an exciting time for us all. Absolutely.

Tim Burnett: Absolutely. Well, Tim, thank you so much.

Rachel Cossar: This has been fascinating. Like I said, it’s it’s a topic that, you know, many people are constantly experiencing, you know, whether they’re going through their academic journey or or have in the past, and, really interesting to understand it more deeply. So thank you for for sharing so much with us today. No. That’s been thank you very much. And, if people would like to follow your work more, I think you also have a podcast. Like, what what’s the best way to kind of follow along and connect with you? So a couple of options.

Tim Burnett: You can search out the It Depends podcast, the one that I do with Amanda Davis. We talk about psychometrics. We just try and help people understand it, more. Well, Amanda helps me understand it more, and everyone hears me, learning, as we go. The test community network, if you wanna kinda connect it with other diverse areas, from, the assessments, you go to test community dot network. And you can always find me on LinkedIn, so Tim Burnett on LinkedIn. I’m sure you’ll find me there, find my picture. Wonderful. Great. Well, thank you again, Tim.

Rachel Cossar: And, as always, thank you to our listeners for tuning in, and definitely, you know, reach out with any questions you have for myself or Tim. Thank you very much. See you next time.