assessment, Be the change, testing

Dear STAR Test, We Need to Talk Again…

Three years ago, almost to this date, I wrote my first blog post about the STAR test, a test sold by Renaissance Learning and employed in thousands of districts across the United States. That post started a discussion with the people behind STAR and while I wish I could say that it created change, isn’t that after all what we always hope for, it didn’t. Three years later, on the eve of my final STAR reading test of the year, I return to those same questions, once again hoping for some clarity, some light to be shed on how this test can be sold as a valid assessment tool.

Because, dear STAR test, it just doesn’t seem like you have evolved much from when we first started together. That in the three years since I last wrote to you hoping for some answers, that you have changed much. I guess, I could count your fancy new interface as change, but really all that has done is cause me to spend more time searching for the things I need in order to try to figure out what my students’ results supposedly are and what they may mean. But the essence of you, a comprehensive reading test that will quickly give me an elaborate understanding of 46 reading skills in 11 different domains remains the same. And much like so many of your cousins, all of the other computer tests who are supposed to be useful in our instruction, I keep feeling like I get the short end of the stick. Like a fool when I tell my students to show off their knowledge, to prove to the computer what we already know; just how much they have grown, just how much stronger they are.

Because according to the tests today, I have pretty much made all of my students worse readers than when they started. Or amazing super readers whose results are so incredible I want to cry tears of joy. It happens every year it seems. That the computer test tells us that they exploded, or that they didn’t grow or in fact reversed their abilities, but the face-to-face tests tell us a different story. The conversations we assess in their book clubs that show deep critical analysis and understanding. The written depth of their knowledge as they explore what it means to think about others’ stories and how it may affect them. How we see them share books, read books, recommend books.

And so that old letter stands the test of time, which is why I am reposting it, because honestly, now three years later into this relationship, I am still wondering why I bother. Why I get my hopes up for reliable, useable date? Why I tell my students to try their hardest? Why we take the time to try to do it right? Because I want to believe in you, STAR, I really do, but at this point, I am just not sure you are worth my time.

So Dear STAR test, we need to talk…again

We first met five years ago, I was fresh out of a relationship with MAP, that stalwart older brother of yours that had taken up hours of my 5th graders time.  They took their time and the results were ok; sometimes, at least we thought so but we were not sure.  But oh the time MAP and I spent together that could have been used for so many better things.

So when I heard about you, STAR, and how you would give me 46 reading skills in 11 different domains in just 30 or so questions, I was intrigued.  After all, 34 timed questions meant that most of my students would spend about 20 or so minutes with you.  You promised me flexibility and adaptation to my students with your fancy language where you said you “…combine computer-adaptive technology with a specialized psychometric test design.”  While I am not totally sure what psychometric means, I was always a sucker for fancy words.   Game on.

With your fast-paced questions, I thought of all the time we would save.  After all, tests should be quick and painless so we can get on with things, right?  Except giving my students only 90 seconds to read a question and answer it correctly meant they got awfully good at skimming, skipping lines, and in general being more worried about timing out than being able to read the whole text.

In fact, every year I have a child in tears who tell me that the timer popped up when they were still reading, that their anxiety is peeking because of that timer.  (Fun fact, if a child times out of a question it is treated as incorrect).  For vocabulary, all they get is 45 seconds because either they know it or they don’t, never mind that some of my kids try to sound words out and double-check their answer all within those precious seconds, just like I have taught them to do.  I watched in horror as students’ anxiety grew.  In fact, your 90 second time limit on reading passages meant that students started to believe that being a great reader was all about speed.  Nevermind, that Thomas Newkirk’s research into reading pace tells us that we should strive for a comfortable pace and not a fast one.  So yes, being a slow reader= bad reader.  

And sure, we could just turn the time off except that is not a decision I am allowed to make as an educator because that is a power given to the administration level, not the individual. On a larger scale, the fact that the product even comes with a time limit should be debated further; what does time have to do with reading comprehension and vocabulary knowledge besides the selling point of being able to administer it quickly or as you say “there are time limits for individual items intended to keep the test moving and maintain test security?” What does that do to bolster the validity of our test? How is that supported by best practice?

And so for some reason, year after year, I keep hoping that this will be the year where the data will truly be useful. Where I will gain knowledge that I can use to shape my teaching, isn’t that, after all, what the whole purpose of collecting data on our students is? But much like previous years, the results are a kaleidoscope of fragmented stories that refuse to fit together into a valid picture.  Students whose scores dropped 4 grade levels and students whose scores jumped 4 grade levels.  Students who made no growth at all.  Once again, I spend the day questioning my capabilities as a teacher because I don’t know what to take credit for.  Is it possible that I am the worst teacher ever to have taught 7th grade ELA or perhaps the best?  You confuse me, STAR, on so many occasions.  

As in previous year, students whose score differences are significant sometimes get to re-test, after all, perhaps they are just having a bad day?  And sure, sometimes they have gone up more than 250 points, all in the span of 24 hours, but other times they have dropped that amount as well.  That is a lot of unmotivated or “bad day” students apparently.   And yet, you tell me that your scores are reliable, and you’re not alone, many studies say you are too, yet that is simply not what we see every day in our classroom.  Although, this study (sponsored by you_did point out that you are most reliable between 1st and 4th grade, so where does that leave my 7th graders?

And last time I dug around your reports, I found that according to your own research at the 7th-grade reading level you only got a score of 0.73 retest reliability which you say is really good but to me doesn’t sound that way (page 54) 0.73 – shouldn’t it be closer to 1.0? If we look at the Cronbach’s Alpha Reliability that is only acceptable. And I guess that’s what I keep coming back to. Is your reliability simply measured as compared to other tests who are also problematic in their assessment methods and who we also know do not give us overly valid results?   Who knows, you would need a math degree to dig through your technical manual to make sense of all of the numbers.

Yet through all of this, you have dazzled me with your data, even know when I dig into your research I keep getting tripped up in your promises of reliable test scores, of comparable rest results, of scores that mean something, but what it is they actually mean, I am not quite sure of.  With all of the reports that I could print out and pour over.  Perhaps you were not accurate for all of my students, but certainly, you had to be for some.  It wasn’t until a long night spent pondering why some of my students’ scores were so low that I realized that in your 0.73 reliability lies my 0.27 insecurity.  After all, who are those kids whose scores are not reliable?   I could certainly guess but the whole point of having an accurate assessment means that I shouldn’t have to.  So it doesn’t feel like you are keeping up your end of the deal anymore, STAR test.  In fact, I am pretty sure that my own child will never make your acquaintance, at least not if we, her parents, have anything to say about it.

So dear STAR test, I love data as much as the next person.  I love reliable, accurate data that doesn’t stress my students out.  That doesn’t make them really quiet when they realize that perhaps they didn’t make the growth.  I love data that I can rely on and it turns out STAR, I just don’t think you fit that description, despite the efforts of those who take you.  Perhaps I should have realized that sooner when I saw your familial relationship with Accelerated Reader.  Don’t even get me started on that killer of reading joy.  You even mention it yourself in your technical manual that there may be measurements errors.  You said,  Measurement error causes students’ scores to fluctuate around their “true scores”. About half of all observed scores are smaller than the students’ true scores; the result is that some students’ capabilities are underestimated to some extent.”  Granted it wasn’t until page 81.  So you can wow me with all of your data reports.  With all of your breakdowns and your fancy graphs.  You can even try to woo me with your trend scores, your anticipated rate of growth and your national percentile rankings.  Your comparability scores to other state testing. But it is not enough, because none of that matters if I can’t count on you to provide me with accurate results. It doesn’t matter if I can’t trust what you tell me about my students.

So I wish  I could break up with you, but it seems we have been matched for the long run for now.  All I can be thankful for is that I work for a district that sees my students for more than just one test, for more than just their points because does anyone actually know what those points mean?  I can be so thankful that I work in a district that encourages us to use STAR as only one piece of the data puzzle, that chooses to see beyond it so we can actually figure out a child’s needs.   But I know I am lucky, not everyone that is with you has that same environment. So dear STAR, I wish you actually lived up to all of your fancy promises, but from this tired educator to you; it turns out I don’t need you to see if my students are reading better because I can just ask them, watch them, and see them grow as they pick up more and more books.  So that’s what I plan on doing rather than staring at your reports, because in the end, it’s not really you, it’s me.  I am only sorry it took me so long to realize it.

Best,

Pernille

PS: In case it needs to be spelled out, this post does not reflect the official view of my employer.

assessment, assumptions, grades, No grades, Personalized Learning

Using the Single Point Rubric for Better Assessment Conversations

A few years ago, I read the following post discussing single-point rubrics from Jennifer Gonzales on her incredible blog Cult of Pedagogy. The post discussed the idea of using a single-point rubric for assessment rather than the multi-point rubrics I was taught to use and how they were not only easier to create, but also offered up an opportunity for students to understand their assessment in a deeper way. Intrigued, we started tinkering with it over the last few years as an English department, developing our process as we went. The other day, I realized that I have never shared that work on here and thought that perhaps if someone had missed Jen’s post or was wondering what this looks like implemented, a blog post may be helpful.

So first of all, what does a single-point rubric look like? Here is an example of one we used with an assessment after finishing the book Refugee for The Global Read Aloud.

We operate on a 1-4 standards-based assessment system, so the difference between multi-point and single-point is the descriptive language found for each score. Where under a multi-point rubric you would fill in the description for 1 through 4, with a single-point rubric you just focus on what you would expect an at grade-level product to contain. This is what sets it apart in my mind; it allows us to focus on what we are specifically looking for and recognizing that students don’t always fall into the other categorizations that we set, no matter how much we broke them down.

This is one of the major reasons why I have loved using single point rubrics; it allows me to leave more meaningful feedback for students when they are either not meeting the grade-level target or are exceeding it. Rather than trying to think of all of the ways a student may not be at grade-level, I can focus on what would place them there assessment-wise and then reflect on when they are not. This has allowed me to leave more meaningful, personalized feedback, while also really breaking down what at grade-level thinking contains.

So what is the process for creating one?

  1. Determine the standards or learning targets that will be assessed. Students should be a part of this process whether through discussion and creation of the rubric or at the very least seeing and understanding the rubric before anything is turned in, after all, we want students to fully understand what we are trying to discover as far as their learning.
  2. Once the standards have been determined, decide what “at grade-level” understanding will contain. While the rubric shown above shows only one box per standard, sometimes our rubrics are broken down further within the standard in order for students to see exactly what it is we are hoping to see from them. (See the example below).
  3. Discuss with students if you haven’t done so already. Do they understand what at grade-level understanding looks like and what it contains? Is the rubric a helpful tool for them to take control of their learning? If not, go back to the drawing board with the rubric.
  4. Add reflective questions for students so that their voice is heard and further ownership is created over the learning process. This is important because too often assessment is something that is done to students rather than a process that allows students to fully see what they are able to do independently, as well as set goals for what they need to work on.
A few reflective questions – to see the original rubric, go here

Using the single-point rubric is a breeze for me compared to the multi-point rubric. First of all, it takes less time to create because we really just focus on that “at grade-level” understanding. Secondly, and this is the big one for me, it allows me to deeply reflect on why my gut or the rubric is telling me that a child is not showing “at grade-level” understanding or above it somehow. I have to really think about what it is within their understanding that moves them into a different category. One that is not limited by the few things that I could brainstorm before I saw their work. I then have to formulate that into written or spoken feedback in order to help that child understand how they can continue to grow. This allows our assessment conversations to change from grades to reflection.

Tips for implementing:

  1. Discuss it with students before using it the first time. Our students had not seen a rubric like this before and so we took the time to discuss it with them before we used it. This would happen for any assessment rubric, but it took a little bit longer because it looked different.
  2. Set the tone for assessment. I have written extensively about my dislike of grades and how I try to shift the focus, and yet I work within a system that tells me I have to assess with numbers attached to it. So there are a few things that need to be in place with the biggest one being the ongoing conversation that assessment is a tool for reflection and not the end of the journey. This is why students always self-assess first in order to reflect on their own journey and what they need from us. This can be messy in the beginning but through the year it gets easier for students to accurately reflect on their own journey and what they need to grow. They then hand that to me in order for me to look at their work and then it culminates in a final discussion if needed.
  3. Break it down. It is easy to get caught up in too many things to assess, using the single-point rubric has allowed us to focus in on a few important things. This is important so that students can work on those skills specifically rather than feel overwhelmed by everything within the process.

What do students think?

Our students seem to like them, or at least that is what they say. They understand mostly what they are being assessed on and they understand the feedback that is given to them. Having them self-assess and reflect prior to our assessment is also huge as it shows students that they are in charge of their assessment and their growth and that we want them to fully invest in their learning. It gives them an opportunity to see how they are growing and what their next step is before I add my opinion in there. This can also help reduce the “shame” factor that is sometimes associated with grades. When we discuss repeatedly with students that there is nothing wrong with being below grade-level and instead let the assessment guide us to the next steps, it shifts the assessment process, as well as the internalization of grades.

Overall, the single-point rubric has been another tool that allows us to help students become more reflective learners, while also helping us get to know the students’ needs more, resulting in a more impactful assessment experience for everyone involved. While we started small, the single-point rubric is now almost exclusively the only type of rubric we use in English and for that I am grateful. If you haven’t tried it yet, I would highly recommend you do. If you have any questions, after all my brain is tired from traveling, please leave them in the comments.

If you are wondering where I will be in the coming year or would like to have me speak, please see this page. If you like what you read here, consider reading my book, Passionate Readers – The Art of Reaching and Engaging Every Child.  This book focuses on the five keys we can implement into any reading community to strengthen student reading experiences, even within the 45 minute English block.  If you are looking for solutions and ideas for how to re-engage all of your students consider reading my very first book  Passionate Learners – How to Engage and Empower Your Students.   

assessment, being a teacher, Literacy, Reading

A Notecard Check – A Simple Way to Check for Understanding

5014990831222784.jpeg
Students’ answers to when I asked when does reading suck…

Ask our students what makes them hate reading and many of them will say the work that comes after.  The reading logs, the essays, the taking notes when reading, the post-its, the to-do’s.  Not the act of reading itself.  They share their truths year after year and year after year, I wonder how I am going to see whether they really are understanding and learning without making them drown in assignments that make them hate reading.  It is a hard balance to find, especially if your students like ours have reading abilities that range from years above grade level to years below.

While the students will be working on other skills with their reading, right now, we are working on increasing stamina and enjoying their books, a skill that some of our students need a lot of work on.  When we introduce too much to them to do, that is when they end up not really working on their reading but rather hunting the text for their answer. This is when they start to dislike reading.  While being able to disseminate a text and do the heavy work with text analysis is important, I cannot have them do that all of the time, not every time they read.  After all, how many adults do that every time they read?

This year, my colleague, Reidun offered up a great idea;  the simple notecard.  The notecard is unassuming.  It is limited in its scope based on its size and it also does not take much time.  Rather than writing anything long, which we only do once in a while, when students have been introduced to a teaching point such as writers using emotive language, we then ask them to return to their own self-selected text and look for an example.  As they read they find a sentence or two, write it down and hand it to us.

5283364949721088.jpeg
A student’s example of descriptive language found within her text.

When I have a moment, I am able to quickly scan through to see who got it and who didn’t, make a note of it and then figure out who needs to be in one of our small groups.  Who gets it, who doesn’t.  The kids spend most of their time reading, rather than taking notes, and I get a chance to peek into their thought process.

As the year progresses, our skill focus will change, our questions will deepen, and yet, offering students time to “simply” read is something that we will continue to protect every single day.  The notecard allows me to peek at skills, to inform my instruction, and to collect data.  All without causing a major interruption in their time with the text.

If you like what you read here, consider reading my newest book, Passionate Readers – The Art of Reaching and Engaging Every Child.  This book focuses on the five keys we can implement into any reading community to strengthen student reading experiences, even within the 45 minute English block.  If you are looking for solutions and ideas for how to re-engage all of your students consider reading my very first book  Passionate Learners – How to Engage and Empower Your Students.      Also, if you are wondering where I will be in the coming year or would like to have me speak, please see this page.

 

 

assessment, being a teacher, Reading

How to Assess Student Reading Skills Without Knowing the Book They Are Reading

A question I am asked often is how do I know what my readers know when they are all reading their own self-chosen books?  How do I keep them accountable for their reading?  How do I know what they can do without knowing the book they are reading?

And I get it.  How do we adequately assess the readers we have for their growth in comprehension when we don’t have them DO stuff all of the time?  Because that’s the thing, one of the biggest reasons that kids report for hating reading is actually the stuff we have them do after they read, not the actual reading itself.  It seems as if we have forgotten a few simple lessons that academics such as Louise Rosenblatt have been trying to remind us of for years.  Kids need relevant reading experiences that not only teach them how to be better readers but also help them grow or protect their love of reading.  I think I just summarized the goal of my book Passionate Readers here.

Yet, when we constantly attach something to the very act of reading, we diminish the act itself.  So how do we assess without harming?

A few things first, our students get ten minutes of protected self-chosen reading every day.  It is the very first thing we do.  If I had more time to teach (darn you 45 minutes), it would be a longer period of time.  During those ten minutes, the only thing our students are working on is reading a great book.  There are no post-its, no jots, no turn-and-talk.  They are working on their reading relationship, nothing else, because that task is big enough in itself.  While they read I check in with them, not to have them do stuff or talk about their book, but to talk about themselves as readers.   To help them establish or continue a connection to the act of reading itself.

Then we move into our mini-lesson, whatever that may be.  If the day is focused on reading then after we read the students then apply the skills as they read their own books if they can, a central tenet to our teaching always being; be able to use the skill when needed but you don’t have to use it all of the time.

We use read aloud, picture books, or short stories to model and discuss what readers need.  If students are asked to do a long-term project with their own books such as focusing on character development, symbolism, or analysis in some capacity,  then I model what that may look like within a read aloud.  I can sort out from their own writing whether they grasped the depth of the skill or if they merely skimmed the surface.  I don’t need to know the plot of their book to do this.  If I have questions I ask or look at the book they used.

If I need to narrow my scope then I will model a skill using a short story or a picture book (aren’t they almost the same?!) and then have either another short story or a stack of picture books to hand to them as they try out the skill for themselves.  Because I know all of the picture books and short stories I can quickly assess whether they understood it or not and use their work to further my own instruction.  Simple and yet it works every time.  Students get to balance reading their own self-selected text with the work of the classroom and we have created a balance between teaching skills and establishing the love.

And there you have it, how I assess my students’ skills without needing to have them do a reading journal – one of the number one things my students attribute to causing a hatred of reading – and without needing to know every single book they are reading.  Sometimes it is the simple things that make the biggest difference.

If you like what you read here, consider reading my newest book, Passionate Readers – The Art of Reaching and Engaging Every Child.  This book focuses on the five keys we can implement into any reading community to strengthen student reading experiences, even within the 45 minute English block.  If you are looking for solutions and ideas for how to re-engage all of your students consider reading my very first book  Passionate Learners – How to Engage and Empower Your Students.      Also, if you are wondering where I will be in the coming year or would like to have me speak, please see this page.

 

assessment, being a teacher, being me, testing

Dear STAR Test, We Need to Talk

Dear STAR,

We first met two years ago, I was fresh out of a relationship with MAP, that stalwart older brother of yours that had taken up hours of my 5th graders time.  They took their time and the results were ok; sometimes, at least we thought so but we were not sure.  But oh the time MAP and I spent together that could have been used for so many better things.

So when I heard about you, STAR, and how you would give me 46 reading skills in 11 different domains in just 30 or so questions, I was intrigued.  After all, 34 timed questions meant that most of my students would spend about 15 or so minutes with you.  You promised me flexibility and adaptation to my students with your fancy language where you said you “…combine computer-adaptive technology with a specialized psychometric test design.”  While I am not totally sure what psychometric means, I was always a sucker for fancy words.   Game on.

With your fast-paced questions, I thought of all the time we would save.  After all, tests should be quick and painless so we can get on with things, right?  Except giving my students only 120 seconds to read a question and answer it correctly meant they got awfully good at skimming, skipping lines, and in general being more worried about timing out than being able to read the whole text.  (Fun fact, a fellow teacher timed out of most of her questions when she took the test in training and still received above 11th-grade level).  For vocabulary, all they get is 60 seconds because either they know it or they don’t, never mind that some of my kids try to sound words out and double-check their answer all within those precious seconds, just like I have taught them to do.  I watched in horror as students’ anxiety grew.  In fact, your 120 second time limit on reading passages meant that students started to believe that being a great reader was all about speed.  Nevermind that Thomas Newkirk’s research into reading pace tells us that we should strive for a comfortable pace and not a fast one.  So yes, being a slow reader= bad reader.  Thanks, STAR.

And yet, maybe it was just my first year with you.  After all, we all have growing pains.  But this year, it didn’t get better, it just got worse.  Students whose scores dropped 4 grade levels and students whose scores jumped 4 grade levels.  Or how about those that made no growth at all.  I didn’t know what to take credit for.  Was it possible that I was the worst teacher ever to have taught 7th grade ELA or perhaps the best?  You confused me, STAR, on so many occasions.  So when students significantly dropped, they sometimes got to re-test, after all, perhaps they were just having a bad day?  And sure, sometimes they went up more than 250 points, all in the span of 24 hours, but other times they dropped that amount as well.  That is a lot of unmotivated or “bad day” students apparently.   And yet, you tell me that your scores are reliable.  Yet, I guess they aren’t always, after all, at the 7th-grade reading level you only got a score of .82 retest reliability which you say is really good but to me doesn’t sound that way.  0.82 – shouldn’t it be closer to 1.0?  In fact, when your company compared you to other recognized standardized tests it dropped to 0.70 for 7th grade, but perhaps it was because of the small sampling size, just 3, 924 students?  Who knows? I suppose I could email you to ask for more updated results like it says in the very small footnote.

Yet through all of this, you have dazzled me with your data.  With all of the reports that I could print out and pour over.  Perhaps you were not accurate for all of my students, but certainly, you had to be for some.  It wasn’t until a long night spent pondering why some of my students’ scores were so low that I realized that in your 0.81 reliability lies my 0.19 insecurity.  After all, who are those kids whose scores are not reliable?   I could certainly guess but the whole point of having an accurate assessment means that I shouldn’t have to.  So it doesn’t feel like you are keeping up your end of the deal anymore, STAR test.  In fact, I am pretty sure that my own child will never make your acquaintance, at least not if we, her parents, have anything to say about it.

So dear STAR test, I love data as much as the next person.  I love reliable, accurate data that doesn’t stress my students out.  That doesn’t make them really quiet when they realize that perhaps they didn’t make the growth.  I love data that I can rely on and it turns out STAR, I just don’t think you fit that description.  Perhaps I should have realized that sooner when I saw your familial relationship with Accelerated Reader.  Don’t even get me started on that killer of reading joy.  You even mention it yourself in your technical manual that there may be measurements errors.  You said,  Measurement error causes students’ scores to fluctuate around their “true scores”. About half of all observed scores are smaller than the students’ true scores; the result is that some students’ capabilities are underestimated to some extent.”  Granted it wasn’t until page 81.  So you can wow me with all of your data reports.  With all of your breakdowns and your fancy graphs.  You can even try to woo me with your trend scores, your anticipated rate of growth and your national percentile rankings.  But it is not enough, because none of that matters if I can’t count on you to provide me with accurate results. It doesn’t matter if I can’t trust what you tell me about my students.

So I wish  I could break up with you, but it seems we have been matched for the long run for now.  All I can be thankful for is that I work for a district that sees my students for more than just one test, for more than just their points because does anyone actually know what those points mean?  I can be so thankful that I work in a district that encourages us to use STAR as only one piece of the data puzzle, that chooses to see beyond it so we can actually figure out a child’s needs.   But I know I am lucky, not everyone that is with you has that same environment. So dear STAR, I wish you actually lived up to all of your fancy promises, but from this tired educator to you; it turns out I don’t need you to see if my students are reading better because I can just ask them, watch them, and see them grow as they pick up more and more books.  So that’s what I plan on doing rather than staring at your reports, because in the end, it’s not really you, it’s me.  I am only sorry it took me so long to realize it.

Best,

Pernille

PS:  I am grateful that Renaissance Learning did reach out to me to discuss my post, here is their response:

Renaissance Learning is deeply committed to teacher success in the classroom. I am the STAR Product Marketer and read your blog regarding our product. I welcome the opportunity to talk with you about your concerns and help you get the best experiences with Renaissance!

I captured two primary issues from your blog:

  1. STAR Reading Time Limits
  2. Reliability

STAR Reading Time Limits

I wanted to make sure you know that you can set an extended time preference in the software to help reduce students’ test anxiety and frustration. The instructions for doing so are on page 217 in our STAR Reading software manual.

On page 12 of our STAR Reading technical manual there’s an overview of testing time by grade that illustrates guidance for timing. This information can be used to assess what is the best time limits for your students (based on analysis of testing conducted in the fall of 2011).

Reliability

Reliability is a far more complex topic. There are three things to look at when discussing this topic: Reliability, Validity and Standard Error of Measurement (SEM).

Reliability is the extent to which a test yields consistent results from one test administration to another. Validity is the degree to which it measures what it is intended to measure and is often used to judge a test’s effectiveness. Standard error of measurement (SEM) measures the precision of a test score. It provides a means to gauge the extent to which scores would be expected to fluctuate because of imperfect reliability, which is a characteristic of all educational tests. These elements are described in detail in the Understanding Why STAR Test Scores Fluctuate.

STAR assessments have been independently reviewed and certified by the National Center on Response to Intervention www.rti4success.org and the National Center on Intensive Intervention http://www.intensiveintervention.org  and received high ratings as a screening and progress monitoring tool based on the criteria set forth to meet exceptional standards.

And my response:

Thank you for your response; the time limit is not something decided by me but by my district, but the fact that the product even comes with one should be debated further; what does time have to do with reading comprehension and vocabulary knowledge besides the selling point of being able to administer it quickly?
 The next point is the reliability; you seem to have missed the major point of the post, which is that when we do not know which child’s scores are reliable or not, then it becomes very hard to use the test for anything.  While I have read the document you linked again (I had read it before the post) it doesn’t yield any new information.   In fact,  it appears that teachers are expected to either assume it is because of something going on with the child or a measurement error.  The reliability for 7th grade as reported by STAR itself is 0.70 as referenced on page 25 in this manual.  According to the technical manual the SEM reported on page 41 in table 12 it is 71.74 for 7th grade.  That is incredibly high error measurement when it comes to kids’ scores, and yet that wouldn’t cover the fluctuation that we see in many students.
While I appreciate your response, I stand by the post; it is a travesty that teachers are being evaluated based upon tests like this, particularly when they are meant to be a diagnostic tool.  And while scores are probably accurate for some students it is hard to figure out who they are accurate for and who they are not.  My only wish for the future is that the test is either more accurate or somehow allows us to better decide which children’s scores are accurate.
assessment, being a teacher, grades, No grades, Student dreams

How to Make Grades About Students Again

http---www.pixteller.com-pdata-t-l-292057.jpg

Our first semester ended last week.  I have been working on grades, both standards scores and letter grades, for the past 3 days.  Pondering.  Wondering.  Pulling my hair out as I try to figure out which box to place my students in as we try to assess the growth that has happened.  In the end of it all, I am reminded of how much I still hate grades.

And yet, within this process of distilling my students down to a single letter comes an amazing opportunity for conversation.  Within the very simplification of all of the learning they have done, we have had an awesome chance to change the notion of what that score or letter means.

So how can we make grades and scores about the students again?

Start with a student definition.  Our very first conversation was the definition of each letter grade; a seemingly simple topic of conversation that shows just what students think grades measure; hard work, effort, being able to follow deadlines.  Very few students really understood that grades are supposed to measure knowledge.  Not work habits.  Not personalities.  In fact, many students said that the teacher is who is in control of grades (which, of course, there is an ounce of truth in) but they had never thought about how they could control the score.  How the choices they make result in the score they get.  The light-bulb moment for some students was tangible.

Have them grade themselves.  And not just for fun, but really.  After each grading period, my students either write which standard score they should get and why or which letter grade and why.  This very simple act – it takes about 5 minutes – become the seeds of conversations we need to have next.

Have them do a semester survey.  I continually want to be a better teacher for my students and that means that I need to face some ugly truths; some students are not getting enough help, some students feel I talk too much, some students are still not reading but getting very good at faking it.  How do I know?  I asked them on our end of semester survey and they told me their truths.  

Ask them what they are proud of.  As students came up one-by-one, this was the very first question I asked.  Not what their grade should be, not what their goal is.  But what are they proud of.  Why that?  What else?  Then what are goals?  How can I help best?  How can I support?  I read their survey and we discussed things right then.  Their voices came first, not their score.

Then discuss their grades.  The majority of students had picked the same letter I would, but some had either scored themselves too high or too low.  And while this is always interesting to see,  the bigger truth can be found in why they have picked that letter to define them.  Those simple statements they use to explain their answer; I am not that smart, I don’t understand, I work hard, I can teach others.  All clues to how they see themselves and their own learning.  All clues we can use to change the very essence of the conversation, because the conversation cannot just be about the grade, it needs to be about them.  About more than what they are distilled to on paper.  About how they see themselves, about their strengths, about their needs, about their dreams.

So while I work in a system that still asks me to define students through a score, we can reclaim that very conversation.  We can change the focus of the grade.  We can go deeper, make it more meaningful but we have to take the time to do it.  We have to assure that students have a voice in the process if we ever want it to be meaningful, if we ever want them to care.  We cannot do that through a letter, we cannot even do it through a report card comment.  It is not enough.  This conversation is the very least we can do to make grades about the students again.

If you like what you read here, consider reading my book Passionate Learners – How to Engage and Empower Your Students.  Also, if you are wondering where I will be in the coming year or would like to have me speak, please see this page.