If I ask you to rate how happy you are today, on a 1 - 5 scale, I'll get a numerical response. If I invite you to "describe the quality of how you are feeling," at the moment, I'll get a description. The intention of asking the first question is to draw conclusions and generalizations about a large group, and the second would be to examine the quality of the experience itself. When someone indicates to you that they are 1 - 5 on a happiness scale, we know that the difference between 1 and 2 and between 2 and 4 is (relatively) arbitrary. We know we cannot "divide" the happiness of someone in such a clean-cut way. When someone describes the quality of wellbeing in words, those words contain a snapshot of the subjective phenomena the researcher is trying to understand.
In both qualitative and quantitative research, the researcher recognizes that all "indicators" are incomplete. They know that the phenomena are not the same as the indicator. Most people operate with a similar heuristic. There is a famous book called 10% happier; most of us understand that this doesn't mean 10% happier; it means "a bit" happier or "somewhat" happier. We also realize that words like "somewhat" or "a bit" are limited in what they tell us. We can translate that 10% into words that have meaning for us as we talk or think about the phenomena of being more or less happy.
However, as well as being limited, in the context of measuring psychological phenomena or psychometric properties, in particular, the relationship between the indicator and the "thing itself" is problematic for another reason. The act of asking a question has the potential to cause a specific psychometric phenomenon to arise.
However, as well as being limited, in the context of measuring psychological phenomena or psychometric properties, in particular, the relationship between the indicator and the "thing itself" is problematic for another reason. The act of asking a question has the potential to cause a specific psychometric phenomenon to arise.
We should not believe that the quantity and quality of "happiness" are "the same" regardless of the question we ask. Yes, the measures are incomplete. To mitigate this, we want to get a more complete "picture," and there are many things to consider. So we measure more, measure differently, use another strategy to "get to" the subject of the study. By doing this, we are trying to capture the subject from multiple angles or indeed, isolate the causal relationship between the phenomena and other phenomena. We are assuming that our act of observation is causally neutral; we think that watching something doesn't change what we are watching. This is based on a classic view of causation that sees causal properties to be linked in time and space. X causes Y because there is a physical causal chain. Genes causally interact with DNA which causally interacts with cells and so on. To use the analogy of a photograph, taking a photograph doesn't change the subject. A photograph captures the subject in a certain way and gives a specific type of information, but taking a photograph of the rock shouldn't change the physical constitution of the rock.
Think about another example, when you look at the ocean on a clear night when the moon is full, the white light pathway of the moon appears to come directly towards the observer. If you lined up 20 people along the beach, they would all experience the same phenomena - the light of the moon makes a pathway directly to them. "Where" is the light in this example? Well, the light is not on the water or on the moon or in your eye, it's caused by the interaction of these three things. The positioning of the subject changes the phenomena. In quantum physics, the role of the observer and how that changes the phenomena is relatively well known. When scientists observed quantum waves, they behave differently with a camera present. Analogous to naughty children who behave differently in the presence of an adult, perhaps? The act of observation is never neutral.
Think about another example, when you look at the ocean on a clear night when the moon is full, the white light pathway of the moon appears to come directly towards the observer. If you lined up 20 people along the beach, they would all experience the same phenomena - the light of the moon makes a pathway directly to them. "Where" is the light in this example? Well, the light is not on the water or on the moon or in your eye, it's caused by the interaction of these three things. The positioning of the subject changes the phenomena. In quantum physics, the role of the observer and how that changes the phenomena is relatively well known. When scientists observed quantum waves, they behave differently with a camera present. Analogous to naughty children who behave differently in the presence of an adult, perhaps? The act of observation is never neutral.
If I ask a particular type of question, then the "idea" of that question provokes something in the subject that produces a systemic response. Example - how scared are you of spiders? Maybe there is no proper answer to the question, "amount of fear," but because I am being asked to give an approximation on a survey, say 1 - 5, then I am creating an answer which itself causes me to feel fear of spiders. To what extent would the simple act of being asked to make me reflect upon something - fear of spiders - that perhaps I have not even considered before? Does this then invite me to consider spiders in a negative context? Did the question cause the fear? You said spider; I'd never considered how much fear I had before. Now I think about it; I am quite scared of spiders. How much do we trust that the thought indicating "I am" isn't a product of the process of being asked and nothing else?
This argument is one reason I don't like excessive summative testing. The assumption is that we are merely "collecting information," but we aren't, we are creating information through the process of assessment. There was a recent debate between Mr Kingsley and Daisy Christodoulou in which Soloman took the new GCSEs to task for including excessive amounts of content that it was unrealistic for everyone to master. Daisy replied to this by indicating that it was necessary to have a range of questions that differentiate the top achievers from the lower achievers. In other words, not everyone should be able to master the content because not everyone can achieve a top grade. This discussion assumes that student mastery of the material is uncovered by having more difficult questions. The increased difficulty will lead to more "accurate" results - if and only if by "accurate" we mean fine levels of distinction. If our purpose is to differentiate achievement in greater detail, having 10% or more of questions that only 1% of students can answer makes sense. And yet, in the absence of this fine-grained differentiation of testing, would the fine-grained differentiation between achievement levels exist?
Whether we need such subtle distinctions is important because we also know students' self-image, their idea of themselves as a student, how "smart" they are, impacts their future success. So if I believe I am smart and capable, and my teacher believes I am smart and capable, then I achieve more. If you create a system that deliberately attempts to distinguish, in ever more subtle ways, who is smarter and more capable, you might create distinctions that would not exist without that act of measurement. There is a great line from the Tao Te Ching, the sharpest knife is the quickest to blunt. As we become more exacting in our measurement apparatus we are increasingly susceptible to subtle, minute changes.
Let's take the most obvious example of a quantitative measure that works well. How tall you are! Imagine I say something ludicrous like - measuring your height more frequently makes you taller. You could falsify that by taking boy A and boy B, measuring boy A fifty times and measure boy B ten times. You could do this in a reasonably large population and do an average. If measured-boys grew more quickly because we measured them, we would know measurement impacts growth. We find out that the frequency of measurement has nothing to do with an increase in height. The lack of growth due to measurement is because my beliefs about how tall I am don't in any way impact how tall I am. In the words of the famous rap-song, I might wish I was a little bit taller, but that isn't going to make me a little bit taller.
There is an oft-touted phrase; you don't fatten a pig by weighing it. A common counter-argument is that we weight the pig to know what to feed it and to know what to change. We also, at some point, need to decide where to send the fatter pigs and where to send the thinner pigs. We assess the output to helps up to understand what inputs to choose. The argument I am trying to make here is that, when it comes to educational outcomes, the act of weighing has the effect of producing either disillusionment or confidence, self-efficacy, or whatever the opposite of that is. The act of testing children isn't a value-neutral thing. In other words, in educational terms, weighing the pig DOES have the potential to make it fatter or thinner because, in our world, the indicator or measurement changes the phenomena.
Comments
Post a Comment