Empiricism assumes reliability and validity in research, and that randomization has occurred in the process. It is also based on the assumption that a researcher defends with rigor any threats to reliability and validity, and designs research to ensure an accepted level of randomization. This is outlined in MacNealy along with an explanation of how these ideals can be affected negatively or properly preserved.
North outlines the roles of the researcher(s) based on the type of research being conducted. His account of empiricism implies that there is much human involvement and therefore a greater margin for error, or greater possibility for error that is not always obvious. While MacNealy presents the ideal with the requisite caveats, North presents the evolution of research methods - the four modes of Experimental, Clinical, Formal and Ethnographic inquiry. North's background of how these different types of inquiry evolved lead to an enhanced understanding of MacNealy's concepts.
Schoen's discussion of causality was particularly helpful to me, since I have observed similar phenomena in organizational behavior. To apply what I have experienced to Shoen's theory, I took away that empiricism is not without flaws but that understanding the potential chinks in the armor are the key to successful research. What struck me about all three readings was the emphasis on empirical research as inquiry based, rather than authoritative. There's a lot of logic involved: if this, then that. Empirical research assumes that there will be gaps, conditions, exceptions, assumptions, and other factors that can impact research. The key to empiricism seems to be anticipating the possible mitigating factors, and planning for those factors, rather than avoidance (although some avoidance can occur, not all factors can be avoided).
I agree with North's assumption that what came before is what we build on currently, and that research is only good until a competing hypothesis proves otherwise. I also agree with Schoen's notion of causality - that causal relationships have much to tell us about the validity of research. Schoen discusses the meaning of Validity on page 85, which struck me as an important distinction. He differentiates between the notion of validity as used in everyday language and the notion of validity in research. His definition of "validity of causal inferences, where a conclusion asserting a causal relationship is said to follow logically from a set of premises asserting evidence for that relationship" captures the notion that empiricism is interpretive. This was a new way to think about empirical research for me - although there is a degree of interpretation inherent in discovery, it is the interpretation brough to research design that intriqued me here. Empiricism assumes a level of causal inference, according to Schoen, which is just fine because the process of indefinite disconfirmation (Schoen, 87) applied by others in the field once research is published makes for a check/balance approach that maintains validity (as much as validity is possible). Schoen's notion of "reflective transfer" seems to provide another check for validity - both during the process if engaged in by the researher, and after the process if used by challengers to disprove findings.
Reliability and validity are relative terms in empirical research, given the complexity of evaluation when causal relationships are considered. It makes empirical research seem less valid until we see the logic of the complex system involved in the interpretation and validation process. The complexity involved in empiricism is the difference between the making of knowledge and speculation or theorizing, which are not empirical at all.
When I finished the bulk of the MacNealy chapter my head was indeed swimming, just as she predicted. As the categories got more and more intense and in-depth I decided to take notes to attempt to separate them out, but in the end, I was still a hot mess. I found myself relieved to hear her end comments about overwhelmed students and the need for a realistic view of empirical research. Overwhelming does not seem to properly describe my current view of empirical research; perhaps insurmountable is better. There are simply so many aspect to consider (reliability, validity—internal and external—randomization, probability, and null hypothesis) it seems inevitable that one’s first few initial (naïve) attempts at empirical work are simply bound to fail. I don’t mean to sound horribly depressing but the whole ordeal sounds like a sort of exercise in futility. The research aims for perfection, the elimination of as many negative variables as possible, for the ultimate generalizability; however, we are only human, thus doomed to consistently fall short of such perfection. I am not conceding that there is no value in such research. Simply because we cannot attain total perfection does not mean there are not varying degrees of accuracy or generalizability; I am just pointing out that the notion of empirical research, in all reality, sets us up for failure. Like an observer at a football game, it seems all too easy to sit back and point out the flaws in another’s work; however, actually attempting this kind of work has to mean that someone will necessarily not find it good enough, in some respect. It seems all empirical researchers can realistically strive for is ensuring that their work and methods warrant more commendations than condemnations.
However, what interests me more is the tension in all three of the pieces this week (implicitly or explicitly) between the work of researchers and practitioners. While I find it implicit in North’s work, Schon is rather overt about the topic. Schon discusses how often social scientists treat practitioners as subjects rather than co-researchers, and in this way fail to make their work and finding useful to practitioners. The tension between generalizable and context-based work is repeatedly mentioned throughout the piece, and the tension made me question whether one type of work/research needs be placed hierarchically above the other. There is certainly value in being able to generalize findings, but with all of the categories and problems MacNealy discusses for empirical research how much can one truly separate findings from context?
What made this thought game all the more interesting is when I came across the MLA call for papers: (http://www.wpacouncil.org/node/1074) “MLA President Gerald Graff has chosen the theme of teaching for the 2008 MLA Conference, encouraging panels on related topics. Professor Graff has noted that ‘I’ve come to think of the MLA conventions themselves as sites of teaching and learning,’ going on (echoing the 1998 Boyer Commission Report) that ‘what the complaints about the research-teaching conflict overlook is how much of our research has come increasingly to contribute to our teaching.’” There is a separate section which discusses “The Scholarship of Teaching: How Writing Programs Support Teaching-Centered Research.” The call further explains “not long ago, a discussion was held in small-college English department about the necessary requirements for scholarly activity to achieve tenure. Within that discussion, one area of dispute was whether teaching-based research should be considered commensurate with other forms of research.” The tension we are discussing in class is most certainly immediately palpable. There are new messages every week on the WPA listserv from people who need help convincing those above them that pedagogical work should be considered tenure-worthy. I have to wonder out loud how things got to this state. As we have read in other historical accounts of the field, composition started out as a service work field, yet now we do not give those who maintain such service work the honor of tenure. Are the apprehensions of Connors and Fulkerson valid when they worry about the state of our field as it specializes itself and rests its head in theory rather than praxis? I have to wonder what amount of praxis, if any, will our own future careers allow for. And most importantly, do we even have a choice in the matter or has the field created a “natural progression” that simply has too much momentum .
Empiricism seeks to establish verifiable truths through the objective observation of phenomena as they react to environmental factors in nature and in controlled experiments. Through such observations, empiricists expect to be able to identify and describe recurring patterns that can then be used to generalize about how to deal with such occurrences in similar situations in the future. Schon suggests that empirical research is supposed to allow us to categorize and delineate the way things work so that methods that cause things to happen can be identified and systematized (91-92). North explains that the empiricist’s confidence that such results can be achieved through empirical studies is based on “the belief that things-in-the-world […] operate according to determinable or ‘lawful’ patterns [...] which exist quite apart from our experience of them, and which are, in addition, accessible to the right kinds of inquiry” (137). It seems to me, however, that the premise that the objective observations of “things-in-the-world” apart from our experience can ever be made is a faulty one. As Schon also points out, even though people might want to be able to observe and understand their experiences without being influenced by personal perspectives, biases, and prejudices, it isn’t possible to do so (92-92). Therefore, even in the empirical projects in which scientists control all the variables they can think of and when the results a scientist achieves are replicated in later, similar studies, the scientist is not evaluating the results objectively. S/he must necessarily focus on certain results while ignoring others, and, without objectivity, the conclusions can only present one of several possible perspectives. Yet these conclusions are often treated as if they are absolute and unquestionable.
To me, this feels dangerous. It provides a mechanism through which people can justify and reify particular perceptions of the world and make these perceptions seem to be verifiable and verified truths when they are, in each case, only one of many possible interpretations. The SAT is a significant example. As MacNealy points out, the SAT is considered very reliable as a measure of scholastic aptitude because the results have been quite similar over time (53). But are these tests really providing objective measures of students’ scholastic aptitude (in other words, students’ natural or inherent academic abilities) as they claim to be doing? Or are they really measuring how closely the students’ cultural backgrounds and experiences mesh with those of the test developers? A number of studies have suggested that there is really no statistically significant correlation between SAT scores and success in college-level programs, yet there is a statistically significant correlation between a student’s family income level and his or her performance on the test; high-income students typically out-perform their less affluent counterparts (see James Crouse and Dale Trusheim’s The Case Against the SAT, for specific studies). Despite the many questions raised about the validity of standardized tests such as the SAT, however, students’ scores are still regularly used as a key component in the high stakes college admissions process because people routinely accept these tests as valid measures of aptitude.
These kinds of events serve to make me uncomfortable with empirical approaches to research. Yet I also know that empirical studies have been responsible for many modern advances. Therefore I cannot escape the belief that empirical research has value. However, I think we should be careful to accept empirical evidence just as we do anecdotal or rational evidence, not as something that will provide us with indisputable answers but rather as just another window through which to get a different, and perhaps broader (but still limited), view of a particular set of phenomena.
MacNealy’s definition of empiricism is set-up for us in her chapter and outlines what she deems a checklist for evaluating preliminary designs (76). MacNealy spends this chapter discussing reliability and validity in research and then the randomization that happens during the research process. Her first two sections, reliability and validity, look at ways in which to reduce threats to these two qualities of quantitative research. She said that these two qualities are ways in which scholars and researchers grant credibility and importance to research (52). I was surprised that she seemed to gloss over reliability and then spent 15 pages explaining reliability. Does this mean in this type of research that reliability is not seen as important as validity? I must say I wanted a little more especially with the examples she used for reliability. She did say that researchers can do much to control the reliability but the methods suggested were mainly standardized tests. To quote Ruth, this seems dangerous and most definitely makes me uncomfortable. It seems too easy to say that a standardized test can set-up the reliability of this type of research. Along with Ruth, I have to offer the drawbacks to using this. Many people do not perform well on these types of test and its not just the SATs (or ACTs) that people don’t do well on. The creative tests she mentions also can affect a test-taker negatively. Let me give you an example (and relying on my own experience), when I was in third, fourth and fifth grade I was asked to take a standardized test to put my in the “gifted” program in my school. Three years I took it and three years I was never asked to join because my test scores did read as “gifted.” So I would always go into the test believing I was a special student but leave knowing that I was not. I never made it into the gifted program. Now am I am not gifted as a student? What did these tests really suggest about me as a student? Years later when I took the SAT, ACT and GREs I confirmed that I was not a good test taker. The scores I received suggested that I was not a promising student or heck even a great student. Yet here I am in a Phd program questioning the very test(s) that could have held me back. Maybe I should go back to those test takers from my childhood and ask if I am now gifted or do I still fall short? North’s view of empiricism is a little different than MCN’s. He sets up his chapter by discussing the four major modes of inquiry in composition (137). They are: experimental, clinical, formal and ethnographic inquiry. He defines each which allows us and gives a few examples of people who he offers as each type of researcher. “Caution applies especially to this section where it can be hard to say at what point methodological variations go from producing differences in degree to differences in kind” (139). I found this quote to be interesting and was wondering if there is more to it then North offers. I like the several of the above people have to question this type of research. Obviously there is some type of value in it since we are discussing it in class, but also because so many different research studies have used it. Natalie said that we are only human and “thus doomed to consistently fall short of such perfection.” She makes a valid point. We are but humans…where does this play into account? And how does it? Since I am a human (obviously) is my fatal flaw my lack of testing taken abilities? And what would this mean if I decided to take up this type of research? Knowing that I am not a great test-taker; would I be able to go into this type of research unbiased in the use of standardized tests? Not really sure I know that answer.
My initial assumptions on empiricism are, well, varying.
On one hand, empiricism—or the act of conducting empirical research—sounds intriguing. It is very hands-on oriented, and it seems like an appealing break from long hours locked in the library or your room with a mound of books (or, if you are like me, web articles off JSTOR). There also seems to be a lot more agency involved in empiricism. Conducting empirical research allows you, in a way, to create your own path rather than retrace others’ steps—of course, you would still need to make sure others did not conduct your experiment, and you would need to know those who have conducted similar ones. Furthermore, in empiricism, you construct the rules—you decide the logistics of the experiment; you get to watch the results unfold. In a sense, it is as if you are the gatekeeper to a certain type of knowledge, and it thus becomes your duty to share that knowledge and the study that produced it with the rest of the world (or field for which it is appropriate).
On the other hand, empiricism scares the bejesus out of me. MacNealy was absolutely spot-on in her inclusion of the “A Few Words of Encouragement” section that ended Chapter 4 – Concepts Basic to Quantitative Research. I could handle the five basic concepts to sound research methodology, but the subsequent breakdown of the threats to validity—both internal and external—was overwhelming (to say the least). She, of course, admits as much in that final section, but saying it is overwhelming does not negate the fact that it is overwhelming—if that makes sense. The room for error seems so large in empiricism that I would be nervous with every step of the process. I can only imagine how disconcerting it would be to put a year or more into a research project only to find out it was invalid or tasted defeat at the hands of the menacing null hypothesis. At least when I make glaring omissions in a paper, I can revise it and (hopefully) turn it around in a short period of time. Revising an entire experiment, however, sounds daunting—both intellectually and financially.
And, while I know this will sound very elementary and middle school-ish, empiricism just seems too, well, math and sciencey (yes, I made that word up). The reasons I was—and still am—so fond of the English field were because (i) I was somewhat decent at it, and (ii) it didn’t involve any math or science, subjects that I was, for the most part, indifferent towards. Empiricism, however, seems to blur the boundaries and conflate the two sides of the brain, and I am not sure I really dig that.
Means? Analyses of Variance? If the book would not have had writing explicitly in the title, and if MacNealy herself did not constantly quote experiments concerning writing, I would have sworn I was reading a math or science book.
Take, for instance, the following article we (well, most of us) read for Designing Writing: “Are Advanced Placement English and First-Year College Composition Equivalent? A Comparison of Outcomes in the Writing of Three Groups of Sophomore College Students.” Hell, even the title is a little intimidating, but that is not my point. What is my point, however, is that I really enjoyed this article until the end—right when Hansen et al. started talking about their research methods. I just simply could not follow along. The terminology, the numbers, the means, the ratios: it was all just too much for me to compute. I felt like I was in “Calculus 2” my freshmen year of undergrad when I hadn’t taken “Calculus 1.”
So in sum, empiricism, for me, involves a dual reaction. Initially, I am intrigued, drawn to this alluring method of study with which I am pretty unfamiliar. However, after really thinking about it—reflecting if you will (ha, ha!)—I tense up. My hands become clammy, I get noticeable back sweat, and I say, “Damn, this is scary; I don’t think I can do this.”
But then again, that is probably a common reaction toward things we don’t know.
The goal of empirical research is to answer the research question through direct observations. Through empirical research, researchers draw conclusions about observable patterns, with the goal being to generalize the findings. Schon and North both reference the use of analyzing patterns when referring to empirical research. On one hand Schon talks about being able to categorize findings and on the other hand North discusses the use of patterns as a source of confidence for researchers. This seems to me one area in which empirical research could be subjective. In order to reduce the subjectivity, the researcher would have to clearly define how the patterns were discovered; otherwise multiple patterns could be uncovered. Who is to say which pattern is the correct one? Furthermore, when looking at these patterns, researchers make conscious choices about which data to include and which data to exclude, yielding only one possible perspective, right? This brings me to one possible assumption about empirical research: when written up, this type of research is often presented as absolute truths, when in reality, I’m not sure “absolute” is the correct qualifier. But, how does one present their findings without discrediting him/herself, yet also acknowledging that these findings present only one perspective?
I did find MacNealy’s chapter helpful though as one possible way for overcoming this struggle with subjectivity and the choices a researcher must make. MacNealy’s advice about using her chapter as a checklist seemed especially helpful as a way to handle the vocabulary (reliability, validity, biases, diffusion, means, etc.) One question I do have about MacNealy’s chapter has to do with the reliability of measuring tools. She suggests finding well-established measurement instruments, The Daly Miller Test, for example, and using these because they have already establish reliability. But, what if no such measurement tool exists for your purposes? How much pilot testing does one have to do to establish reliability? If the best dissertation is a done one, how much pre-testing of an instrument is required? It seems as though one could spend years and a fortune simply trying to establish the reliability of their measurement tool.
Echoing Rory’s concern, I think what makes me most uneasy with this kind of research is the huge amount of error that is possible (I’m ignoring all the numbers involved, because the numbers alone make me want to run like the wind!) Many of the posts refer to how we are all human, thus not possible of perfection. Creating a study, conducting the study, and then analyzing the study seem daunting, when such a great deal of energy and time is focused on validity and reliability. It seems as though all of this “other stuff” is the primary focus and not the actual question under investigation. How does one find balance, so as not to lose focus of the question? Finally I have some general questions about designing research studies. Not sure if this is the place to present them, but what the heck?! Could someone differentiate between methods and methodology? Sometimes I hear these words used interchangeably. Are they? I know we just spent some time talking about how to approach our questions from a theoretical perspective, but how does one go about creating a theoretical frame for their study? Hmmm, in re-reading, I think I have more questions this week than anything else.
Empirical research, according to MacNealy, has three definable characteristics: it is planned in advance, data is systematically collected, and it produces a body of work that can be examined by others. One might be led to believe that this simple definition is reflected in the simplified nature of empiricism, but this is not the case. Empiricism is anything but trouble-free, as it takes great liberties in its assumptions. For one, empiricism assumes that all of our knowledge of the world and how things operate come from experience (a posteriori). This assumes that we all experience things in the same manner, thus creating the same knowledge of the world. Reason would tell us this is not so. Empiricism also assumes that phenomenon is always determinate, and not only hard facts, but that there is a constant connection and correspondence between subjects. These patterns that are “discovered” are seen as a point of replication where others can theoretically reproduce the same results in the same manner. I say “theoretically” because although a systematic approach may be followed step-by-step, this does not account for the variables. The replication could prove problematic because there are too many variables to really determine a “constant” group in which to study. The innumerable factors to consider have already been referenced by my classmates above, things that interfere with the integrity of an empirical study. Particularly of note in MacNealy’s work is the factor and importance of randomization to empirical study. We must recognize the effect randomization could have in altering the outcome, quite possibly to an undesired outcome, not consistent with the former. Empiricism also operates on the notion that in order to study something, it must be measurable. This idea is troubling because certain things cannot be measured—signs that might *point* to a larger thing can be measured, but not the thing itself. What do I mean by this? A great example would be love. One cannot measure love. One can only measure the “symptoms of love”…things would point to the greater concept. I’m not sure if everything is completely measurable. I tend to think that everything is not.
Another aspect of empiricism to consider is the idea of context. We discover things in context, illuminating ideas against a backdrop. Why then would empiricism lend itself to the idea that if there is an empirical study, the study produced results, and that study can be reproduced into similar results? This seems extremely problematic because it seems to ignore the contextual aspect. It is difficult to maintain the same context in separate studies. After all, you would have different researchers, different variables. Even though a systematic approach is followed, there is not guarantee on a patented result.
But yet, it is amazing how much stock we place in empirical research. MacNealy points out that empiricism has the “power to persuade” and I would completely agree with that. Just look to the examples of “believable research” that we provided at the beginning of the semester for this class. Many of the examples given were based on empirical research. I think this is because empirical research is something tangible. We can use our senses, not just our thoughts, to prove it. If it happened in reality, it must be so.
MacNealy provides a rubric outlining the qualities of an effective empirical study. Elements such as reliability, validity, randomization, and probability are constraints on empirical research. While we can try to maintain internal validity by working diligently to create a balanced study, external factors will constantly conflict with a study’s stasis. MacNealy says, “Threats to external validity…are those factors which affect the generalizability of findings (i.e., the degree to which one can generalize from the behavior of the research subjects to the population of interest or to other populations)” (MacNealy 55). These threats that MacNealy identifies are the very problems with humanities-based empirical research. People are never a controlled population, yet people are the basis of any humanities study. While people are indeed creatures of habit, researchers dealing with numbers or controlled environments seem to have their research received much more readily by the public. Perhaps this is a struggle that humanities will perpetually face, but as a fledgling member of the humanities research community, I see myself hesitant to believe the evidence provided by theorists such as Flower and Hayes.
Our experience with empirical research in rhetoric and composition focuses mainly on studies that record how writing is constructed and achieved. These studies for example, one where researchers had participants speak their writing through processes, providing dialogue to communicate the mind’s working, and one where researchers studied how learned participants in a study communicate with less-learned participants via written letters, provide insight into the field of composition that certainly enhances the discipline’s validity. Assuredly these empirical studies provide controlled environment insights into the way writing takes place, yet it seems to me that in a field where numbers may as well be a foreign language, empirical studies fail to fulfill compositionists’ needs.
To return to MacNealy, it’s not the testing that troubles me. Rather, it’s the overwhelming list of threats that creep into an empirical study. How do researchers seek to combat bias, achieve randomness, and balance control when the subject matter is so variable? I don’t want to come across as anti-empirical research, because it is an effect place to identify and develop material that may lead to theory, yet I think it is beneficial to keep reservations in regard to the effectiveness and ultimate purpose surrounding empirical studies within our discipline. When I read the studies presented in our Composition Theory class, I found myself fighting against the findings. I was quick to make notes of the flaws I saw in the studies, and used those notes to invalidate the researchers’ efforts. I want to take my own skepticism and transform it into motivation to eliminate as many research design problems as possible so that my research will be undeniably valid.
Although I am posting late (MY BAD), I have taken ample time to read through the previous posts. I think the assumptions of empiricism are idealistic and often unrealistic, agreeing with those who have said that identifying all the variables in a study is impossible. Tiffany pointed out that researchers will always exclude certain variables in their statitics. She also asked how a study could offer absolute truth while also acknowledging other possible perspectives. A solution might be for researchers to explain what information has been excluded and why (or why the statistics they use are significant). Now, although I think an ideal empirical study in R/C is impossible (identifying all possible variables), it's still quite valuable. I think people want to read empirical types of research. Empiricism is based on knowledge that is reliable (can be replicated with similar results) and pretends toward conclusiveness. People are more likely to believe a study with heavy empirical data more readily than narratives.
But what is the role of narrative in empirical research? Our Schon reading for the reflection suggested that no communication escapes a narrative form (true--empirical research usually has the form of an ABSTRACT, INTRODUCTION, METHODS, RESULTS, and DISCUSSION. See "Reading Empirical Research Studies: The Rhetoric of Research). But besides the narrative form of the empirical article, I was wondering how ETHNOGRAPHY differs from the kind of "empirical research" we're defining as having validity, reliability, variables, and so on. Ethnography--observing others--is straight narrative! How is that reliable? It's context-specific and the level of reliability is pretty low I'd think. Then again, maybe not. I found Anne Beaufort's book, Writing in the Real World, highly instructive; and it's a complete ethnography. Ethnography sounds fun, too. I'd love to sit in a college class posing as a "student" so that I could write up a study of that class as an ethnographer.
Finally, I can relate with the frustration of realizing that an English-based field like Rhetoric and Comp "reaches across the aisle," as it were, to take on statistics and math. That's where we draw the line! When I first learned about the role statistics might play in R/C I felt pretty stand-offish, but with time I'm more open to it. It's a pain to try to identify all the control variables and independent variables in a given study, but at BSU we had fun brainstorming some "possible empirical studies" like how would a room with blue and orange walls (BSU's colors) influence the ways students write? Would students in this class write better? We talked about what times the classes would be taught and if that would make a difference; what sizes the rooms were; if having a class on the second floor made any difference; how the teaching would be observed; and how the papers would be evaluated. And the list could go on. I don't think the point is to control for all possible variables, but rather to decide which variables are most important and why. Creating the study might be more fun than implementing it, but I imagine it would be a big accomplishment to achieve a good empirical study.
For graduate students in English, I think empirical research is like Miranda's brave new world with creatures in it we've never seen.
Finally, I'm still stuck on this: if reliability means basically that a study can be replicated, what does validity mean? I can't seem to tease that out yet as something different from reliability.
9 comments:
Empiricism assumes reliability and validity in research, and that randomization has occurred in the process. It is also based on the assumption that a researcher defends with rigor any threats to reliability and validity, and designs research to ensure an accepted level of randomization. This is outlined in MacNealy along with an explanation of how these ideals can be affected negatively or properly preserved.
North outlines the roles of the researcher(s) based on the type of research being conducted. His account of empiricism implies that there is much human involvement and therefore a greater margin for error, or greater possibility for error that is not always obvious. While MacNealy presents the ideal with the requisite caveats, North presents the evolution of research methods - the four modes of Experimental, Clinical, Formal and Ethnographic inquiry. North's background of how these different types of inquiry evolved lead to an enhanced understanding of MacNealy's concepts.
Schoen's discussion of causality was particularly helpful to me, since I have observed similar phenomena in organizational behavior. To apply what I have experienced to Shoen's theory, I took away that empiricism is not without flaws but that understanding the potential chinks in the armor are the key to successful research. What struck me about all three readings was the emphasis on empirical research as inquiry based, rather than authoritative. There's a lot of logic involved: if this, then that. Empirical research assumes that there will be gaps, conditions, exceptions, assumptions, and other factors that can impact research. The key to empiricism seems to be anticipating the possible mitigating factors, and planning for those factors, rather than avoidance (although some avoidance can occur, not all factors can be avoided).
I agree with North's assumption that what came before is what we build on currently, and that research is only good until a competing hypothesis proves otherwise. I also agree with Schoen's notion of causality - that causal relationships have much to tell us about the validity of research. Schoen discusses the meaning of Validity on page 85, which struck me as an important distinction. He differentiates between the notion of validity as used in everyday language and the notion of validity in research. His definition of "validity of causal inferences, where a conclusion asserting a causal relationship is said to follow logically from a set of premises asserting evidence for that relationship" captures the notion that empiricism is interpretive. This was a new way to think about empirical research for me - although there is a degree of interpretation inherent in discovery, it is the interpretation brough to research design that intriqued me here. Empiricism assumes a level of causal inference, according to Schoen, which is just fine because the process of indefinite disconfirmation (Schoen, 87) applied by others in the field once research is published makes for a check/balance approach that maintains validity (as much as validity is possible). Schoen's notion of "reflective transfer" seems to provide another check for validity - both during the process if engaged in by the researher, and after the process if used by challengers to disprove findings.
Reliability and validity are relative terms in empirical research, given the complexity of evaluation when causal relationships are considered. It makes empirical research seem less valid until we see the logic of the complex system involved in the interpretation and validation process. The complexity involved in empiricism is the difference between the making of knowledge and speculation or theorizing, which are not empirical at all.
When I finished the bulk of the MacNealy chapter my head was indeed swimming, just as she predicted. As the categories got more and more intense and in-depth I decided to take notes to attempt to separate them out, but in the end, I was still a hot mess. I found myself relieved to hear her end comments about overwhelmed students and the need for a realistic view of empirical research. Overwhelming does not seem to properly describe my current view of empirical research; perhaps insurmountable is better. There are simply so many aspect to consider (reliability, validity—internal and external—randomization, probability, and null hypothesis) it seems inevitable that one’s first few initial (naïve) attempts at empirical work are simply bound to fail. I don’t mean to sound horribly depressing but the whole ordeal sounds like a sort of exercise in futility. The research aims for perfection, the elimination of as many negative variables as possible, for the ultimate generalizability; however, we are only human, thus doomed to consistently fall short of such perfection. I am not conceding that there is no value in such research. Simply because we cannot attain total perfection does not mean there are not varying degrees of accuracy or generalizability; I am just pointing out that the notion of empirical research, in all reality, sets us up for failure. Like an observer at a football game, it seems all too easy to sit back and point out the flaws in another’s work; however, actually attempting this kind of work has to mean that someone will necessarily not find it good enough, in some respect. It seems all empirical researchers can realistically strive for is ensuring that their work and methods warrant more commendations than condemnations.
However, what interests me more is the tension in all three of the pieces this week (implicitly or explicitly) between the work of researchers and practitioners. While I find it implicit in North’s work, Schon is rather overt about the topic. Schon discusses how often social scientists treat practitioners as subjects rather than co-researchers, and in this way fail to make their work and finding useful to practitioners. The tension between generalizable and context-based work is repeatedly mentioned throughout the piece, and the tension made me question whether one type of work/research needs be placed hierarchically above the other. There is certainly value in being able to generalize findings, but with all of the categories and problems MacNealy discusses for empirical research how much can one truly separate findings from context?
What made this thought game all the more interesting is when I came across the MLA call for papers: (http://www.wpacouncil.org/node/1074)
“MLA President Gerald Graff has chosen the theme of teaching for the 2008 MLA Conference, encouraging panels on related topics. Professor Graff has noted that ‘I’ve come to think of the MLA conventions themselves as sites of teaching and learning,’ going on (echoing the 1998 Boyer Commission Report) that ‘what the complaints about the research-teaching conflict overlook is how much of our research has come increasingly to contribute to our teaching.’” There is a separate section which discusses “The Scholarship of Teaching: How Writing Programs Support Teaching-Centered Research.” The call further explains “not long ago, a discussion was held in small-college English department about the necessary requirements for scholarly activity to achieve tenure. Within that discussion, one area of dispute was whether teaching-based research should be considered commensurate with other forms of research.” The tension we are discussing in class is most certainly immediately palpable. There are new messages every week on the WPA listserv from people who need help convincing those above them that pedagogical work should be considered tenure-worthy. I have to wonder out loud how things got to this state. As we have read in other historical accounts of the field, composition started out as a service work field, yet now we do not give those who maintain such service work the honor of tenure. Are the apprehensions of Connors and Fulkerson valid when they worry about the state of our field as it specializes itself and rests its head in theory rather than praxis? I have to wonder what amount of praxis, if any, will our own future careers allow for. And most importantly, do we even have a choice in the matter or has the field created a “natural progression” that simply has too much momentum .
Empiricism seeks to establish verifiable truths through the objective observation of phenomena as they react to environmental factors in nature and in controlled experiments. Through such observations, empiricists expect to be able to identify and describe recurring patterns that can then be used to generalize about how to deal with such occurrences in similar situations in the future. Schon suggests that empirical research is supposed to allow us to categorize and delineate the way things work so that methods that cause things to happen can be identified and systematized (91-92). North explains that the empiricist’s confidence that such results can be achieved through empirical studies is based on “the belief that things-in-the-world […] operate according to determinable or ‘lawful’ patterns [...] which exist quite apart from our experience of them, and which are, in addition, accessible to the right kinds of inquiry” (137). It seems to me, however, that the premise that the objective observations of “things-in-the-world” apart from our experience can ever be made is a faulty one. As Schon also points out, even though people might want to be able to observe and understand their experiences without being influenced by personal perspectives, biases, and prejudices, it isn’t possible to do so (92-92). Therefore, even in the empirical projects in which scientists control all the variables they can think of and when the results a scientist achieves are replicated in later, similar studies, the scientist is not evaluating the results objectively. S/he must necessarily focus on certain results while ignoring others, and, without objectivity, the conclusions can only present one of several possible perspectives. Yet these conclusions are often treated as if they are absolute and unquestionable.
To me, this feels dangerous. It provides a mechanism through which people can justify and reify particular perceptions of the world and make these perceptions seem to be verifiable and verified truths when they are, in each case, only one of many possible interpretations. The SAT is a significant example. As MacNealy points out, the SAT is considered very reliable as a measure of scholastic aptitude because the results have been quite similar over time (53). But are these tests really providing objective measures of students’ scholastic aptitude (in other words, students’ natural or inherent academic abilities) as they claim to be doing? Or are they really measuring how closely the students’ cultural backgrounds and experiences mesh with those of the test developers? A number of studies have suggested that there is really no statistically significant correlation between SAT scores and success in college-level programs, yet there is a statistically significant correlation between a student’s family income level and his or her performance on the test; high-income students typically out-perform their less affluent counterparts (see James Crouse and Dale Trusheim’s The Case Against the SAT, for specific studies). Despite the many questions raised about the validity of standardized tests such as the SAT, however, students’ scores are still regularly used as a key component in the high stakes college admissions process because people routinely accept these tests as valid measures of aptitude.
These kinds of events serve to make me uncomfortable with empirical approaches to research. Yet I also know that empirical studies have been responsible for many modern advances. Therefore I cannot escape the belief that empirical research has value. However, I think we should be careful to accept empirical evidence just as we do anecdotal or rational evidence, not as something that will provide us with indisputable answers but rather as just another window through which to get a different, and perhaps broader (but still limited), view of a particular set of phenomena.
MacNealy’s definition of empiricism is set-up for us in her chapter and outlines what she deems a checklist for evaluating preliminary designs (76). MacNealy spends this chapter discussing reliability and validity in research and then the randomization that happens during the research process. Her first two sections, reliability and validity, look at ways in which to reduce threats to these two qualities of quantitative research. She said that these two qualities are ways in which scholars and researchers grant credibility and importance to research (52). I was surprised that she seemed to gloss over reliability and then spent 15 pages explaining reliability. Does this mean in this type of research that reliability is not seen as important as validity? I must say I wanted a little more especially with the examples she used for reliability. She did say that researchers can do much to control the reliability but the methods suggested were mainly standardized tests.
To quote Ruth, this seems dangerous and most definitely makes me uncomfortable. It seems too easy to say that a standardized test can set-up the reliability of this type of research. Along with Ruth, I have to offer the drawbacks to using this. Many people do not perform well on these types of test and its not just the SATs (or ACTs) that people don’t do well on. The creative tests she mentions also can affect a test-taker negatively. Let me give you an example (and relying on my own experience), when I was in third, fourth and fifth grade I was asked to take a standardized test to put my in the “gifted” program in my school. Three years I took it and three years I was never asked to join because my test scores did read as “gifted.” So I would always go into the test believing I was a special student but leave knowing that I was not. I never made it into the gifted program. Now am I am not gifted as a student? What did these tests really suggest about me as a student? Years later when I took the SAT, ACT and GREs I confirmed that I was not a good test taker. The scores I received suggested that I was not a promising student or heck even a great student. Yet here I am in a Phd program questioning the very test(s) that could have held me back. Maybe I should go back to those test takers from my childhood and ask if I am now gifted or do I still fall short?
North’s view of empiricism is a little different than MCN’s. He sets up his chapter by discussing the four major modes of inquiry in composition (137). They are: experimental, clinical, formal and ethnographic inquiry. He defines each which allows us and gives a few examples of people who he offers as each type of researcher. “Caution applies especially to this section where it can be hard to say at what point methodological variations go from producing differences in degree to differences in kind” (139). I found this quote to be interesting and was wondering if there is more to it then North offers.
I like the several of the above people have to question this type of research. Obviously there is some type of value in it since we are discussing it in class, but also because so many different research studies have used it. Natalie said that we are only human and “thus doomed to consistently fall short of such perfection.” She makes a valid point. We are but humans…where does this play into account? And how does it? Since I am a human (obviously) is my fatal flaw my lack of testing taken abilities? And what would this mean if I decided to take up this type of research? Knowing that I am not a great test-taker; would I be able to go into this type of research unbiased in the use of standardized tests? Not really sure I know that answer.
My initial assumptions on empiricism are, well, varying.
On one hand, empiricism—or the act of conducting empirical research—sounds intriguing. It is very hands-on oriented, and it seems like an appealing break from long hours locked in the library or your room with a mound of books (or, if you are like me, web articles off JSTOR). There also seems to be a lot more agency involved in empiricism. Conducting empirical research allows you, in a way, to create your own path rather than retrace others’ steps—of course, you would still need to make sure others did not conduct your experiment, and you would need to know those who have conducted similar ones. Furthermore, in empiricism, you construct the rules—you decide the logistics of the experiment; you get to watch the results unfold. In a sense, it is as if you are the gatekeeper to a certain type of knowledge, and it thus becomes your duty to share that knowledge and the study that produced it with the rest of the world (or field for which it is appropriate).
On the other hand, empiricism scares the bejesus out of me. MacNealy was absolutely spot-on in her inclusion of the “A Few Words of Encouragement” section that ended Chapter 4 – Concepts Basic to Quantitative Research. I could handle the five basic concepts to sound research methodology, but the subsequent breakdown of the threats to validity—both internal and external—was overwhelming (to say the least). She, of course, admits as much in that final section, but saying it is overwhelming does not negate the fact that it is overwhelming—if that makes sense. The room for error seems so large in empiricism that I would be nervous with every step of the process. I can only imagine how disconcerting it would be to put a year or more into a research project only to find out it was invalid or tasted defeat at the hands of the menacing null hypothesis. At least when I make glaring omissions in a paper, I can revise it and (hopefully) turn it around in a short period of time. Revising an entire experiment, however, sounds daunting—both intellectually and financially.
And, while I know this will sound very elementary and middle school-ish, empiricism just seems too, well, math and sciencey (yes, I made that word up). The reasons I was—and still am—so fond of the English field were because (i) I was somewhat decent at it, and (ii) it didn’t involve any math or science, subjects that I was, for the most part, indifferent towards. Empiricism, however, seems to blur the boundaries and conflate the two sides of the brain, and I am not sure I really dig that.
Means? Analyses of Variance? If the book would not have had writing explicitly in the title, and if MacNealy herself did not constantly quote experiments concerning writing, I would have sworn I was reading a math or science book.
Take, for instance, the following article we (well, most of us) read for Designing Writing: “Are Advanced Placement English and First-Year College Composition Equivalent? A Comparison of Outcomes in the Writing of Three Groups of Sophomore College Students.” Hell, even the title is a little intimidating, but that is not my point. What is my point, however, is that I really enjoyed this article until the end—right when Hansen et al. started talking about their research methods. I just simply could not follow along. The terminology, the numbers, the means, the ratios: it was all just too much for me to compute. I felt like I was in “Calculus 2” my freshmen year of undergrad when I hadn’t taken “Calculus 1.”
So in sum, empiricism, for me, involves a dual reaction. Initially, I am intrigued, drawn to this alluring method of study with which I am pretty unfamiliar. However, after really thinking about it—reflecting if you will (ha, ha!)—I tense up. My hands become clammy, I get noticeable back sweat, and I say, “Damn, this is scary; I don’t think I can do this.”
But then again, that is probably a common reaction toward things we don’t know.
The goal of empirical research is to answer the research question through direct observations. Through empirical research, researchers draw conclusions about observable patterns, with the goal being to generalize the findings. Schon and North both reference the use of analyzing patterns when referring to empirical research. On one hand Schon talks about being able to categorize findings and on the other hand North discusses the use of patterns as a source of confidence for researchers. This seems to me one area in which empirical research could be subjective. In order to reduce the subjectivity, the researcher would have to clearly define how the patterns were discovered; otherwise multiple patterns could be uncovered. Who is to say which pattern is the correct one? Furthermore, when looking at these patterns, researchers make conscious choices about which data to include and which data to exclude, yielding only one possible perspective, right? This brings me to one possible assumption about empirical research: when written up, this type of research is often presented as absolute truths, when in reality, I’m not sure “absolute” is the correct qualifier. But, how does one present their findings without discrediting him/herself, yet also acknowledging that these findings present only one perspective?
I did find MacNealy’s chapter helpful though as one possible way for overcoming this struggle with subjectivity and the choices a researcher must make. MacNealy’s advice about using her chapter as a checklist seemed especially helpful as a way to handle the vocabulary (reliability, validity, biases, diffusion, means, etc.) One question I do have about MacNealy’s chapter has to do with the reliability of measuring tools. She suggests finding well-established measurement instruments, The Daly Miller Test, for example, and using these because they have already establish reliability. But, what if no such measurement tool exists for your purposes? How much pilot testing does one have to do to establish reliability? If the best dissertation is a done one, how much pre-testing of an instrument is required? It seems as though one could spend years and a fortune simply trying to establish the reliability of their measurement tool.
Echoing Rory’s concern, I think what makes me most uneasy with this kind of research is the huge amount of error that is possible (I’m ignoring all the numbers involved, because the numbers alone make me want to run like the wind!) Many of the posts refer to how we are all human, thus not possible of perfection. Creating a study, conducting the study, and then analyzing the study seem daunting, when such a great deal of energy and time is focused on validity and reliability. It seems as though all of this “other stuff” is the primary focus and not the actual question under investigation. How does one find balance, so as not to lose focus of the question? Finally I have some general questions about designing research studies. Not sure if this is the place to present them, but what the heck?! Could someone differentiate between methods and methodology? Sometimes I hear these words used interchangeably. Are they? I know we just spent some time talking about how to approach our questions from a theoretical perspective, but how does one go about creating a theoretical frame for their study? Hmmm, in re-reading, I think I have more questions this week than anything else.
Empirical research, according to MacNealy, has three definable characteristics: it is planned in advance, data is systematically collected, and it produces a body of work that can be examined by others. One might be led to believe that this simple definition is reflected in the simplified nature of empiricism, but this is not the case.
Empiricism is anything but trouble-free, as it takes great liberties in its assumptions. For one, empiricism assumes that all of our knowledge of the world and how things operate come from experience (a posteriori). This assumes that we all experience things in the same manner, thus creating the same knowledge of the world. Reason would tell us this is not so. Empiricism also assumes that phenomenon is always determinate, and not only hard facts, but that there is a constant connection and correspondence between subjects. These patterns that are “discovered” are seen as a point of replication where others can theoretically reproduce the same results in the same manner. I say “theoretically” because although a systematic approach may be followed step-by-step, this does not account for the variables. The replication could prove problematic because there are too many variables to really determine a “constant” group in which to study. The innumerable factors to consider have already been referenced by my classmates above, things that interfere with the integrity of an empirical study. Particularly of note in MacNealy’s work is the factor and importance of randomization to empirical study. We must recognize the effect randomization could have in altering the outcome, quite possibly to an undesired outcome, not consistent with the former.
Empiricism also operates on the notion that in order to study something, it must be measurable. This idea is troubling because certain things cannot be measured—signs that might *point* to a larger thing can be measured, but not the thing itself. What do I mean by this? A great example would be love. One cannot measure love. One can only measure the “symptoms of love”…things would point to the greater concept. I’m not sure if everything is completely measurable. I tend to think that everything is not.
Another aspect of empiricism to consider is the idea of context. We discover things in context, illuminating ideas against a backdrop. Why then would empiricism lend itself to the idea that if there is an empirical study, the study produced results, and that study can be reproduced into similar results? This seems extremely problematic because it seems to ignore the contextual aspect. It is difficult to maintain the same context in separate studies. After all, you would have different researchers, different variables. Even though a systematic approach is followed, there is not guarantee on a patented result.
But yet, it is amazing how much stock we place in empirical research. MacNealy points out that empiricism has the “power to persuade” and I would completely agree with that. Just look to the examples of “believable research” that we provided at the beginning of the semester for this class. Many of the examples given were based on empirical research. I think this is because empirical research is something tangible. We can use our senses, not just our thoughts, to prove it. If it happened in reality, it must be so.
MacNealy provides a rubric outlining the qualities of an effective empirical study. Elements such as reliability, validity, randomization, and probability are constraints on empirical research. While we can try to maintain internal validity by working diligently to create a balanced study, external factors will constantly conflict with a study’s stasis. MacNealy says, “Threats to external validity…are those factors which affect the generalizability of findings (i.e., the degree to which one can generalize from the behavior of the research subjects to the population of interest or to other populations)” (MacNealy 55). These threats that MacNealy identifies are the very problems with humanities-based empirical research. People are never a controlled population, yet people are the basis of any humanities study. While people are indeed creatures of habit, researchers dealing with numbers or controlled environments seem to have their research received much more readily by the public. Perhaps this is a struggle that humanities will perpetually face, but as a fledgling member of the humanities research community, I see myself hesitant to believe the evidence provided by theorists such as Flower and Hayes.
Our experience with empirical research in rhetoric and composition focuses mainly on studies that record how writing is constructed and achieved. These studies for example, one where researchers had participants speak their writing through processes, providing dialogue to communicate the mind’s working, and one where researchers studied how learned participants in a study communicate with less-learned participants via written letters, provide insight into the field of composition that certainly enhances the discipline’s validity. Assuredly these empirical studies provide controlled environment insights into the way writing takes place, yet it seems to me that in a field where numbers may as well be a foreign language, empirical studies fail to fulfill compositionists’ needs.
To return to MacNealy, it’s not the testing that troubles me. Rather, it’s the overwhelming list of threats that creep into an empirical study. How do researchers seek to combat bias, achieve randomness, and balance control when the subject matter is so variable? I don’t want to come across as anti-empirical research, because it is an effect place to identify and develop material that may lead to theory, yet I think it is beneficial to keep reservations in regard to the effectiveness and ultimate purpose surrounding empirical studies within our discipline. When I read the studies presented in our Composition Theory class, I found myself fighting against the findings. I was quick to make notes of the flaws I saw in the studies, and used those notes to invalidate the researchers’ efforts. I want to take my own skepticism and transform it into motivation to eliminate as many research design problems as possible so that my research will be undeniably valid.
Although I am posting late (MY BAD), I have taken ample time to read through the previous posts. I think the assumptions of empiricism are idealistic and often unrealistic, agreeing with those who have said that identifying all the variables in a study is impossible. Tiffany pointed out that researchers will always exclude certain variables in their statitics. She also asked how a study could offer absolute truth while also acknowledging other possible perspectives. A solution might be for researchers to explain what information has been excluded and why (or why the statistics they use are significant). Now, although I think an ideal empirical study in R/C is impossible (identifying all possible variables), it's still quite valuable. I think people want to read empirical types of research. Empiricism is based on knowledge that is reliable (can be replicated with similar results) and pretends toward conclusiveness. People are more likely to believe a study with heavy empirical data more readily than narratives.
But what is the role of narrative in empirical research? Our Schon reading for the reflection suggested that no communication escapes a narrative form (true--empirical research usually has the form of an ABSTRACT, INTRODUCTION, METHODS, RESULTS, and DISCUSSION. See "Reading Empirical Research Studies: The Rhetoric of Research). But besides the narrative form of the empirical article, I was wondering how ETHNOGRAPHY differs from the kind of "empirical research" we're defining as having validity, reliability, variables, and so on. Ethnography--observing others--is straight narrative! How is that reliable? It's context-specific and the level of reliability is pretty low I'd think. Then again, maybe not. I found Anne Beaufort's book, Writing in the Real World, highly instructive; and it's a complete ethnography. Ethnography sounds fun, too. I'd love to sit in a college class posing as a "student" so that I could write up a study of that class as an ethnographer.
Finally, I can relate with the frustration of realizing that an English-based field like Rhetoric and Comp "reaches across the aisle," as it were, to take on statistics and math. That's where we draw the line! When I first learned about the role statistics might play in R/C I felt pretty stand-offish, but with time I'm more open to it. It's a pain to try to identify all the control variables and independent variables in a given study, but at BSU we had fun brainstorming some "possible empirical studies" like how would a room with blue and orange walls (BSU's colors) influence the ways students write? Would students in this class write better? We talked about what times the classes would be taught and if that would make a difference; what sizes the rooms were; if having a class on the second floor made any difference; how the teaching would be observed; and how the papers would be evaluated. And the list could go on. I don't think the point is to control for all possible variables, but rather to decide which variables are most important and why. Creating the study might be more fun than implementing it, but I imagine it would be a big accomplishment to achieve a good empirical study.
For graduate students in English, I think empirical research is like Miranda's brave new world with creatures in it we've never seen.
Finally, I'm still stuck on this: if reliability means basically that a study can be replicated, what does validity mean? I can't seem to tease that out yet as something different from reliability.
Post a Comment