Baking Bread Without The Yeast

Among my son's favorite books are the ones in Richard Scarry's Busytown series. In What Do People Do All Day?, Able Baker Charlie puts too much yeast in the dough, resulting in a gigantic, explosive loaf of bread that the bakers (and Lowly Worm) need to eat their way out of.

The opposite problem -- a lack of yeast -- is present in Michelle Rhee's recent op-ed in Education Week. In it, she limits her call to "rethink" teaching policy to "how we assign, retain, evaluate, and pay educators" and to "teacher-layoff and teacher-tenure policies." (And she casts the issue of retention purely as one about so-called "last-in, first-out" employment policies rather than about school leadership, collaboration or working conditions.)

The utter absence of any focus or mention of teacher development either in this op-ed or in her organization's (StudentsFirst) expansive policy agenda leaves me wondering if Rhee believes that teachers are capable of learning and improving. If Rhee indeed does believe that new teacher induction and career-long professional development have value, then why does she consistently ignore it in her public statements and in her organization's strategic priorities? The alternative, of course, is a view that teachers are static beings, incapable of improvement. They are either born effective or ineffective. "Mr. Anderson's value-added score is an 18, thus he is an ineffective teacher and should be fired because his inability to teach cannot be ameliorated." We, of course, know this not to be the case. This alternative view also involves a strategy of simply trying to hire and fire our way to success. From research and international exemplars, I think most of us understand such a narrow approach to be ineffectual, albeit politically attractive in some quarters, especially among the Republican governors that Rhee is assisting exclusively.

High-quality development opportunities for teachers are like the yeast that helps the bread to rise. Comprehensive teacher induction has been shown to accelerate new teacher effectiveness and increase their students' learning. Likewise, personalized and purposeful professional development also can strengthen teaching skills and classroom impact.

It seems to me that a stated policy goal should be to ensure that as many as teachers as possible successfully pass educator evaluations being developed across the nation. Too many advocates such as Rhee appear to be eager to fire more teachers rather than make investments and restructure schools to maximize their effectiveness. A critical role for policy then would be to re-define teacher development in a way that raises the quality bar and invests public dollars in programs and approaches shown to have the desired impact on teaching and learning. Isn't that something we all can agree with?

Teachers are tremendously influential -- and we should do everything we can to unleash their full power. On teacher effectiveness, I'm unwilling to settle for half a loaf.

Jumping to Conclusions

In a recent post on Education Week's Inside School Research blog, Debra Viadero offers a caution about President Obama's support for community colleges. Pointing to her recent article on community college research that indicated how much more we need to know about how best to improve completion rates in that sector, she questions whether the president would be wiser to place his bets on career colleges. She says that a recent study by the Educational Policy Institute (EPI) and an ongoing program of research by James Rosenbaum and colleagues support her contention that community colleges ought to take cues from career colleges.

In my opinion, this talented reporter is jumping to conclusions.

Yes, the graduation rates at two-year for-profit colleges exceed those at two-year public colleges. No one disputes that. That does not necessarily mean, however, that career colleges are outperforming community colleges, or that community colleges should take steps to become more like career colleges. The plausible alternative explanations for the differences in results are numerous. For example, the students attending the two types of colleges may differ in important yet unmeasured ways, ways that are associated with chances of graduation (what researchers refer to as selection bias). Is one group more economically or educationally advantaged? More motivated? More apt to have a family, nighttime work, or receive tuition support from an employer? It's also possible that the differences in graduation rates stem from constraints that community colleges face but career colleges do not-for example, inadequate resources or a lack of control over mission or governance. It's one thing to point to differences in practices between the community colleges and for-profit colleges, and another thing to attribute those differences to variation in the "will" or intentions of practitioners, rather than attribute them to under-funding and all that comes with it.

Establishing that community colleges have poorer graduation rates than career colleges for reasons they can and should do something about requires evidence that the two differ on one or more key aspects that is causally linked to college completion. Say we knew that smaller classes caused better student retention-and community colleges have larger classes than career colleges. We'd then be able to say, there's something community colleges ought to fix. But we don't have evidence that that's the case.

Instead, research simply establishes that (a) career and community colleges have different graduation rates and (b) career and community colleges (sometimes) employ different institutional practices. Rosenbaum and his colleagues have done a nice job, as Viadero notes, of documenting the latter-using qualitative methods, mostly at colleges in the Chicago area. But they have not shown that those practices cause observable differences in graduation rates. Moreover, while they've produced one paper indicating that differences in the student populations at career and community colleges do not appear to account for disparities in outcomes, that analysis is based solely on a limited set of observable characteristics-and therefore don't rule out the possibility that different levels of student motivation, for example, are really the culprit. Just think about how students get to college-many at career colleges are actively recruited (sometimes off their living room couches) while many at community colleges effectively wander in the door. Why would we think, then, that career and community colleges are serving the same kinds of people and producing different results?

There's another consideration Viadero neglects, and that's college costs. Students at career colleges leave with far more debt than students at community colleges. Data from the 2007-2008 National Postsecondary Student Aid Study reveal that 61% of community college students graduate with less than $10,000 in debt, compared to only 22% of students graduating from 2 year for-profits. In contrast, 19% of graduates from 2 year for-profits have $30,000 or more in student loans, compared to only 5% of community college graduates. Nearly all students (98%) finishing at 2 year for-profit colleges have taken on a loan, compared to just 38% of community college graduates. Is that a problem? Is it offset by higher rates of graduation? The answers are far from clear. Absent better ones we shouldn't be relying on evidence like EPI's---a study of career colleges' high graduation rates that was supported by the Imagine America Foundation, formerly the Career College Foundation, established in 1982 as the research, scholarship and training provider for the nation's career colleges. Full-text of that study wasn't even placed online for researchers to fully vet!

Community colleges have a long, rich history of serving this nation. Sure, there's room for improvement, but without more solid evidence of which changes are needed let's not jump to conclusions and tout the for-profits as a model to which they ought to aspire. We might end up in a bigger mess than we're already in.

POSTSCRIPT:

I have now obtained a copy of the full EPI report. My suspicions were correct: the authors use nothing more than simple descriptive comparisons of students' characteristics and degree completion rates (calculated using NCES's DAS system, likely without propering weighting) to support their causal claims about the "benefits" of attending community college. For example, they write "The report suggests that career colleges work harder to provide appropriate student services and support" but present no data on institutional services or effort expended, particularly any tied to student outcomes. Their final conclusion -- "statistically, not only do students attending career colleges perform as well as or better than many other students attending comparative public institutions, but they persist in and complete their education while typically being more economically, educationally and socially challenged than other students"-- is based on nothing more than comparisons of sample means (no regression, no nothing). C'mon folks, this ain't the kind of research any consumer ought to be taking seriously. Glad to see Kevin Carey agrees.

"No Effects" Studies

Education Week's got this article out about randomized trials producing "no effects." According to the article, these null findings are raising eyebrows and "prompting researchers, product developers, and other experts to question the design of the studies, whether the methodology they use is suited to the messy real world of education, and whether the projects are worth the cost, which has run as high as $14.4 million in the case of one such study."

Wow, is that ever a disappointing reaction. Here's why:

1. We should be psyched, not upset, that studies with null effects are being released. That is not always the case. Publication bias, anyone? I've often thought that studies demonstrating null effects need to be publicized even more widely than those that find positive or negative impacts. Too many places out there are at the behest of funders, and can't release null findings. Too many assistant professors don't get tenure because they "didn't find anything." Are you kidding me? If it's a current practice and you learn it doesn't produce any effects, either way, it needs to be out there. We should learn as much from null findings and "worst practices" as we do from "statistically significant" impacts and "best practices."

2. Saying that experimentation isn't suited to the "messy real world" is a cop out. It lumps many different kinds of experiments into one category-- the good, the bad, and the ugly. Field experiments, lab settings, cluster-randomized trials with volunteer districts, and student-level randomized experiments with participants selected via administrative data-- these are very different animals. Each approach has a differential potential for generalizable results (external validity) and varying levels of challenges to internal validity as well. I'll grant you, experiments that rely on volunteer samples probably can't help us much in education-- since in real life programs aren't applied to students, families or schools who volunteer--they apply to everyone. This is especially a problem when we try interventions to close achievement gaps-- African-Americans who volunteer for studies are very, very different from those who do not (Tuskegee anyone?).

3. Doing experiments well costs a LOT of money. Putting trials on tight budgets helps to ensure they aren't run well--PIs cannot build the kinds of relationships that promote treatment fidelity, cannot collect high-quality data, and cannot get inside the black box of mechanisms--and instead are stuck simply estimating average treatment effects. No drug works for everyone, and no drug works in the exact same way for everyone-- the medical community knows this, and uses larger samples to make identifying differential and heterogeneous effects possible. When is Education going to catch up?

4. One thing I do agree with this article on. The model IES is using needs some revisions. I heard William T. Grant president Bob Granger give a great talk at SREE recently, where he made the point that the usual 'try small things then scale them up' model isn't going anywhere fast. We need to know how current policies work as currently implemented-- at scale. Go after that, spend what's necessary to conduct experiments with higher internal AND external validity, and support researchers to do this who reject old models and try new things. I promise you, we'll get somewhere.

Measurement Is Not Destiny

Stephen Sawchuk has written an excellent cover story ("Stimulus Bill Spurs Focus on Teachers") in this week's edition of Education Week. It discusses the federal stimulus legislation which directs states to abide by the equitable teacher distribution provisions in the No Child Left Behind Act -- as well as to improve teacher effectiveness -- in exchange for state stabilization funds and the opportunity to apply for competitive grants as part of Secretary Duncan's "Race To The Top Fund."

With regard to teacher effectiveness, there's just one little problem. There's no definition in federal law -- let alone in state laws -- about what that actually means. From Education Week:

Several states, and some districts, now endorse performance-based teacher evaluations that define good teaching, determine which teachers exhibit such practices, and identify those who fall short for assistance. Others are reorienting professional development toward sustained school-based approaches that researchers say are more likely to change teacher behavior and improve student achievement than “one shot” workshops.

Some efforts to improve teacher effectiveness have proved politically challenging. The federal Teacher Incentive Fund, a performance-pay program, has promoted interest in using test scores to estimate teacher effectiveness. That approach has generally not been favored by teachers’ unions. The tif program received an additional $200 million in the stimulus.

Additionally, a limited number of states have the ability to match teacher records to student data, and even those with the technical capacity have not always used their data to estimate teacher effectiveness. The unions fear such links could ultimately be used to establish punitive policies, and they have successfully lobbied legislators to curb the use of “teacher effect” data in some states. ("Growth Data for Teachers Under Review," Oct. 12, 2008.)

But the possibilities of “value added” are enticing to policymakers. Officials in Tennessee, the lone state that has incorporated teacher-effect data into personnel decisions, are awaiting new data that will reveal whether efforts to attract effective teachers to the most challenged schools have improved results, said Julie McCargar, the state director of federal programs.

This is a huge issue – and it will be interesting to see if the U.S. Department of Education focuses its regulatory definition and its expectations of states – like so many others – simply on measuring and identifying and perhaps rewarding effective teachers. The logical and more purposeful next step, of course, is to look at what behaviors, characteristics, or knowledge make certain educators more effective and then determine how to scale up approaches to initial training or on-going professional development programs to help make the vast majority of teacher candidates, beginning teachers and veteran teachers better. I have no insider knowledge about the Department's thinking around all this, but I’m always astonished at the wealth of policymakers, policy organizations, and foundations that never seem to get past square one on this topic.

Measurement is not destiny.

If all we do is use value-added metrics to determine who the best teachers are and pay them more money for being better, we will be sacrificing the quality of public education for a short-sighted reform. While more money might keep some effective educators from leaving a particular school or district, or from leaving the profession entirely, it won't do anything to make existing and future teachers a whit better.

The teacher effectiveness conversation must be about more than value-added measurement and performance pay, although it can certainly include those elements. It can't be simply about rewarding the good and getting rid of the bad. Fundamentally, it must be about a concerted human capital strategy to use existing knowledge as well as future data and research to strengthen teacher preparation, induction and professional development to improve the skills and abilities of all teachers. Hopefully, the Department's focus on teacher effectiveness will impel such an effort.

We can do better -- by learning from the best teachers and finding ways to replicate their success. Now, that would be effective.