Showing posts with label statistics. Show all posts
Showing posts with label statistics. Show all posts

Saturday, October 18, 2014

Merit’s Liquidity

Over the past year, multiple major media outlets and a powerful university president or two have been agitating about the correlation between parents’ income and SAT scores. Over that same time period, editors of various psychology journals were rejecting my regression study to determine the true influence of parents’ income on SAT scores while controlling for other possible factors because the study did not “make clear what the gap or problem is in the literature and exactly how the present study fills that gap.” If I cannot convince a journal that a lack of an examination on the influence of socioeconomic status in the entire 87-year history of America’s most important academic test is a gap in the literature, then what hope remains for my other project on the interplay of body-mass index and perspiration on navel lint accumulation?

Part of the problem resulted from my need to rely heavily on years of state averages, rather than the scores of individual students, but I am convinced that anyone who manages to gain access to individual data will merely corroborate my work at much greater expense. One editor conceded, “I agree that there is something we can learn from these data,” despite rejecting my work anyway. Ultimately, the new open-access journal, Open Differential Psychology published the study.

Last week, the College Board released the latest year’s SAT results. The organization cleverly released its “benchmarks” report the day before their usual, more detailed report. Reporters were too lazy to report on the subject two days in a row, let alone do any analysis. Blasé headlines declared scores “flat,” as they would for any two-consecutive-year analysis. I found the numbers fascinating. They dovetail with my study and other previous writings and flesh out some trends the media missed.

In my study I addressed the overlooked fact that correlations between parents’ education and (individual) SAT score and between parents’ income and SAT score have not been constant over time.
Family educational advantage seems to evince virtually undeviating growth as a predictor of SAT scores, but financial advantage seems to grow as the economy worsens. Rather than postulate that times of economic difficulty almost immediately make wealthy people smarter, one should focus on the exclusivity of the income category.


2014 brought new record advantages to students whose parents have bachelor’s or graduate degrees on the critical-reading and mathematics subtests. This fact can inform discussion about the notion of an “education bubble.” The expansion of college access has raised concerns over indebtedness, falling standards, and underemployment, but, while these concerns are likely to grow, education appears increasingly to benefit the following generation as if to demonstrate that the ability of higher education to sift and inculcate basic skills remains intact, or at least it did one generation ago.

The difference between the yearly trends for the SAT-score benefit of education to a family compared to the benefit of coming from a home with a higher income startled me, especially given the sharp drop in income advantage coinciding with the 2008 economic downturn. So, I graphed income’s SAT advantage with US gross domestic product (GDP) growth rate. The graphs almost perfectly coincide when the income divide compares families above $100,000 to those below that.


My regression study included a control for year, but an even more detailed study of the influence of family income could benefit by controlling for the state of the economy, as I suggested in my concluding remarks.
Perhaps additional or alternative variables could be identified. For instance, year was significant in some regression iterations but had small β values. Perhaps year is a proxy variable for other factors like the state of the economy.

The graph provides convincing evidence that there is some kind of causal relationship between the economy and SAT scores despite the relatively low influence of parents’ income on scores in the regression analysis. Student stress or mood seems more plausible as a mechanism than radical yearly changes in educational quality, but my point about exclusivity concerned whether economic slowdown weeded out low test performers from higher economic strata due to the heritable component of cognitive ability shared between the parents and the student. However, for that to be the mechanism, one would expect GDP to look more like a leading indicator of test performance or for the slope of the growth rate to tend to be the negative of the slope of test scores over time. This graph actually seems to indicate that poor economies weed out high test performers from higher economic strata. In The Bell Curve, Richard Herrnstein and Charles Murray claimed that the twenty-first century would continue “the emergence of a cognitive elite.”
The isolation of the brightest from the rest of society is already extreme; the forces driving it are growing stronger rather than weaker.
Of course, the economy of the 1990’s, when they wrote that, usually appeared to be growing stronger rather than weaker.

I can imagine two mechanisms for economic contraction weeding out smart families from the upper class: circumstances cause the power elite to lose interest in the power of ideas, and some upstarts’ ideas were not that good from the start. I suspect both are at work. The first mechanism supports a leftist or Marxist vision of ensconced aristocracy holding the privileges of power and leisure. If recessions interspersed with periods of anemic growth are “the new normal,” then The Bell Curve was partly wrong, and economic populism should be of interest to the so-called cognitive elite. On the other hand, when bad ideas create bad economies, perhaps the cognitive elite receive their just desserts. Some people who are obsessed with IQ research engage in some paradoxically simplistic thinking. As anyone who has surveyed the differential psychology literature about liberals versus conservatives or even the results for the SAT Student Descriptive Questionnaire can attest, the best indicator of dimness is indifference. Wrong ideas also sometimes require intelligent formulation. Spectacularly wrong ideas receive their negative appraisal in retrospect because they were compelling enough to do damage. Moreover, misbegotten hype might serve as a more commonly available vehicle of upward mobility for smart upstarts than truly transformative ideas.

Wisdom and top-down analytic ability sadly receive fewer lauds than the clichés “street smarts” and “common sense.” To illustrate the distinctiveness of wisdom from intelligence so as to support the notion that smart people might promote unwise ideas, I would like to analogize with my personal experience working with engineers and doctors. Many of the engineers whom I have known work hard on mundane projects but like to engage in fascinating discussions about politics, metaphysics, and, of course, the possibility of alien life forms. They contrast with neurologists and neurosurgeons whom I have met who regularly work with cases of alexia without agraphia and amygdalectomy but remain willfully indifferent to any profound questions these phenomena raise about the soul or free will. If emphasis on analytic ability (as opposed to memory) is analogous to “wisdom,” then the engineers are the aristocrats in this analogy, although engineers often zealously advocate for their upstart ideas. I think most people assume that doctors are smarter than engineers because becoming a doctor is harder and better reimbursed, but I would say engineers are deeper conversationalists, which manifests a certain type of wisdom to me.


Merit’s Mobility

Though the 90’s are long gone, Bill Clinton might still point out that this is not “midnight in America.” The American economy usually does expand, and smart people disproportionately tend to be the proverbial movers and shakers. Hints of this appear as evidence for their geographical mobility based on yearly maps of SAT and ACT data. Racial demographics also relate to test scores and have their own trends of time and place.


Last year, I graphed all data from state SAT reports for SAT scores of different family income brackets according to the proportion of each state’s population that belongs to the two highest scoring racial groups, whites and Asians. As the updated version above shows, students from the wealthiest category have the lowest association between state racial demographics and SAT score. One reader considered this “evidence against a genetic explanation.” Of course, homes earning less than $80,000 per year are not all dealing with “typical environmental problems of poverty.” Part of my study’s regression analysis looked at the effects on SAT scores that were common to both income and race. These effects existed when the income cutoff was $100,000 but not $20,000. Without completely dismissing the environmental hypothesis, I pointed out alternative factors that could be at work. An alternative explanation for the graph could be that higher income families with good test performance are more likely to live in racially diverse states compared to other income levels, since the racial proportions pertain to the states, not the income brackets, themselves. It turns out that SAT data does offer some support for my previous contention that “racially diverse states like California have industries or attractions that pull in successful, educated whites.”

Before revealing the map that demonstrates this, I shall review the latest SAT and ACT data by race. Composite ACT-SAT scores by race are very similar to the year before.


Native-American scores remain just slightly above those of Hispanic Americans after a significant drop in Native-American ACT scores from 2010 to 2013.


Overall, Asian scores have continued their amazing progress. The Asian SAT mathematics subtest advantage over whites rose to 64 points.


Asians are not closer to surpassing whites on the critical-reading subtest only because white scores rose by the same amount this year.


In contrast to all other groups, Asian SAT writing subtest scores rose in seven of the past nine years.


However, Asian progress on the ACT did appear to stop. Their scores on the reading and math subtests even slightly dropped. The average score drop for Pacific Islanders, for whom scores are available only in recent ACT data, was quite severe this year: almost a full composite ACT point. Fortunately for Pacific Islanders, this probably only resulted from a massive, 19-percent increase in ACT participation. Hawaiian students, who were 83 percent Asian-American, according to SAT numbers, bumped up their ACT participation from 40 percent to 90 percent, while their SAT participation rate barely budged at 63 percent. Asian students have been the one racial group who heavily favor taking the SAT rather than the ACT. This is likely due in large measure to foreign students. As the following graph shows, ACT participation surpassed that of the SAT, driven mostly by white students.


It would be tempting to surmise that foreign students are causing the continued progress of Asians on the SAT, but, as I previously discussed, the scores of foreign students (represented on the following graph as colored lines of a negative advantage for American students) still closely resemble Asian SAT scores (represented as gray lines for a negative white advantage), while the number of foreign students could not overwhelm the number of Asian students.


Indeed, Asian progress has been so impressive that it calls into question some assumptions of experts in differential psychology and adherents to the philosophy behind so-called “human biodiversity.” Rather than reveal a unitary Asian-white general intelligence gap, Asians have always had a large mathematics advantage. Rather than maintain a constant mathematics gap as Asians improve their English skills, Asian mathematics, reading, and writing skills improved in tandem. Rather than have SAT scores that coincide with research on the secular IQ gains, known as the Flynn effect, Asian SAT gains have been large at the same time as their IQ gains have been small, and white SAT gains have been small, as large IQ gains in Western societies have continued unabated. Frey and Detterman famously called the SAT an IQ test in 2004, but they offered no explanation for why their IQ-estimation equation essentially eliminated the reading subtest. Murray recently defended The Bell Curve by pointing out how little the black-white test score gap has changed. So, what is the precise meaning of that, given that the Asian-white gap has changed so greatly?

What should be clear from the preceding review of new data is that participation rates greatly matter. Hawaii’s ACT scores fell this year by almost two points on a scale from 11 to 36, following its participation increase. In order to appropriately map composite ACT-SAT scores, I must follow my previously described methodology for adjusting scores according to state SAT and ACT participation rates.

For comparison, here is the map of the percentage of white and Asian SAT-taking students over the years:

 photo sat aw map.gif


















Student diversity increased especially among coastal states. I previously claimed, “Demographic changes correspond to falling test scores, and one can see it, at least in terms of a North-South divide, on these maps.” The most recent years of this participation-controlled composite SAT-ACT score map make me want to amend that assessment.

 photo sat-act part cont.gif


















One can more easily notice the change by just looking at the oldest and newest maps without animation.


The earliest year does suggest a North-South divide, but the coastal states of California, Georgia, South Carolina, and North Carolina improved, while some Mountain states declined. The trend could be a fluke, and a few states buck the trend, but it fits with my previous explanation of diverse states attracting relatively well scoring students from wealthy families. The upper class wishes to live near beaches and in high-status states with impressive cultural and educational institutions. Many of the cognitive elite actually might like some kinds of racial diversity. Southern states like Louisiana have not improved, or, in the case of Mississippi, scores improved but were already extremely low. Perhaps the cognitive elite would be attracted to this Southern coast but find Southern culture too alienating, and maybe such a feeling of alienation from otherwise attractive settings makes liberal condescension slightly more understandable.

In an era that made “big data” a catchphrase, the colossal data pool that describes the colossal sample who took these tests inspires elite news outlets to make bar graphs of simple correlations and reports of flat scores. I call that flat reporting.



ResearchBlogging.org






nooffensebut (2014). Parents’ Income is a Poor Predictor of SAT Score Open Differential Psychology, 1-19

Friday, July 4, 2014

Parents’ Income Poorly Predicts SAT Score


Abstract
Parents’ annual income lacks statistical significance as a predictor of state SAT scores when additional variables are well controlled. Spearman rank correlation coefficients reveal parents’ income to be a weaker predictor of average SAT scores for each income bracket within each state than parents’ education level as a predictor of average SAT scores for each education level within each state. Multiple linear regression of state SAT scores with covariates for sample size, state participation, year, and each possible combination of ordinal variables for parents’ income, parents’ education, and race shows income to lack statistical significance in 49% of the iterations with greater frequency of insignificance among iterations with higher explained variance. Cohen’s d comparisons of the yearly individual SAT advantage of having educated parents show a fairly consistently increasing positive relationship over time, whereas similar analysis of the yearly individual SAT advantage of having high-income parents shows variability somewhat coinciding with the business cycle.

Read the whole study at Open Differential Psychology.

See below for important excerpts and extra super-awesome graphs.

“Sackett et al (2009) recounted a series of accusations that the SAT merely measures family wealth. The College Board’s announcement of 2016 SAT reforms has stirred anew claims that 'the only persistent statistical result from the SAT is the correlation between high income and high test scores' (Botstein, 2014). Thus, income as an important predictor of SAT scores somewhat fits a view critical of the SAT, which is that financial resources and class privilege unduly enable higher SAT achievement. If the education component of socioeconomic status dominates over the income component, then the relationship between socioeconomic status and scores might instead more accurately reflect a family’s values towards education and a hereditary influence shared between test performance and educability.”

“This study seeks to thoroughly parse the effects of multiple covariates, including family income, parents’ highest education level, and potential confounding variables specific to state or multiple-year comparisons. To do this, full advantage will be taken from all sixteen years of state data.”


Income p-values without race as a covariate (p-values are shown on an inverse logarithmic scale)


Income p-values with race as a covariate (p-values are shown on an inverse logarithmic scale)

“The racial variable was the most consistently significant variable of these three ordinal variables for composite scores and subtests, which speaks to its independence from socioeconomic status. Race also explained much of the SAT advantage that appeared to be attributable to parents’ income prior to the addition of the racial variable in iterations with low income thresholds simultaneous with the education cutoff being graduate degree.”


Critical-reading and mathematics standardized coefficients compared to adjusted R2 values, organized by education first, race within each educational category, and income within each racial category

“Parents’ income has a significant association with SAT scores, but parents’ education is consistently stronger, and regression with effective controls for race, education, and other factors, usually suppresses the income variable to insignificance. The income variable achieved significance when the education threshold was high school diploma most likely because so few parents were dropouts that education was no longer effectively controlled, and parents’ income became a proxy variable for parents’ education…. Part of this dominance could result from heritability in test performance corresponding to parents’ educational attainment, given the high heritability estimates from twins studies for high-stakes standardized exams in the UK and the Netherlands (Bartels et al, 2002; Shakeshaft et al, 2013).”

“Figure 1 seems to contradict Dixon- Román et al in finding that the racial variable had its greatest influence at the highest education level and at high income levels.”

“Asian Americans have historically high average mathematics subtest scores but lower verbal/critical-reading average scores than the white majority…. Despite their likely small average verbal disadvantage and small population in many states, this study’s consistent regression results for Asian race match verifiable individual SAT-score phenomena. A study with fewer observations, a much smaller represented sample, or fewer or poorly chosen covariates might not have achieved that level of definition, but, fundamentally, states do not take the SAT; people do.”


Cohen’s d SAT advantage of having parents’ annual income above $60,000

“Family educational advantage seems to evince virtually undeviating growth as a predictor of SAT scores, but financial advantage seems to grow as the economy worsens. Rather than postulate that times of economic difficulty almost immediately make wealthy people smarter, one should focus on the exclusivity of the income category…. The declining relative income advantage on the mathematics subtest compared to the critical-reading subtest also could be related to structural changes to the economy since the decline of the high-technology boom of the 1990’s, which also fits this interpretation of persistence within families.”

For those readers who do not have a heart condition, I recommend the spirited and colorful statistics debate in the open peer-review forum. One may also find there data supplements of state data that required many months of typing out the data from 816 state reports into a database, which makes a fun toy.



ResearchBlogging.org






nooffensebut (2014). Parents’ Income is a Poor Predictor of SAT Score Open Differential Psychology, 1-19

Ariel Investments. (2010). The Ariel Investments 2010 black investor survey: Saving and investing among higher income African-American and white Americans. Retrieved April 1, 2014 from http://www.arielinvestments.com/landmark-surveys/

Balf, T. (2014). The story behind the SAT overhaul. New York Times. Retrieved March 25, 2014 from http://www.nytimes.com/2014/03/09/magazine/the-story-behind-the-sat-overhaul.html.

Bartels M, Rietveld MJ, Van Baal GC, & Boomsma DI (2002). Heritability of educational achievement in 12-year-olds and the overlap with cognitive ability. Twin research : the official journal of the International Society for Twin Studies, 5 (6), 544-53 PMID: 12573186

Botstein, L. (2014). College president: SAT is part hoax, part fraud. Time. Retrieved March 25, 2014 from http://time.com/15199/college-president-sat-is-part-hoax-and-part-fraud/.

Buchmann, C., Condron, D., & Roscigno, V. (2010). Shadow Education, American Style: Test Preparation, the SAT and College Enrollment Social Forces, 89 (2), 435-461 DOI: 10.1353/sof.2010.0105

Dixon-Román, E.J., Everson, H.T., & McArdle, J.J. (2013). Race, poverty and SAT scores: Modeling the influences of family income on black and white high school students’ SAT performance. Teachers College Record, 115 (4), 1-33

Duckworth AL, Quinn PD, Lynam DR, Loeber R, & Stouthamer-Loeber M (2011). Role of test motivation in intelligence testing. Proceedings of the National Academy of Sciences of the United States of America, 108 (19), 7716-20 PMID: 21518867

Duncan, J., Seitz, R.J., Kolodny, J., Bor, D., Herzog, H., Ahmed, A., Newell, F.N., & Emslie, H. (2000). A Neural Basis for General Intelligence Science, 289 (5478), 457-460 DOI: 10.1126/science.289.5478.457

Everson, H.T., and Millsap, R.E. (2004). Beyond individual differences: Exploring school effects on SAT scores. (RR-2004-3). New York: College Board.

MacCallum RC, Wegener DT, Uchino BN, & Fabrigar LR (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological bulletin, 114 (1), 185-99 PMID: 8346326

Marioni RE, Davies G, Hayward C, Liewald D, Kerr SM, Campbell A, Luciano M, Smith BH, Padmanabhan S, Hocking LJ, Hastie ND, Wright AF, Porteous DJ, Visscher PM, & Deary IJ (2014). Molecular genetic contributions to socioeconomic status and intelligence. Intelligence, 44 (100), 26-32 PMID: 24944428

Neter, J., Wasserman, W., and Kutner, M.H. (1983). Applied Linear Regression Models. Homewood, IL: Richard D. Irwin, Inc.

Prescott, B.T., and Bransberger, P. (2012). Knocking at the College Door: Projections of High School Graduates (eighth edition). Boulder, CO: Western Interstate Commission for Higher Education.

Sackett PR, Kuncel NR, Arneson JJ, Cooper SR, & Waters SD (2009). Does socioeconomic status explain the relationship between admissions tests and post-secondary academic performance? Psychological bulletin, 135 (1), 1-22 PMID: 19210051

Sackett PR, Kuncel NR, Beatty AS, Rigdon JL, Shen W, & Kiger TB (2012). The role of socioeconomic status in SAT-grade relationships and in college admissions decisions. Psychological science, 23 (9), 1000-7 PMID: 22858524

Schmitt N, Keeney J, Oswald FL, Pleskac TJ, Billington AQ, Sinha R, & Zorzie M (2009). Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact on demographic status of admitted students. The Journal of applied psychology, 94 (6), 1479-97 PMID: 19916657

Shakeshaft NG, Trzaskowski M, McMillan A, Rimfeld K, Krapohl E, Haworth CM, Dale PS, & Plomin R (2013). Strong genetic influence on a UK nationwide test of educational achievement at the end of compulsory education at age 16. PloS one, 8 (12) PMID: 24349000

Trzaskowski M, Harlaar N, Arden R, Krapohl E, Rimfeld K, McMillan A, Dale PS, & Plomin R (2014). Genetic influence on family socioeconomic status and children's intelligence. Intelligence, 42 (100), 83-88 PMID: 24489417

US Census Bureau. (1990). Asians and Pacific Islanders in the United States. Retrieved March 23, 2014 from https://www.census.gov/prod/cen1990/cp3/cp-3-5.pdf.

US Census Bureau. (2013). Asian/Pacific American Heritage Month: May 2013. Retrieved March 24, 2014 from https://www.census.gov/newsroom/releases/pdf/cb13ff-09_asian.pdf.