Do you want to understand the A-F accountability system that we use to evaluate Texas public schools without reading through 180+ pages of formulas and scaling tables?
Well, sit back. Relax. Grab some coffee.
It's time to explore the A-F system.
The Texas A-F Accountability System utilizes an incredibly complex, somewhat controversial method for evaluating the academic performance of districts and campuses. The purpose of this system is to "simplify" school evaluation into terms that the general public can understand: a grade of A, B, C, D, or F.
While this sounds perfectly awesome (and it was no doubt well-intentioned), the final letter grade that schools and districts receive overly simplifies a massively complex set of data that cannot be accurately summarized with a single letter.
In addition to providing an overall letter grade to every campus and district, the A-F system also breaks that single letter grade into three areas, called domains, each with their own letter grade. Those three domains are: Student Achievement, School Progress, and Closing the Gaps.
For the visual people out there:
Each of these three domains have various metrics that are used to determine these letter grades.
In this post, I'm going to explain the methodology used to calculate the first domain: Student Achievement.
DOMAIN 1: STUDENT ACHIEVEMENT
As you can see above, the state explains that this domain "shows how much students know and are able to do at the end of the school year."
And while that summary is technically correct, as with the overall letter grade, it oversimplifies what this domain actually measures.
The first thing to understand about the Student Achievement Domain: districts and high schools are evaluated differently than elementary schools and middle schools.
For districts and high schools, there are three components to the Student Achievement Domain:
- STAAR Performance
- College, Career, and Military Readiness
- Graduation Rate
For elementary campuses and middle schools, this domain is limited to only STAAR Performance.
So what do these three components measure?
STAAR Performance Component
This component measures exactly what you'd expect: student performance on STAAR assessments. However, as you'll probably guess, how this is calculated is fairly complex.
This conversation starts with performance levels.
There are four performance levels that a student can earn on every STAAR exam that they take: Did Not Meet Grade Level, Approaches Grade Level, Meets Grade Level, and Masters Grade Level.
To put numbers to that, for the 3rd grade reading STAAR test, a student earns Approaches after answering 53% of the questions correctly, Meets at 76%, and Masters at 85%. Any student scoring below 53% did not meet grade level on their assessment.
Under STAAR Performance, there is an assumption that every student on every campus in every district could (in theory) earn Masters Grade Level.
This assumption has a direct impact on how this component is calculated.
For STAAR Performance, every district and every campus has 300 possible points they could earn: 100 points representing the possibility that 100% of students will score at Approaches on their STAAR test, 100 points for 100% of students scoring at Meets, and 100 points for 100% of students scoring at Masters.
If a district were to have 65% of their students score at Approaches, 30% score at Meets, and 15% at Masters, their STAAR Performance component score would be 37 (36.66 rounded up), representing the fact that the district earned 37% of the possible 300 points they could have earned.
This component score is then "scaled" using a scaling table/formula, a process that I will explain in a moment.
In short: The STAAR Performance component of the Student Achievement Domain is calculated by adding up the total percent of students who scored at Approaches, Meets, and Masters performance levels and dividing by 300. This calculation is based on the assumption that all students can score at the Masters level on every STAAR test. For elementary and middle school campuses, this is the only calculation that is used for the Student Achievement Domain.
QUICK NOTE ABOUT SCALING
One of the key "benefits" of the A-F accountability system is that the grades districts and campuses receive are supposedly easily understood by the majority of the general public because most people experienced an A-F grading scale during their childhood.
However, in order to convert (or scale) the raw component scores to a letter grade (A-F), the state uses a series of complex formulas that are based on thresholds determined via a forced distribution method of the 2017 accountability data...bla bla bla....
Let's just say...
..it's complicated.
All you really need to know about the scaling process is that districts and campuses receive a raw score for every component of the A-F system, and then this raw score is converted to a scaled score (via complicated formulas) in order to create a letter grade of A, B, C, D, or F.
On to our next component...
College, Career, and Military Readiness (CCMR) Component
The CCMR component only applies to districts and high schools.
This component is fairly straightforward: take the number of graduates from your district or high school that met one of the state's CCMR criteria and divide it by the total number of graduates.
The CCMR criteria include:
- Graduating with an associate's degree
- Earn an industry-based certification or Level I / II certification
- Enlist in the military
- Earn a qualifying score on an AP or IB exam
- Earn 3+ dual-credit hours in English or Math, or 9+ hours in all subjects
- Meet the TSI criteria via a qualifying score on ACT, SAT, TSI, or passing a college prep course
- College credit via OnRamps course
Under this component, the assumption is that every student is capable of meeting one of these criteria; therefore, the calculation works out to a simple percentage of total graduates who met one of the CCMR criteria.
For example, if a district had 200 of their 250 graduates meet one of the above criteria, their CCMR component score would be 80, representing the fact that 80% of their graduates are college, career, or military ready.
As with STAAR Performance, the score for this component is also "scaled" using the process outlined above.
In short: The College, Career, and Military Readiness component of the Student Achievement Domain is calculated by determining what percentage of a district's or high school's graduates met one of the state's CCMR criteria.
Graduation Rate Component
You would think that the graduation rate component would be fairly straightforward; however, as with almost everything else with the A-F system, there is some hidden complexity here as well.
In a simple world, the graduation rate component would simply be a district's or high school's graduation rate from the school year for which they are being evaluated. This is what the state would call a "4-year graduation rate."
And while this is one of the metrics that is used in the Graduation Rate component of the Student Achievement Domain, there are two other metrics the state looks at in addition to the 4-year rate.
Those two metrics are the 5-year graduation rate and the 6-year graduation rate.
All three of these rates--4-year, 5-year, and 6-year---are identical in practice: follow a cohort of students beginning in 9th grade and calculate what percentage of students from that cohort graduate 4 years later, 5 years later, or 6 years later.
The state then looks at these three rates, selects the highest rate, and then that is your district's or high school's graduation rate under the Graduation Rate component.
For example, if a district has a 4-year graduation rate of 97.5%, a 5-year rate of 98.2%, and a 6-year rate of 96.5%, their Graduation Rate component will be 98.2% (the highest of the three).
This percentage is then scaled using a scaling table that is unique to the Graduation Rate component...
...fun times.
In short: The Graduation Rate component of the Student Achievement Domain is calculated by taking the highest of a district's or high school's 4-year, 5-year, or 6-year graduation rates and then scaling it using a complicated scaling table.
Calculating the Final Domain 1: Student Achievement Score
Once all three components of the Student Achievement Domain have been calculated and scaled, you still have to combine these scores into a single score.
First, remember that for elementary and middle school campuses, the first component of this domain---STAAR Performance---is the only component that is used to calculate the final Student Achievement Domain score.
So, for an elementary or middle school campus, if their STAAR Performance component is 41% (again, representing that they earned 41% of the possible 300 points available from their combined Approaches, Meets, and Masters performance levels), then their raw Student Achievement Domain score is a 41. This value is then scaled using the scaling formulas, resulting in a final STAAR Performance value of 70. This value---70---is supposed to indicate that an elementary campus that earns 41% of their possible STAAR Performance points has a performance level equivalent to C (more specifically, a 70).
For a district or high school, the scaled scores from all three components---STAAR Performance, CCMR, and Graduation Rate---are weighted and then combined to calculate the Student Achievement Domain score.
This calculation works out as follows:
- STAAR Performance (weighted at 40%)
- CCMR (weighted at 40%)
- Graduation Rate (weighted at 20%)
If a district had the following component scores:
- STAAR Performance = 37 (raw) = 64 (scaled)
- CCMR = 80 (raw) = 95 (scaled)
- Graduation Rate = 98.2 (raw) = 95 (scaled)
...then the district's Student Achievement Domain score would be calculated using the following formula:
- (64 x .40) + (95 x .40) + (95 x .20) = 25.6 + 38 + 19 = 82.6 (or 83 after rounding up)
So, if a district's students earned 37% of their possible Approaches / Meets / Masters points on their STAAR tests, 80% of their graduates met one of the CCMR criteria, and the district had a 98.2% 4-, 5-, or 6-year graduation rate, then that district would be a B (more specifically, an 83).
What you just read was the calculation process for the Student Achievement Domain (Domain 1) of the Texas A-F Accountability System, which is only one of three domains.
We still have two more domains to discuss....
A couple things to note:
1) Obviously, this system is complicated. While the resulting grades may see simple---I mean, who doesn't understand basic letter grades?---the data and calculations used to get to those letter grades are quite intricate.
2) This illusion of simplicity is a double-edge sword. On the one hand, this simplicity enables multiple districts to earn the same letter grade via multiple pathways. On the other, this same-letter-multiple-pathways system deceives the general public into thinking a single letter can encapsulate everything that is going on, good or bad, within a district.
For example, review this table of Domain 1 calculations for five different districts:
From the general public's view, all five of these districts are a solid B, each with a rating of 85.
However, in reality, the component scores for each district are very different.
While all five districts have the same overall score for Domain 1, District 2 is struggling with STAAR Performance yet doing well with CCMR, District 3 needs to address their graduation rate, and District 5 is doing a poor job of getting their students ready for life after high school.
As I said above, this masked complexity is a double-edged sword. Providing districts with multiple pathways for success is very helpful; however, at the same time, the fact that these ratings are still very much tied to the poverty level of a district should cause some concerns over the long-term ramifications of this overly simplified rating system.
Again, I've only covered the first domain with this post. Sadly, the second and third domains are even more complex, and therefore, I'm going to dedicate separate posts to explaining the Student Progress and Closing the Gaps domains.
Until then...
