Florida botches release of teacher evaluation data

The Florida Department of Education yesterday released teacher evaluation data to the public and then held a conference call for reporters. Within about an hour of the release, the data quickly got pulled down, The Tampa Bay Times reports.

It seems FDOE published the data before checking it, data designed to inform the public about the percentage of teachers at public schools that were rated as “highly effective,” “effective,” “needs improvement,” “developing” or “unsatisfactory.” More on this checking part later …

Anyway, when the Times asked FDOE spokesperson Cynthia Sucher how such a mistake could happen, on such a high-profile project, she reportedly pointed out that it was a brand-new system, with preliminary data that is scheduled to be finalized later. Also, individual school districts had some latitude in creating their own evaluation systems, so there are differences from county to county.

Therefore, when looking at data supplied by the school districts, “you’re not comparing apples to apples, you’re comparing apples to grapes,” the Times quoted her as saying.

The FDOE was expected to give it another try today.

Data managed (and reported) by non-educators

I haven’t checked how today’s version worked out, but even if it was smooth, this incident underscores a growing problem among boards and departments of education, especially at the state level. As a former database and system administrator for a mortgage company, I recognize this problem, and I would like to share with you some experiences from my past and current life.

First, let’s all agree that mortgage companies in the 1990s and state departments of education today share a problem they have to solve: the behemoth databases required to keep track of what they do make it absolutely necessary to hire people who know databases, not mortgage pricing or education.

When I would receive technical specifications for a new database table or a screen on a client program I was developing for some business unit at the mortgage company, I had to do a lot of translation in order to express the “technical” requirements from the business unit in terms programmers and database administrators could understand. I had no idea if I was translating correctly.

About a year and a half ago, I was reviewing test items for Maryland’s No Child Left Behind test for third or fourth grade (the exact details will not be reported for obvious reasons). I came across a test question that I believed had no correct answer choice among the four options. Worried that I must be losing my mind, I showed the question to a co-worker, not an educator by training but one of the computer programmers we have had to hire.

The question itself asked students to identify which answer option showed two congruent shapes. I believe she didn’t know what “congruent” meant, since the answer she gave me indicated she was confusing “congruent” with “similar.”

Sketchy reports from Florida—and believe me, they won’t get any clearer—indicate that the error occurred in the released data because some schools entered more than one “job code” for several teachers. This is a common database programming error made by inexperienced database administrators—or simply those that didn’t ask or have the opportunity to ask the questions they needed to ask of those who spelled out the requirements. The error will go unnoticed in Florida, because the people managing the programmers are probably educators who have no idea what referential integrity is. They probably know less about the SQL “select” statement than my co-worker knew about congruent shapes. That’s my guess.

What we suspect happened, based on the evidence

In all likelihood, FDOE’s database was thrown together quickly in order to get the data out to the public, and it wasn’t tested by actual educators who were relying on the system. Sure, it was no doubt tested by several programmers and database administrators, but I suspect their “translation” from the specifications provided by the educators was faulty or incomplete. And no one’s knowledgeable in both the education requirements and the database requirements to know there was any problem.

FDOE should have asked actual educators to review the data, as many noticed within minutes that their numbers looked high. Let’s look at a simplified version of how I would imagine FDOE’s database should be set up.

First, there’s probably a database table or collection for teachers. Each teacher would have one record in this table, and the fields associated with that teacher might be the following:

• last_name
• effectiveness_rating
• school
• teacher_id_number

Then, there’s probably a separate table identifying the job code(s) for each teacher. This table would be “linked” to the teacher table above by referring to the “Teacher ID Number” in the teacher table. The fields in the “jobcode” table might include the following:

• teacher_id_number
• job_code

Here’s where things got messed up. The programmers didn’t understand that any given teacher may have more than one job code. Some may be listed as “department chair” and “teacher,” while others are listed only as “teacher.”

Now, let’s imagine that the report FDOE wanted to produce would list all the teachers at a given school, showing their job code and their effectiveness rating. In SQL, the most common database language for the last several decades, this was probably written as follows:

SELECT teacher.last_name, teacher.effectiveness_rating, jobcode.job_code
FROM teacher INNER JOIN jobcode
ON ( jobcode.teacher_id_number = teacher.teacher_id_number )
WHERE teacher.school = 'My Elementary School';


Of course, My Elementary School would be replaced once the person running the report selected an actual school from Florida’s database.

This would produce one record for every job code, not one record for every teacher, and the total number of teachers would be reported high. Let me fix this for FDOE. It should be programmed as follows (it’s a little fancier, but nothing a contract database administrator worth a job at FDOE shouldn’t be able to read in his sleep):

SELECT teacher.last_name, teacher.effectiveness_rating,
MIN( jobcode.job_code ) , teacher.teacher_id_number
FROM teacher LEFT JOIN jobcode
ON ( jobcode.teacher_id_number = teacher.teacher_id_number )
WHERE teacher.school = 'My Elementary School'
GROUP BY teacher.teacher_id_number ORDER BY teacher.last_name;


Now, of course, this only results in one job code being listed for each teacher, but at least the number of teachers at the school is correct. This was what got Florida’s good teachers in an uproar.

The way this happened at FDOE shines an embarrassing light on what’s happening across the country: The talent level of programmers needed to maintain the referential integrity and perfect normalization of huge databases exceeds the computer ability of educators who understand what it all means—and, in this case, why it is absolutely illogical to write (and release) a program that assumes teachers have only one job code.

Let’s test our code, just to make sure I’m not giving FDOE bad programming advice. Not that they would recognize it as such, as this amateur programming hour in the Sunshine State has so clearly demonstrated, but just for my own peace of mind:

I created the tables described above with four teachers at “My Elementary School.” Their names are Owens, Mills, Roberts, and Wilson. Wilson has no job code, and the others have an assorted array of jobs. Running the first SELECT statement, we can see what those job codes are and how they match up with my teachers:

last_name, effectiveness_rating, job_code
Owens, Highly Effective, Teacher
Owens, Highly Effective, Coach
Owens, Highly Effective, Department Chair
Mills, Highly Effective, Teacher
Roberts, Effective, Department Chair


That says we’ve got five teachers, and it doesn’t even count Wilson. BIG problem! Now, let’s run it how it should be run. We get this:

last_name, effectiveness_rating, job_code, teacher_id_number
Mills, Highly Effective, Teacher, 3
Owens, Highly Effective, Coach, 2
Roberts, Effective, Department Chair, 4
Wilson, Highly Effective, NULL, 1


Finally, I want to point out that any programming task has an infinite number of solutions. The one I showed is not likely to match Florida’s solution. For example, I could have extended my WHERE clause to test jobcode.job_code = ‘Teacher’ or something. That also would have limited the report to one record per teacher but it would not have included teachers who had no job code entered, another possible user interaction issue. Real educators would have spotted this problem right away, and programmers could have then changed how the tables (or collections) were joined before the public spectacle.

Easy. What on Earth are we spending our citizens’ money on down there, folks?!