Graham v MOA #8: The Exams 1: How The Process Works

[The Graham v MOA tab above lists all the posts in this series and gives some overview of the case and why I think it's important.]

The Exams - How The Process Worked In 2012

The exams firefighter Jeff Graham sued the Municipality of Anchorage over were to determine who would be promoted from the entry level Anchorage Fire Department (AFD) position - firefighter - to the next level - engineer.  A firefighter has to promote to engineer to move up in the AFD.  If you passed the exams, you would go on a list based on your score.  Then, as there were openings for engineers, names would be taken from the top of the list.  If the list was used up before the next scheduled exam - in two years - an interim exam could be held.  If you were on the list, but not called in two years, then you had to take the exams all over again.  

In 2012, the exams had three parts:  1) a written exam, 2)  a practical exam, and 3)  an oral exam.  It was the first year an oral exam was part of the engineer promotion exams.   It was the oral exam that Graham objected to.  You had to pass ALL THREE parts to get on the engineer list.  The first two exams were relatively objective.  The oral exam was problematic in many ways, giving graders lots of leeway to pass or fail candidates without much accountability.   Let’s look at them all before going into the details of the oral exam in the next posts.  

The Written Exam 

This was based on a standard set of questions from a national bank of engineer test questions about technical knowledge.  It’s multiple choice.  Test makers can choose from many questions. This allows them to modify the test to be appropriate to local conditions.  Overall, the bank of questions has been validated nationally- the questions are related to what a fire engineer should know and this national association determines the  correct answers.  

Graham passed this exam with a score of 85.  He needed 70 to pass.  

The Practical Exam

The practical exam is made up of a series of 'evolutions,' as they call them, that test the candidates' abilities to handle the trucks and equipment as needed on duty.   The evolutions (think of them as scenarios) involve actually driving vehicles, hooking up hoses, responding to different types fires, etc.  This exam was designed by local test makers.  Casey Johnson was in charge of this in 2012 and he followed the basic model that had been used in previous exams, but creating his own specific scenarios. 

The different evolutions on the exam are supposed to be unknown to the test takers until they take the test.   But the exams are given outside on consecutive days and people taking it on the second or third day can learn from others what events they will be asked to respond to.

A related issue that came up has to do with training outside the Academy.  Senior AFD officers often assist firefighters in their stations when there are no emergency calls.  So different candidates will get different coaching on different possible evolutions at their stations.  In one case, it was argued that one of the people who helped prepare the practical exam gave his subordinates, at the station, training on a new process that hadn’t been used at AFD yet, but was on the exam.  Questions were raised whether they had gotten advance information to prepare for that event.  The suggestions were denied.  

This exam was not validated professionally.  Jeff Graham has not challenged the events on this exam - they are related to what people have to do as engineers, but whether successfully completing the events on this test is the best, or even a good, predictor of success as engineer is not known.  

Graham did have some questions about the reliability of the exam.  Scores on the first day of the exam were low and the fail rate was very high.  By the third day, the fail rate dropped significantly.  Why might this be?

The exams are done out in the open where they can be seen by anyone.  The first people to take the exam do not know what they will be asked to do.  By the third day, people have been able to see what events were used, plus people who took the test can talk to friends who haven’t taken the test, so the later test takers can be better prepared.  

There was also some unconfirmed discussion at trial about whether the grading standards were loosened by the third day because the success rate was so low.  Which would raise questions about how the grading criteria were established.  

A reliable test is one where a test taker’s score should not vary regardless of the conditions of the test - which includes what day they took the test.  All test takers must face the same test conditions for the test to be reliable.

Jeff Graham passed the practical test comfortably.

The Oral Exam

The oral exam was created especially for the 2012 exam, by Casey Johnson.  Oral interviews had been held for higher level positions, like captain, but not for the technical job of engineer.  The exam consisted of two parts:  1)  a technical part and 2)  a “peer review.”  This is the part that Jeff Graham failed, by one point.  This was the part that Graham complained about before the exams even began, after he was told he failed the exam, and later to the Alaska Human Rights Commission, and finally in court.  

The technical part consists of ten questions, supposedly about technical issues, though the 2012 exam had two questions about how to prepare for the test and some that were more AFD policy and administration questions.  

The ‘peer review’ consists of five questions that seem to be designed to determine whether someone’s character is good enough to become an engineer.  

There are five testers for the oral exam.  Engineers are asked to volunteer to assist with various parts of the Academy (the training program designed to prepare people for the exam).  The Academy administrators, in this case Chad Richardson and his assistant Casey Johnson, decide who will perform what duties at the Academy and in the exams.  They can also encourage people to apply, which at least a couple of the testers said happened to them.  

Before the exam takes place, the testers pre-grade the peer review part of the exam.  That means, they give each test taker a score based on their knowledge of that person.  There was mention of reviewing the application for promotion, but graders had different responses about whether these were considered.  If they have no knowledge of that person, they can leave that part blank. So, even though they, theoretically, had access to someone’s application, they could skip the pre-score, which suggests that either the application wasn’t important, they didn’t look at the applications, or prior personal knowledge of the candidate was the key factor.  There was conflicting testimony about when this pre-grading was done. Graders were asked to come in early and do things like score candidates on the pretest.  But testimony showed  that didn’t necessarily happen.  Pre-grade scoring could be done in the morning before the testers came in, before anyone was tested, or before each individual came in to be tested.  

The Peer Review test process

The candidate walks into the room.  He’s given his pre-scores on the peer review.  He then has an hour to answer 15 questions - the ten technical questions and the five character questions.  That gives someone about four minutes per question.  The questions are projected on a screen and the candidate begins answering them.  

If the candidate got a passing score on the peer review pre-score, he can elect to skip any or all of the peer review questions and spend more time on the technical questions.  This, obviously, gives an advantage to people who were pre-scored well.  

Jeff Graham’s pre-score grades were below the needed 70. He got 69.  He was surprised by this.  

Overall Test Scoring

To pass the engineer exam, candidates have to pass all three parts of the exam.  That means that if they fail any of the three parts of the exam (get less than 70%), they fail the whole exam - even if their overall average on all the exams was 71% or 80% or 89%.  

Since you have to get 70% or better on ALL three exams, if someone gets a 69 on the first exam (the written exam), they do not go on to take the practical exam.  Those who pass the written and then  pass the practical exam can go on to take the oral exam.  

I don’t have the exact numbers available, but a large number took the written exams and fewer took each succeeding exam.  At trial, Deputy Chief Hetrick said people who made it to the orals had good scores - around 85 or more - on the written exams.   

From Exam To Promotion

Passing the exam doesn’t mean a firefighter will be promoted to engineer.  Those who pass go on a list based on their scores.  The higher the score, the higher they are on the list.  When there is an opening for an engineer, the top person on the list is promoted.  The list is good for two years.  If all the people on the list are promoted before the two years are up and they have new openings, they can have an interim Academy and test.  

Anyone left on the list after two years is no longer eligible and has to take the whole exam again. 

The cost of Academies is quite high in money and in time. It is the full time job for at least two people (in 2012 Chad Richardson and Casey Johnson) for a period of time, plus the time of all the volunteers and all the candidates.  Then there is the equipment and other things used.  The practicing on various rigs and gas that takes.  One figure I heard was about $60,000 but I don’t have confirmation of that.  

The Meaning of the High Fail Rate

A lot of people go to the Academy and a relatively small number make it onto the promotion list each time.  We can’t be sure why so many fail, but there are several possible explanations that come to mind.  

1. People take the Academy to learn more about the promotion process and they might take the written test just to see how they do.  Sort of a  trial to gauge how difficult it is and how much they’ll have to study when they take it seriously.  People mentioned this was the case for some.
2. The quality of the firefighters is low.  Only a high school degree or a GED is required.  They may not be particularly good at studying and/or test taking.
3. The training at the academy is inadequate to prepare most people to pass the exams.  
4. The tests are necessarily rigorous to make sure only the most technically competent are promoted from firefighter to engineer.
5. The tests are unnecessarily difficult or harshly graded. 
6. Fewer engineers means more overtime for those who are engineers 

I suspect there is an aspect of all six reasons (and perhaps some I haven’t thought of.)  

Let me explain number six a little more.  Because of the 24 hour shifts several days a week, AFD line employees have a lot of time away from work.  Many use this time to run other businesses.  But for many others this is an opportunity to work overtime.  Not only does the overtime give them time-and-a-half pay, it also raises the annual pay that their retirement benefits will be based on.  Some have argued that by making the testing for engineer difficult, the pool of engineers is kept small, and those who become engineers can work more overtime.  At trial, the MOA’s expert witness hired to calculate possible compensation for Graham in the chance he won, testified that Graham had very little overtime compared to many who had 1000-2000 hours of overtime.  Firefighters work three days of 24 hour shifts per week.  1500 hours of overtime would be the equivalent of 6 weeks.  That’s a lot of overtime and a lot of pay at time and a half.  One has to ask whether hiring more employees would make overtime less necessary and save the MOA money.    

I realize this bit on overtime goes beyond just an overview of the exams, but I’m not sure where else it will fit in, and it gives context to questions that come up about the exams overall.  I’m raising this issue because it came up. I don't know how significant it is.  I haven’t studied it, but it seems like something that should be followed up on.  

The Devil is in the Details

The next posts on Graham v MOA will focus on the details of the oral exams.  

Crimes of violence tend to be very tangible, very graphic.  We can imagine someone with a gun threatening a mugging victim or a bank teller.  We can imagine a stabbing.  We can see it vividly.  These are crimes that involve people on people violations.  

But administrative crimes are much harder to imagine.  They are structural crimes that are less visible and easier to hide.  They are tied up in details, rows of numbers, pages of text. Easier to conceal.  

How a scoring sheet for a test is designed, can make the difference between whether someone passes the exam or not.  This is why we hear stories of people who have embezzled money for years and years before they were caught.  A petty thief can face much stiffer legal obstacles than a white collar criminal who has bilked people of millions.  The latter crime seems less problematic because it's so abstract and harder to visualize.  

The details can be tedious.  One reason I’m slow in getting these posts out is that I’m trying to make them as easy to read and understand as possible.  'Interesting' is a goal, but that's more elusive.  I keep revising and revising and eventually I say, ok, enough, just post it already.  Even though I’m sure it’s still hard for the average person with a busy schedule to read, let alone digest.