Seward's Day Begins With Fire Trucks

Before the fire trucks, in fact yesterday, Sunday, we were at the Anchorage Botanical Garden Spring Conference downtown at the Dena'ina Center.  I'd never been to one of these before.  I was a bit underwhelmed, but I did get some ideas and tips and inspiration.  In this session (on the right) we learned how to make a liquid to spray on plants to get them the calcium, and boron they need to flourish.  

Most useful, I think, was meeting someone from the Anchorage Soil and Water Conservation District who will come to my house next fall and test the soil and make suggestions.  We've got some areas where only the hardiest plants survive.  I'm hoping that can be changed.  

But today I woke up to see two fire trucks across the street.  I was worried that a neighbor was having an health emergency, since there didn't seem to be a fire anywhere.  When I went out, I saw there were actually four AFD vehicles.  

Since I was out, I decided to walk around the neighborhood and get some blood moving in my veins. I kept wondering about why they needed so many vehicles for a paramedic call.  When I got back, the firefighters/paramedics (there are far more paramedic calls than fire calls) were walking back to the vehicles.  Not from the building across the street, but from around the corner.  

I asked one of them what was happening and he told me they had been viewing the house around the corner that had burned.  Which was when I realized that I'd read about a fire nearby while we were visiting out granddaughter Outside, but had forgotten about it.  And I was reminded again that it's always good to ask rather than assume.  

I also found out today that my very low carb diet, of the last four months, did indeed make a difference on my A1c blood test.  That was gratifying.  I'd thought that it hadn't made a difference based on another test result I got last week.  But this test wasn't in among the results until today.  

I also went to pick up a book on hold at the library.  The door I normally go in was locked, so I went over to the main entrance where I saw the sign that said the library was closed for Seward's Day.  I had gone to the library website to see how long they were going to hold the book, but there was nothing there that I saw to say the library was closed.  Oh well.  

This evening I walked over to see which house had burned.  It was an apartment building.  What is odd is that another house almost next door, burned down in  March 2016.  The red circle is the recently burned house.  The purple circle is the new house built where the 2016 house burned.  

Here's the building a little closer up.  Another neighbor came out to see what I was doing near the

burnt house.  He said he'd called the fire department that night and helped to get another family out.  There was a man who went back in to get his wife.  Both died. It was arson he said.  

I noticed that both news articles were written by the same reporter.  I'm guessing that he didn't visit the site this time because he should have noticed that it was practically next door to the previous fire.  

Hope you had a good Seward's Day and thought about the man who negotiated the purchase of Alaska from the Russians - who actually only occupied a relatively small portion of the land.  

Panhandling, Inflation, Clouds

 Despite three different topics in the title, this isn't going to be a long post.

1.  It's ok for firefighters, but not for the hungry

Lake Otis and Tudor is one of the busiest intersections in the city.  I also have to get across it on a couple of my regular bike rides.  

Two weeks ago it was crawling with firefighters raising money for charity.  Though collecting money in Firefighters' boots seems a little gross.  They didn't look like new, unused boots.  

That's an admirable activity.  But they were doing it standing in the intersection.  Some in the middle, others between the right turn lanes and the through traffic lanes.  

Photo by ADN photographer Marc Lester

Eighteen months ago, signs like this caused a stir in Anchorage.  

The ADN article tells us:

"The municipality spent more than $8,000 to post anti-panhandling signs at dozens of Anchorage’s busiest intersections in December — but the city law cited on the sign was found unconstitutional by a state court years ago."

"Corey Young, a spokesman for Mayor Dave Bronson, said the signs are meant to 'keep pedestrians away from dangerous situations in the roadway.'” 

It appears from the article that this was done by the mayor's office without consulting affected  departments like the Police Department.  I don't think anyone disputes the idea that there's an element of danger involved in walking the lines of cars at busy intersections, but the courts had said it couldn't be prohibited. 

If the mayor's office thinks this is dangerous, why are they letting the Fire Department do this?  Did the mayor's office even know the Fire Department was doing this?  

Or maybe we should ask if the original signs were an attempt to make those experiencing homelessness less visible to the general public, and danger wasn't the real issue.  

2.  Who's responsible for inflation

I like seaweed.  I don't eat it everyday, but I do now and then.  Last week I went to the Korean grocery story on Fireweed and Eagle to get some more seaweed.  Here's last year's empty package.

And the new one I got last week.  

The weight and number of servings are both the same.  It's at least a year since I bought the first package of seaweed there.  But the price of both is still the same!  

While national chain groceries have been rapidly raising their prices, this local Korean grocery is charging the same amount as they did a year ago - $9.99.  A similar product at Carr's, for instance, is advertised:

This is a total of .92 ounces for $8.99.   The Korean store seaweed is 65 servings at .07 ounces per serving, or 4.55 ounces!  One is $9.77 per ounce  and the other is $2.20 per ounce.

But my point isn't that you can get seaweed much cheaper at the local Korean grocery than at the chain store.  

It's really about inflation.  We know prices have gone up rapidly in the chain store groceries.  But on this item, the Korean grocery has kept the price the same for over a year.  No blaming inflation to raise the price, and adding further to inflation.  [But it's true that I don't know how much the Gimme packages were selling for a year ago.  It's possible that no one increased the price of seaweed.]

3.  Clouds

Anchorage has been having weather this month.  By that I mean wind and rain and sun all fighting it out.  I put up some cloud pictures two weeks ago.  Here are from one this week's bike rides.

Same corner as top pic but with little traffic and no fire department panhandlers. 

Taku Lake

4.  Biking.  And since I've mentioned bike rides, I reached my 1000 km goal for the summer (since April) and then got to 1100.  Getting most of my rides done on the local bike trails and getting regular views of places like Taku Lake make the riding a pleasure.  For lots of folks 600 miles is not that much, but it's kept me out exercising regularly all summer.  

Graham v MOA #12: Fire Chief LeBlanc Retires

First paragraphs from an ADN article:

"Anchorage Fire Chief Denis LeBlanc retired Friday after nearly three years on the job, officials said.
A successor to LeBlanc will be named in a few weeks, said Kristin DeSmith, spokeswoman for Mayor Ethan Berkowitz. Jodie Hettrick, the fire department's deputy chief of operations, is serving as interim chief.
LeBlanc, who is 70, said he told the administration of his decision in early May. He said he has loved the job, but he's been working on and off for 53 years, including a few decades in the oil industry. . ."

I'm including this hear because LeBlanc was fire chief while the Graham suit went to court.  He was never a fire-fighter (usually required for the chief's job).  He came from the oil industry, was City Manager for Begich, then went to CM2HHill (the company that bought VECO).

He was asked to get the AFD budget under control.  In his deposition he expressed no interest in reaching out to the community to increase the number of women or minorities in the AFD.  My hope is that the Diversity Mayor will be able to find a new chief for whom increasing the number of women (about 2% now) and people of color (about 12% now) in the fire department will be a major priority.

I'd note that LeBlanc was on the public administration advisory board when I was chair of the department at UAA and he is a very affable person who was always supportive of the program.

Graham v MOA #10: The Exams #3: The Oral Technical Exam

[All of the posts in this series are indexed at the "Graham v MOA' tab under the blog header above.  Or you can just click here.]

The details are complicated so here's an overview of some key problems; details below.

• Overall, oral exams (including interviews) are highly invalid and unreliable predictors of success on the job, despite the fact that they are used a lot.
• Imagine  a question about how to do something fairly complex like, "tell me what you know about brakes."  Where do you start?  You have four minutes.  Go.  
• The graders may or may not understand the answer, but they have four or five pages taken from a textbook,  they can scan through, with nothing to indicate what's important and what isn't.  
• The grading scale has uneven intervals - 5, 4.5, 3.5, 2.5, 1.5, 0 - with 3.5 being the lowest possible pass.  This odd scale is not what people are used to and skews the grades downward.
• You have to answer each question, they have to assess your answer, all in about four minutes.
• Lack of Accountability.  There's no recording, no way to check to see what a candidate actually said.  

That should give you an idea of the kinds of problems that exist with the oral technical exam that Jeff Graham took.

The Details

As I mentioned in an earlier post, firefighters first have to pass a written exam on technical issues.  Questions come from a national bank of questions that have been shown to be relevant to being an engineer (the next step above fire fighter).

Then comes the the practical exam.  Here the firefighters have to get onto trucks and use equipment based on 'evolutions'  (AFD's term) which are more or less scenarios that the firefighters have to role-play.  This is the most hands-on part of the exam process.

If you pass these two tests, then you can go on to the oral exams - made up of ten questions of the technical oral exam and then of five questions of the 'peer review' oral exam.  That's fifteen questions in an hour, or four minutes per question.

Jeff Graham passed the written and the practical exams.  His dispute was over the oral exams which he complained about as being subjective and easy to manipulate.  So let's, in this post, look at the problems with the technical part of the oral exams.

All Oral Exams, Including Interviews, Are Suspect

Before we even start on the oral technical exam,  I need to reiterate this point I made in an earlier post.

Despite the popularity of job interviews, experts agree that they are among the most biased and result in the least accurate predictions of candidate job performane.  

 You can read the whole article or search the internet on this topic and find I'm not exaggerating or cherry picking.  It's pretty much only people who conduct interviews who think they know what they're doing.

" interviewers typically form strong but unwarranted impressions about interviewees, often revealing more about themselves than the candidates."
"Research that my colleagues and I have conducted shows that the problem with interviews is worse than irrelevance: They can be harmful, undercutting the impact of other, more valuable information about interviewees."

Or see this link.  I offer a few quotes.

"Consider the job interview: it’s not only a tiny sample, it’s not even a sample of job behaviour but of something else entirely. Extroverts in general do better in interviews than introverts, but for many if not most jobs, extroversion is not what we’re looking for. Psychological theory and data show that we are incapable of treating the interview data as little more than unreliable gossip. It’s just too compelling that we’ve learned a lot from those 30 minutes."

Some of these comments are made about unstructured interviews.  The AFD engineer exam was a structured interview which researchers find to be a little better.  But still not that good.  From one of the most well known academics writing on human resources management, Edward E. Lawler III:

"Years of research on job interviews has shown that they are poor predictors of who will be a good employee. There are many reasons for this, but perhaps the key explanation is that individuals simply don’t gather the right information and don’t understand what the best predictors of job success are. A careful analysis of the background of individuals and their work history and work samples are more accurate predictors of success on the job than are the judgments which result from interviews."

The graders in the AFD oral exam did not have background information on the candidates' performance, they didn't review the performance reviews done by the candidates' supervisors.

Excuse me for reiterating this point.  It's one that is counter to people's perception and to practice.  But it's important to make this point loud and clear from the beginning.

My Problems With the Exams

On the positive side, the AFD oral technical and peer review exams were structured.  But there were numerous problems with the structure.

The Questions

There were ten questions on technical topics.  My understanding of the conditions of my reviewing the  exam questions themselves, is that I can't talk about the specific questions publicly.  Only things that were discussed publicly in court.

There were ten technical questions.  The MOA provided no evidence - though Graham's lawyer asked the MOA to provide documentation of how the exams were validated or even that the exam was validated.  That is, we have no evidence to show that the ten questions were predictors of a candidate's likely success in the position of engineer.  There is content related to the job, but AFD has produced no evidence showing that, for example, a candidate with a score of 90 will be a better engineer than a candidate with a score of 65. It may very well be that AFD may only be testing who has the best oral test taking skills.

Two of the ten were about how to prepare for the exam itself.  The test creator said these were intended to be softball questions to relax candidates.  Graham was scored low on them.  The other questions were about things like how equipment worked and about AFD procedures for different things.

The Answers

For tests to be reliable, the graders need to be able to compare the candidate's answer to the ideal answer.  The key points should be listed with the value of each point.  The graders were given a package of answers.  Some questions had no real answers attached.  Most looked like they were cut and pasted together from text books or AFD policy and procedure manuals.  For one question, I timed myself reading the four pages of answer that were provided.  It took me 11 minutes and 30 seconds just to read the answer.  But there are only about four minutes available per question. How can a grader reliably a) listen to the candidate's answer, b) take notes, and c) read through the answer sheet to match what the candidate said to the expected correct answer?  He can't.  Graham appears to have done better on the few questions that had bullet points on the provided answers rather than several pages raw material.

Assuming that the question itself is valid, the answer sheets graders had should have had a list of key points that the candidate should mention for each question.  The best such answer sheets would identify the most important points that should be in the response with a score for each.  There was nothing like that.  Instead the graders had to balance the answer the candidate gave, the pages of 'answer' copied out of the text or manual, and give the candidate a score on a completely different score sheet.

The Grading Scale

Rubric For Oral Technical Exam 5  - Outstanding, exemplary performance far exceeds expectations 4.5 - Above average performance exceeds level of basic skills/abilities 3.5  - Adequately demonstrated the basic abilities/skills 2.5 - Needs improvement, falls short of what is expected 1.5 - Unsatisfactory, performance is substandard 0 - Unacceptable, does not demonstrate comprehensive and/or application of required skills sets

Note:  There's a half point difference between "Outstanding" and "Above average".  Then it drops, not .5 like from 5 to 4.5,  but  1 point to 3.5 "Adequate".  So a grader could think, ok, this is good enough, it's adequate.   But 3.5 isn't 'adequate' it's really 'just barely adequate."  It's the lowest possible passing score.  The top two scores are very high marks.  The next one is barely passing.  1.5 is "unsatisfactory", but 3.5 (the same interval from 5 as 1.5 is from zero

But on the bottom of the scale, it goes from 0 to 1.5 - both of which are almost equivalent failing grades.  But the 1.5 is the same interval as from 5 to 3.5.



1.  This scale has uneven intervals. That is, the distance between the points on the scale are not the same.  Look closely.  The top two scores are both strong passing scores and the bottom two scores are both very poor failing scores.  But the top two are separated by .5 points, while the bottom two are separated by 1.5 points.   That's the same interval as from 5 to 3.5.

If 1.5 is 'unsatisfactory' the 3.5 should be "satisfactory' but it's only 'adequate' and more accurately 'barely adequate' because 3.5 is the lowest you can get and still pass. 3.4 is a failing grade.  (It's less than the 70% (20*3.5) needed to pass.)

The scale skews the scores down.  The points graders can mark go from 100% (5) to 90% (4.5) to 70% (3.5) which is the lowest possible passing score.  Why isn't there an 80% (4)?  That's what normal scales with equal intervals would have next.

2.  Passing on this test is 3.5. It took me a bit to figure that out, but 70% is the passing score, so each of these numbers need to be multiplied by 20 to get the actual score.  70% (3.5*20) is the minimal passing score.  Let's compare that to other scoring forms you know.  Say the standard Likert scale on most surveys:

"How do you feel about the Principal's performance this year?
5 – strongly approve
4 – somewhat approve
3 – neutral/no opinion
2 – somewhat disapprove
1 – strongly disapprove" (from Brighthub Education though it had the scale reversed)

Note that here a score of 3 is in the middle and is neutral, whereas in the AFD rubric, 3.5 is the lowest passing score.  The 3 or neutral would have a range from 3.5 to 2.5.  So a 3.5 would be on the high end, not the low end  as it is on the AFD scale.

I chose this 5 point scale, because the AFD calls its scale a five point scale.  But if you look at it, it's really a six point scale, since it has six possible scores from 0, 1.5, 2.5, 3.5, 4.5, and 5.  They didn't even realize they had a six point scale!   In most six-point Likert scales, there are three positive and three negative options. On such a scale (5, 4, 3, 2, 1, 0)  a 3.5 would be a strong passing score, not barely passing.

Let's look at it again from a different angle.  In the common academic scoring system, 90-100 is an A, 80-89 is a B, 70-79 is a C, 60-69 is a D, and below 60 is an F.

But in the AFD system, below 70 is an F.   On the AFD scale the first two numbers (a ten percent interval) are both very good scores.  The next score (20 percent less) is barely passing.  All the rest are failing scores.

Why does this matter?  First, it matters because the intervals are not equal.  It's not common to have uneven intervals  so that the distance between one score and another can vary from 10% (5 to 4.5) to 20% (4.5 to 3.5) to 30% (1.5 to 0).

It matters because the scale isn't like any scale we normally see and so the graders don't have a good sense of how skewed it is toward failing candidates.  The 3.5 isn't 'good', it's barely passing.  Yet most people are used to the academic scoring system and would assume that 70% would be something like a low C.  There is no 'good' on this scale.  There's 'walks on water,' 'really outstanding,' then 'barely passing'.  It took me a while to understand why the scale seemed so off.  I don't imagine an AFD grader would figure this out in the time they have to grade.  There isn't enough time to even figure out what the right score should be, let alone analyze the scale to see how skewed it is.

Reliability

Tests have to be both valid and reliable.  Valid means a high score predicts success (in this case, 'will perform the job well') and a low score predicts (in this case) 'will perform the job poorly'.  Even if the test itself is valid (and the AFD never did have anyone test for validity), they also have to be reliable if they will be accurate predictors.  Without reliability, the scores aren't accurate, and so the test is no longer a good predictor (no longer valid.)
Reliable means that the same candidate taking the test under different conditions (different time, different place, different graders) would essentially get the same score.

Given the lack of clear answers for the graders to use to compare the candidate's answer to, and given the lack of a clear rating system, it's not likely that this is a reliable test.

And in fact, when I looked at the scores individuals got on the same question by different graders, there were some big differences in score.  Some were a point off, some were 2 points off and even 2.5 points off.  There were even a few that were 3 points off.

That may not sound like much, but on a six point scale, it's a lot.  Even though they actually used a six point scale, they said that 3.5 was equal to 70% - the lowest passing grade.  So each point was worth 20%.  Therefore, two points equals a 40% difference.  2.5 equals a 50% difference on the score.  3 equals 60%.  If this test were reliable, the scores by different graders shouldn't vary by more than 10% or so.  That would be half a point on any question.

On this exam,  graders gave scores that were up to 60% (3 points) different for the same candidate on the same question!  That's the difference - in an academic setting - between a 95% (A) and 35% (low F).  But even a one point difference is a 20% difference - that could be a 90% and a 70% or the difference between an A- and a C-.   That's a huge spread, and it's there because the answer sheets don't tell the graders what a good answer looks like, and the grading scale easily tricks graders into giving lower ratings than they realize they are giving.

The two people in charge of setting up the tests (and the overall training director at the time) were actually not qualified to prepare the tests.  They all had Alaska Fire Safety Training Instructor Certificate Level 1.  They were eligible to administer training and tests that had been prepared by someone else, someone with a Level II certificate.  Here's the earlier post on the certifications.

Accountability

The exam is an hour long.  Some of the graders wrote brief notes on their score sheets, but there isn't much room.  Those notes are the only record of what actually happened in the oral exam.  There's no way for the candidate or the graders to check what was actually said.  Even as they review the pages that contain the answer, they can't replay the tape to see if the candidate covered something or not.  And years later?  At trial?  Well I know that five minutes later my wife and I have totally different recollections of what each of us said.  And in trial, there were a number of occasions when Jeff Jarvi, Jeff Graham's attorney, asked the graders about something they had said at their depositions or at the State Human Rights Commission hearing.  They'd answer.  Then Jarviv would have them read the transcripts, and the transcripts were not at all what they remembered.

Video or audio recordings of the oral exam probably could have averted this whole trial.  The tapes would have demonstrated whether Jeff Graham had covered the points as he said he did, or that he didn't as some of the graders said.

Not having recordings means no one can check how well the graders did their jobs.  Or whether people with low scores really did answer less accurately than people with high scores.  And it means that candidates can't get feedback on how they did so they can better prepare for the next exam.  (I'd note, that as a professor, the most convincing way to demonstrate to students that his paper wasn't very good, was to have them read the best papers.  It worked every time.)

Unequal Conditions

One other twist here.  This takes us ahead to the oral peer review, but it affects how people do on the technical as well.  Prior to the oral peer review, candidates are 'pre-scored' by the graders based on their personal knowledge of the candidates.  Imagine random co-workers some who might work closely with you and others who don't, evaluating you based on, well, whatever they want.  No one asks the basis.  It's not based on written materials they have about your work history, because there is the option of N/O - not observed.  If there were some written documentation, they wouldn't need the N/O option.

When the candidate walks into the room, he (it's almost always a he) is given his pre-scores.  If you pass (get over 70%) you can choose to spend more time on the technical questions and skip the peer-review questions altogether.  You can gain 20 minutes.

Imagine the emotional impact of being told, before you even start the oral technical exam, that the graders had failed you on the pre-scoring of the peer-review.  Or that they passed you with a high score.  In Jeff's case, he was stunned to learn he'd been failed by the graders in the pre-scoring of the peer-review.  It set him back as the technical oral began.


You've Made It To The End

If you made it this far, congratulations.  Going through details like this is necessary to truly get a sense of how badly these exams were designed and implemented.  But it does take work.  Thanks for hanging in there.  These are the kinds of details that jury had to sit through over the three week plus trial.

This post covered the technical oral exam problems and the next post will cover the oral 'peer review' exam that was part of the one hour along with the technical oral exam.  

Graham v MOA #8: The Exams 1: How The Process Works

[The Graham v MOA tab above lists all the posts in this series and gives some overview of the case and why I think it's important.]

The Exams - How The Process Worked In 2012

The exams firefighter Jeff Graham sued the Municipality of Anchorage over were to determine who would be promoted from the entry level Anchorage Fire Department (AFD) position - firefighter - to the next level - engineer.  A firefighter has to promote to engineer to move up in the AFD.  If you passed the exams, you would go on a list based on your score.  Then, as there were openings for engineers, names would be taken from the top of the list.  If the list was used up before the next scheduled exam - in two years - an interim exam could be held.  If you were on the list, but not called in two years, then you had to take the exams all over again.  

In 2012, the exams had three parts:  1) a written exam, 2)  a practical exam, and 3)  an oral exam.  It was the first year an oral exam was part of the engineer promotion exams.   It was the oral exam that Graham objected to.  You had to pass ALL THREE parts to get on the engineer list.  The first two exams were relatively objective.  The oral exam was problematic in many ways, giving graders lots of leeway to pass or fail candidates without much accountability.   Let’s look at them all before going into the details of the oral exam in the next posts.  

The Written Exam 

This was based on a standard set of questions from a national bank of engineer test questions about technical knowledge.  It’s multiple choice.  Test makers can choose from many questions. This allows them to modify the test to be appropriate to local conditions.  Overall, the bank of questions has been validated nationally- the questions are related to what a fire engineer should know and this national association determines the  correct answers.  

Graham passed this exam with a score of 85.  He needed 70 to pass.  

The Practical Exam

The practical exam is made up of a series of 'evolutions,' as they call them, that test the candidates' abilities to handle the trucks and equipment as needed on duty.   The evolutions (think of them as scenarios) involve actually driving vehicles, hooking up hoses, responding to different types fires, etc.  This exam was designed by local test makers.  Casey Johnson was in charge of this in 2012 and he followed the basic model that had been used in previous exams, but creating his own specific scenarios. 

The different evolutions on the exam are supposed to be unknown to the test takers until they take the test.   But the exams are given outside on consecutive days and people taking it on the second or third day can learn from others what events they will be asked to respond to.

A related issue that came up has to do with training outside the Academy.  Senior AFD officers often assist firefighters in their stations when there are no emergency calls.  So different candidates will get different coaching on different possible evolutions at their stations.  In one case, it was argued that one of the people who helped prepare the practical exam gave his subordinates, at the station, training on a new process that hadn’t been used at AFD yet, but was on the exam.  Questions were raised whether they had gotten advance information to prepare for that event.  The suggestions were denied.  

This exam was not validated professionally.  Jeff Graham has not challenged the events on this exam - they are related to what people have to do as engineers, but whether successfully completing the events on this test is the best, or even a good, predictor of success as engineer is not known.  

Graham did have some questions about the reliability of the exam.  Scores on the first day of the exam were low and the fail rate was very high.  By the third day, the fail rate dropped significantly.  Why might this be?

The exams are done out in the open where they can be seen by anyone.  The first people to take the exam do not know what they will be asked to do.  By the third day, people have been able to see what events were used, plus people who took the test can talk to friends who haven’t taken the test, so the later test takers can be better prepared.  

There was also some unconfirmed discussion at trial about whether the grading standards were loosened by the third day because the success rate was so low.  Which would raise questions about how the grading criteria were established.  

A reliable test is one where a test taker’s score should not vary regardless of the conditions of the test - which includes what day they took the test.  All test takers must face the same test conditions for the test to be reliable.

Jeff Graham passed the practical test comfortably.

The Oral Exam

The oral exam was created especially for the 2012 exam, by Casey Johnson.  Oral interviews had been held for higher level positions, like captain, but not for the technical job of engineer.  The exam consisted of two parts:  1)  a technical part and 2)  a “peer review.”  This is the part that Jeff Graham failed, by one point.  This was the part that Graham complained about before the exams even began, after he was told he failed the exam, and later to the Alaska Human Rights Commission, and finally in court.  

The technical part consists of ten questions, supposedly about technical issues, though the 2012 exam had two questions about how to prepare for the test and some that were more AFD policy and administration questions.  

The ‘peer review’ consists of five questions that seem to be designed to determine whether someone’s character is good enough to become an engineer.  

There are five testers for the oral exam.  Engineers are asked to volunteer to assist with various parts of the Academy (the training program designed to prepare people for the exam).  The Academy administrators, in this case Chad Richardson and his assistant Casey Johnson, decide who will perform what duties at the Academy and in the exams.  They can also encourage people to apply, which at least a couple of the testers said happened to them.  

Before the exam takes place, the testers pre-grade the peer review part of the exam.  That means, they give each test taker a score based on their knowledge of that person.  There was mention of reviewing the application for promotion, but graders had different responses about whether these were considered.  If they have no knowledge of that person, they can leave that part blank. So, even though they, theoretically, had access to someone’s application, they could skip the pre-score, which suggests that either the application wasn’t important, they didn’t look at the applications, or prior personal knowledge of the candidate was the key factor.  There was conflicting testimony about when this pre-grading was done. Graders were asked to come in early and do things like score candidates on the pretest.  But testimony showed  that didn’t necessarily happen.  Pre-grade scoring could be done in the morning before the testers came in, before anyone was tested, or before each individual came in to be tested.  

The Peer Review test process

The candidate walks into the room.  He’s given his pre-scores on the peer review.  He then has an hour to answer 15 questions - the ten technical questions and the five character questions.  That gives someone about four minutes per question.  The questions are projected on a screen and the candidate begins answering them.  

If the candidate got a passing score on the peer review pre-score, he can elect to skip any or all of the peer review questions and spend more time on the technical questions.  This, obviously, gives an advantage to people who were pre-scored well.  

Jeff Graham’s pre-score grades were below the needed 70. He got 69.  He was surprised by this.  

Overall Test Scoring

To pass the engineer exam, candidates have to pass all three parts of the exam.  That means that if they fail any of the three parts of the exam (get less than 70%), they fail the whole exam - even if their overall average on all the exams was 71% or 80% or 89%.  

Since you have to get 70% or better on ALL three exams, if someone gets a 69 on the first exam (the written exam), they do not go on to take the practical exam.  Those who pass the written and then  pass the practical exam can go on to take the oral exam.  

I don’t have the exact numbers available, but a large number took the written exams and fewer took each succeeding exam.  At trial, Deputy Chief Hetrick said people who made it to the orals had good scores - around 85 or more - on the written exams.   

From Exam To Promotion

Passing the exam doesn’t mean a firefighter will be promoted to engineer.  Those who pass go on a list based on their scores.  The higher the score, the higher they are on the list.  When there is an opening for an engineer, the top person on the list is promoted.  The list is good for two years.  If all the people on the list are promoted before the two years are up and they have new openings, they can have an interim Academy and test.  

Anyone left on the list after two years is no longer eligible and has to take the whole exam again. 

The cost of Academies is quite high in money and in time. It is the full time job for at least two people (in 2012 Chad Richardson and Casey Johnson) for a period of time, plus the time of all the volunteers and all the candidates.  Then there is the equipment and other things used.  The practicing on various rigs and gas that takes.  One figure I heard was about $60,000 but I don’t have confirmation of that.  

The Meaning of the High Fail Rate

A lot of people go to the Academy and a relatively small number make it onto the promotion list each time.  We can’t be sure why so many fail, but there are several possible explanations that come to mind.  

1. People take the Academy to learn more about the promotion process and they might take the written test just to see how they do.  Sort of a  trial to gauge how difficult it is and how much they’ll have to study when they take it seriously.  People mentioned this was the case for some.
2. The quality of the firefighters is low.  Only a high school degree or a GED is required.  They may not be particularly good at studying and/or test taking.
3. The training at the academy is inadequate to prepare most people to pass the exams.  
4. The tests are necessarily rigorous to make sure only the most technically competent are promoted from firefighter to engineer.
5. The tests are unnecessarily difficult or harshly graded. 
6. Fewer engineers means more overtime for those who are engineers 

I suspect there is an aspect of all six reasons (and perhaps some I haven’t thought of.)  

Let me explain number six a little more.  Because of the 24 hour shifts several days a week, AFD line employees have a lot of time away from work.  Many use this time to run other businesses.  But for many others this is an opportunity to work overtime.  Not only does the overtime give them time-and-a-half pay, it also raises the annual pay that their retirement benefits will be based on.  Some have argued that by making the testing for engineer difficult, the pool of engineers is kept small, and those who become engineers can work more overtime.  At trial, the MOA’s expert witness hired to calculate possible compensation for Graham in the chance he won, testified that Graham had very little overtime compared to many who had 1000-2000 hours of overtime.  Firefighters work three days of 24 hour shifts per week.  1500 hours of overtime would be the equivalent of 6 weeks.  That’s a lot of overtime and a lot of pay at time and a half.  One has to ask whether hiring more employees would make overtime less necessary and save the MOA money.    

I realize this bit on overtime goes beyond just an overview of the exams, but I’m not sure where else it will fit in, and it gives context to questions that come up about the exams overall.  I’m raising this issue because it came up. I don't know how significant it is.  I haven’t studied it, but it seems like something that should be followed up on.  

The Devil is in the Details

The next posts on Graham v MOA will focus on the details of the oral exams.  

Crimes of violence tend to be very tangible, very graphic.  We can imagine someone with a gun threatening a mugging victim or a bank teller.  We can imagine a stabbing.  We can see it vividly.  These are crimes that involve people on people violations.  

But administrative crimes are much harder to imagine.  They are structural crimes that are less visible and easier to hide.  They are tied up in details, rows of numbers, pages of text. Easier to conceal.  

How a scoring sheet for a test is designed, can make the difference between whether someone passes the exam or not.  This is why we hear stories of people who have embezzled money for years and years before they were caught.  A petty thief can face much stiffer legal obstacles than a white collar criminal who has bilked people of millions.  The latter crime seems less problematic because it's so abstract and harder to visualize.  

The details can be tedious.  One reason I’m slow in getting these posts out is that I’m trying to make them as easy to read and understand as possible.  'Interesting' is a goal, but that's more elusive.  I keep revising and revising and eventually I say, ok, enough, just post it already.  Even though I’m sure it’s still hard for the average person with a busy schedule to read, let alone digest.   

Graham v MOA #5: A Much Better Overview Of Why This Case Is Important

Graham v MOA - Overview of Why It’s Important

[This is an overview to explain what I see as a major problem.

MOA = Municipality of Anchorage AFD = Anchorage Fire Dept

 My evidence in this post is minimal.  This series of posts will offer detailed evidence.]

1. There was a wrongdoing.  Jeff Graham was intentionally and unfairly denied promotion.   This was proven in court to the satisfaction of the jury.  
2. The problem is structural and individual.
1. The exam process did not follow the MOA Charter requirement that it comply with ‘merit principles’.  Instead the oral exam (particularly) was so subjective that graders could give whatever score they wanted.  The only remaining evidence was the sketchy notes of the graders.  No recordings, audio or video.  This subjective exam was abused to prevent Jeff Graham from promoting.
2. The exam was part of a good-old-boy system that demanded loyalty and punished those who didn't toe the line.
3. The wrong doing was aggravated by the response to Graham’s protest of the exam.  It was not taken seriously, there was no investigation.  Constructive critiques of the exam by two other firefighters were also ignored.
4. In court, evidence was presented (and not refuted by the Municipality) that the head of the training academy said he would never let Jeff Graham promote.

 ******************************************

After Jeff Graham was told he had failed the oral exam, he filed a complaint.  The complaint was never followed up on by AFD beyond a quick rejection.  In addition, two other fire fighters also filed more generalized constructive critiques of the exams that also were ignored. 

The correct response, in 2012,  to Graham’s complaint about the exam would have been to look into the complaint, to look at the exam.  Anyone with professional testing and training knowledge and experience would have recognized the problem with the test immediately.  

Another correct response (given AFD demographics - about 2% female and about 85% white) would have been to seriously examine why the demographics are so different from Anchorage demographics.  They could have, for example, hired objective outside experts to meet with people of color and women in the department to find obstacles to entry and promotion so that more underrepresented groups could get into and promote within the system.  

Instead of investigating and correcting the problem, the department saw Jeff Graham as the problem and the department pulled all its forces together to fight him when he got no response and finally sued the MOA.  Organizations count on their greater resources to keep people like Graham from taking them to court.  But Jeff Graham felt seriously wronged and wanted to correct the injustice he had suffered.  

Why did this happen?

1. People in charge of testing and promotion were not qualified.  There is a state certification program. The test makers only had Level 1 certification and thus were not certified to make a test, only to administer a test created by someone with Level 2 certification.
2. Accountability systems - like the Human Resources Department - did not do their work to insure the exams complied with merit principles.
3. The system has aspects of a good old boys system where if you fit in with the crowd and you stay quiet and loyal, you might get ahead.  This is not unusual, and we are currently seeing nationally the consequences of such systems in the sexual assault scandals.  If people are afraid to speak up, wrongs go on forever.  

Therefore, the highly subjective oral exam allowed the good old boys to select who they wanted to promote and in Graham's case, keep out the ones they didn't want to promote.  And this promotion exam was the only passage way up the ladder from the position of fire fighter.  

Why should you believe what I say here?  Lots of reasons.

1. My expertise.  I have academic training and practical experience in the field. I have a PhD in public administration, with a masters level specialization in personnel management.  I worked as special assistant to the director of Employee Relations at the MOA for two years (1982-84).  I was chair of the University of Alaska Anchorage grievance committee for two years and I was later the campus and statewide grievance rep for my faculty union. I sat on numerous search committees.  I taught public personnel administration for 30 years.  I’ve also published in this field.
2. My access to the case - I served as an expert witness for the plaintiff.  This allowed me to see much of the documentation of the testing and other activities.  I also attended the trial so I was able to hear the evidence offered by the MOA.  
3. The demographics of the Anchorage Fire Department compared to a) the rest of the MOA and b) the population of Anchorage are abysmal.  The AFD - according to the deposition of the Deputy Chief - has about 2% women and about 85% whites.
4. A superior court jury, after a three week trial, found the MOA guilty of bad faith and unfair treatment in Jeff Graham’s case.  They awarded him $660,000.  (The judge later approved additional legal fees.)
5. A fire fighter eyewitness testified at the trial that the head of the engineer promotion and testing academy told him and some others that “As long as I’m in charge of promotion, Jeff Graham will never promote.”  The city did not refute this testimony in court.  
6. The PowerPoint used to prepare fire fighters for the oral peer review covered slides which point out - unintentionally - some of the problems with the test:
1. “Is This A Popularity Contest?”  - they say they talked about this to dispel rumors, but obviously the rumors must have been pervasive enough that they felt the need to officially address them.  Their proof that it wasn’t?  They simply asserted it wasn’t. And blamed sore losers who'd failed the exam for the rumors.
2. Candidates were told to expect ‘skeletons’ to be brought up at the oral exam.  I'd note the MOA has disciplinary procedures with proper investigations for dealing with behavior that violates MOA regulations.  There were no disciplinary issues with Jeff Graham.  The promotion exam is not the proper place to raise unfounded and unproven "skeletons" against people. 
3. Candidates were told to prepare ‘nuggets’ or stories about themselves that would prove their character and that the nuggets could be ‘irrelevant’ to the job of fire fighter.  I'd note again, that the merit principles require that all criteria for selection and promotion be directly job related and predictive of success on the job.  
7. The oral exams were not validated even though the national fire safety instructor training manual the department uses says that high impact exams such as promotion exams should be validated professionally.  
8. Merit principles require tests that predict who will be good at the job, or what is technically known as 'valid.'   A valid test in this case would mean test takers who score higher will be more successful engineers and those who score lower will be less successful.  There are technical ways to validate a test.  Another key factor is reliability - that the conditions of the test - location, time, graders, etc. - don't bias the outcome.  Tests need people with technical training to ensure a test is valid and reliable.    A number of key people, including the test makers, were unable in depositions to explain:
1. merit principles
2. validity
3. reliability
9. Aside from not being validated, the oral exams were so obviously subjective that they allowed graders great leeway.  There are too many serious problems with the exams to address them here.  But I will in other posts.

As I said at the beginning, a wrongdoing happened and was compounded by the fire department uniting to fight Jeff Graham instead of investigating the problem professionally.  They strongly supported the people who caused the problem and fought against the victim of the problem.  The dynamics of this are the same as the dynamics that have kept so many women (and men) from reporting sexual assaults and even rape.  Fear of retribution kept many fire fighters silent about problems, even though they signed statements that required them to report any violations of the integrity of the exams.  

Thus closing this particular case doesn’t solve the problem.  It’s a systemic problem that still exists.  

Remedies

1. The most basic remedy already exists.  In fact the MOA Charter requires it be used. The Charter requires the application of merit principles in all personnel decisions.  Unfortunately, no one in the fire department involved in the case knew anything about merit principles.  Thus, the first recommendation is to train all the top level executives and all the employees involved in personnel actions where decisions are made about employees (selection, promotion, discipline, etc.) in merit principles and how to apply them.   
2. The MOA needs to hire enough professionals skilled in testing, to 
1. review selection and promotion testing in all departments to be sure tests are valid and reliable and there is accountability if they are not 
2. develop new testing procedures in those departments where tests are not valid and reliable, starting with those most out of compliance
3. train relevant people in the departments 
4. have professionals investigate complaints and suggestions about testing procedures
5. establish procedures to record all oral exams
3. Get demographic data for all departments to determine how close the MOA reflects the demographics of the working age population of Anchorage.
1. This should include age, gender, ethnicity, at a minimum, at all levels of each department
2. Departments below a to-be-determined percent of the general population demographics must develop plans for making their departments more welcoming and for developing pools of qualified candidates for future openings - if necessary, this could use the UAA ANSEP model where recruitment goes as far as preparing  high school students to develop the skills needed in key areas.  
4. The merit principles required in the MOA Charter need to be resuscitated by
1. extensive training of employees and the public on what merit principles are and how they work,
2. reviewing how many current executive level positions should actually be civil service positions with civil service protection from arbitrary firing.
5. Departments need to be much more open to their employees’ suggestions and complaints.  If necessary this means publicizing the role of the ombudsman office more and increasing the ombudsman staff skilled in administrative investigation.  Or setting up an investigation team in Human Resources.
1. This includes the legal department, which should be reviewing the meaning of ‘duty to client’ when they are government employees.  Is their duty to the employee in the department who has the authority to call them in?  Is it to the MOA as a whole?  Is it to the public?  What should they do when these duties conflict?  If the legal department had considered the fairness of the system and what happened to Jeff Graham instead of whether they thought they could beat a discrimination case, this never would have happened.  I'd note this case followed closely a discrimination case in the Police Department that the MOA also lost. Department heads who have done wrong or have allowed wrong to be done on their watch, shouldn’t be able to use the MOA attorneys to cover up their mistakes and persecute employees who are the victims of the wrong doing.  
6. It would also be nice to promote Jeff Graham.  He actually passed the test back in 2008, but a day or two later he was told he hadn’t been eligible to take the test.  It’s true he hadn’t been a fire fighter for the required five years, but he had been a mechanic in the fire department for eight years (and the duties greatly overlap with engineer)  and the HR department and Fire Department waived the requirement because of ‘equivalency’ as they did for other fire fighters.  And in 2012 he passed the written and practical tests with high scores.  But a third exam - the subjective oral exam - was added.  If you fail any one of the three exams, you can’t promote.  Graham failed the oral exam by one point after a ‘skeleton’ in the form of a rumor he’d never heard about before was sprung on him in one question.  (Each question had about four minutes to be addressed.) [UPDATE May 21, 2018:  Jeff retook the exams recently and passed.  That puts him on a list of people who will be promoted if engineer positions come open in the next two years.  Read more here.]

Jeff Graham initially filed a racial and age discrimination complaint.  Those are hard to prove.  It was the only legal grounds he was aware of for such a complaint.  He didn't know anything about the merit principles in the Charter.  The MOA claimed there was no racial discrimination, but in the trial, these other issues came out and the jury found the MOA violated its obligation of good faith and fair treatment.  

The beauty of the merit principles is that they protect everyone from discriminatory practices - whites as well as people of color, men as well as women.  They require tests to, as much as possible, discriminate solely on whether someone is likely to be successful on the job.  

My Goal

My goal in these posts is to raise awareness of the problems that this case has raised.  It got almost no media coverage and many of the issues are 'invisible' in that they are very detailed issues of policy and administration.  I hope to be able to make them visible and explain them and their significance in these posts.  

Ultimately I hope the MOA will seriously review the internal structures that increase the likelihood that employees will be treated fairly, like the merit principles in the charter require. Hiring and promoting employees based on their qualifications for the job and not their connections, generally leads to a more effective and efficient government.  So does an administration that takes complaints seriously instead of punishing the complainers.  

This is the overview.  The other posts will fill in the details.  As I post them, I'll add links to them here.  

You can see an index of all my posts about this case here - or at the Graham v. Municipality of Anchorage tab just below the "What Do I Know?" banner on top of the blog.