Civil Service - Who Are These People ET Are Firing? - Part II

INTRO:  Part I is here.  

If you find this topic dry and hard to get your head around, then you are half way there.  Because some of the most important things to know about government are dry and hard to get one's head around.  And that makes it easy for politicians to bamboozle voters with falsehoods and misinformation.  

So if you want to understand why ET's firing of civil servants (most of government employees) is a violation of law and various regulations, you'll have to buck up and read carefully.  Even take notes.  

This content is based on testimony I gave in a local discrimination case.  So I had to pare it down to as simple an explanation as possible so that I didn't lose the jury.  The attorney was nervous that his expert would talk over their heads, but when I was done he was relieved that I'd made it very easy to understand.  And the jury said the local government was guilty.

So good luck.   [I explained ET in the Intro to Part I, but it's not critical.]

From a February 19, 2018 post: 

Graham v MOA #9: Exams 2 - Can You Explain These Terms: Merit Principles, Validity, And Reliability?

The Municipality of Anchorage (MOA) Charter [the city's constitution] at Section 5.06(c) mandates the Anchorage Assembly to adopt

“Personnel policy and rules preserving the merit principle of employment.”   AMC 3.30.041 and 3.30.044 explain examination types, content, and procedures consistent with these merit principles.  

Âs defined in the Anchorage Municipal Code Personnel Policies and Rules,

“Examination means objective evaluation of skills, experience, education and other characteristics demonstrating the ability of a person to perform the duties required of a class or position.” (AMC 3.30.005)

[OK, before I lose most of my readers, let me just say, this is important stuff to know to understand why the next posts will look so closely at the engineer test that Jeff Graham did not pass.  But it's also important to understand one of the fundamental principles underlying government in the United States (and other nations.)  And I'd add that the concepts behind merit principles are applied in most large private organizations to some extent, though they may have different names. 

Jeff Graham's attorney made me boil this down to the most basic points to increase the likelihood I wouldn't put the jury to sleep.  So bear with me and keep reading. 

And, you can see an annotated index of all the posts at the Graham v MOA tab above or just link here.]  

Basic Parts of Government In The United States

Governments can be broken down into several parts.

• The elected politicians who pass the laws and set the broad policy directions (legislature)
• The elected executive who carries out the laws.
• The administration is led by the elected executive - the president, the governor at the state level, and the mayor at the city level.
• Civil Service refers to the career government workers who actually carry out the policies.  There are also appointed officials at the highest levels who are exempt from some or all of the civil service rules.

Merit principles are the guidelines for how the career civil servants are governed.  

So What Are Merit Principles?

Probably the most basic, as related to this case, are:

• Employees are chosen solely based on their skills, knowledge, and abilities (SKAs) that are directly related to their performance of the job. 
• The purpose of this is to make government as as effective and efficient as possible by hiring people based on their job related qualities and nothing else.  
• That also means other factors - political affiliation, race, color, nationality, marital status, age, and disability should not be considered in hiring or promotion.  It also means that arbitrary actions and personal favoritism should not be involved. 
• Selection and promotion criteria should be as objective as possible.   

So Steve, what you're saying, this sounds obvious.  What else could there be?

Before the merit system was the Spoils System.  Before merit principles were imposed on government organizations, jobs (the spoils) were given to the victors (winning politicians and their supporters)   The intent of the Merit System is to hire the most qualified candidates.

In 1881, President Garfield was assassinated by a disgruntled job seeker, which spurred Congress to set up the first version of the federal civil service system - The Pendleton Act.

Only a small number of federal positions were covered by this new civil service act, but over the years more and more positions were covered and the procedures improved with improvements in the technology of testing.  The merit system, like any system can be abused, but it's far better than the spoils system.  Objective testing is a big part of applying merit principles.

What does 'objective criteria' mean? 

Objectivity has a couple common and overlapping meanings:

• Grounded on facts.  Grounding your understanding or belief on something concrete, tangible.  Something measurable that different people could 'see' and agree on.
• Unbiased.  A second, implied meaning from the first, is that you make decisions neutrally, as free as you can be from bias, preconceived ideas.  That’s not easy for most people to do, but there are ways to do it better. 

What Ways Can Make  Tests More Objective And Free Of Bias?

I think of objectivity as being on one end of a continuum and subjectivity being on the other end.  No decision is completely objective or subjective, nor should it be.  But generally, the more towards the objective side, the harder it is to introduce personal biases.* 

objective ...............................................................................................subjective

First Let's Define "Test"

In selection and promotion, we have tests. Test is defined as any thing used to weed out candidates, or rank candidates from poor to good.  So even an application form can be a test if it would lead to someone being cut out of the candidate pool.  Say candidates are required to have a college degree and someone doesn’t list one on an application.  They would be eliminated already.  

Again,  how do you make tests more objective?

There are two key terms we need to know:  validity and reliability.

What’s Validity?

Validity means that if a person scores higher on a test, we can expect that person to perform better on the specific job.  

Or saying it another way, the test has to truly test for what is necessary for the job.  So, if candidates without a college degree can do the job as well as candidates with a degree, then using college degree to screen out candidates is NOT valid.  

And what is reliability?

Reliability means that if  a person takes the same test at different times or different places, or with different graders, the person should get a very similar result.  Each test situation needs to have the same conditions, whether you take the test on Monday or on Wednesday, in LA or Anchorage, with Mr. X or Miss Y administering and/or grading the test.  

How Validity and Reliability Relate To Each Other

To be valid, the selection or promotion test must be a good predictor of success on the job. People who score high on the exam, should perform the job better than those who score low.  And people who score low should perform worse on the job than people who score high. 

BUT, even if the test is intrinsically valid, the way it is administered could invalidate it.  If the test is not also reliable (testing and grading is consistent enough that different test takers will get a very similar score regardless of when or where they take the test and regardless of who scores the test) the test will no longer be valid.  This is because the scores will no longer be good predictors of who will do well on the job. 

How do you go about testing for validity and reliability?

This can get complicated, especially for  factors that are not easy to measure.  I didn't go into this during the trial.  I wanted to point out some pages in a national Fire Safety Instructor Training Manual used by the Municipality of Anchorage, but I was not allowed to mention it.  It talks about different levels of validity and how to test for them.  It also says that for 'high stakes' tests, like promotion tests, experts should be hired to validate the test.  The jury didn't get to hear about this. But it's relevant because as I wrote in an earlier post, the people in charge of testing, and specifically in charge of the engineer exam, only had Level I certification, which allows them to administer training and testing designed by someone with Level II certification.  It's at Level II that validity and reliability are covered.  

There really wasn't need to get detailed in the trial, because the oral exam was so egregiously invalid and unreliable that you could just look at it and see the problems.  And we'll do that in the next posts. 

That should be enough but for people who want to know more about this, I'll give a bit more below.

-----------------------------------------------------------------------

Extra Credit

*"the harder it is to introduce bias"  There are always ways that bias can be introduced, from unconscious bias to intentionally thwarting the system.   When civil service was introduced in the United States, there was 'common understanding' that women were not qualified for most jobs.  That was a form of bias.  Blacks were also assumed to be unqualified for most jobs.  Over the years many of these sorts of cultural barriers have been taken down.  But people have found other ways to surreptitiously obstruct barriers.  

Merit Principles

If you want to know more about merit principles I'd refer you to the Merit System Protection Board that was set up as part of the Merit System Reform Act of 1978.  

A little more about reliability problems (because these are important to understand about the engineer promotion exam)

In the main part of this post I wrote that all the important (could affect the score) conditions of the test need to be the same no matter where or when or with whom a candidate takes the test.  Here are some more details

• Location - If one location is less comfortable - temperature, noise, furniture, lighting, whatever - it could skew the scores of test takers there.
• Time -  could be a problem in different ways.  
• All candidates must have the same amount of time to take the test.  
• Instructions - all instructions have to be identical
• Security of the test questions - if some applicants know the questions in advance and others do not, the test is not reliable.

The scoring, too, has to be consistent from grader to grader for each applicant. 

And there are numerous ways that scoring a test can go wrong.

• Grader bias  - conscious and unconscious.   Raters who know the candidates may rate them differently than people who don’t know them at all. 
• The Halo effect means if you have a positive view of the candidate, you’re likely to give him or her more slack.  You think, 'I know they know this.' 
• The Horn or Devil Effect is the opposite - If you already have a negative opinion about a candidate, you consciously or unconsciously give that a candidate less credit.  These are well documented biases.
• Testing order bias affects graders and candidates.  
• After three poor candidates, a mediocre candidate may look good to graders.  

• Grading Standards - Is the grading scale clear and of a kind that the graders are familiar with?
• Are the expected answers and how to score them clear to the graders?
• Do the graders have enough time to calculate the scores consistently?
• Grader Training -
•  If they aren't well trained, it could take a while to figure out how to use their scoring techniques, so they score different at the end from the beginning. 

How Do You Overcome the Biases In More Subjective Tests Like Essays, Interviews, and Oral Exams?

Despite the popularity of job interviews, experts agree that they are among the most biased and result in the least accurate predictions of candidate job performane.  Or see this link.

You have to construct standardized, objective rubrics and grading scales - this is critical, particularly for essay and oral exams.

On November 9, 2016 when the electoral college vote totals were tallied, everyone saw the same facts, the same results.  But half the country thought the numbers were good and half thought they were bad. When evaluating the facts of a job or promotion candidate, the organization has to agree, before hand, what ‘good’ facts look like and what ‘bad’ facts look like. Good ones are valid ones - they are accurate predictors of who is more likely to be successful in the position.   Good and bad are determined by the test maker, not by the graders.  The graders merely test whether the performance matches the pre-determined standard of a good performance.

What’s a rubric?

It’s where you describe in as much detail as possible what a good answer looks like.  If you’re looking at content, you identify the key ideas in the answer, and possibly how many points a candidate should get if they mention each of those ideas.  It has to be as objective as possible. The Fire Safety Instructor Training Manual has some examples, but even those aren't as strong as they could be. 

Good rubrics take a lot of thought - but it's thought that helps you clarify and communicate what a good answer means so that different graders give the same answer the same score.

Here are some examples: 
UC Berkeley Graduate Student Instructors Training
Society For Human Resource Management - This example doesn't explicitly tell graders what the scores (1,2, 3, 4, 5) look like, as the previous one does.
BARS - Behaviorally Anchored Rating Scales - This is an article on using BARS to grade Structured Interviews.  Look particularly at Appendices A & B. 
How Olympic Ice Skating is Scored - I couldn't find an actual scoring sheet, but this gives an overall explanation of the process.

My experience is that good rubrics force graders to ground their scores on something concrete, but they can also miss interesting and unexpected things.  It's useful for graders to score each candidate independently, and then discuss why they gave the scores they did - particularly those whose scores vary from most of the scores.  Individual graders may know more about the topic which gives their scores more value.  Or may not have paid close attention.   Ultimately, it comes down to an individual making a judgment.  Otherwise we could just let machines grade.  But the more precise the scoring rubric, the easier it is to detect bias in the graders. 


Accountability

Q:  What if a candidate thinks she got the answer right on a question, but it was scored wrong?

Everything in the test has to be documented.  Candidates should be able to see what questions they missed and how they were scored.  If the test key had an error, they should be able to challenge it. 

Q:  Are you saying everything needs to be documented?

If there is going to be any accountability each candidate’s test and each grader’s score sheets must be maintained so that if there are questions about whether a test was graded correctly and consistently from candidate to candidate, it can be checked. 

In the case of an oral exam or interview, at least an audio (if not video) record should be kept so that reviewers can see what was actually said at the time by the candidate and the graders. 

Q:  Have you strayed a bit from the Merit Principles?

Not at all. This all goes back to the key Merit Principle - selecting and promoting the most qualified candidates for the job.  There won’t be 100% accuracy. But in general, if the test is valid,  a high score will correlate with a high job performance.  But unless the test is also reliable, it won’t be valid. The more reliable the test, the more consistent the scores will be under different conditions and graders.  The best way to make tests more reliable is to make them as objective as possible.

Cancelling My LA Times Subscription [Updated]

[UPDATES:  Here's the link to the second post on this topic. The Nov 3 UPDATE is at the bottom of the original article]

Overview:  I'm giving context to why I cancelled my subscription.  I look back to heroic actions taken by the  New York Times and the Washington Post during the Vietnam war to compare to what appears to be the cowardly action of the Post and the LA Times owners today.  

I'd note that while other papers have discussed the LA Times' decision, the LA Times as so far not had any article about this issue. 

So we start with the Pentagon Papers story.  Then we go to the vetoing of editorials supporting Kamala Harris for president by the owners of the two newspapers this week.  

Then I mention an important article by Vaclav Havel that directly addresses what happens when owners of businesses voluntarily comply to pressure from authoritarian governments.  But I'll save that discussion for the next post.  

 In 1971, The New York Times and the Washington Post were given copies of "The Pentagon Papers."  This was a classified report on the Vietnam War.  .  

One of the researchers, Daniel Ellsberg, was disturbed that the research showed that the US government was lying to the people of the United States about major aspects of the Vietnam war.  

Student protests had been going on constantly.  In spring of 1970, four students at Kent State were shot dead by National Guardsman called to quell the protests on campus.  This led to huge protests all over US campuses.  

While I was a young adult during the times of the Pentagon papers and it is all still vivid in my mind, I'm writing all this because I realize that every US citizen under the age of 53, was not even born then.  Even though they may have heard about the Pentagon Papers, most are probably have a very fuzzy understanding of the significance.  I know that was my experience of current events that took place in recent history but before I was born. I'm just summarizing some highlights.  You can read more at Wikipedia.  Their article starts with the contents of the Papers.  You have to scroll down to learn about the politics of publishing them in the newspapers.  

Ellsberg copied the Pentagon Papers.  In those days you generally had to copy page by page.  He took them to Kissinger (who he knew) and to  key Members of Congress, but didn't get the support he needed.  Then he went to the New York Times and shared them.  The Times began publishing excerpts on June 13.

The Nixon Administration tried to stop the publication by the Times with an injunction.  The Washington Post then began to publish the documents.  Also, Alaska US Senator Mike Gravel placed the full Pentagon Papers into the public record.

The Supreme Court decided 6-3 that

"Only a free and unrestrained press can effectively expose deception in government. And paramount among the responsibilities of a free press is the duty to prevent any part of the government from deceiving the people and sending them off to distant lands to die of foreign fevers and foreign shot and shell.

— Justice Black[56]"  [Wikipedia]

Unfortunately the court's decision doesn't appear to be a compelling value to the owners who quashed the endorsements in their papers.  

[Another interesting comparison to today: the Times published the first piece on June 13.  The US Supreme Court announced its decision on June 30!]

Ellsberg was personally charged but was not found guilty.  

I offer you this because this week the owners of both the Los Angeles Times and the Washington Post overruled their editorial boards' decisions to endorse Kamala Harris for president.  There have been resignations by editors of both papers over this.  

We can speculate why the owners took these actions.

The MSNBC headline was:

"The Billionaire Owners of the Washington Post and LA Times Just Capitulated to Trump"

NPR's headline didn't attribute a motive to the Washington Post's decision, 

"Washington Post' won't endorse in White House race for first time since 1980s"

but quoted former Washington Post former Executive Editor Martin Baron:  

"This is cowardice, a moment of darkness that will leave democracy as a casualty," Baron said in a statement to NPR. "Donald Trump will celebrate this as an invitation to further intimidate The Post’s owner, Jeff Bezos (and other media owners). History will mark a disturbing chapter of spinelessness at an institution famed for courage."

This is, of course, why I have included the story of the Pentagon Papers.  This is a far different action this week by  the owner of the Washington Post than we saw from Katherine Graham, the owner of the Post in 1971.  

Jeff Bezos, of course, is the owner of Amazon and one of the richest men in the world.  

Patrick Soon-Shiong is a billionaire doctor who got rich based on medical technology he developed.  His parents fled China during the Japanese occupation in WW II and Soon-Shiong was born in South Africa in 1952.  I don't know exactly what his situation was, but here's a description of the status of Chinese in South Africa in Wikipedia:

"In 1966 the South African Institute of Race Relations described the negative effects of apartheid legislation on the Chinese community and the resulting brain drain:

No group is treated so inconsistently under South Africa's race legislation. Under the Immorality Act they are Non-White. The Group Areas Act says they are Coloured, subsection Chinese ... They are frequently mistaken for Japanese in public and have generally used White buses, hotels, cinemas and restaurants. But in Pretoria, only the consul-general's staff may use White buses .. Their future appears insecure and unstable. Because of past and present misery under South African laws, and what seems like more to come in the future, many Chinese are emigrating. Like many Coloured people who are leaving the country, they seem to favour Canada. Through humiliation and statutory discrimination South Africa is frustrating and alienating what should be a prized community.[5]: 389–390"

One would think that both Bezos and Soon-Shiong are rich and powerful enough to be able to stand up to Trump.  But I'm guessing they both have goals and ambitions about what they still want to do with their companies.  And they have put these ambitions above risking the possibility of retribution from Trump if he gets elected.  

And I'm guessing Soon-Shiong, while treated as a non-white in South Africa, also took some solace that he wasn't treated as Black.  It would be interesting to know how he felt when Nelson Mandela was freed from prison and eventually became the president of South Africa and won a Nobel Prize.  

His behavior in this matter suggests those events didn't really register with him positively.  He's certainly now showing Mandela's courage in fighting an authoritarian government.

This post is long enough.  I wanted to also talk about Vaclav Havel's essay, "The Power of the Powerless" which is highly relevant to the actions of actions of these two wealthy newspaper owners.  I'll do that in another post.  For those who want to get ahead, here's a link to the essay.  It's very good.

Here's the link to the follow up post on Havel's essay.

Cancelling the LA Times subscription was a clear choice, though not an easy one.  I grew up in LA and when my mother died, I inherited the house that I lived in from 6th grade through the beginning of college.  It's the house my mother lived in for 65 years, that we visited often, and that my children spent time when they visited their grandmother.  In addition to getting reasonably good news coverage, I also got local news that was relevant to owning a house there and visiting.  

But various social media folk have suggested other newspapers to switch to and I'll look into that.  Though I won't get  the local LA and California news.  I'd note that when you cancel, you get a list of one or two word reasons to let them know why you cancelled.  The best I could do was 'editorial policy' or something like that.  Leaving comments elsewhere limits you to very few words.  

[UPDATE Sunday November 3]

From an October 25, 2024  article in the LA Times, we learn what Soon-Shiong, the billionaire owner, said about the decision not to endorse anyone for president, even though the editorial board was about to endorse Harris:

“'I have no regrets whatsoever. In fact, I think it was exactly the right decision,' he said in an interview with The Times on Friday afternoon. 'The process was [to decide]: how do we actually best inform our readers? And there could be nobody better than us who try to sift the facts from fiction' while leaving it to readers to make their own final decision."

Today's LA Times editorial page seems to belie that policy.  Instead of "leaving it to readers to make their own decisions," the LA Times has a long list of ballot measures and candidates they endorse for other offices from local and state to federal.    

"Election 2024

The Times’ electoral endorsements for Nov. 5

STATEWIDE BALLOT MEASURES

Proposition 2: Yes

Proposition 3: Yes

Proposition 4: Yes

Proposition 5: Yes

Proposition 6: Yes

Proposition 32: Yes

Proposition 33: No

Proposition 34: No

Proposition 35: No

Proposition 36: No

LOS ANGELES CITY

City Council District 2: Adrin Nazarian

City Council District 10: Heather Hutt

City Council District 14: Ysabel Jurado

Charter Amendment DD: Yes

Charter Amendment LL: Yes

Charter Amendment HH: Yes

Charter Amendment II: Yes

Charter Amendment ER: Yes

Charter Amendment FF: No

LOS ANGELES COUNTY

District attorney: George Gascón

Measure A: Yes

Measure E: Yes

Measure G: Yes

LOS ANGELES COMMUNITY COLLEGE DISTRICT

Seat 1: Andra Hoffman

Seat 3: David Vela

Seat 5: Nichelle Henderson

Seat 7: Kelsey Iino

LOS ANGELES UNIFIED SCHOOL DISTRICT

District 1: Sherlett Hendy Newbill

District 3: Scott Schmerelson

District 5: Karla Griego

Measure US: Yes

LOS ANGELES COUNTY SUPERIOR COURT JUDGES

Office No. 39: Steve Napolitano

Office No. 48: Ericka J. Wiley

Office No. 97: Sharon Ransom

Office No. 135: Steven Yee Mac

Office No. 137: Tracey M. Blount

STATE LEGISLATURE

Assembly District 52: Jessica Caloza

Assembly District 54: Mark Gonzalez

Assembly District 57: Sade Elhawary

Senate District 35: Michelle Chambers

U.S. HOUSE AND SENATE

U.S. Senate: Adam B. Schiff

27th Congressional District: George Whitesides

30th Congressional District: Laura Friedman

45th Congressional District: Derek Tran

47th Congressional District: Dave Min

Read the full endorsements online at latimes.com/opinion."

Graham v MOA #9: Exams 2 - Can You Explain These Terms: Merit Principles, Validity, And Reliability?

The Municipality of Anchorage (MOA) Charter [the city's constitution] at Section 5.06(c) mandates the Anchorage Assembly to adopt

“Personnel policy and rules preserving the merit principle of employment.”   AMC 3.30.041 and 3.30.044 explain examination types, content, and procedures consistent with these merit principles.  

Âs defined in the Anchorage Municipal Code Personnel Policies and Rules,

“Examination means objective evaluation of skills, experience, education and other characteristics demonstrating the ability of a person to perform the duties required of a class or position.” (AMC 3.30.005)

[OK, before I lose most of my readers, let me just say, this is important stuff to know to understand why the next posts will look so closely at the engineer test that Jeff Graham did not pass.  But it's also important to understand one of the fundamental principles underlying government in the United States (and other nations.)  And I'd add that the concepts behind merit principles are applied in most large private organizations to some extent, though they may have different names.

Jeff Graham's attorney made me boil this down to the most basic points to improve the likelihood I wouldn't put the jury to sleep.  So bear with me and keep reading.

And, you can see an annotated index of all the posts at the Graham v MOA tab above or just link here.]  

Basic Parts of Government In The United States

Governments can be broken down into several parts.

• The elected politicians who pass the laws and set the broad policy directions (legislature)
• The elected executive who carries out the laws.
• The administration is led by the elected executive - the president, the governor at the state level, and the mayor at the city level.
• Civil Service refers to the career government workers who actually carry out the policies.  There are also appointed officials at the highest levels who are exempt from some or all of the civil service rules.

Merit principles are the guidelines for how the career civil servants are governed.  

So What Are Merit Principles?

Probably the most basic, as related to this case, are:

• Employees are chosen solely based on their skills, knowledge, and abilities (SKAs) that are directly related to their performance of the job. 
• The purpose of this is to make government as as effective and efficient as possible by hiring people based on their job related qualities and nothing else.  
• That also means other factors - political affiliation, race, color, nationality, marital status, age, and disability should not be considered in hiring or promotion.  It also means that arbitrary actions and personal favoritism should not be involved. 
• Selection and promotion criteria should be as objective as possible.   

So Steve, what you're saying, this sounds obvious.  What else could there be?

Before the merit system was the Spoils System.  Before merit principles were imposed on government organizations, jobs (the spoils) were given to the victors (winning politicians and their supporters)   The intent of the Merit System is to hire the most qualified candidates.

In 1881, President Garfield was assassinated by a disgruntled job seeker, which spurred Congress to set up the first version of the federal civil service system - The Pendleton Act.

Only a small number of federal positions were covered by this new civil service act, but over the years more and more positions were covered and the procedures improved with improvements in the technology of testing.  The merit system, like any system can be abused, but it's far better than the spoils system.  Objective testing is a big part of applying merit principles.

What does 'objective criteria' mean? 

Objectivity has a couple common and overlapping meanings:

• Grounded on facts.  Grounding your understanding or belief on something concrete, tangible.  Something measurable that different people could 'see' and agree on.
• Unbiased.  A second, implied meaning from the first, is that you make decisions neutrally, as free as you can be from bias, preconceived ideas.  That’s not easy for most people to do, but there are ways to do it better. 

What Ways Can Make  Tests More Objective And Free Of Bias?

I think of objectivity as being on one end of a continuum and subjectivity being on the other end.  No decision is completely objective or subjective, nor should it be.  But generally, the more towards the objective side, the harder it is to introduce personal biases.* 

objective ...............................................................................................subjective

First Let's Define "Test"

In selection and promotion, we have tests. Test is defined as any thing used to weed out candidates, or rank candidates from poor to good.  So even an application form can be a test if it would lead to someone being cut out of the candidate pool.  Say candidates are required to have a college degree and someone doesn’t list one on an application.  They would be eliminated already.  

Again,  how do you make tests more objective?

There are two key terms we need to know:  validity and reliability.

What’s Validity?

Validity means that if a person scores higher on a test, we can expect that person to perform better on the specific job.  

Or saying it another way, the test has to truly test for what is necessary for the job.  So, if candidates without a college degree can do the job as well as candidates with a degree, then using college degree to screen out candidates is NOT valid.  

And what is reliability?

Reliability means that if  a person takes the same test at different times or different places, or with different graders, the person should get a very similar result.  Each test situation needs to have the same conditions, whether you take the test on Monday or on Wednesday, in LA or Anchorage, with Mr. X or Miss Y administering and/or grading the test.  

How Validity and Reliability Relate To Each Other

To be valid, the selection or promotion test must be a good predictor of success on the job. People who score high on the exam, should perform the job better than those who score low.  And people who score low should perform worse on the job than people who score high.

BUT, even if the test is intrinsically valid, the way it is administered could invalidate it.  If the test is not also reliable (testing and grading is consistent enough that different test takers will get a very similar score regardless of when or where they take the test and regardless of who scores the test) the test will no longer be valid.  This is because the scores will no longer be good predictors of who will do well on the job.

How do you go about testing for validity and reliability?

This can get complicated, especially for  factors that are not easy to measure.  I didn't go into this during the trial.  I wanted to point out some pages in a national Fire Safety Instructor Training Manual used by the Municipality of Anchorage, but I was not allowed to mention it.  It talks about different levels of validity and how to test for them.  It also says that for 'high stakes' tests, like promotion tests, experts should be hired to validate the test.  The jury didn't get to hear about this. But it's relevant because as I wrote in an earlier post, the people in charge of testing, and specifically in charge of the engineer exam, only had Level I certification, which allows them to administer training and testing designed by someone with Level II certification.  It's at Level II that validity and reliability are covered.  

There really wasn't need to get detailed in the trial, because the oral exam was so egregiously invalid and unreliable that you you could just look at it and see the problems.  And we'll do that in the next posts.

That should be enough but for people who want to know more about this, I'll give a bit more below.

-----------------------------------------------------------------------

Extra Credit

*"the harder it is to introduce bias"  There are always was that bias can be introduced, from unconscious bias to intentionally thwarting the system.   When civil service was introduced in the United States, there was 'common understanding' that women were not qualified for most jobs.  That was a form of bias.  Blacks were also assumed to be unqualified for most jobs.  Over the years these many of these sorts of cultural barriers have taken down.  But people have found other ways to surreptitiously obstruct barriers.  

Merit Principles

If you want to know more about merit principles I'd refer you to the Merit System Protection Board that was set up as part of the Merit System Reform Act of 1978.  

A little more about reliability problems (because these are important to understand about the engineer promotion exam)

In the main part of this post I wrote that all the important (could affect the score) conditions of the test need to be the same no matter where or when or with whom a candidate takes the test.  Here are some more details

• Location - If one location is less comfortable - temperature, noise, furniture, lighting, whatever - it could skew the scores of test takers there.
• Time -  could be a problem in different ways.  
• All candidates must have the same amount of time to take the test.  
• Instructions - all instructions have to be identical
• Security of the test questions - if some applicants know the questions in advance and others do not, the test is not reliable.

The scoring, too, has to be consistent from grader to grader for each applicant.

And there are numerous ways that scoring a test can go wrong.

• Grader bias  - conscious and unconscious.   Raters who know the candidates may rate them differently than people who don’t know them at all. 
• The Halo effect means if you have a positive view of the candidate, you’re likely to give him or her more slack.  You think, I know they know this?  
• The Horn or Devil Effect is the opposite - If you already have a negative opinion about a candidate, you consciously or unconsciously give that a candidate less credit.  These are well documented biases.
• Testing order bias affects graders and candidates.  
• After three poor candidates, a mediocre candidate may look good to graders.  

• Grading Standards - Is the grading scale clear and of a kind that the graders are familiar with?
• Are the expected answers and how to score them clear to the graders?
• Do the graders have enough time to calculate the scores consistently?
• Grader Training -
•  If they aren't well trained, it could take a while to figure out how to use their scoring techniques, so they score different at the end from the beginning. 

How Do You Overcome the Biases In More Subjective Tests Like Essays, Interviews, and Oral Exams?

Despite the popularity of job interviews, experts agree that they are among the most biased and result in the least accurate predictions of candidate job performane.  Or see this link.

You have to construct standardized, objective rubrics and grading scales - this is critical, particularly for essay and oral exams.

On November 9, 2016 when the electoral college vote totals were tallied, everyone saw the same facts, the same results.  But half the country thought the numbers were good and half though they were bad. When evaluating the facts of a job or promotion candidate, the organization has to agree, before hand, what ‘good’ facts look like and what ‘bad’ facts look like. Good ones are valid ones - they are accurate predictors of who is more likely to be successful in the position.   Good and bad are determined by the test maker, not by the graders.  The graders merely test whether the performance matches the pre-determined standard of a good performance.

What’s a rubric?

It’s where you describe in as much detail as possible what a good answer looks like.  If you’re looking at content, you identify the key ideas in the answer, and possibly how many points a candidate should get if they mention each of those ideas.  It has to be as objective as possible. The Fire Safety Instructor Training Manual has some examples, but even those aren't as strong as they could be.

Good rubrics take a lot of thought - but it's thought that helps you clarify and communicate what a good answer means so that different graders give the same answer the same score.

Here are some examples:
UC Berkeley Graduate Student Instructors Training
Society For Human Resource Management - This example doesn't explicitly tell graders what the scores (1,2, 3, 4, 5) look like, as the previous one does.
BARS - Behaviorally Anchored Rating Scales - This is an article on using BARS to grade Structured Interviews.  Look particularly at Appendices A & B.
How Olympic Ice Skating is Scored - I couldn't find an actual scoring sheet, but this gives an overall explanation of the process.

My experience is that good rubrics force graders to ground their scores on something concrete, but they can also miss interesting and unexpected things.  It's useful for graders to score each candidate independently, and then discuss why they gave the scores they did - particularly those whose scores vary from most of the scores.  Individual graders may know more about the topic which gives their scores more value.  Or may not have paid close attention.   Ultimately, it comes down to an individual making a judgment.  Otherwise we could just let machines grade.  But the more precise the scoring rubric, the easier it is to detect bias in the graders.


Accountability

Q:  What if a candidate thinks she got the answer right on a question, but it was scored wrong?

Everything in the test has to be documented.  Candidates should be able to see what questions they missed and how they were scored.  If the test key had an error, they should be able to challenge it.

Q:  Are you saying everything needs to be documented?

If there is going to be any accountability each candidate’s test and each grader’s score sheets must be maintained so that if there are questions about whether a test was graded correctly and consistently from candidate to candidate, it can be checked.

In the case of an oral exam or interview, at least an audio (if not video) record should be kept so that reviewers can see what was actually said at the time by the candidate and the graders.

Q:  Have you strayed a bit from the Merit Principles?

Not at all. This all goes back to the key Merit Principle - selecting and promoting the most qualified candidates for the job.  There won’t be 100% accuracy. But in general, if the test is valid,  a high score will correlate with a high job performance.  But unless the test is also reliable, it won’t be valid. The more reliable the test, the more consistent the scores will be under different conditions and graders.  The best way to make tests more reliable is to make them as objective as possible.

Graham v. MOA #3: Following The Merit Principles In The MOA Charter Could Have Prevented All This

[This is post #3 of a series on Graham v MOA.  You can get an overview and index of all the posts here. Or just go to the Graham v. Municipality of Anchorage tab up above.]

Understanding the merit system and how to measure people's skills for a job are important to understanding why this case is important.  So I beg your indulgence here.  I've tried to make this pretty easy to digest.

Briefly, before the Merit System (and principles) governments were run on the spoils system - you got a government job if you helped get a candidate elected.  Loyalty, not skill and public spirit, were the key job qualification.  Merit principles don't guarantee fair hiring and promotion, but they go a long way in that direction.

Below is excerpted from the expert witness report I wrote up October 2016.  The judge did not allow the plaintiff to use the MOA (Municipality of Anchorage) charter because he said the promotion process is part of the collective bargaining agreement.  Collective bargaining agreements are approved (on the MOA side) by the Assembly.  But the Charter can only be amended by a vote of the people of Anchorage.  So I still don't understand how the contract would trump the Charter.  But I'm not a lawyer.

Here's from my report:

Background and Purpose of Merit Principles and Systems  

Race and age bias, in the context of promotion of a public employee, is an important issue. But the bigger issue is merit system principles.  Modern human resources departments in reasonably sized organizations both public and private use what is known as a merit system.  The system stems from the 19th Century when governmental structures were evolving from feudal systems based on loyalty to the ruler, to more modern ones, based on rationality.  Scholar Max Weber noted that a new form of organization was emerging which he called a bureaucracy, that was based on rational rules rather than the arbitrary decisions of a ruler.
The point was that organizations that hired people based on their ability to do specific jobs and not on their relationship and loyalty to the ruler were more effective, more efficient, and more permanent.
These ideas of selecting the best person for the job were also promoted in the factory in the early 20th Century by Frederick Taylor and his idea of Scientific Management became widely adopted.  Over time, the ideas underlying Weber and Taylor - the idea that rational, scientific analysis can be applied to management - took hold in private companies.  Applicants would be evaluated by their qualifications to hold their jobs, though personal connections and other biases still were a factor.
In government the change took place both on the federal level  and state and local levels.  In 1883, the Pendleton Act established the US Civil Service after President Garfield was shot by a disgruntled job seeker.  It only applied to a small percentage of jobs at first, but over the years, it has come to cover most federal positions in the career civil service.
On the local level, reformists pushed for merit systems as a way to combat the big city political machines like Tammany Hall that recruited immigrants into their political party with promises of government jobs, as they arrived in the US from Europe.
So, both in government and in the private sector these ideas of rational rules to develop a competent workforce took hold.  But the biases of the times often got written into the rules.  Job tests were written so that new immigrants wouldn’t pass.  Women were assumed ineligible for most jobs and fired from those they could take - like teacher - if they got married.  The societal structure which kept people of color in segregated housing, deficient schools, in poverty, prevented most people of color from getting the needed qualifications, or even from knowing about job openings.  And overt racism prevented those who could qualify from being hired in most cases.
The civil rights movement changed that.  Brown v. Board of Education struck down segregated schools.  This was supposed to lead to African-Americans (particularly) getting better high school educations, then into universities, and then into good jobs.  But many communities opposed busing and set up all-white private schools, leaving the public schools for African-Americans and the poor.
The Voting Rights Act was intended to prevent laws that kept Blacks from voting.  Griggs v. Duke Power was a groundbreaking case in terms of job discrimination.  Black workers traditionally got the lowest level jobs and got paid less than white workers.  When they were required to put in tests for employees seeking supervisory positions, Duke Power created tests that were unrelated to the position and intended to keep blacks from passing. The Supreme Court struck this down saying that the tests for the jobs had to be related to the work that would be done.  They also said that the plaintiffs didn’t have to prove intentional discrimination, only that the test had a disparate impact on the minority candidates.
The merit system was an outgrowth of science being applied to management to ensure more qualified employees got hired.  Businesses developed measures that focused on someone’s ability to successfully do the job.  The civil rights movement fit perfectly into this theoretical ideal.  Job requirements should focus on qualifications, not race or gender.  Griggs v. Duke Power drew back the curtain on the hidden biases that were blocking access to better employment for women and minorities.
Today we’ve come a long way, but we are still a society that sees minority actors in movie roles as criminals or maids or chauffeurs much more than as doctors or lawyers or accountants.  Many people still cringe at the idea of their daughter marrying someone of a different race or religion.  When those feelings spill over into the workplace, into hiring, it’s illegal discrimination.
Unconscious racial bias perpetuates discrimination through assumptions about people based on their race or other characteristics.  Conscious bias attempts to set up barriers that seem legitimate, but are actually intended to keep out undesired applicants.
The merit system is one of the best ways to thwart discrimination so that the most qualified candidates, not the most ‘like us’ candidates, get hired.  It’s the best antidote we have to cronyism, racism, and other forms of discrimination in hiring and promoting employees.

Merit Principles and Systems at Municipality of Anchorage 

The MOA Charter at Section 5.06(c) mandates the Anchorage Assembly to adopt “Personnel policy and rules preserving the merit principle of employment.”   AMC 3.30.041 and 3.30.044 explain examination types, content, and procedures consistent with these merit principles.
Âs defined in the Anchorage Municipal Code Personnel Policies and Rules, “Examination means objective evaluation of skills, experience, education and other characteristics demonstrating the ability of a person to perform the duties required of a class or position.” (AMC 3.30.005)
According to the Firefighters collective bargaining agreement, the conduct and administration of the Anchorage Fire Department, including selection and promotion of employees, are retained by the Municipality. (IAFF-MOA CBA Section 3.1)

Application of Merit Principles to Making And Evaluating Objective Examinations 

In practice, the term merit principles means using procedures that ensure that decisions are made rationally to select and promote those people who are most suited for a job.  They mean that organizations do their best to identify the factors that best predict which applicant is most likely to succeed in the position.  Factors that are irrelevant to someone’s success on the job should not be part of the process.
A test (or examination as used by the MOA) is any process used to evaluate an applicant’s suitability for a position.  An application form can be thought of as a test to the extent that information is used to distinguish between applicants who qualify and those who do not.  A written exam, a practical exam, an interview are all tests when it comes to activities like selection and promotion.
Two basic factors are important when evaluating tests used in personnel decisions.  First, is the test valid?  Second, is the test reliable?
Validity means that the test, in fact, tests what it is supposed to test.  In employment that generally means it is useful in separating those applicants most likely to do well in the position from those less likely to do well.  For example, if a college degree is required for a position, but those without college degrees do was well as those with a degree, then that is not a valid factor to consider, because it doesn’t predict success on the job.  It is common to give applicants a written or practical test or an interview.  These are scored and applicants with higher scores are selected over people with lower scores.
Such tests are valid only if it is true that people with higher scores are more likely to be successful in the position than those with lower scores.  That is, people with higher scores are more likely to do well AND people with lower scores are more likely to do poorly.  If that is not the case, the test is not valid.

Employment tests can be validated by checking scores against actual performance of employees, though this does require selecting employees with low scores as well as with high scores to determine if the lower scoring employees really do perform poorly compared to the higher scoring employees.  This can be expensive and many organizations use ‘common sense.’  But common sense may not be accurate and if an employer is accused of discrimination, they will have to defend the validity of the test.
For rare, specialized positions, validation is difficult to do.  For common positions that are similar across the nation, such as fire fighters, there are often companies that prepare, validate, and sell, and even administer employment tests.
Reliability means that the way a test is administered is consistent.  The same applicant, taking the test at different times or locations or with different testers, would have basically the same result every time.  When people take the college entrance exams, for instance, the conditions are standardized.  No matter where someone takes the test, they get exactly the same instructions, the physical conditions of the test room are within certain parameters (desk size, temperature, noise level, etc.) and they all have exactly the same amount of time to complete the exam.  The scoring of the exams is also the same for everyone.
To ensure reliability of the test taking, all conditions that could affect the outcome must be the same.  To ensure reliability of scoring, the way points are calculated must be as objective and measurable as possible.  Often tests are designed with scales that help a rater know how to give points or how to put applicants in the correct category.
At the most basic level you might just have a scale of 1 - 5 for instance, with ‘good’ at one end and ‘poor’ at the other end.  But how does the rater determine what’s good or bad?
Better would be to have a more objective descriptor such as “successfully completed task with no errors” on one end and “failed to complete the task” at the other end.  Even better would be to have descriptors for each point on the scale.  The more that the descriptor describes an actual objectively testable level of achievement, the more likely it is that different raters would come up with the same score.  For example, ‘meets expectations’ is not as objective as “accomplished the task within 2 minutes with no errors that compromised the outcome.”
Basically, the greater the objectivity of the scoring system, the greater the likelihood of reliability, because there is a clear standard attached to each number in the scale. And with a more objective system, discrepancies can be more easily spotted.  A biased evaluator has a harder job to select favored applicants or disqualify disfavored candidates.  Also, a candidate who was graded unfairly has a better chance of challenging the score.
Another way to increase reliability is to train evaluators on how to use the scoring system.  It is also helpful to have raters who do not have personal relationships with the applicants.
Given the need for validity and reliability, interviews, while frequently used, have been found to be prone to many biases unrelated to the job. There are ways to improve the validity and reliability of interviews.  The questions asked must be clearly tied to ability to be successful in the position, recognizing that being able to perform a task is not the same as being able to describe how one would perform a task.  If personality and speaking ability are not being tested, then interviews can become treacherous employment tests for the applicant and for the employer.  The more subjective a test and the rating system, the easier it is to bias the outcome, whether unintentionally or intentionally.
Since proving intent to discriminate requires overhearing private conversations or emails, this is an impossible hurdle for most applicants.  The courts have recognized this and have allowed ‘impact’ to be used in lieu of intent.  But employment tests can often give us evidence of intent if they are subjective and there is little or no validity or reliability.

Conclusion 

I have seen no materials that offer any information on the validity or reliability of the tests used in the engineer promotional examinations which Jeff Graham has taken.  The exam score sheets I have seen lack rigorous descriptors for raters (or proctors) to calculate scores for applicants and appear extremely subjective.  The materials I’ve seen that were used to train the raters were lacking in detail and substance.
Without evidence to show the exams are valid and reliable, one must assume that the exams do not comply with the Municipality’s mandate to follow merit principles. [Such proof of validation had been requested from but not provide by the MOA.] The point of merit systems is to identify the most qualified candidates for each position and to prevent the introduction of personal biases into their scoring of candidates.  The tests themselves may or may not be discriminatory.  But when they are subjective as the oral board/peer reviews are, biases of the raters are easily introduced into the scoring of candidates. The type of bias could be racial, sexual, age based, or personal depending on the rater.
It is my understanding that MOA has not produced all requested materials and that depositions still remain to be done in this case.  I therefore reserve the right, should additional materials and information become available, to modify or supplement this report.  

Because merit principles were ruled out as the measure the jury would use to evaluate the case, this report was not introduced in court or given to the jury.  However, I was allowed to testify on merit principles in general, but not allowed to relate them to the facts of the case, or even to the MOA.

I was also not allowed to refer to the Fire Safety Instructor Training Manual that the MOA uses which talks about validity in some detail and also talks about 'high stakes' tests - like a promotion test - needing to be professionally prepared and validated.

I was allowed to talk about, again in general terms and not relating what I said to the AFD exams, subjectivity and objectivity.  I acknowledged there is no such thing as 100% objective or subjective, but that there is a continuum from some theoretical total subjectivity to theoretical total objectivity.  The goal of test makers is to have tests as far to the objective side of the continuum as possible.  The more subjective a test, the easier it is to introduce bias, conscious or unconscious.

Genre: Legislative Fiction - Story: Alaska Legislature Selling UAA to Charter College

This story idea popped into my head recently.  Probably because of all the stories about huge budget cuts to the University of Alaska plus bills to make it legal to carry guns on campus.  Along with the legislature's reluctance to end subsidies for the oil companies and all the mega-projects which are, in effect, subsidies for construction companies.

We've already passed April 1, so I can't just put this up straight.  Although it's far fetched, some of the people I've mentioned this story to said things like, "Oh, I didn't hear that yet."  They just took it for real without blinking.  An Irony icon (*I*) might get overlooked.

So I want you to consider this genre of literature:  Legislative Fiction.  Like science fiction, which imagines a world changed by future developments in science and technology, legislative fiction imagines a world in which the wildest desires of some legislators are fulfilled.  In this case, I'm pushing to the limits conservative desires to privatize government functions that they think could be done as well by the private sector, their concern about radical left-wing faculty brainwashing their students, and their desire to reward private sector supporters and funders.

So here's my short story.

Alaska's Majority coalition legislators have announced they are working to sell the University of Alaska Anchorage to Charter College.  The deal is being handled by developer Mark Pfeffer, whose commission should more than make up for any losses at the LIO.  In the tradition of the Alaska Republican majority, not only do they propose to sell the campus, they are giving Charter College a $500 million zero interest loan,  so Charter can afford to make the purchase.  The sale will also effectively cancel all union contracts, pension obligations, and health benefits.



Reporters noted Charter College's questionable record*, according to College Factual:

Among the Worst Graduation Rates
Only 23.6% of students graduate from Charter College - Anchorage on-time (two or four years depending on the degree) and only 25.4% graduate at all, ranking this school among the worst in the country in both categories.
Graduating From College Isn't for Everyone.
The Majority of Non-Grads at this School Dropped Out. 74.6% of students at Charter College - Anchorage failed to graduate within 150% of the expected time. The majority did so because they dropped out.

Senator D, said he thought they could also achieve those levels with the University.

*This part, unfortunately, isn't fiction.