Category Archives: Metrics

Improvement, Metrics

Quality Metrics Monday – June 2015

Leave a comment

Quality Metrics
Quality metrics are a vital part of any management system. They allow for data-driven decision making. They allow the individuals within an organization to be able to answer the question, “How’s it going?” – without resorting to “gut feel”.

Gut feel is horribly unreliable, our brains lie to us (read the very good You Are Not So Smart: Why You Have Too Many Friends on Facebook, Why Your Memory Is Mostly Fiction, and 46 Other Ways You’re Deluding Yourself, for examples).

I’ve covered quality metrics in earlier posts, particularly with regard to goals and objectives. That two-parter grew into some specific metrics for a Human Resources department and I’d like to add more departments as well.

Frankly, it got away form me a little. Hey, it happens… Let us begin again.

What are Quality Metrics?

Quality metrics, to summarize and perhaps pull a different tack, are the basis for continual improvement. Continual improvement is a large component of what compliance to the standard means. To use an oft quoted phrase, “If you don’t know where you are, a map won’t help. ” (attributed to software engineering legend Watts Humphrey).

  • Metrics let you know where you are – goals, objectives tell you where you want to be.
  • Metrics are measurable, and they are often numeric.
  • Metrics are sometimes called KPI’s (Key Process Indicators).
  • Metrics can be called Measurables (because they should be measurable).

The Minimum Set

When we audit, we’re looking for a minimal set of these measurables. I want to let you know that set, so your life – and mine, can be a little easier. And no, these aren’t set in stone, most of them aren’t anyway – but you should have something like these.

You should have something.

The other influence on the minimum set is going to be the Quality Policy of the company. Generally however, the most basic quality policy invokes the necessity for the metrics I will discuss below.

If your quality policy has something particular, like “… happy employees” then part of the set should have some way to measure that (Employee Turnover, for example). Or if the policy addresses “… a vital part of the community”, then your collection of metrics should include a measurement for that (participation in community outreach programs, perhaps).

To be fair, the standard is only specific about one metric, and metrics are only one  way to measure an objective. And where 9001 isn’t specific, it is so general that justifying the absence of a metric or the objective it measures get’s very… convoluted.

And this measuring of objectives is important because the gospel, according to 5.4.1, says that objectives shall be established at all relevant functions and levels within an organization. Here’s the chapter and verse:

5.4.1 Quality objectives
Top management shall ensure that quality objectives, including those needed to meet requirements for product…, are established at relevant functions and levels within the organization. The quality objectives shall be measurable and consistent with the quality policy.

Objectives shall be measurable… okay, I can play devil’s advocate too; “accomplished or not accomplished” – that’s a measurement.

Fine, that works – and sometimes it makes complete sense. But sometimes it is, at best, a short-cut, and as my dear-old-dad used to say, “You’re only hurting yourself”.

Metrics can heal. I said the standard only specifies one metric, so here it is:

8.2.1 Customer satisfaction
As one of the measurements of the performance of the quality management system, the organization shall monitor information relating to customer perception as to whether the organization has met customer requirements. The methods for obtaining and using this information shall be determined.

NOTE Monitoring customer perception can include obtaining input from sources such as customer satisfaction surveys, customer data on delivered product quality, user opinion surveys, lost business analysis, compliments, warranty claims and dealer reports.

So, an auditor wants to see that. What is your customers’ perception of how your company is meeting their requirements?

Not, “Our customers love us!” verbally expressed.

Not, “We hardly get any complaints.”

And no, not the blank deer-in-the-headlights stare followed up with, “So, would you like another doughnut?” (Yes. Yes, I would very much like another doughnut).

It can be a list of customers with a percentile score next to their name.

It can be a list of customers with a stoplight metric; green/yellow/red. Or little smiley faces. But behind the pretty lights and the unblinking eyes should be data.

How do you know? Like the notes says after the requirement in 9001:2008, “input from sources such as customer satisfaction surveys, customer data on delivered product quality, user opinion surveys, lost business analysis, compliments, warranty claims and dealer reports” – whatever your creative genius comes up with. But something.

I’ve seen Survey Monkey put to good use for this. Reply cards sent with the product generally have a horrible return rate and they are almost not worth the effort. On-line options tend to do better.

Have an overall score covering all customers, or individual scores – but be prepared to justify your choices. If you exclude some customers then explain why.

Action Plans to Meet Goals

Along with that metric there should be a Goal. And – this is important – if the goal is not being met then there should be an action plan to achieve the goal.

The requirement for this is manifold and at the core of the standard. Here’s one:

8.2.3 Monitoring and measurement of processes
The organization shall apply suitable methods for monitoring and, where applicable, measurement of the quality management system processes. These methods shall demonstrate the ability of the processes to achieve planned results. When planned results are not achieved, correction and corrective action shall be taken, as appropriate.

Here’s another:

8.5.1 Continual improvement
The organization shall continually improve the effectiveness of the quality management system through the use of the quality policy, quality objectives, audit results, analysis of data, corrective and preventive actions and management review.

One more:

8.4 Analysis of data
The organization shall determine, collect and analyse appropriate data to demonstrate the suitability and effectiveness of the quality management system and to evaluate where continual improvement of the effectiveness of the quality management system can be made. This shall include data generated as a result of monitoring and measurement and from other relevant sources.

I could go on…

Back to the Minimum Set

So, customer satisfaction is one. “Check.”

Here are some others we expect to see (depending on your business, of course).

On Time Delivery – I could be part of customer satisfaction. Have a way to measure it that is consistent.  Often MRP systems have a problem with making a sensible number out of the data it collects. There is absolutely no problem with normalizing the data afterwards. Do what you have to do to get a number that is consistent. It’s value is that it is a means toward continual improvement.

And, please, don’t tell me it is 100% unless you’re selling unrealistic expectations as an actual business. I will be happily surprised if you can achieve that goal – though yes, it does happen routinely for some companies. Make it a stretch goal, but start low and change it gradually as your process can consistently reach it.

Scrap – how much material are you not using. Some folks use this along side a “re-use” metric. This is typically a dollar value, but for some commodities it is a weight or volume metric to save frequent currency or value conversions.

Machine down-time – Can be tied with a “Preventive Maintenance tasks on time” report as they are often related.

The rest of the organization.

So far I’ve listed mostly production and operation-related metrics and those should be a “slam dunk” for most organizations.

The other departments, such as Sales, Purchasing, QA and HR have a typical set as well (covered HR pretty well in a  previous post).

Again, I am pointing out basic and typical minimum expected sets.

Sales

Conversion Rate – converting quotations into purchase orders

Time to quote – in days, usually.  Add your own fine print.

Purchasing

Time to place – Time between purchase request and placing of the order. This might include a sub-metric; Time to approve.

HR

I provide an extensive list here, but at a minimum, there should be a focus on the timing-related items, such as Time to Indoctrinate/Perform initial training.

QC

Hold times – inspection wait times, for example – we often find Cp, Cpk, etc. – but I think the product metrics are pretty well understood. I could cover them in another post.

QA

All of the the QMS related items, particularly:

Internal Audits on Time

Open NCR/CA/PA aging

Until Next Time

Please, don’t get the idea that all of these metrics are expected all of the time, other than customer satisfaction metrics.

But, most organizations could use a few more, rather than a few less. Certain metrics come and go. They get crushed, and another one takes its place.

At the end of the day they should provide value to the company, and generally that can’t be determined until they are tried on for a while.

Quality metrics are the lenses by which we see how the organization is working; make them work for you. Empty frames don’t do anyone any good.

 

 

 

Annex SL, Human Resources, Metrics, Quality Tools

Metrics 2 – How To’sday

Leave a comment

Mouse MetricsLast How To’sday we introduced the topic of metrics and half of the Why of it – namely, “because it makes sense”.

This week, the other Why, “because it’s required” is covered – it means putting a little more focus on Objectives, but metrics are at the core of that.

Lastly, we’ll make a good start at examples of metrics by looking at a typical suite of HR measureables.

Where are the Requirements?

Within 9001:2008

Let’s start with ISO 9001:2008 as it is representative of the requirement within several other standards. Mainly, there’s this:

4.2 Documentation requirements
4.2.1 General
The quality management system documentation shall include

a) documented statements of a quality policy and quality objectives,
b) a quality manual…

This section requires that objectives be documented, and by documented this means according to the methods described in 4.2.3 Control of documents.

So, objectives are in the same documentation category as your Quality Manual, the Quality Policy and all of the required procedures as stated in the standard.

I thought we were talking about metrics? As covered in the last installment, metrics are the method by which performance to an objective is measured. Can’t discuss objectives, without touching metrics.

5.4.1 Quality objectives
Top management shall ensure that quality objectives, including those needed to meet requirements for product…, are established at relevant functions and levels within the organization. The quality objectives shall be measurable and consistent with the quality policy.

One key point here is, “established at relevant functions and levels within the organization“. So, if a function doesn’t have at least one objective, an organization would have to make the claim that the function isn’t relevant. If it isn’t relevant – why is it there?

And to underscore the point of what a metric is, the section also contains, “The quality objectives shall be measurable“.

I do realize that may present a slight contradiction to what is stated in ISO 9000:2005 (Definitions) as it says, under the term (3.5.1) distinguishing feature (a component of an objective), “A characteristic can be qualitative or quantitative.”

We’ll have to assume that a qualitative characteristic is measurable. I guess that’s what adjectives are for. Don’t worry, that potential disconnect is the least of your worries.

These Metrics are Out of Control

Objectives are built into PDCA

Also, within 9001:2008 is a discussion of the Plan-Do-Check-Act (PDCA) process. In the description of “Plan” it states:

“establish the objectives and processes necessary to deliver results in accordance with customer requirements and the organization’s policies.”

And for “Check”, it adds:

“monitor and measure processes and product against policies, objectives and requirements for the product and report the results.”

Clearly, for an organization to be compliant with ISO 9001:2008, it must conform to the model presented by PDCA – a major component of the current version. Establishing objectives and monitoring performance against them via metrics is then a key management system component.

Annex SL

For the future, it’s wise to look toward Annex SL as a guide as this will cover many sectors – and it is built into the next version of ISO 9001.

Where is says, “XXX” it means “Insert management system nomenclature of choice here” – such as “Environmental” or “Quality” or “Safety“, etc.

6.2 XXX objectives and planning to achieve them
The organization shall establish XXX objectives at relevant functions and levels. The XXX objectives shall

  • be consistent with the XXX policy
  • be measurable (if practicable)
  • take into account applicable requirements
  • be monitored
  • be communicated, and
  • be updated as appropriate.

Nearly the same as 9001:2008, right? Those bullets are, with the exception of the last, accounted for within the current text in several places.

There’s a few helpful Notes further down the section as well:

NOTE 1 to entry:   An objective can be strategic, tactical, or operational.

NOTE 2 to entry: Objectives can relate to different disciplines (such as financial, health and safety, and environmental goals) and can apply at different levels (such as strategic, organization-wide, project, product and process (3.12)). 

NOTE 3 to entry:   An objective can be expressed in other ways, e.g. as an intended outcome, a purpose, an operational criterion, as an XXX objective or by the use of other words with similar meaning (e.g. aim, goal, or target).

Some Actual Metrics

Here’s a look at the possibilities. Not all of these are suitable for all organizations and the list isn’t exhaustive.

And, just like more rivets won’t make an airplane stronger, the full collection isn’t something to “strive for”. Some of these may even be mutually exclusive.

In no particular order – Here we go!

Human Resources

Timing related

  • Days to orientation training: Assumes must be done within X days,
  • Other required training completed on-time
  • Reviews done: (percentage)
  • Reviews late

Training Related

  • Percentage of required training completed: Forces a clearer definition of training requirements
  • Pass/Fail rate for verification of effectiveness: Are some methods better than others; can they be made better? Should everyone pass every time?
  • Trainer/Training satisfaction indexes: Are some trainings better than others… Ask the students.

Recruitment related

  • Interviews to offers ratio: Provides a measure of how efficient recruitment is. Track by manager to get data across departments, and which managers might need help.
  • Referral rates by department: Part of the new hiring process or even the exit interview (in some cases) – “Would you recommend [your company] to a friend?” Better to make it a scale from 1-5, for example.
  • Percentage of hires by source: For your business, is Monster better than LinkedIn? And so-on.

Retention related

  • Retention rate by type of employee: What are the  mission critical roles, or the day-to-day “getting it done”, or the “can find this role anywhere” types? Are you better at retaining some levels over others?
  • Resignation rates by department:  Not necessarily a reflection of a bad manager, there may be other factors at work.
  • Overall Monthly Turnover Rate: The standard calculation goes:  (number of departures during month divided by the average number of employees during month) x 100 to get the rate).

Performance Related

  • Revenue per Employee: Simply total revenue divided by total number of employees.
  • Human Capital Cost: Pay + Benefits + Contingent Labor Cost / Full Time associates.
  • HR to Staff Ratio: Employees / Human Resources Team Members. Essential if you ever want to justify departmental expansion.
  • Promotion Rate:  Promotions / Headcount.
  • Overtime per Individual Contributor Headcount: Overtime Hours/Individual Contributor Headcount.
  • Employee Absence Rate: Number of days in month / (average number of employees during month x number of days).

Demographic Related

Hires that meet certain demographic characteristics can reflect a broad spectrum of requirements, policies and philosophies. These metrics are not a means toward discrimination, but as a tool to determine progress toward a goal or objective.

  • Percentage [Females, Military Service, Race] by Management Level:  Can be essential when bidding some government contracts and to fortify positions with a given quality policy.
  •  Average Age of Employees: Consider breaking this down by department, management level – or both.

Until Next Week!

I hope you were able to find something useful in the HR list of metrics. And, with some luck, you’ve gained a broader understanding of why we need metrics at all.

Next week we’ll look at the other main departments within a typical organization – along with additional rationale behind their implementation.

Thanks again for reading.

Sal

Improvement, Metrics, Quality Tools

Metrics – How To’sday

1 Comment
W. Edwards Deming

Collecting Metrics and tracking them is widely considered to be key to improvement – but the specifics are often elusive, particularly for relatively small organizations.

And, naturally, acting appropriately to the results is critical to improvement as well. We’ll touch on that, too. Eventually.

These posts (there will be at least two of them on the topic), while not providing an exhaustive list of departmental metrics, will provide at least a starting point for many.

The terms themselves are often used interchangeably, even if they clearly aren’t synonymous; metrics, KPIs, measurables, goals and objectives, dashboards…  where to begin?

Define Your Terms

Beyond dictionary and accepted terminology, ISO 9000:2005 “Quality management systems — Fundamentals and vocabulary” has formal definitions for a few of the key terms we’ll be using.

3.2.5 quality objective – something sought, or aimed for, related to quality (3.1.1)

3.1.1 quality – degree to which a set of inherent characteristics (3.5.1 – “distinguishing feature”) fulfils requirements (3.1.2 – “need or expectation that is stated, generally implied or obligatory”)

The next question is, “How does one measure progress against an objective?”

Fortunately, the 2nd note in ISO 9000:2005 under the term (3.5.1) distinguishing feature (a component of an objective) says, “A characteristic can be qualitative or quantitative.”

Helpful or not, Note 3 goes on to state:

NOTE 3 There are various classes of characteristic, such as the following:

  • physical (e.g. mechanical, electrical, chemical or biological characteristics);
  • sensory (e.g. related to smell, touch, taste, sight, hearing);
  • behavioral (e.g. courtesy, honesty, veracity);
  • temporal (e.g. punctuality, reliability, availability);
  • ergonomic (e.g. physiological characteristic, or related to human safety);
  • functional (e.g. maximum speed of an aircraft).

Essentially, you can measure performance against an objective any way you wish – that performance measurement is what I am calling a metric.

Why have Metrics?

If an organization is going to spend time doing anything, particularly with metrics, then there should be a reason.

So, as with most Management System realities the two main reasons tend to be “Because it’s required” and “Because it makes sense”.

Let’s look at the latter first.

Because it makes sense.

Does it? You may be surprised to learn that not everyone agrees with managing via metrics, or at least, they would say that not everything of importance can be measured.

W. Edwards Deming (October 14, 1900 – December 20, 1993) was one who said it, in fact. Don’t know who Deming is?  Complicated man – almost legendary, possibly self-contradictory (the lens of history often does that to great men). He is thought to be one of the key figures helping Japan rise from the literal ashes after WWII. He gained popularity by expanding and championing concepts of Statistical Process Control, Plan-Do-STUDY-Act (PDSA) – and much, much more. I’d need a book to do him justice.

Side note – Deming was not a fan of the now common “Plan-Do-Check-Act” (PDCA). Notice the difference; “Study” vs “Check”. He called PDCA a “corruption”. His PDSA takes Study literally as deductive and inductive learning – it is built into a learning and improvement cycle. We’re getting a little far from the main topic, but I could expand on this more if anyone would like me to – leave a comment.

Here is a list of Key Deming Books. If you only have time to read one, many would recommend “Out of the Crisis“.

While Deming understood that not everything is measurable, and was quick to point out the dangers of slogans, thoughtless quotas and targets (such as “zero defects”) he still believed a system must be managed and that “There is no substitute for knowledge.” 

For the mere mortals among us, however, a reasonable place to start is by carefully selecting a set of solid process measurements to determine if something is improving – or not.

Another popular way to put it, “If it can’t be measured, it can’t be improved.” (sounds a bit like a slogan, doesn’t it? My apologies to Mr. Deming). That quote, by the way, isn’t attributed to Deming – it is a quote from Lord Kelvin (actually “William Thomson” of Belfast, 1824-1907). He was a mathematical physician and father of the first and second Laws of Thermodynamics. Next time you hear someone say Deming said it, please do let them know how unlikely that would be.

Anyway, what I’m saying is, “A map doesn’t help you if you don’t know where you are.” Another slogan… That one, by the way is probably based on a similar quote by Lewis Carroll, an English writer, mathematician, logician, Anglican deacon and photographer. His went, “If you don’t know where you are going, any road will get you there.”

Did I make my point? What was the point? Do metrics make sense? I’m leaning toward “Probably”. Let’s circle back after we look at some metrics – see if you think they’d make sense.

We’ll address the first reason, “Because it’s a Requirement” – next Tuesday. I’ve got more Deming to read – I can’t decide if he was a genius or if he was insane… typical.

Thanks.

Sal