Design of Experiments (DOE) for Manufacturing

Girish P Kelkar, Ph.D., WJM Technologies, Cerritos, CA 90703

Abstract

Design of Experiments (DOE) is a powerful tool to understand and improve manufacturing processes. With the wide-spread use of computers and DOE software, one would have expected that majority of manufacturing engineers would be using DOE techniques. However, use of DOEs is not as prevalent as one would expect. The lack of penetration of DOE techniques could perhaps be attributed to a typical first exposure that does not meet expectations. In such situations, DOE is regarded as unreliable, needing expertise in statistics, and unsuitable for the particular process in question. The paper discusses common concerns with DOE including basic concepts of statistics, limitations of DOE, misuse of DOE. A brief roadmap for a successful implementation of DOE is presented.

Introduction

Manufacturing processes are quite complex operations even though some might seem fairly simple at the first glance. For example even a simple operation such as turning can have a variety of factors affecting the cut quality including cutting tool geometry, cutting tool hardness, rotation speed, cutting fluid, etc. A one-factor-at-a-time approach to understand the process would require a large number of experiments and may still not provide the optimal settings. DOE techniques allow the engineer to change multiple factors simultaneously; such an approach considerably reduces the number of experiments required and also allows the engineer to investigate interactions and higher order effects. If the engineer chooses a model where factor settings are chosen such that all possible combinations are tested, then the model is called the full-factorial design. A full-factorial design allows the analysis of effects of main factors, interactions, and, depending on the factor levels, higher order effects. An interaction can be thought of as a new factor which is a combination of two or more factors. Interactions are not intuitive and their effects are hard to predict. Even if a DOE is more efficient than a one-factor-at-a-time approach, the matrix can still be very large and may not be suitable for a variety of reasons including lack of necessary materials, lack of time available on the machine, lack of man-hours, or all of the above. For example, a process with 5 factors evaluated at 2 levels will require 2⁵ = 32 experiments for a full-factorial. With three repetitions at each setting, we need 32x3 = 96 experiments. Depending on the time required for each experiment, completing the matrix can take anywhere from half a day to a few weeks.

To perform DOE studies without a full factorial matrix, statisticians have devised fractional factorial matrices where a certain set or combinations are not tested; the reduced set of experiments is called a fractional-factorial. For the example discussed above, a half-fraction would have 16 (2^5-1) experiments and a quarter fraction will have 8

Girish P. Kelkar, Ph.D. (562) 743-7576 girish@welding-consultant.com

www.welding-consultant.com

5-2

) experiments; much less than 32. This reduction in matrix size considerably reduces the number of experiments and the corresponding resources required. However, the compromise is not without drawbacks. A fractional-factorial experiment cannot clearly identify all effects independently. Depending on the size of the fraction, a certain number of factors (main factors and interactions) are confounded, i.e., their effects cannot be distinguished form each other. Confounding introduces uncertainty to the results and makes interpretation more difficult.

In addition to conducting the experiments, there is a certain level of understanding of statistical concepts that is required for DOE analysis. Unfortunately, statistics thrives on uncertainty whereas most engineers have a hard time coming to grips with statistical nature of test results. They are more comfortable with plugging in numbers into a calculator and getting a single number for an answer. Statistical concepts required to work on DOE includes basic statistics and inferential statistics.

Concepts of basic statistics are the same as those used in Statistical Process Control (SPC). Basic statistics is necessary to identify and quantify the “problem” that the manufacturing engineer is supposed to solve with a DOE. Problem definition and quantification is quite challenging and a key step in the DOE process. Basic statistics will help the engineer to isolate the location of the problem either in terms or where and when it is occurring. The problem may be related to a specific machine, a particular shift, or caused by manual error. Selection of the right population is key. If multiple populations are incorrectly identified as a single population, the variation between populations may drown out critical information. Once the right population has been identified, selection of correct sample size, histogram, and process control charts will help in defining the problem. The engineer should be careful at this stage to precisely quantify the problem and expected goals. Just expecting an improved yield may once again leave an uncertainty in expectations from a DOE on behalf of the engineer, QA personnel, and management.

While basic statistics are important in defining the problem, inferential statistics are required for interpretation of test results. Analysis of a screening DOE does not require any substantial understanding of statistics. However, interpretation of outputs from regression analysis does require understanding of α-β errors, confidence levels, F-statistic, and p-values. Interpretation of F-statistic and corresponding p-values will help eliminate certain factors that are not making a significant contribution. This is an opportune time for the manufacturing engineer to rationalize the results with the underlying physics. Any decision to drop a factor should be consistent with process knowledge. If the p-values indicate that the factor is not significant, what does that mean? Is it possible that the factor levels selected were too close? Is it likely that factor levels were not properly controlled thus adding a lot of noise to the results? If the factor is considered important based on p-values, could it be that confounding interaction is contributing to the results. Once the engineer has satisfactorily answered all the questions based on process knowledge, he/she can make the appropriate decision.

Girish P. Kelkar, Ph.D. (562) 743-7576 girish@welding-consultant.com

www.welding-consultant.com

DOE experiments also require a certain rigor for the data to be worthwhile. The factors should be definable, controllable, and measurable. For consistency, manual inputs should be minimized as much as possible. If part alignment is important, a fixture should be devised so as to reduce operator input in the alignment process. Measurements can sometimes be on a relative scale or 1 to 10; experienced and conscientious personnel are able to do such tests. The factors have to be properly selected and set during the experiment. Environmental effects of temperature and humidity can often play a major role and should be taken into account.

Perhaps the most common difficulty facing the manufacturing engineer is the resource called time. On one hand, the manager is expecting the engineer to maintain (or most likely) increase production. At the same time, the engineer is also expected to improve the process results either by reducing scrap or by improving a performance metric. To improve the process, the engineer may be interested in conducting DOE but to do so he has to stop production but that goes against increase of production quantities. In this era of just-in-time, maintaining production often becomes the mantra. In this situation, the engineer is forced to find a compromise that could severely limit his ability to conduct a large number of experiments. With limited time and resources available, the engineer has to be very particular in selecting a DOE that will provide him with the information he needs in the time available.

The rigor of properly conducting the tests also can prove a deterrent and can affect the test results such that they become difficult to interpret. Additionally, DOE is often seen as a miracle tool that, once applied, can solve all problems. It is the over-selling of DOE techniques without proper understanding of the DOE process that leads to DOE failures and distrust among engineers. There are several reasons why DOE techniques often fail to deliver and meet expectations. The next section discusses common causes that can be attributed to DOE failures.

Common Causes of DOE Failures

Most physical phenomena related to manufacturing are fairly well understood by the scientific community. However, often the understanding has not been transferred to the operating engineers who may consider the process an art form. Results from poorly designed tests and anecdotal evidence are then used to set up governing rules that do not correlate with engineering or scientific principles. It is not surprising that a DOE performed under such circumstances ends up in failure. The reasons for a lack of satisfactory outcome from the DOE can be traced to one or more of the following events:

1. DOE used to gain process knowledge One of the limitations of any experiment is that the results are valid only in the space where the test is conducted. If the results are used as an indication of trends in underlying physical process, then one might draw the wrong conclusions. Figure 1 shows

Girish P. Kelkar, Ph.D. (562) 743-7576 girish@welding-consultant.com

www.welding-consultant.com

the trends for change in weld strength as a function of welding current. Quite often, the results will show a maximum on either side of which the strength values decrease. Now if two independent DOE studies are conducted on either side of the maxima then conclusions drawn from the two would be vastly different. One would conclude that strength increases with current; the other would conclude that strength decreases with current. Anyone who tries to draw process knowledge from either study would obviously be misled.

Selection of DOE matrix and size A lack of process knowledge leads to inclusion of too many factors. A large number of factors and limited resources (time, materials, man-power, etc.) then require selection of matrices that are two-level and highly fractioned. Such a matrix will produce multiple interaction schemes thus becoming difficult to interpret the data. At the end of the lengthy experiment, if the DOE does not produce a “miracle” it is considered a failure.
Factor levels selection Selection of meaningful factor level settings is also a key to successful experimentation. Without proper process knowledge, factor levels selected could either be too close or too far apart. Either way the conclusions drawn could be misleading. Figure 2 shows a schematic to make the point. Additionally, if the factor levels are set too close, the user should make sure that those levels can actually be set and be differentiated. Often the noise level at the two settings may essentially nullify any actual effect, if any.

DOE Roadmap

In order to avoid the common pitfalls, a DOE roadmap is proposed that will guide an engineer through the DOE process. The 10 step process outlined below can help the engineer to be efficient with resources as well as be able to meet his process improvement goals. The individual steps are listed and discussed below:

1. Acquire process knowledge The first, and the most important step, is to acquire a strong fundamental understanding of the process under consideration. Knowledge gained from experience or by conventional wisdom may not be technically correct and may actually be misleading. Additionally, new developments in that field may provide alternative solutions and should be investigated prior to launching a DOE. Process knowledge could include understanding of materials involved and their characteristics that affect the process, understanding of process physics that helps to shape or change the materials, and methods to identify and compensate for typical process variations. A good in-depth process understanding will also allow the engineer to identify critical factors that he needs to maintain in close tolerance. Process knowledge will play a critical role in identifying important factors, fixing factors, setting factor levels, and interpreting results. It is recommended that the engineer attend a training class to get a good in-depth understanding of the process that is being investigated before launching into a DOE.

Girish P. Kelkar, Ph.D. (562) 743-7576 girish@welding-consultant.com

www.welding-consultant.com

Identify/Quantify the problem In order of importance, this would be the second most important step since if the problem is not properly defined, it is almost impossible to find the correct solution. It is important to define the problem in quantifiable terms so that there is no ambiguity at a later stage. For example, goals could be to have mean weld strength of 50 lbs with minimum strength no less than 30 lbs tested over 25 consecutive welds. A Cpk number could be used as well. The engineer should be realistic in setting the goals since it is easy to call out a 6sigma process but may be difficult to get to in the near term.
Acquire DOE expertise A basic knowledge of statistics and DOE can be learned from books or by attending a training class. Statistics is a difficult subject for engineers and may require some time to become comfortable with. As discussed earlier, basic statistical concepts are important to identify and define a problem and inferential statistics are important in interpreting DOE results.
Identify important factors At this stage, the engineer should assemble a team of personnel involved with this project including those from QA, design, and manufacturing. Input from all participants during a brainstorming session will bring out important issues and may help in diffusing any misunderstandings about the process or the goals.
Reduce variable factors Based on the process knowledge, it would be prudent to fix the settings for as many factors as possible. This step helps in keeping the design matrix small and reduces the time required on the machine. The factors that are chosen to be fixed at a certain setting should be such that they can indeed be maintained during the DOE and later in actual production. If the factors cannot be controlled they should be part of a DOE or should be included in a noise matrix.
Setup and conduct a screening DOE At this stage, the engineer should have narrowed down the number of factors to 7 or less. A fractional matrix with 8 runs can be used for 7 factors at 2 levels each. With 3 repeats for each run will result in 24 experiments. Of the 7 factors there could be one or two factors whose settings are difficult to switch between “low” and “high.” If there are such factors, then randomization of these factors should be kept to a minimum. For example, one of the factors could be chemistry of a plating bath or composition of solder in a solder pot that is difficult to change. The experiment conducted in this manner will not be completely randomized as required per strict statistical guidelines. Interpretation of factor effects of this particular factor should be conducted accordingly. A portion of the factor effects observed may be attributed to time-dependent effects; examples could be tool wear or change in ambient conditions.

Girish P. Kelkar, Ph.D. (562) 743-7576 girish@welding-consultant.com

www.welding-consultant.com

Analyze screening DOE results The engineer should be aware of the confounding that exists in such a screening matrix and should take great care in interpreting the results. Confounding makes it difficult to separate effects of some of the factors and hence interpretation should be based on process knowledge. If interpretation becomes difficult, the engineer may have to conduct a simple side experiment to get a better understanding. Based on the results of this matrix, the number of factors should be reduced to 2 or 3 factors. The others should be fixed at an appropriate level that can be maintained within tolerance during actual production.
Run small 2 or 3 factor factorial DOE A small factorial DOE with either 2 or 3 factors should be conducted at this stage to get an understanding of the trends. The factor levels should be tweaked based on results of the screening DOE. The number of levels should be chosen based on type of response expected. If factor response is likely to be linear, then a 2-level DOE would work fine. If response is likely to plateau or has a maximum, a 3-level DOE would be more appropriate. Once again, the tradeoff is between time available and results required.
Analyze results of factorial DOE Analysis of a full-factorial DOE is easier since there is no confounding. All factors, interactions, and higher order terms can be independently estimated. At this point, the focus should on the F-statistics (or p-values) for individual factors, adjusted R-squared for total fit of the model, and analysis of residuals to check if further tweaking is required. Residual analysis is essentially a means of understanding the variation that has not been explained by the regression equation. If residual variation is random and within limits, the regression equation will accurately reflect the underlying physical phenomena.
Draw conclusions At this point, the engineer has two options. One is to conclude the DOE and select a particular set of factor level settings to check validity of the results over a larger batch of parts. If the factorial DOE results indicate that the process has not produced the desired results, either in terms of metrics or variation, then the results of the factorial experiment can be used for the next level DOE to move towards an optimum. Alternatively, it may be time to get back to the drawing board and take another hard look at the process, factors, factor settings, and the analysis.

Summary

Design of Experiments is a powerful tool for the manufacturing engineer to have in his arsenal. However, the engineer should have realistic expectations of what a DOE can do for them and be aware of the common pitfalls that prevent successful implementation of

Girish P. Kelkar, Ph.D. (562) 743-7576 girish@welding-consultant.com

www.welding-consultant.com

such techniques. A road map is presented that gives a simple step-by-step approach to implementing DOE on the shop floor.

Bibliography

“Statistical Quality Control,” Douglas C. Montgomery, 2nd Edition, John Wiley and Sons.
“Introduction of Probability and Statistics,” J.S. Milton and Jesse C. Arnold, 2nd Edition, McGraw-Hill.
“Understanding Industrial Designed Experiments,” Stephen R. Schmidt and Robert

G. Launsby, 4^th Edition, Air Academy Press.

4. “Fundamental Concepts in the Design of Experiments,” Charles R. Hicks, 4^th Edition, Oxford University Press.

www.welding-consultant.com

Figure 1. Figure shows a schematic of variation on weld strength as a function of welding current. To the left of the maxima, the welding strength increases with current; towards the right, it decreases with current.

Figure 2. Figure shows a schematic with effects of factor settings. If factor settings are set either too close or too far, their effect on the output may not be significant and without process knowledge could lead to misleading results.