Schlagwort-Archive: quality

Test Approches for Green and Brown Field Projects

The picture  on the right side show the relationships between the different test levels. The tests are a kind of test stages in green field projects. Usually, the stages are done from bottom to top and from different parties or members of the time.

 

 

Green Field Projects (and also new code in brown field if possible)

Here it is to be pointed out, that the usual order of testing in Green Field Projects (and code writing) is done in the following direction:

  1. Writing Code (developer)
  2. Checking Code (developer and CI system with plugins)
  3. Writing Unit Tests (performed as block in TDD with steps 1 and 2)
  4. Writing Component Tests (developer and QA)
  5. Writing Integration Tests (developer and QA)
  6. Setting up automated GUI tests (QA)
  7. Manual Testing (QA and if needed, developers conducted by QA)
  8. Endurance Tests (QA)
  9. Performance Tests
  10. Load Tests (Stress Tests)
  11. Decoupled: Penetration Tests (continuously by all, but systematically by QA)

The first three steps which go hand in hand and in TDD it seems that the numbering is reorder to 3, 1 and 2, but the code of the tests is also code.

The tests become from one step to the next less detailed, but go more into use cases and real world scenarios. The tests become also more and more realistic in the configuration and setup in relationship to the target environment. Even patch levels need to be controlled, if needed.

Brown Field Projects

In Brown Field Projects it is the other way around. A legacy code base is available and not under test. The approach can not go here to write unit tests first, because it is too much work to do and in most cases it takes a long time to understand the old legacy code. To get a good unit test suite, one calculates roughly to have at least 1 unit test for every 100 lines. In a legacy code base for only 100,000 lines it would mean to write at least 1,000 meaningful unit tests. This work would really suck…

The approach goes here to write integration tests first to assure the basic functionality. The basic use cases need to be right and the functionality is assured. With integration tests not all paths of working can be checked, but the basic behavior, the correct results for the most important use cases and the error handling for the most common error scenarios can be checked.

As soon as new functionality is developed, functionality changed or extended, new integration tests are to be implemented first to assure the still needed functionality stays unchanged and the new functionality is tested as it is required. Unit test and other tests are implemented as needed and practical.

As soon as an integration check fails, the failure analysis will reveal the details of the issue. These details are then check with additionally developed component and unit tests. As long as this procedure is used, more and more meaningful unit tests are developed and the more test coverage is reached.

The same is with GUI tests. At first one has only a chance to perform manual GUI tests. Later on the most common use cases and the general GUI behavior can be tested automatically. This also leads to a better test coverage.

In Brown Field Projects the tests in different tests groups are also developed in parallel. Integration tests are developed in parallel with the performing of manual testing which assures the current ability to deliver of the software. The current situation at hand dictates what is to be done.

Programming is both: A Craft and Science

There is a latent conflict in software development. I experienced it several times in different contexts. It is no always obvious, but nevertheless it is there… There are two different approaches to software development: An academic and a craftsman approach. One needs to be a good craftsman to write solid, maintainable, and efficient code, but one also needs to be an academic to understand IT systems and their specialties.

The academic approach is all about theory. Students in universities get told that theory. These students become very good and they are very knowledgeable, but they lack the experience and practice to develop real world software. They also learn a lot of mathematics like calculus, numerical mathematics, statistics, and linear algebra. For academics it is often difficult to realize that real world software cannot be perfect (too expensive), needs to be maintainable (software needs to be maintained and bug fixed and also developed on in future) and cost efficient, robust solutions might not be modern, cool or state-of-the-art. Academics also tend to fall in love with some problems and try to solve them in the most beautiful way possible. This is economically meaningless and too expensive.

The other approach is, that software is a craft. Real craftsmanship needs a lot of time, practice and guidance by a senior. It is like building stuff. The first trials will be messy, unusable, ugly… But, the more practice a craftsman get the better the outcome is. But, good code does not solve hard and complex issues. These need an academic approach. For example, some problems need to be converted into problems of Linear Algebra to get them performed efficiently. A pure programmer who only learn to program is not able to do that.

The problem is now: Students are intellectually trained very well, but they are no craftsman. Pure craftsman can write good code, but may not be intellectually prepared for complex systems. As a consultant I see both sites. I train scientists and engineers which are experts in their specific domain to write good, maintainable code, but I also see people who write good code, but need some input on how to tackle some hard problems.

What can be done!?

Academics: Science and engineering is ruled by laws and intellectual challenges. It is fine, but there is more to be done in Software. For Academics I recommend to read some books like „The Pragmatic Programmer“ by Andrew Hunt et. al. (http://astore.amazon.de/pureso-21/detail/020161622X) and think about initiatives like Software Craftsmanship (http://manifesto.softwarecraftsmanship.org) and observe the own daily doing. Academics in most cases are very good in self reflection. It is easily understood what needs to be done to not hassle later on with bad code, unmaintainable software. (For questions, I can be booked, too… ;-)) The main concern should be: How can the job be done in a way with maximum of automation, minimum of repetitive work and as maintainable as possible.

Craftsman: A craft is doing, evaluation and redoing until perfection is reached. It is a good approach, but some problems cannot be solved so easily. The easiest way to proceed is to team up with a domain expert, the second best thing is to read about the domain of expertise which is missing to become an expert in this field to a certain level which is needed. The main concern should be: How can the problem be solved efficiently? The most problems are already solved to a certain level. This can be copied and studied. Only the remaining parts need to be invented.

The best software developers combine skills of academics and craftsman. Both sites are challenging on its own, but the combination needs to be developed over time. The result is worth it: The software developed by these people is amazing: Maintainable, understandable, simple, efficient…

The Curse of Cp and Cpk

In factory automation the calculations of Cp (Process capability index) and Cpk (measure of process capability) values are a good tool to monitor and proof performance and quality. I do not give here the full theory for the calculation of Cp and Cpk and all the math behind it. I want to show some issues from my professional experience which arise in understanding and also in expectations.

Some Theory for Recapitulation

For a given process measurements are performed and for a single parameter the average (Avg) and the Standard Deviation (StdDev) are calculated. The Cp and Cpk values are calcualted in the following way:

Cp = (USL – LSL) / (6 * StdDev)

Cp,upper = (USL – Avg) / (3 * StdDev)

Cp,lower = (Avg – LSL) / (3 * StdDev)

Cpk = Min(Cp,upper, Cp,lower)

The Cp value gives the theoretical capability of the process. It assumes a centered production and evaluates the best possible outcome for the given process deviation. This is the target to reach for process centering. The Cp value it self should be maximized with minimizing process deviation.

The Cpk value gives the current process performance and takes also into account if the process is not centered. If the process is centered the equation Cp = Cpk is valid.

For a Six Sigma production, values of Cp=2.0 and Cpk=1.5 are expected. The theoretical process capability should show a process deviation which matches six times into the process specification. A normal variation of the process around its center is always present and can not be avoid completely. Therfore the long time Cpk value is allowed to be smaller due to experiences of process variations in field of 1.5 sigma around the target.

If everything is Six Sigma ready, one gets only 3.4 violations of the specification into each direction out of one million samples. That means 6.8 violations in total for one million samples.

The Issue of Normal Distribution

The first issue and seldom questioned is: „Are my parameters normally distributed?“

In reality I have seen processes with Cp and Cpk values smaller than expected or wanted, but without any fail. The reason is that the parameters used are not normally distributed and the caluculations and statistics go fail. One can think of a rectangular distribution which is always within specification, e.g. between LSL/2 and USL/2. The calculation of Cp and Cpk give a Sigma level which tells to have a messy process. But in reality everything goes fine and no violation is to be expected. Only the math gives numbers out of wrong expectations.

A gate oxid growth for example can not be normally distributed due to the impossibility of negative gate oxid thicknesses. Processes which are under SPC (Statistical Process Control) control where corrective actions are performed, can also not be normally distributed. The influence of the corrective actions destroy the normal distribution if it were present before-hand.

One should always do a test for normal distribution first before calculation Cp and Cpk. In reality only a few processes are quite well normally distributed and for them the normal Cp and Cpk measurements are suitable. In most processes one should use different distributions for calculations or better do a counting of violations and a calculation back to a Sigma level. Those gives better information about real processes.

Hunting for Cp and Cpk

In a lot of industries, quality is one of the most important values and in some even life saving like in aerospace, automotive and medical industry. The pressure to reach and proof quality is very strong and necessary to stay in business and to challenge ones competitors. Cp and Cpk values are crucial for all purposes.

Cp and Cpk strongly depend on the specifications used. This leads to the next section…

Setting the Limits

When it comes to customer satisfaction, good Cp and Cpk values are important goals. It’s sometimes hard to reach them technically, but choosing the right limits can help. I heard, that the specification limits can be adjusted to meet Cp and Cpk value requirments and that these limits do not have any restrictions due to the fact that the process is controlled trougth its control limits. I knew afterwards I have to write about the right limits. Even a professional consultant on this mentioned something like that and I could only explain why there are technical limits for process and quality engineering.

At first we should define which limits are available for process control and what the purposes of them are.

Functional Limits

These limits define within which limits the product is suspected to work. Outsite these limits the product is just waste, not to be used and therefore to be dumped. There is nothing more to say… 😉

Specification Limits

These limits are negotiated with the customers or internal limits which are defined to decide whether a product is to be delivered or not. Specification limits can be defined multiple times for different customers, markets or functions, because not all product purposes need the same quality. The only requirement is, that the specification limits are equal to the functional limits or smaller within the functional limit range. A product which does not work at all is not suitable for any market except for selling to waste recycling.

Control Limits

Control limits are used for triggering corrective actions. If the process runs out of the control limits, a production system should signal this event immeditially and someone responsible has to start corrective actions to get the process back to target or the deviation back to a normal level. The control limits need to be defined left and right (or above and below) the target to make sense for corrective actions. For the purpose to start corrective actions, the control limits also have to be defined between the specification limits. If the control limit triggers an event outsite specification, it’s too late for any meaningful actions. Several control limits can be defined if necessary to separated different levels of escalation or different levels of inversive corrective actions.

Relationship of Limits

Regarding the facts above there is a simple relationship between all these limits which can be written as:

UFL >= USL > UCL > TARGET > LCL > LSL >= LFL

Shortcut Name Meaning
UFL Upper Functional Limit The upper limit for the product to work properly
USL Upper Specification Limit The upper limit of the product specification as delivery criterium
UCL Upper Control Limit The upper control limit as trigger for corrective action
TARGET Target The production target and optimum goal for production
LCL Lower Control Limit The lower control limit as trigger for corrective action
LSL Lower Specification Limit The lower limit of the product specification as delivery criterium
LFL Lower Functional Limit The lower limit for the product to work properly

The Curse Revealed

Where is the curse now in all this? Everything is well defined and we do the right statistics!?

In reality, most parameters are not normally distributed and the calculation of Cp and Cpk give numbers implying a worse process than really present. The numbers do not look very well because of wrong asumptions of normal distributions. The correct calculation is difficult and only statistics based on event couting is really accurate, but the number of events for counting need to exceet the 1 million mark, which is not really practical.

Customers also want Cp and Cpk values which do have a well defined sigma level like 3 (Cp = 1.0), 4 (Cp = 1.33), 5 (Cp = 1.67) or 6 (Cp = 2.0). The specification limits can only be expanded to functional limits and the process has to have a deviation and variation to meet this goal which is sometimes not really meetable due to physical constraints of machines or processes. Sometimes the process capability can not be proofed accurately with Cp and Cpk values.

In semiconductor industry for example one has to deal with physical issues. For an 8nm tunnel oxid for example one has to grow a layer of roughly 30 atoms to meet the thickness. The specification is set to 8nm +/- 1nm. 1nm is roughly 4 atoms. To meet the Six Sigma level one has to have a process deviation of a 6th of this 4 atoms, which is 2/3 of an atom. Therefore, to meet the Six Sigma level one has to prepare with a process variation of 2/3 of an atom and that in a process which processes a whole 8 inch wafer. For the whole area only a 2/3 of an atom of variation is allowed. These kind of tunnel oxids are prepared in 0.6um technologies. The current technologies in CMOS go down to smaller than 0.03um and the tunnel oxids become thinner and thinner… We obviously meet the physical limitations here.

What can be done, when customers have their expectations and the physics is at its end? Sometimes there is a real gap between needs, expectations and physical capabilities.

Solution

Sometimes we have to slow down, to sit together and to discuss in detail and technically correct what’s going on. If the Cp and Cpk values are used for production control, it’s a good step, but the blind hunt for numbers is not leading to the final target: Quality. The numbers are not always correct and the expectations have to be adjusted to the reality. Every production process should be optimized for best possible and effortable quality, but this has to be made transparent to external and internal customers of manufactoring parameters.

For managers, engineers and customers, there should be open discussions and if needed trainings to get background knowledge to get into constructive discussions and decision making procedures. Otherwise there is a large potential for professional conflicts…