Understanding Why Proper Product Sampling Provides Credible Results

Sampling is the process of selecting material to be tested. The process used for collecting test
samples directly affects the conclusions that can be supported by the test results. Statisticians
use the term “inference” to describe the extension of test results from a sample to a broader
group, or population, of product. An appropriate sampling process allows the ensuing test
results to be inferred to the desired group. Conversely, inference of the results beyond the
group that can be justified by the sampling process will lead to inappropriate conclusions.
From time to time, roofing industry members collect shingle, roll goods, or other product
samples to have their properties tested and publish the results. This may be for a magazine
article, a product rating on a website, or a blog post, among other uses. The conclusions that
may be reached from those results are dependent upon the process used to collect the product
samples.
Before starting a product test program, consider the intended use of the results. This first step
is essential to creating an appropriate sampling program. If the desire is to make broad
statements about a product, the number of samples to be collected will be much larger than if
a specific statement is to be made about a defined collection of product. Figuring out the
question to be answered is critically important.
A key aspect of any sampling program is the statistical concept of a “lot.” One definition of a lot
is found in ASTM E456 – Standard Terminology Relating to Quality and Statistics which states,
“a definite quantity of a product or material accumulated under conditions that are considered
uniform for sampling purposes.” Examples of a lot might include:
• All the shingles of the same brand and style in a distributor’s warehouse
• All the modified bitumen cap sheet rolls delivered to a job site for application on a roof
• All the packages of a specific ventilation product on a single truckload shipping from a
roofing manufacturer
“Random” is another key concept in sampling. Grabbing the first five bundles or rolls available
is not random sampling. Random selection requires that each unit (e.g., bundle, roll, package) in
the lot has the same chance (i.e., statistical probability) of being selected. This includes the
bundle on the bottom row of the least accessible pallet and the roll in the middle of the pallet.
Proper sampling is essential. If a carefully designed sampling plan is followed, the tester can have reasonable confidence that similar results would be obtained if different bundles or rolls
were selected and tested from the same “lot” of material. Results from testing properly
selected samples can be inferred over the entire lot of material.
Suppose five asphalt ply sheet rolls are purchased at a retail outlet, without being randomly
selected, and are subjected to testing. Results from testing the rolls are applicable only to those
five rolls. The tester cannot properly infer those results to the pallet or pallets from which the
rolls were taken, to the stock of those rolls in the retail outlet’s warehouse, or to the specific
brand and style of roll good, because the randomness requirement of the sampling
methodology was not satisfied. All that can be concluded is that the five-roll sample met or
failed to meet the criteria being evaluated.
As another example, consider a situation in which 10 bundles of shingles are purchased from a
roofing distribution outlet and evaluated for one or more properties. What can be concluded
from the results? As in the previous example, the sampling process determines the limitations.
In this case, the sampling process permits the distributor to select the shingle bundles. If the
distributor selects and ships the most easily accessible 10 bundles, the conclusions of the
testing are constrained to the 10 bundles and should not be inferred to apply to any larger
group or population of shingles. If the distributor selects the shingle bundles randomly from
stock of that product in its warehouse, the associated test results may be inferred to apply to
the stock in the warehouse, but not to a larger group.
Conclusion
When considering test results for shingles, roll goods, or any roofing product, remember to
consider the sampling process. Use of proper sampling techniques ensures confidence in
conclusions drawn from testing of product samples.