Imagine coming home from a day of outdoor activities and changing into new clothing, only to discover that you have a sunburn. You applied sunscreen before heading out, so what gives? Well last month, Consumer Reports reported, of the 60 sunscreens tested in 2017, almost 40% demonstrated less than half the advertised sun protection factor (SPF). This is concerning given the number of diagnosed cases of skin cancer in the US is more than all other cancers combined. The main cause of skin cancer is the overexposure of skin cells to ultra violet (UV) rays from sunlight and tanning beds. These rays can cause DNA damage in cells, causing them to mutate and become cancerous. One way to protect our cells from this damage is to use sunscreens, but given the data from Consumer Reports, the average person may not be able to trust the product is going to be effective. But before we panic, let’s unpack this a bit, as it is more complicated than it seems.

The US Food and Drug Administration (FDA) is the federal agency charged with reviewing and approving a variety of products including sunscreen. In the US, sunscreens are considered Over-The-Counter (OTC) Drugs. The OTC approval process is focused on the ingredients and formulation of specific sunscreen ingredients and does not specifically approve specific products. This is important because it puts a lot of faith in companies to ensure their products meet the advertised SPF rating and does not individually verify every product's SPF claim.

It is important to understand how experiments can vary, and while scientific research depends upon reproducibility, meaning that two scientists from different parts of the world should be able to perform the same experiment with the same tools to obtain the same or similar data, there can be variation in the experimental system. Depending on the sensitivity or design of the experiment, the data averages should be similar, but specific data points may not. For example, if I were to run 100 meters three times, it is unlikely I would get the same time, but I would have an average number I could use. This is an example of acceptable variability within an experiment.

The FDA regulations provide guidelines for specific experiment. This method relies on human researchers experimenting on human test subjects. This creates natural variation both in application, as the testers could have slight variations in amount of product applied and how it is applied, as well as variations of the reactions in the skin being tested on the human subjects. Even with this variability, repeated testing should provide an average SPF rating sufficient enough to guarantee an SPF. However, the lack of oversight by the FDA could result in differing SPF as a company could test their product at the time of initial release and not perform follow-up tests to ensure each lot maintains the promised SPF. This assumes the manufacturing process or the combination of ingredients never changes the overall SPF of the product. It is important to note, that a company may retest if they reformulate their product, but again, ongoing testing is up to the company.

To further complicate this SPF testing and labeling process, it is important to note independent testing can use variations on the prescribed experiment or take a completely different experimental approach. This is a problem when one tries to compare the data. Take the example of running 100 meters, if instead, this time I ran backwards, my final time would be very different, as I am quite uncoordinated, but experimentally I am still measuring how fast I could run 100 meters. This is the same concept in the laboratory. You may have two different experiments designed to measure the same thing, but because of the difference in methodology, you get two different non-comparable results. This is the primary response from the sunscreen manufacturers, stating the results cannot be compared and their internal testing is accurate.

Variability between identical experiments and comparisons between experiments with different methods is one of the greatest challenges scientists and the science-consuming public face. These intricacies can be teased apart by experts familiar with the experimental design, In the scientific community this is resolved by the development of a scientific consensus. However, in this case a consensus is not appropriate, as SPF monitoring is not something done by a large number of independent groups. However, one alternative would be to identify an independent group whose testing is accepted by the manufacturers, the FDA and consumer groups.

What can a consumer do? The FDA and other organizations suggest the key is in application. Start with sunscreen with SPF of at least 30, or 60 if you are worried about the SPF rating on the label, apply liberally, at least a full shot-glass amount, and often, every 2-hours. Importantly, don’t buy a high SPF, e.g. SPF 100, and assume you don’t have to reapply as often, 2 hours is the longest you should go without reapplying.

What is the take home message? Because of the inherent variability in scientific research don’t assume all sunscreens are created equal.