Skip to main content

To Qualify as ‘Scientific,’ Evidence Has to Be Reproducible

We can never be sure if the interstellar object ‘Oumuamua, for example, was artificial—but we could be ready to answer that question for such a visitor in the future

Alien spaceships? We'll never know.

In various conversations with reporters leading to the publication of my new book, Extraterrestrial, I encountered difficulties getting across one essential aspect of the scientific process. Science relies on reproducibility of results. In order to believe a phenomenon, it must be possible to reproduce it as an outcome of similar circumstances. Stories about one-time miracles are the foundation of myths and could be believed by the public, but they do not stand up to the standards of science.

According to the biblical story of the binding of Isaac, Abraham heard the voice of God asking him to sacrifice his only son. Today, with a voice memo app on his cell phone, Abraham could have recorded the voice of God and convinced all of humanity to believe in the reality of his experience. But without a recording device, hearsay evidence is not scientifically sufficient.

Even when recording devices are used, one-time events like the “Wow!” or BLC1 radio signals, are of dubious scientific value because they cannot be distinguished from environmental noise or instrumental malfunction.


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


One-time astronomical events like the big bang, however, are of great value since they allow reproducible scientific studies of their consequences. For example, conditions in the early universe resulted in the brightness patterns of the cosmic microwave background that seeded the formation of large-scale structure in the universe. These outcomes can be studied repeatedly, in detail, and used in concert with theoretical calculations to determine the constituents of the cosmic mass budget.

The situation gets more complicated with eyewitness testimonies of one-shot events. Recently, the Pentagon was asked by lawmakers to disclose all it knows about Unidentified Flying Objects (UFOs) by mid-2021. But this focus on past eyewitness reports is misguided. It would be prudent to progress forward with our finest instruments, rather than examine past reports. Instead of declassifying documents that reflect decades-old technologies used by witnesses with no scientific expertise, it would be far better to deploy state-of-the-art recording devices, such as camera or audio sensors, at the sites where the reports came from, and search for unusual signals. A scientific expedition focused on reproducing old reports would be more valuable to unraveling the mysteries behind them. A conversation I had recently with podcast host Joe Rogan about my new book triggered a grassroots initiative to fund such experiments. 

My long-term collaborator and friend from Princeton University, Ed Turner, has noted that the number of cameras around the world increased by several orders of magnitude over the last couple of decades. They include security cameras, phone cameras, car cameras, drone cameras, dedicated astronomical observatories and arrays of cameras installed on satellites and aircraft. This led to a vast increase in the numbers of images, often videos, of car and aircraft crashes, meteors, natural disasters, crimes and so on. 

A vivid example is provided by the many clear video and sound recordings of the superbolide Chelyabinsk meteor event over Russia in 2013, obtained by different types of cameras from different directions. At the same time, there had been no noticeable increase in the rate at which UFOs are recorded or in the quality of the recordings. “We had fuzzy, intriguing but not compelling, images and occasional videos way back in the 1960s at a time when no one had ever photographed a commercial airplane crash,” points out Ed, “but we don't have much better evidence now.”

Scientists view reproducibility as a virtue, but others in the public consider it as a limitation. Ed remembers from his early career days that the famous physicist Richard Feynman liked to remark on the limitations of science as a tool for understanding many of our experiences of the world, when talking to graduate students like Ed over beer on Friday afternoons at the Caltech faculty club bar. Perhaps reproducibility is one of them.

As discussed in my book, our inability to continue collecting data on `Oumuamua, which is now a million times fainter than it was during its close passage to the sun, limits our ability to make further progress in deciphering its nature. Our best path forward is therefore to use survey telescopes like Pan-STARRS or the forthcoming Vera C. Rubin Observatory to identify other weird interstellar objects as they approach us, and then deploy a small spacecraft equipped with camera along their predictable path and take a close-up photograph of them. As the saying goes, “a picture is worth a thousand words.” In this case, it would allow us to distinguish between a rock and a technological artifact. 

The experience would be similar to walking on the beach and finding mostly seashells and rocks that are naturally produced, but every now and then stumbling over a plastic bottle that implies the existence of a civilization out there. Whereas the traditional search for radio signals is equivalent to speaking on the phone with a living counterpart, finding a “message in a bottle” resembles receiving a letter in the mail, which could occur after the sender had died. It offers a much more powerful peek into the past existence of dead civilizations, and provides the space analog of an archaeological search for relics left behind by cultures that are not around anymore on Earth. 

Ultimately, the assembly of reproducible results by instruments is essential for separating the subjective impressions of humans from objective data gathering. This does not imply that Abraham did not hear the voice of God. Instead, it highlights the fact that he would have needed a recording device to make the Biblical report convincing beyond a reasonable scientific doubt.