The answer is in there

Palaeontologists glean the workings of ancient ecosystems from a single fossilized leaf.  Archaeologists derive whole civilizations from a fragment of a tooth and a few charred animal bones. Astronomers track down our planetary cousins from the periodic twinkle of a distant star. Physicists spend a lifetime studying particles that they can only imagine and infer.

Most of science, it seems, is searching for truths amongst a paucity of data. From the most meager of evidence, with a rigor that is designed to disprove rather than prove, they are able to show us profound insights into time, space and the nature of all things.

For those of us in IT we are often blessed with an abundance of data. Our purpose was once to collect data, aggregate it, present it and make informed decisions based upon it. That has been our task for the past 7 decades (happy 70th birthday to Colossus by the way). But now we are being asked to see the data in a very different way.

We walk though the world mostly oblivious to the detail. We don’t feel the sub-atomic particles that pass through us every day, we don’t recognize the colorful rock as fossil, we think the star is merely pretty as it twinkles. Unless our job is to analyze the fundamental structure of the world we only see the surface of it.

The same is true of data. We have to force ourselves to find meaning in the data beyond what the data documents. And now we have lots of data. Decades of data.

There is a wonderful chart of Facebook status-changes that shows most breakups happen around the holidays. Google has a great map of people checking on cold symptoms that presages flu outbreaks. Amazon has had a patent issued (appropriately it was issued on December 24th) that shows they can ship our orders before we place them because they know when we are browsing and when we are buying.

This year will see the emergence of modern shaman. These new mystics will discover hidden meaning in the random collisions of the 0’s and 1’s. They will uncover new truths, break uncrackable codes and answer questions not yet asked.

When the seemingly random language of the Lorenz code was cracked 70 years ago it took an extraordinary human insight and mighty technology breakthrough. The insight was provided by Tommy Flowers, Max Newman and the amazing team at Bletchley Park and the technology the world’s first electronic computer, Colossus.

Big data will deliver this year because of some very smart people with a unique way of looking at the world supported by some extraordinary technology that is not bounded by limits of power and time.

What this means for business is the chance to shift to new markets and new products ahead of the competition, ahead of the economy, ahead of the consumer. What this means for technologists is new problems, new solutions and new careers. We are witnessing a pivotal moment: it’s very exciting.