Colossal data


The answer is in there

Palaeontologists glean the workings of ancient ecosystems from a single fossilized leaf.  Archaeologists derive whole civilizations from a fragment of a tooth and a few charred animal bones. Astronomers track down our planetary cousins from the periodic twinkle of a distant star. Physicists spend a lifetime studying particles that they can only imagine and infer.

Most of science, it seems, is searching for truths amongst a paucity of data. From the most meager of evidence, with a rigor that is designed to disprove rather than prove, they are able to show us profound insights into time, space and the nature of all things.

For those of us in IT we are often blessed with an abundance of data. Our purpose was once to collect data, aggregate it, present it and make informed decisions based upon it. That has been our task for the past 7 decades (happy 70th birthday to Colossus by the way). But now we are being asked to see the data in a very different way.

We walk though the world mostly oblivious to the detail. We don’t feel the sub-atomic particles that pass through us every day, we don’t recognize the colorful rock as fossil, we think the star is merely pretty as it twinkles. Unless our job is to analyze the fundamental structure of the world we only see the surface of it.

The same is true of data. We have to force ourselves to find meaning in the data beyond what the data documents. And now we have lots of data. Decades of data.

There is a wonderful chart of Facebook status-changes that shows most breakups happen around the holidays. Google has a great map of people checking on cold symptoms that presages flu outbreaks. Amazon has had a patent issued (appropriately it was issued on December 24th) that shows they can ship our orders before we place them because they know when we are browsing and when we are buying.

This year will see the emergence of modern shaman. These new mystics will discover hidden meaning in the random collisions of the 0’s and 1’s. They will uncover new truths, break uncrackable codes and answer questions not yet asked.

When the seemingly random language of the Lorenz code was cracked 70 years ago it took an extraordinary human insight and mighty technology breakthrough. The insight was provided by Tommy Flowers, Max Newman and the amazing team at Bletchley Park and the technology the world’s first electronic computer, Colossus.

Big data will deliver this year because of some very smart people with a unique way of looking at the world supported by some extraordinary technology that is not bounded by limits of power and time.

What this means for business is the chance to shift to new markets and new products ahead of the competition, ahead of the economy, ahead of the consumer. What this means for technologists is new problems, new solutions and new careers. We are witnessing a pivotal moment: it’s very exciting.

About Kevin

In the past year Kevin has spoken at 20 conferences and seminars on a range of leading IT topics, including methodologies, business analysis, quality assurance techniques, governance, open source issues, tool interoperability, release management, DevOps, Agile, ITIL, from the mainframe to distributed platforms to the web, mobile, wearable and embedded systems. He is a much sought after speaker, recognized around the world for his provocative and entertaining style. Kevin is a 40 year industry veteran, holder of three technology patents and today is VP of Worldwide Marketing and Chief Evangelist at leading Application Development and Deployment vendor Serena Software. In the past decade he has been crossing the globe and has met with over 4,000 people. At Serena he works closely with industry analysts, the press, customers, partners and employees to exchange ideas about industry direction and business issues. He was born and educated in the UK and lives and works in the Bay Area, California.
This entry was posted in Business and Technology and tagged , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s