Our world is ruled by numbers; commerce, time, travel, health, clothing, food, sports, are all, by way of example, defined, gauged and administered by numbers.  You could say we are slaves to numbers.  The old adage of white lies, bold lies and statistics might today be changed to ‘numbers, statistics and Big Data’!

Big Data is a much-used term with varied interpretations.  Essentially, it’s simply used to describe data sets so large, that traditional processing applications are inadequate. The data is typically incomplete, loosely structured and not readily accessible.

To help manage all those numbers there’s ‘Hadoop’, an open source software platform that provides enormous storage and processing capabilities which allows the user to handle virtually limitless tasks simultaneously.  How does it work? Well for the uninitiated (and the faint-of-heart), trying to comprehend it in one article would leave your brain as exhausted as an Amish horse, so we won’t try   Instead, let’s focus on what really matters, the fact that Big Data improves our lives.

Big Data – by the people, for the people!

At the end of the day, it isn’t about grasping the intricacies of Hadoop or SAP HANA (another user framework) for that matter, what makes Big Data so exciting, is that it’s benefits are often tangibly demonstrated.  Sounds counterintuitive but it’s true.  For example, did you know that Big Data is used to; design, maintain and position wind turbines (which generate clean energy), to help the United Nations promote health and wellbeing in third world countries and, it gives Woolworths, Coles et al, the ‘intel’ they need to better price and position their product lines.  There’s a good reason there’s chocolates and drinks at the checkout!

Still dubious?  Don’t be.  Even though it sounds like an oxymoron to say it, Big Data also brings people together. Just ask Facebook, Instagram and LinkedIn, heavy collectors and users of Big Data, and, whilst reading about people running late for meetings because they feel the need to share the fact they can’t find their favourite shoes, or that the dog has left a landmine behind the lounge might not seem like anything too significant, in the grand scheme of all things ‘Big Data’, Facebook does function as a the world’s most significant social media barometer.  Want to know what’s trending, popular, or out-of-favour?  Look at Facebook.

Numbers are important, but they don’t exist without people

We love big data and why wouldn’t we?  We’re an IT recruitment company, so we provide the data scientists and specialists that understand the tools and procedures used to collect, store, manage and manipulate terabytes, exabytes and petabytes, to the people who need and use the data.  Health, education, retail, tourism, urban planning, environment, energy and the financial sector all benefit from the availability and use of, Big Data.

While the benefits of Big Data are undeniable, it is still something of a ‘new frontier’ and there are many technical challenges yet to be addressed.  The logistical trials of capturing and managing such vast quantities of data present an ongoing and major challenge in their own right, let alone those presented by the semantic interpretation of same.  There are recognized phases in the collection, annotation, aggregation, analysis and interpretation of Big Data, and of course, there’s a timeline to all of it.  Do it too slowly, and by the time you finish, well, its time to start again.

In God we trust. All others must bring data – W. Edwards Deming

Big Data will continue to be big news for a long time to come and privacy issues aside (both a technical and sociological issue), the advantages conveyed are best demonstrated by…you guessed it…numbers.  In 2013 it was estimated that PriceWaterhouseCoopers generated $312 million from Big Data revenues, whilst IBM turned over circa $1.37 billion; seriously big numbers and a clear indicator of things to come (source: Wikibon).

Numbers are vital.  The ancient Egyptians recognized that, but they also knew, as we do, that it takes specialists to collect, manage and interpret data…especially Big Data.