Big Data is not the next Big Thing – It is Here and Now!

Big Data originally referred to data sets so large and heterogeneous that making any use of them was a technological challenge of colossal proportions. Big Data is now a term used in business and marketing to refer to the untapped value of information lying in servers of governments and companies of all sorts.

Humans and machines create 2.5 Exabytes (2.5 x 1018 bytes) of data every day. Individuals create around 70% of that information. Your mobile phone and your web browsers are generating loads of data. Your debit card too. Only Facebook stores and makes available 100 Petabytes (100 x 1015 bytes) of data online.

Graph showing how a single tweet propagates on Twitter networks as retweets. Source: www.paradigmadigital.comSensors employed in power grids, medical, meteorological, biological and genomics research and military surveillance are creating Petabytes of data that need to be stored and processed. This is Big Data. It will grow 50x in the next 10 years.

The 3 Vs: Volume, Velocity and Variety

There is more to it than just size. Big Data are data sets whose value depends on two other dimensions in addition to volume: variety and velocity. Information is generated as text, media, log files, transaction records, etc, in unstructured data sets. Their value is time-sensitive, often real-time. The 3 V's explain the obsession of many CTOs and IT managers for computation and speed.

Harnessing the power of Big Data requires storing information in a scalable way, tools to process it and technologies for querying in non-relational database systems, analysing, modelling and visualizing it.

IT being courted by Business

The technologies that turned Big Data into a business are still growing in popularity. Ever more powerful infrastructure (servers, storage, networks), data management (NOSQL), analytics and discovery (R software) are still frantically evolving. Queries with terms like Hadoop, MapReduce and Cassandra are growing in web searches as their adoption spreads.

There is however a new trend rocketing up; the searches “Big Data” are booming in frequency in the last 12 months. They grow faster than the popularity of the technologies involved in it. Business and marketing are discovering the new buzzword. Big Data is overflowing the IT departments and spreading over business and marketing.

What is this hype all about?

Data science is the new black - Tim O’Reilly, March 2012.

Imagine business intelligence on steroids. Some sources estimate the size of the market between $16.9 billion (IDC) to $32.1 billion (Wikibon) in 2015 from $3.2 billion in 2010.

You can make more money... or save it. Governments can use Big Data to improve the collection of tax revenues. Companies can use Big Data to predict customer lifetime, decide how, where and exactly when to spend the marketing budget, manage stocks, hire staff, launch new products, hedge risks or finance investments. New applications and uses are found every day.

It’s not the data, it’s what you do with them

More data beats better algorithms, but better questions beat more data, Gregory Piatetsky, KDnuggets, October 2012

Big Data is also a force of change of organizations. Information silos need to open up to sharing environments. This is for many a big cultural leap into a whole new world. It is highly likely that your company or government think of "their" data as their best kept secret. For them, the value data reside in the balance in some limbo between intangible assets, intellectual property and goodwill.

Businesses which depend on data-driven decisions to survive and compete should appoint a Chief Data Officer (CDO). Reporting to the CEO, the CDO should define identify new business opportunities pertaining to data and optimizing revenue generation -or contention of OPEX- through data.

Big Data is not for everybody

Cloud computing attracts big investments and justly so. Storage and sharing fuels the gold rush to provide ever more processing muscle and memory. Big money loves tangible investments. Building huge data centers with massive number crunchers is one of those building blocks that everyone understands. Software is also popular among investors. Couchbase and 10gen are among the hottest start-ups ranked as the Next Big Thing.

Despite the interest in infrastructure and software, the limiting resource of Big Data is talent. Practically all skills come short in Big Data but Analysts are the one bottleneck that is slowing down the entire industry. The hottest job of the new industry is Data Analyst. The profile of the job is of a technically competent quantitative analyst with a natural curiosity and ability to ask the right questions to data sets of all skins and origins.

Paradigma and Big Data

Paradigma Tecnológico is helping to promote Big Data Spain 2012, a conference offering an introductory review of the technologies involved in cloud computing, High Performance Computing (HPC) and Big Data.

The organizers of the conference are an informal group of universities, communities of developers and companies who think that there is a demand for an international conference for developers hosted in Spain covering all areas of Big Data. The audience will be packed with senior developers, IT managers and technology strategists. Join us in Madrid on November 16th 2012 and meet top speakers from O’Reilly Media, DataStax, Hortonworks, Google, 10gen (MongoDB), GigaSpaces, Bull and many others. In the meantime, subscribe to our newsletter and check the official hashtag of the conference, #BDSpain