Planning for Big Data Success

BY Prashant Nair · April 27, 2018 · Leave a Comment


Most organizations are either starting or have already started on their Big Data journey.  As most other technology hypes, Big Data has also followed the hype cycle and there was a drop in interest in past years after the initial frenzy.  What we are seeing now is the second round of influx in interest around Big Data after the initial peak some years back. With increasing maturity in Big Data products and business use cases, we are inclined to believe that Big Data is here to stay and will prove to be a differentiator by providing the competitive advantage needed in this age.

With this move, we should see more successful adoptions of Big Data technologies. There is a definite opportunity to cash in early on this technology and get the early bird advantage. But, let’s not get carried away! The industry wide success rate for Big Data project still remains between 22-27% (In 2017, Gartner had got this figure at 40% which most other studies found this figure optimistic and peg it at 15%). So, even if you start on your Big Data journey early, there is no guarantee on the returns UNLESS there is a way to increase the probability of success.

This blog post lists some key takeaways from our experience with Big Data initiatives across several industry segments. These insights should assist in better planning and increased control on your Big Data initiative.

Think Big, Act Small

An organization’s Big Data roadmap is going to be one with continuous enhancements and improvements; it is not a one-time effort. There are numerous possibilities to leverage Big Data technologies across the organization and chasing all of them is not feasible. Today, organizations face the challenge of disruptions and major changes in their respective industries more frequently than ever before. This challenge poses as a major threat to long term planning, but, can be offset by a fluidic approach for adapting to changing business circumstances. Acting on short term goals aligned to the long term vision enables organizations to retrospect regularly, prioritize their business needs better and be fluidic.

Key Takeaway 

Treat a Big Data roadmap as a multi-phase initiative with a long-term vision realized in iterations by short-term projects. The short-term projects delivering short-term goals aligned to the long-term goals.  

Data is the new oil?

Oil Companies across the world get more or less similar results and they get the same by-products by processing crude oil. There are established industry wide standards around processes to extract crude oil and further processing it. Unfortunately, there are no industry wide standards or use cases around processing of data yet. Organizations find varied levels of success in Big Data projects even when they start with the same business use cases, same set of technologies and similar data sets. The various approaches, numerous technologies, umpteen business use-cases around Big Data further complicate this issue. It does help looking at industry trends and looking at what the competition is doing, but, these should not be the sole inputs to planning. Every business is different and the problems that the competition is trying to solve might not be the ones high up on your priority list.

Key takeaway  

Don’t hurry, plan well. The goal is to not to provide business teams with a working solution but rather to provide a solution that works for your business.

Pointers for better planning of Big Data projects

  • Clearly identify and list the business objectives and use cases involved in the project
  • Clearly define success criteria and key outcomes from the project
  • Document the estimated metrics around the success criteria which could be used as the benchmark measure for success of the project

Stakeholder buy in

Educating stakeholders about the benefits of Big Data is a must as some benefits from a Big Data implementation are not evident as others and some benefits may take time to showcase their true potential. Getting a buy-in from key stakeholders, right from the start, is critical. Usually, the Big Data foundation project is capital intensive and the rewards of the foundation project are not fully realized immediately but reaped manifold over time as future Big Data initiatives build on top of the foundation.

We no longer live in an age where the ‘Bigger companies eat up the smaller ones’, rather in today’s world the ‘faster’ companies eat up the ‘slower’ ones. Faster actionable insights across the organization are the need of the hour and Big Data implementations hit the sweet spot here as they enable faster discovery of insights than ever before. Along with faster actionable insights, Big Data enables organizations in uncovering hidden business insights which would have remained hidden without the capability to process huge chunks of data.

Key takeaway

Educate the stakeholders about the long term roadmap and the short term project goals. If possible, quantify the potential benefits over time in monetary terms for each project as part of planning.

Technology – means to an end

Another classic debate is which Big Data technologies to choose when there are so many. Technology shouldn’t be the biggest concern for initial planning which focuses on identifying the problems to be solved rather than ‘How’ to solve the problem.  Once the business objectives are identified, you can focus on choosing the technologies based on the following criteria

  • Cost
  • Ease and effectiveness of integration with legacy source systems
  • Scalability
  • Skill gap
  • Security

POCs can provide a feasibility check on the technology as well as reaffirm the business objectives of the project

Key takeaway

Cloud based solutions offer more flexibility, scalability and are less capital intensive. It is beneficial to go with established technologies than the latest ‘next big thing’ as established technologies and products usually have a better support system and there is a usually larger pool of people having the skills related to established technologies.

Data quality

Consider the Big Data solution as an engine which runs on the fuel – Data. Higher the quality of the fuel, better the performance.  When you are trying to uncover insights from data or planning/ forecasting based on data, bad data can lead you in the wrong direction. Data scientists, today, spend more than 60% of their time in cleaning the data to make it usable. A good solution requires good data to produce good results.

Key takeaway

Data profiling for Quality Assessment as part of planning allows factoring in the additional time required to cleanse the data before it can be processed to provide correct insights or even relooking at the overall approach based on the credibility of data.

The takeaways listed above are based on the major root causes for failure of Big Data projects that we have seen over the years. Even though this is not a comprehensive list, these are the most widely seen issues and should help in increasing the success rate for your Big Data initiative.


Even though the current success rate of Big Data projects is low, Organizations cannot afford to ignore Big Data for long. It is inevitable that this success rate will improve in the near future and organizations will start seeing returns and realize the true potential of their Big Data programs. For a Big Data solution to be of competitive advantage, Organizations will have to build solutions centered on their business needs rather than waiting to copy what the competition is doing. Given that the Big Data roadmap is going to be a multi-phased effort, it is recommended to start building the Big Data foundation sooner than later.

About the Author

Prashant is a Project Manager at Mastech Infotrellis with 7+ years of experience. His interest in Data Visualization and Business Intelligence lead him to believe in the power of Big Data and the umpteen possibilities for Organizations to use Big Data solutions for making more informed and data driven decisions.


Big Data, Big Data Challenges, Big Data Project Planning, Big Data Success, Big Data technologies

Leave a Reply

Your email address will not be published. Required fields are marked *