Chaos and Confusion

I have worked as an IT consultant for nearly 20 years and at the beginning of my career a mentor said to me “Chaos and Confusion is the environment we work in”.  Our job is to bring sane practical perspectives to address the hype surrounding new IT tools and projects.  This practical perspective sometimes conflicts with marketers or sales teams.

Recently a colleague shared a WSJ report about Hadoop.  The report spoke about the hype surrounding Big Data and tools like Hadoop which process massive amounts of data.  The article stated that 60% of Hadoop projects fail, and many other projects never deliver valuable results or insights.  Every couple of years we witness this type of hype cycle.  It reminds me of the ‘90s and the dotcom bust or AI or self-driving cars.  We have been here before.  A hyped new technology and the opportunity to build new Information Systems that companies leverage for business advantages.  When new technology is introduced and media stories promote the promises these tools present, we all seem to gravitate to the story and want to get involved.  Of course there are risks but we tell ourselves that the benefits far outweigh them.  There seems to be a strong sense of urgency to move first and exploit the technology quickly, at the expense of diligent IT planning.

In the mid-1990s, The Standish Group conducted a study detailing the current state of software application development. The study compared the development process of software and that of building bridges. This study surveyed hundreds of CIO’s from multiple industries and found the number one reason for software application failure is lack of clear realistic requirements.  The study found that a whopping one out of every three development projects will fail and never be used.  Another 30% of projects will be over budget and over schedule.  The Standish Group conducts a follow-up on this study every year and the numbers are getting better.  Software projects fail less often today;  an estimated one out of every five projects fail.  The number one reason for the improvement is advancements in shared development methodologies (such as Agile), improved project management skills, as well as solid executive sponsorship.

Business environments change fast and software application requirements change.  At Avalon Consulting, LLC, we manage the development process, ask a lot questions, involve everyone, and consistently check for authentication.  This leads toward support, excitement, and ultimately success. Software development does not entirely involve the most difficult technologies; most importantly, it involves people and it involves getting the right information to the right people.

Before listening to the naysayers or promoters of Hadoop, conduct your own analysis.  Do you have a Big Data opportunity aligned with your company’s strategic goals?  Technology is not the issue with these failures.  Where failure occurs it is rooted in the development environment, project management, and the business reasons for initiating the project.  Here are 5 important tips before starting a project:

Filter half-baked ideas

  • What are the reasons for the project?  Which specific business strategy is it linked to?  What are the specific objectives and benefits?  What are the measures of success?  How realistic is the delivery schedule?  Under what conditions is the project abandoned?

Performing due diligence

  • Assess the technical and business complexity of the project.  Identify stakeholders and champions.  Identify any opponents and address their concerns.

Ensure solid executive sponsorship

  • Empower project managers and manage the scope.

Develop skilled project managers

  • Resolve conflicts, approve plans, remove obstacles, manage expectations, and negotiate.

Monitor the vital signs

  • Is the project on schedule and budget?  Are milestones slipping?

Humanity created more data in the last two years than seen by anyone since the beginning of time in the form of social media, sensors, websites, and smartphones.  Hadoop is a tool that helps process and ultimately supports erudition from that data.  The need exists for Hadoop and every day that necessity increases.  For these reasons, I think the software will obviously mature and become better technically but the real challenge is for us to do a better job of developing solutions on the platform.

Today, Hadoop is ready for the big time and many companies already have proven its viability.  The caveat is to avoid taking on an IT project for the sake of the technology.  The prerequisite of a clear business reason for starting the project still exists.  It’s not about the technology, Technology is a tool for achieving a goal, solve a problem, satisfy a need.

Will Thayer About Will Thayer

Will Thayer is a Principal Consultant Technologist at Avalon Consulting, LLC. Will has more than 20 years’ experience in planning, strategy, development, and training. His expertise includes web application development, big data analytics, NoSQL, management information systems, and system development life cycle practices. As an Adjunct Professor at the University of Denver, Will taught graduate and undergraduate students for 5 years. His research in Open EDI and XML EDI has appeared in books as well as trade periodicals. Will lives in Evergreen, Colorado where he enjoys skiing, hiking, biking, and camping in the Rocky Mountains with his wife and children.

Leave a Comment