Data Data Everywhere

How NoSQL complements traditional relational databases

I like working with traditional relational database management systems (RDBMS).  There is something elegant about good table design and the physical implementation of a database.  I like running SQL statements and looking at data along with figuring out how to get specific result sets.  I’m not sure why this pleases me, but it has something to do with order in a chaotic world.  Traditional relational databases have a long history, and I’ve been working with them since the early 90s.  My first ten years as a consultant I built portal applications that were focused on reporting and analytics with features like financial dashboards and reports delivered via secured personalized websites.  During those ten years, I worked with databases every day.

How We Got Here (And Where is Here?)

In 1974 IBM released “System R” based upon the work of EF Codd.  Codd worked at IBM and he proposed theories on data relationships and invented the relational model for data management.  Larry Ellison, inspired by relational theories, launched Oracle in 1979 – then known as Relational Software Inc.

Relational databases are mature and dependable: they support ACID compliant transactions, they have tables and keys and support relationships among tables, they support mini programs stored procedures, they support built-in table constraints and integrity rules, they have indexes for faster lookups of data, and they are modeled using normalization rules and sometimes denormalized for speeding up queries.  They also support a standard language – SQL – structured query language.

There is a high level of confidence with relational databases: they have many built-in functions, good logging, and good support.  BUT – they remain EXPENSIVE to own and operate.  Furthermore, relational databases are running into scale issues with performance thresholds where SLOW response rates are considered disastrous for some companies and their business.  A slow response rate contributes to user abandonment of applications.  The faster a page loads, the better engagement.  A general rule is that any response rate greater than 2 seconds for a web page is too slow.  Another issue facing traditional databases is the open ended nature of user generated data.  For example a Twitter update may contain movies, images, comments and hash tags depending on the user’s whim.  Relational databases are setup with predefined table structures where user input is controlled and guided.

The Organizational Challenges of RDBMS

Part of the expense with databases is self-inflicted by companies because of organization structures which separate operations, DBAs, and developers.  DBAs perform many tasks.  First and foremost they make sure databases are up and running.  They set up new machines: constantly go through upgrade planning, tune any clustering they might have, monitor performance and throttling, and try to keep up with their ongoing education.  Look at what it takes to become an Oracle certified DBA – it’s monumental.  DBAs spend a lot of time troubleshooting data issues for high priority transactional applications, “business critical” type of applications where bad data was somehow loaded from another system or entered mistakenly.

Frequently implementing small iterative changes, common for Agile projects, isn’t given priority when considering all the ongoing responsibilities a DBA confronts.  Part of the expense with traditional databases is the hidden costs of application deployment delays and failures from lack of DBA expertise along with inexperience of data modeling skills.

Introducing NoSQL

Or “not only SQL” as SQL-like languages are supported

NoSQL databases support the storage and retrieval of data modeled differently than traditional tabular relations used in relational databases.  They store key-value pairs, JSON or XML documents.  NoSQL has been around for a while but has gained in popularity because of accelerating increases in data volumes and slowing response times that Web companies such as Facebook encounter.  Other traditional businesses are also dealing with data overload and slowing response times with their current database systems.  Consider airlines and flight schedules, or Walmart and retail sales across various channels, or automobile companies and measurements on every component in your new car.  Digital measurements that companies track, and the corresponding volumes of data, are simply exploding.  Consider data produced by IoT “Internet of Things” or manufacturing sensors.  Also consider that most consumers are expecting information via their smartphone.  Today we’re producing gobs of data, but getting information out of that data and making it useful is the ultimate objective.  The value of providing information and gleaning insights makes the gobs of data valuable, accomplishing that objective is difficult.

NoSQL databases offer an alternative for managing vast amounts of data.  They are fast, flexible and relatively easy to start working with and for these reasons many companies are complementing their traditional database systems with a NoSQL cluster.  NoSQL databases have a cache layer and support large numbers of concurrent users.  They have built-in capability to scale out on commodity cheap servers.  Clustering is simplified and even dynamic with servers added and removed during variable load periods.  NoSQL reduces organizational obstacles, noted previously, because there are no enforced schema rules in the database.  More responsibility is shifted to the development team to form a logical and physical design for their application data while building the application itself.

Benefits

  • Cache – data is in memory
  • Highly responsive
  • Supports large numbers of concurrent users
  • Scale – up and out on commodity cheap servers. This is particularly hard with traditional databases, they scale up easily but not out.
  • Setup / deployment ease – easy to get started
  • Data replication – always available, no downtime. Data is distributed and replicated to every node.
  • Flexible – adapts to changing requirements and changing applications, there are no schema rules enforced by the database.  This alleviates the obstacles of working with disparate groups of people while iteratively changing schemas.

NoSQL presents a tremendous opportunity to rethink how we store and retrieve mountains of data quickly.

So, Is NoSQL Right for Your Needs?

My experience with analytics and traditional databases reinforces my perspective that they aren’t going away anytime soon.  And yet, my visibility into the dozens of enterprise implementations of NoSQL databases that have been done by Avalon Consulting, LLC (primarily on MarkLogic and Couchbase) for our clients and my own personal experience with NoSQL (Couchbase in particular) have motivated me to think broadly about how to enrich application response times with a complementing cost effective technology.  Especially when working with vast amounts of data and time sensitive applications.

NoSQL databases do not eliminate the necessity for cross department collaboration.  In fact, they encourage more cross discipline team dynamics.  It is true that NoSQL allows the development team to make changes rapidly to evolving applications and data models, but the necessity for good data modeling in the form of JSON document types or logical key naming conventions is still paramount as the age old mantra goes “garbage in, garbage out”.  Fortunately, NoSQL enables an easier course correction than traditional RDBMS because there are no schema rules enforced by the database.

New auto deployment technologies, test automation tools, and application data layer technologies like NoSQL are pushing the boundaries of traditional separation of work among System Admins, DBAs, QA, and Development. It’s becoming more and more apparent that cross departmental collaboration is essential while newer technologies require a broader range of skills for successful applications.  NoSQL provides an exciting complement to existing RDBMS with a speedy alternative to storing and accessing data, but it is not a replacement for all types of applications.  At Avalon, we’ve used NoSQL as a complement and sometimes a replacement to relational databases.  We can help jump-start your plans for leveraging NoSQL, visit our contact us page.

Will Thayer About Will Thayer

Will Thayer is a Principal Consultant Technologist at Avalon Consulting, LLC. Will has 18 years’ experience in planning, strategy, development, and training. His expertise includes web application development, management information systems, and system development life cycle practices. As an Adjunct Professor at the University of Denver, Will taught graduate and undergraduate students for 5 years. His research in Open EDI and XML EDI has appeared in books as well as trade periodicals. Will lives in Evergreen, Colorado where he enjoys skiing, hiking, biking, and camping in the Rocky Mountains with his wife Robyn and daughter Sophia.

Leave a Comment

*