In this essay, I will be focusing on the topics of Big Data, NoSQL, the relationship Big Data has with NoSQL and the challenges they face. NoSQL is a non-relational database that is often used for storing Big Data and this new type of database is becoming more and more popular among web companies due to its versatility. The term Big Data is linked to the handling, storing and processing of large volumes of data that can be stored in a database.
From my understanding, the term Big Data is commonly thought of on the same wavelength as the 5 V’s. The 5 V’s are shown in the diagram above and help us manage our continuously growing data. Volume refers to the mass amounts of data we are producing every day. Velocity refers to our need for information to flow near as quick as real-time as possible. Variety refers to how data can be obtained from different sources of devices. Veracity refers to the quality of this data.
Value refers to the ability for this piece of data to turn into something more such as meaningful information. There is a great need for society today to store meaningful data efficiently. Traditional databases such as relational database management systems (RDBMS) are finding it a difficult task to process mass amounts of large structured and unstructured data. From my point of view, it seems as though the volume of data is simply too big, moves too fast and exceeds current processing capacity.
The term Big Data holds a great deal of challenging problems behind it. These statistics I got online gives us a bigger picture into the problems with Big Data we are being faced with: Weather channels receiving 18,055,555 forecast requests every minute, Netflix users streaming 97,222 hours of video every minute and skype users can make 176,220 calls every minute These sets of records are easily too large for us to even imagine dealing with today but still we need a way to manage it efficiently.
And to add to the severity of these staggering numbers, these numbers are only ever going to increase from here on into the future with the knowledge that there will be a rise in the number of people using the internet and the increase in the production of electronic devices. This is where NoSQL enters the big picture to try to deal with the problem encompassing managing Big Data.
It is clear to me that the NoSQL database was developed to the demands presented in building modern-day applications. A developers’ challenge today is to find a solution to handle, process and store structured, unstructured and polymorphic data. NoSQL is an approach to databases that represents a shift away from RDBMS databases which relied on tables, columns, rows, and schemas. The demand for applications to now be delivered as services that must always be accessible anytime and deliver across the globe to millions of users is high. NoSQL is used for Big Data and real-time applications.
In Contrast to RDBMS, I found that NoSQL databases do not rely on these structures and use more flexible data models. NoSQL’s ability to do most things RDBMS has failed to do in terms of performance, scalability, and flexibility-needs has pushed many enterprises to adopt it. NoSQL databases are particularly useful for storing rapidly-growing data. The same cannot be said for the RDBMS databases. The types of data NoSQL is capable of handling include video, images, chat, messaging and log data. Some of the top services NoSQL provides include fraud detection in financial service organizations and digital communications in the enterprise world.
Throughout my research on the topics of Big Data and how the NoSQL database was cleverly designed for today, I found out an obvious correlation between Big Data and NoSQL, the correlation being their relationship – the fact that Big Data is very much dependent on NoSQL databases right now. NoSQL is essential for flexible Big Data applications. Flexibility is a major factor to take into consideration when dealing with real-time sets of data. NoSQL just so happens to meet this standard, providing the user required flexibility as it is non-relational and document-oriented. I found that with this model, duplication can become an issue if you’re not careful, but the flexibility factor isn’t compromised because of its unlimited storage.
Through doing some research online, I discovered that NoSQL databases don’t have the reliability functions in which RDBMS databases hold such as ACID. This is a set of properties that guarantee database transactions are processed reliably. An ACID-compliant database ensures that the data in the database remains accurate and consistent despite any such failures.
To combat the problem of other databases that happen to not hold this property, developers will have to implement code that will make their systems more complex. NoSQL is not compatible with SQL meaning we need a manual query language making things slower and more complex. A factor to take into consideration is that NoSQL is quite new compared to RDBMS so it is far less stable and may hold a lot less functionality. Big Data also faces some drawbacks. These drawbacks circle big data analysis violating principles of privacy and speedy updates in big data mismatching real figures. Big data analysis is not useful in the short run but if analyzed for a duration of time, we can reap its benefits.
