Before Hadoop arrives into the picture
there are several database techniques and server mechanism handled to solve the
problems like existing RDBMS, grid computing and few.
A typical RDBMS or relational database
management system is the master of real-time queries of structured data, which
makes it ideal for real-time online transaction processing or OLTP. But the businesses of all stripes is rely on
this kind of functionality to transact an important business. RDBMS has
supported by the biggest companies of the software industry and its used by the
every mid-to large sized IT organization in the world. But analyzing unstructured
data in RDBMS is like mix oil into water, also when processing the huge
unstructured data is hard to be processed that is where Hadoop comes in to the
picture. Hadoop is a purpose-built to handle enormous volumes of unstructured
data. Where RDBMS are usually run on costly proprietary servers whereas Hadoop
is running on commodity hardware servers and Hadoop splits all the data queries
into various nodes, making it relatively fault-tolerant.
What is RDBMS in depth?
RDBMS is relational database management
system. Database management system (DBMS) stores data in the form of tables,
which comprises of columns and rows. The structured query language (SQL) will
be used to extract necessary data stored in these tables. The RDBMS which
stores the relationships between these tables in different forms such as one
column entries of a table will serve as a reference for another table. These
column values are known as primary keys and foreign keys. These keys will be
used to reference the other tables so that the appropriate data can be related
and be retrieved by joining these different tables using SQL queries as needed.
The tables and the relationships can be manipulated by joining appropriate
tables through SQL queries.
The most important attribute of a relational database system is that a single
database system generally has several tables and relationships between these
tables so that the information is classified into tables of independent
entities. They are also stored independently in a normalized or simplified way
and a relationship is maintained within these tables using primary/foreign key
constraints. This is different from a flat file or data structure. The data on
a database could be stored in a single data file or multiple data files. The
data file size will grow or the new data files will be added as the new records
are added and the size of the database is increased. These all files are commonly
shared by the database server. In high availability systems, these data files
are shared so that each node will have access to the same data file. Generally,
all popular database systems are relational database management systems. In
order to give some quick and easy navigation to related data, some logical
views are created from the actual tables. There will be a physical existence
for every table in the database whereas a view is a virtual table, which does
not exist physically rather a logical creation from the existing physical
table. IBM DB2, Microsoft SQL Server, Sybase, Oracle, MySQL and PostgreSQL are
some examples for RDBMS.
However, RDBMS only work with better
when an entity relationship model (ER model) is defined perfectly and
therefore, the database schema or structure can grow and unmanaged
otherwise and RDBMS works well with structured data.
How Hadoop better than RDBMS?
RDBMS database technology proven,
consistent, matured and highly supported by best companies. This works better only
when the data is definitions such as data types, relationships among the data,
constraints and etc. so, this is more appropriate for real time processing. As
already mentioned, RDBMS only work with better only when ER model is defined
perfectly, so the database schema or structure can grow and unmanaged
otherwise. Especially, where the data size is too large for complex processing
also where not easy to define the relationships between the data, then it
becomes very difficult to save the extracted information in an RDBMS with some coherent
relationship but when it comes to large, unstructured data Hadoop would be the
right choice.
Hadoop framework fits any kind of data
such as structured, semi structured and unstructured data also this supports
variety of data formats such as XML, JSON and text based flat file formats as well.
For example, when we start analyze the Internet data published by various
websites. Out of those existing thousands of millions of websites, each website
is having different types of contents and the relationships between them will
not unique. In those cases, Hadoop is a right choice to analyze them. Since the
exposure of those capabilities will increase, Today the companies are choosing
Hadoop not only for help handling the historically grown huge amount of data,
but also using Hadoop for meeting high performance needs for the applications.
For example, analyzing a monthly energy usage of a customer by comparing
between previous months, between their neighbors or even between customers on
their friends. This may bring some awareness, but running such complex
comparison by analyzing large set of data takes many hours of processing time, but
introduction of Hadoop help improving the computing performances from 10 times
to 100 times or even more.
In late 1999, Ebay scaled out across a
cluster by Logically partitioning their databases for user data, item, data,
purchase data. However, this SQL option did not scale enough for ebay, they
have now
moved their items catalogue to HBase. Facebook
paired complex sharding and caching to MySQL. Facebook split its MySQL database
into ~4,000 shards and 9,000 instances of memcached in order to handle the
site’s massive data volume. This became very difficult to maintain and scale
and now Facebook has moved their messaging to hbase. There are several videos
on the internet about this. Will discuss more about HBase in coming chapter.
MapReduce
RDBMS
|
1.
Size of data
Petabytes
Gigabytes
|
2.
Integrity of data
Low
High
|
3.
Data schema
Dynamic
Static
|
4.
Access method
Interactive and Batch
Batch
|
5.
Scaling
Linear
Nonlinear
|
6.
Data structure
Unstructured
Structured
|
7.
Normalization of data
Not Required
Required
|
Grid
computing:
Most people are familiar with the concept of a power grid,
where various sources of electricity are linked together to supply power to a
certain geographical location. The concept of grid computing is very similar,
where computers are linked together in a grid to provide a greater
computational resource.
Grid computing is an arrangement of computers, connected by
a network, where unused processing power on all the machines is harnessed to
complete tasks more efficiently. Tasks are distributed amongst the machines,
and the results are collected to form a conclusion. The advantage of grid
computing is that it reduces the time taken to complete tasks, without
increasing costs.
Computers on a grid are not necessarily in the same
geographical location, and can be spread out over multiple countries and
organizations, or even belong to individuals.
What
is Grid Computing in depth?
These days’ computers have great processing power, even on
the lowliest of machines. During an average working day, most of this
computational potential lies unutilized by a user. So In a grid computing
environment, computers are linked together, so that a task on one machine could
utilize the unused processing power on another machine to execute their tasks
faster. This arrangement minimizes wasted resources and increases efficiency
considerably, as a task split over multiple machines takes significantly less
time to compute.
Serial
Computing vs. Parallel Computing
Each processor uses a queue system to execute the tasks. Many
algorithms are implemented in the system, but, in the essence, there is a task
queue to perform the tasks. Basically a single processor can handle only one
task at a time and, as a result, since then the programming of software has grown
up to execute each task sequentially. For example, if task ABC needs to be
executed before task XYZ, the programmer has to ensure that order is maintained
in the program, this is known as serial computing.
Even though sequence techniques are playing an important
role in computing, there are certain tasks that are mutually exclusive, that
can be performed simultaneously. For example, if two tasks can be performed
independently of each other, and assume they can those be assigned to two
different machines. Now, each of those machine will perform the task
independently which was assigned and generating the results substantially faster
than the one machine was performing both tasks one after the other. This
processing is known as parallel computing.
Hadoop is also inspired by this
parallel programming concept and is a mechanism for processing large
amounts of data from various sources. For example, web clickstream data, social
network logs, etc. This data is so large and it must be distributed across multiple
machines in the cluster in order to be processed in a reasonable time limit.
This distribution among the cluster implies the parallel computing with the
different dataset those are available on the different machines in the cluster.
So dataset is not depending on each other while executing since each dataset is
distributed across the machines in the cluster and each machine will start
processing on the dataset in parallel. Here, mapReduce is an abstraction that
allows engineers to perform simple computations while hiding the details of
parallelization, data distribution, load balancing and fault tolerance in
nature.
