The Effect Of Hadoop On The Retail Banking Data Processing

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

This paper proposes backup task mechanism to improve the straggler tasks which are the final set of MapReduce tasks that take unusually longer time to complete. The simplified programming model proposed in this paper opened up the parallel computation field to general purpose programmers. This paper served as the foundation for the open source distributing computing software – Hadoop as well as tackles various common error scenarios that are encountered in a compute cluster and provides fault tolerance solution on a framework

MapReduce Parallel programming model if we ever get a chance. In Hadoop, there are two nodes in the cluster when using the algorithm, Master node and Slave node. Master node runs Namenode, Datanode, Jobtracker and Task tracker processes. Slave node runs the Datanode and Task tracker processes. Namenode manages partitioning of input dataset into blocks and on which node it has to store. Lastly, there are two core components of Hadoop: HDFS layer and MapReduce layer. The MapReduce layer read from and write into HDFS storage and processes data in parallel.

The Hadoop employs MapReduce paradigm of computing which targets batch-job processing. It does not directly support the real time query execution i.e OLTP. Hadoop can be integrated with Apache Hive that supports HiveQL query language which supports query firing, but still not provide OLTP tasks (such as updates and deletion at row level) and has late response time (in minutes) due to absence of pipeline

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

For analysis, advanced statistical tools are used and the experimenter can draw necessary conclusions and inferences. In business sectors, they have a huge amount of scattered data in terms of profits, loss, demand, supply, sales and production. The industrial, insurance, agricultural, banking, information technology, food industry, telecommunication, retail, utilities, travel, pharmacy and many more have challenges to manage their data. As the coin always has two sides, there are both advantages and a few disadvantages of data analysis. The key advantages of data analysis are- The organizations can immediately come across errors, the service provided after optimizing the system using data analysis reduces the chances of failure, saves time and leads to advancement. It is also used to compare strategies between two companies so as to reduce the prices and gaining attention of target customers, ultimately leading to maximization of profit and minimization of cost(as done in Game Theory). However, big data analysis sometimes becomes more tedious and disadvantageous because it uses software Hadoop which requires special provisions in the computers. For now use of Hadoop for real-time analysis is not available. The manner in which the data is collected and the decision making view can vary from one person to another. Here, the quality of data gets affected and leaves the data insufficient or inefficient. In order to tackle this problem, the researcher must be professional, well experienced and should have deep knowledge about the characteristic under study. Also, we need to update data from time to time so as to avoid the changes in trend caused by the past data especially, for the rapidly growing

Faster, better decision making: With the speed of Hadoop and in-memory analytics, combined with the capability to analyze new sources of data, businesses are able to analyze data immediately and make decisions based on what they’ve learned.

In modern times, the amount of data being stored is terrifically large. Companies must deal with such abundance of data on a daily basis in both storing and analyzing as fast as they can. One such company that not only store data is Google, they also analyze data from each user using their product. The platform used by google for this database management called BigQuery, which runs in the cloud and provides real time information. In this survey, the inner working of BigQuery is glossed over to show how this platform manages to do the job it is supposed to accomplish.

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

Hadoop is one of the most recognized terms in an ocean of big data buzzwords. In short, it is the most efficient tool available to store and analyze data.

In modern times, the amount of data being stored is terrifically large. Companies must deal with such abundance of data on a daily basis in both storing and analyzing as fast as they can. Google is one such company that not only store data but they analyze data from each user using their product. The platform used by google for this database management is called BigQuery, which runs in the cloud and provides real time information. In this survey, the inner working of BigQuery is glossed over to show how this platform manages to do the job it is supposed to do.

With the rise of Hadoop in the workplace, along with it comes a rise of vendors offering various platforms to store, access, and analyze vast amounts of data. These platforms vary in their functionality, cost, and ease of use, among other factors. Three of the more popular vendors are Amazon Web Services, Map-R, and Cloudera. While each of these is based on Apache Hadoop’s open source offerings, it is the applications and reach that differentiates them. Although price will always be a factor as well, this comparison seeks only to explore the vendors themselves, not the price of admission.

Data has always been analyzed within companies and used to help benefit the future of businesses. However, the evolution of how the data stored, combined, analyzed and used to predict the pattern and tendencies of consumers has evolved as technology has seen numerous advancements throughout the past century. In the 1900s databases began as “computer hard disks” and in 1965, after many other discoveries including voice recognition, “the US Government plans the world’s first data center to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape.” The evolution of data and how it evolved into forming large databases continues in 1991 when the internet began to pop up and “digital storage became more cost effective than paper. And with the constant increase of the data supplied digitally, Hadoop was created in 2005 and from that point forward there was “14.7 Exabytes of new information are produced this year" and this number is rapidly increasing with a lot of mobile devices the people in our society have today (Marr). The evolution of the internet and then the expansion of the number of mobile devices society has access to today led data to evolve and companies now need large central Database management systems in order to run an efficient and a successful business.

We also studied and compared new emerging NoSQL databases like Cassandra, Accumulo, CouchDB, Hbase, MongoDB etc. to find the best solution for organizations in accordance with their requirements.