Hadoop Eco System and YARN

Hadoop Eco System and YARN:

Hadoop Ecosystem components

Hadoop Ecosystem is a platform or a suite that provides various services to solve big data problems. It includes Apache projects and various commercial tools and solutions. There are four major elements of Hadoop i.e. HDFS, MapReduce, YARN, and Hadoop Common. Most of the tools or solutions are used to supplement or support these major elements. All these tools work collectively to provide services such as absorption, analysis, storage and maintenance of data etc.

Following are the components that collectively form a Hadoop ecosystem:

• HDFS: Hadoop Distributed File System
• YARN: Yet Another Resource Negotiator
• MapReduce: Programming based Data Processing
• Spark: In-Memory data processing
• PIG, HIVE: Query-based processing of data services
• HBase: NoSQL Database
• Mahout, Spark MLLib: Machine Learning algorithm libraries
• Solar, Lucene: Searching and Indexing
• Zookeeper: Managing cluster
• Oozie: Job Scheduling

Hadoop-Schedulers

1. FiFO scheduler

As the name suggests FIFO i.e. First In First Out, therefore the tasks or application that comes first are going to be served first. This is the default Schedular we use in Hadoop. The tasks are placed during a queue and therefore the tasks are performed in their submission order. In this method, once the work is scheduled, no intervention is allowed. So sometimes the high priority process has got to wait an extended time since the priority of the task doesn't matter during this method.

2.Capacity schedulers

In Capacity Scheduler we've multiple job queues for scheduling our tasks. The Capacity Scheduler allows multiple occupants to share an outsized size Hadoop cluster. In Capacity Schedular corresponding for every job queue, we offer some slots or cluster resources for performing job operations. Each job queue has its own slots to perform its task. just in case we've tasks to perform in just one queue then the tasks of that queue can access the slots of other queues also as they're liberal to use, and when the new task enters to another queue then jobs in running in its own slots of the cluster are replaced with its own job.

Capacity Scheduler also provides A level of abstraction to understand which occupant is utilizing the more cluster resource or slots, so that the only user or application doesn’t take disappropriate or unnecessary slots within the cluster. The capacity Schedular mainly contains 3 sorts of the queue that are root, parent, and leaf which are wont to represent cluster, organization, or any subgroup, application submission respectively.

3. Fair Scheduler

The Fair Scheduler is very much similar to that of the capacity scheduler. The priority of the job is kept in consideration. With the help of Fair Scheduler, the YARN applications can share the resources in the large Hadoop Cluster and these resources are maintained dynamically so no need for prior capacity. The resources are distributed in such a manner that all applications within a cluster get an equal amount of time. Fair Scheduler takes Scheduling decisions based on memory, we can configure it to work with CPU also. 

As we told you it is similar to Capacity Scheduler but the major thing to notice is that in Fair Scheduler whenever any high priority job arises in the same queue, the task is processed in parallel by replacing some portion from the already dedicated slots.