IBM Big Data Strategy

IBM Big Data Strategy :

IBM, a US-based computer hardware and software manufacturer, had implemented a Big Data strategy.

Where the company offered solutions to store, manage, and analyze the huge amounts of data generated daily and equipped large and small companies to make informed business decisions.

The company believed that its Big Data and analytics products and services would help its clients become more competitive and drive growth.

Issues :

·        Understand the concept of Big Data and its importance to large, medium, and small companies in the current industry scenario.

·       Understand the need for implementing a Big Data strategy and the various issues and challenges associated with this.

·       Analyze the Big Data strategy of IBM.

·       Explore ways in which IBM’s Big Data strategy could be improved further.

 

Introduction to InfoSphere :

InfoSphere Information Server provides a single platform for data integration and governance.

The components in the suite combine to create a unified foundation for enterprise information architectures, capable of scaling to meet any information volume requirements.

You can use the suite to deliver business results faster while maintaining data quality and integrity throughout your information landscape.

InfoSphere Information Server helps your business and IT personnel collaborate to understand the meaning, structure, and content of information across a wide variety of sources.

By using InfoSphere Information Server, your business can access and use information in new ways to drive innovation, increase operational efficiency, and lower risk.

BigInsights :

 

BigInsights is a software platform for discovering, analyzing, and visualizing data from disparate sources.

The flexible platform is built on an Apache Hadoop open-source framework that runs in parallel on commonly available, low-cost hardware.

 

Big Sheets :

 

BigSheets is a browser-based analytic tool included in the InfoSphere  BigInsights Console that you use to break large amounts of unstructured data into consumable, situation-specific business contexts.

These deep insights help you to filter and manipulate data from sheets even further.

 

Intro to Big SQL :        

 

 

IBM Big SQL is a high performance massively parallel processing (MPP) SQL engine for Hadoop that makes querying enterprise data from across the organization an easy and secure experience.

A Big SQL query can quickly access a variety of data sources including HDFS, RDBMS, NoSQL databases, object stores, and WebHDFS by using a single database connection or single query for best-in-class analytic capabilities.

Big SQL provides tools to help you manage your system and your databases, and you can use popular analytic tools to visualize your data.

Big SQL's robust engine executes complex queries for relational data and Hadoop data.

Big SQL provides an advanced SQL compiler and a cost-based optimizer for efficient query execution.

Combining these with massive parallel processing (MPP) engine helps distribute query execution across nodes in a cluster.

Zookeeper

Zookeeper :

ZooKeeper is a distributed coordination service that also helps to manage a large set of hosts.

Managing and coordinating a service especially in a distributed environment is a complicated process, so ZooKeeper solves this problem due to its simple architecture as well as API.

ZooKeeper allows developers to focus on core application logic.

For instance, to track the status of distributed data, Apache HBase uses ZooKeeper.

They can also support a large Hadoop cluster easily.

To retrieve information, each client machine communicates with one of the servers.

It keeps an eye on the synchronization as well as coordination across the cluster

There is some best Apache ZooKeeper feature :

·       Simplicity: With the help of a shared hierarchical namespace, it coordinates.

·       Reliability: The system keeps performing, even if more than one node fails.

·       Speed: In the cases where ‘Reads’ are more common, it runs with the ratio of 10:1.

·       Scalability: By deploying more machines, the performance can be enhanced.

HBASE

HBase Concepts :

HBase is a distributed column-oriented database built on top of the Hadoop file system.

It is an open-source project and is horizontally scalable.

HBase is a data model that is similar to Google’s big table designed to provide quick random access to huge amounts of structured data.

It leverages the fault tolerance provided by the Hadoop File System (HDFS).

It is a part of the Hadoop ecosystem that provides random real-time read/write access to data in the Hadoop File System.

One can store the data in HDFS either directly or through HBase.

Data consumer reads/accesses the data in HDFS randomly using HBase.

HBase sits on top of the Hadoop File System and provides read and write access.

 

HBase Vs RDBMS :

RDBMS	HBase
It requires SQL (structured query language)	NO SQL
It has a fixed schema	No fixed schema
It is row-oriented	It is column-oriented
It is not scalable	It is scalable
It is static in nature	Dynamic in nature
Slower retrieval of data	Faster retrieval of data
It follows the ACID (Atomicity, Consistency, Isolation and Durability) property.	It follows CAP (Consistency, Availability, Partition-tolerance) theorem.
It can handle structured data	It can handle structured, unstructured as well as semi-structured data
It cannot handle sparse data	It can handle sparse data

Schema Design :

HBase table can scale to billions of rows and any number of columns based on your requirements.

This table allows you to store terabytes of data in it.

The HBase table supports the high read and writes throughput at low latency.

A single value in each row is indexed; this value is known as the row key.

The HBase schema design is very different compared to the relational database schema design.

Some of the general concepts that should be followed while designing schema in Hbase:

·       Row key: Each table in the HBase table is indexed on the row key. There are no secondary indices available on the HBase table.

·       Automaticity: Avoid designing a table that requires atomicity across all rows. All operations on HBase rows are atomic at row level.

·       Even distribution: Read and write should be uniformly distributed across all nodes available in the cluster. Design row key in such a way that, related entities should be stored in adjacent rows to increase read efficacy.

MapReduce Scripts in Hive / Hive Scripts

MapReduce Scripts in Hive / Hive Scripts :

 

Similar to any other scripting language, Hive scripts are used to execute a set of Hive commands collectively.

Hive scripting helps us to reduce the time and effort invested in writing and executing the individual commands manually.

Hive scripting is supported in Hive 0.10.0 or higher versions of Hive.

Joins and SubQueries :

JOINS :

Join queries can perform on two tables present in Hive.

Joins are of 4 types, these are :

• Inner join: The Records common to both tables will be retrieved by this Inner Join.
• Left outer Join: Returns all the rows from the left table even though there are no matches in the right table.
• Right Outer Join:  Returns all the rows from the Right table even though there are no matches in the left table.
• Full Outer Join: It combines records of both the tables based on the JOIN Condition given in the query. It returns all the records from both tables and fills in NULL Values for the columns missing values matched on either side.

 

SUBQUERIES :

A Query present within a Query is known as a subquery.

The main query will depend on the values returned by the subqueries.

Subqueries can be classified into two types :

• Subqueries in FROM clause
• Subqueries in WHERE clause

When to use :

• To get a particular value combined from two column values from different tables.
• Dependency of one table values on other tables.
• Comparative checking of one column values from other tables.

Syntax :

              

Subquery in FROM clause               SELECT <column names 1, 2...n>From (SubQuery) <TableName_Main >               Subquery in WHERE clause               SELECT <column names 1, 2...n> From<TableName_Main>WHERE col1 IN (SubQuery);

HiveQL

HiveQL :

Even though based on SQL, HiveQL does not strictly follow the full SQL-92 standard.

HiveQL offers extensions not in SQL, including multitable inserts and create table as select.

HiveQL lacked support for transactions and materialized views and only limited subquery support.

Support for insert, update, and delete with full ACID functionality was made available with release 0.14.

Internally, a compiler translates HiveQL statements into a directed acyclic graph of MapReduce Tez, or Spark jobs, which are submitted to Hadoop for execution.

Example :

DROP TABLE IF EXISTS docs; CREATE TABLE docs (line STRING);

Checks if table docs exist and drop it if it does. Creates a new table called docs with a single column of type STRING called line.

LOAD DATA INPATH 'input_file' OVERWRITE INTO TABLE docs;

Loads the specified file or directory (In this case “input_file”) into the table.

OVERWRITE specifies that the target table to which the data is being loaded is to be re-written; Otherwise, the data would be appended.

CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM (SELECT explode(split(line, '\s')) AS word FROM docs) temp GROUP BY word ORDER BY word;

The query CREATE TABLE word_counts AS SELECT word, count(1) AS count creates a table called word_counts with two columns: word and count.

This query draws its input from the inner query (SELECT explode(split(line, '\s')) AS word FROM docs) temp".

This query serves to split the input words into different rows of a temporary table aliased as temp.

The GROUP BY WORD groups the results based on their keys.

This results in the count column holding the number of occurrences for each word of the word column.

The ORDER BY WORDS sorts the words alphabetically.

Tables :

 Here are the types of tables in Apache Hive:

Managed Tables :

In a managed table, both the table data and the table schema are managed by Hive.

The data will be located in a folder named after the table within the Hive data warehouse, which is essentially just a file location in HDFS.

By managed or controlled we mean that if you drop (delete) a managed table, then Hive will delete both the Schema (the description of the table) and the data files associated with the table.

Default location is /user/hive/warehouse.

The syntax for Managed Tables :

CREATE TABLE IF NOT EXISTS stocks (exchange STRING, symbol STRING, price_open FLOAT, price_high FLOAT, price_low FLOAT, price_adj_close FLOAT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ;

 

External Tables :

An external table is one where only the table schema is controlled by Hive.

In most cases, the user will set up the folder location within HDFS and copy the data file(s) there.

This location is included as part of the table definition statement.

When an external table is deleted, Hive will only delete the schema associated with the table.

The data files are not affected.

Syntax for External Tables :

CREATE EXTERNAL TABLE IF NOT EXISTS stocks (exchange STRING, symbol STRING, price_open FLOAT, price_high FLOAT, price_low FLOAT, price_adj_close FLOAT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/data/stocks';

 

Querying Data :

A query is a request for data or information from a database table or a combination of tables.

This data may be generated as results returned by Structured Query Language (SQL) or as pictorials, graphs or complex results, e.g., trend analyses from data-mining tools.

One of several different query languages may be used to perform a range of simple to complex database queries.

SQL, the most well-known and widely-used query language, is familiar to most database administrators (DBAs)

 

User-Defined Functions :

In Hive, the users can define their own functions to meet certain client requirements.

These are known as UDFs in Hive.

User-Defined Functions written in Java for specific modules.

Some of UDFs are specifically designed for the reusability of code in application frameworks.

The developer will develop these functions in Java and integrate those UDFs with the Hive.

During the Query execution, the developer can directly use the code, and UDFs will return outputs according to the user-defined tasks.

It will provide high performance in terms of coding and execution.

The general type of UDF will accept a single input value and produce a single output value.

We can use two different interfaces for writing Apache Hive User-Defined Functions :

1.     Simple API

2.     Complex API

Sorting And Aggregating :

 

Sorting data in Hive can be achieved by use of a standard ORDER BY clause, but there is a catch. 

ORDER BY produces a result that is totally sorted, as expected, but to do so it sets the number of reducers to one, making it very inefficient for large datasets.

When a globally sorted result is not required and in many cases it isn’t, then you can use Hive’s nonstandard extension, SORT BY instead. 

SORT BY produces a sorted file per reducer.

If you want to control which reducer a particular row goes to, typically so you can perform some subsequent aggregation.

This is what Hive’s DISTRIBUTE BY clause does.

Example :

·       To sort the weather dataset by year and temperature, in such a way to ensure that all the rows for a given year end up in the same reducer partition :

Hive> FROM records2 > SELECT year, temperature > DISTRIBUTE BY year > SORT BY year ASC, temperature DESC;

·       Output :

                1949    111

                1949    78

                1950    22

                1950    0

                1950    -11