2015
Pluralsight
Elton Stoneman
4:13
English
Storm is a distributed compute platform which you can plug into Azure Event Hubs and use to power event stream processing. You can scale Storm to read tens of thousands of events per second and build a reliable workflow so that every event is guaranteed to be processed. HBase is a No-SQL database which is easy to get started with and can store tables with billions of rows and millions of columns. It's for real-time data access and it has a REST interface so you can read and write HBase data from a .NET Storm app. Hive is a data warehouse that provides a SQL-like interface over Big Data - HBase tables, and other sources. With Hive you can join across multiple sources and run queries from PowerShell and .NET. In this course, we use all three technologies running on Microsoft Azure to build a race timing solution and dive into performance tuning, reliability, and administration.
Architecting a Solution with HDInsight
Module Introduction
Big Data and the Three (or Four) Vs
Where Big Data Technologies Fit
About the Demo Solution
Course Overview
Storing Race Data in HBase
Module Introduction
HBase Table Design
Modeling Relationships
Row Key Design
Column Families
HBase on Azure
HBase Shell Scripts
The Stargate REST API
.NET Clients for Stargate
Storing Timing Events
Storing Sector Times
Module Summary
HBase Deep Dive
Module Introduction
HBase Cluster Nodes
HFiles and Regions
HBase Data Structure in Azure
Meta Tables and Region Splits
Splitting and Pre-splitting Regions
.NET and HBase Best Practices
Integration Testing with Docker
Load Balancing Stargate
Performance Analysis
Scaling and Compaction
Module Summary
Processing Timing Events with Storm
Module Introduction
Storm Application Architecture
Processing Race Timing with Storm
Saving Events and Defining Bolt Schemas
Local Memory Caches in Bolts
Buffering Writes in Bolts
Flushing Buffers with the Tick Stream
Designing the Topology
Building the Topology
Deploying to HDInsight
Running Race Simulations
Module Summary
Storm Deep Dive
Module Introduction
Storm Cluster Architecture
Runtime Compute Components
Approaches to Performance Testing
Performance Tuning Storm
Scaling the Storm Cluster
Guaranteed Messaging
Implementing Tuple Trees
Logging and Monitoring
Custom Component Logging
Unit & Integration Testing
Module Summary
Querying Race Data with Hive
Module Introduction
HiveQL, the Hive Query Language
Mapping HBase Tables in Hive
Hive Data Types and HBase Column Families
Querying Race Results
Mapping Flat Files in Hive
Joining HBase Tables and CSV Files in HiveQL
Writing Data to Azure from Hive
Recalculating Race Results
Hive Views and Functions
Collections, Joins, and Ranking
Hive and Big Data
Module Summary
Hive Deep Dive
Module Introduction
Hive and YARN
Parallelism for Hive Queries
HiveQL Execution Plans
Filtering HBase Tables
Parameterising Hive Queries with PowerShell
Running Parallel Hive Queries with PowerShell
The Hive ODBC Connector
Connecting to Hive from .NET Apps
Writing Hive UDFs in C#
Customizing the HBase Cluster for Hive
Course Summary
Download File Size:791.45 MB
|
