SSSamarth Sinha

Case study

Running a 4.2 PB Hadoop Platform

Set up and ran a 4.2 PB Cloudera Hadoop platform end to end, with ETL, reporting and automation, plus two Cassandra clusters for Customer 360.

Role
Set up and ran the platform end to end
Organisation
Bharti Airtel
Period
2018 — 2019
Status
Delivered, in production
4.2 PBCloudera Hadoop platform
15+cluster-utilisation reports
2Cassandra clusters for Customer 360
Stack
  • Cloudera Hadoop
  • Hive
  • YARN
  • Ab Initio
  • NiFi
  • Cassandra
Platform components and reporting Source feeds to ETL. ETL to HDFS · 4.2 PB. HDFS · 4.2 PB to Hive on YARN. Hive on YARN to Reports. Operations to HDFS · 4.2 PB (automates). Monitoring to Hive on YARN (observes). automates observes Source feeds telecom data ETL Ab Initio, NiFi HDFS · 4.2 PB Cloudera Hive on YARN query and batch Reports billing, 15+ utilisation Operations Ansible, ServiceNow Monitoring Grafana, Kibana
IllustrativePlatform components and reportingScroll →

Context

Airtel needed a petabyte-scale Hadoop platform set up and then run day to day, with usage and cost visible to the business.

What I did

I set up and ran the platform end to end.

Architecture and decisions

  • Core platform: Cloudera Hadoop (HDFS, Hive, YARN), with Oracle and Exadata performance tuning.
  • Ingestion: Ab Initio and NiFi ETL.
  • Visibility: billing reports and more than 15 cluster-utilisation reports.
  • Operations: automation and monitoring with Ansible, Grafana, Kibana and ServiceNow.
  • Customer 360: two Cassandra clusters stood up for the programme.

Outcome

  • A 4.2 PB platform in production, with utilisation and billing reporting.