In this hands-on project, learn how to set up real-time monitoring for a high-performance data pipeline processing 1.2 billion records per hour! We cover everything from configuring Grafana and Prometheus to creating a comprehensive Kafka and Spark monitoring dashboard. Watch to see how modern data engineering tools come together for effective system monitoring.
PART 1: https://youtu.be/I2K6uCpxL8E
Timestamps:
0:00 Introduction
1:08 Setting up Grafana
5:36 Linking Prometheus and Grafana
9:52 Grafana Dashboard Setup
13:50 Grafana, Kafka Realtime Monitoring Dashboard
19:42 Outro
Like this video? Support us: https://www.youtube.com/@CodeWithYu/join
👀 Don't just watch it, build it! 🚧
👍 Like, Comment, & Subscribe for more cutting-edge data engineering content!
Resources:
Full Source Code:
https://buymeacoffee.com/yusuf.ganiyu/full-source-code-monitoring-high-performance-architecture-systems
Kafka Documentation: https://kafka.apache.org/documentation/
Apache Spark Documentation: https://spark.apache.org/documentation.html
JMX Exporter Agent - https://github.com/prometheus/jmx_exporter/releases
#ApacheKafka, #Grafana, #Prometheus, #DataEngineering, #BigData, #RealTimeProcessing, #BigDataAnalytics, #DataPipeline, #StreamingData, #KafkaMonitoring, #SparkStreaming, #DataArchitecture, #HighPerformanceComputing