Overview

Harness the power of continuous data processing by building scalable, stateful stream-processing applications with Apache Kafka® Streams and ksqlDB. Learn to filter, transform, enrich, aggregate, and join real-time data streams while leveraging low-latency, fault-tolerant architectures. Develop the skills to design, test, secure, deploy, and monitor event-driven applications that deliver actionable insights and unlock immediate business value.

Audience

This course is designed for application developers, architects,DevOps engineers, and data scientists who need to interact with Kafka clusters to create real-time applications to filter, transform, enrich, aggregate, and join data streams to discover anomalies, analyze behavior, or monitor complex systems.

Prerequisites

Attendees should be familiar with developing professional apps in Java (preferred), .NET, C#, Python, or another major programming language. Participants are required to provide a laptop computer with unobstructed internet access to fully participate in the class.

Objective

Course Objectives

The lessons and activities in this course enable participants to build the skills to:

• Identify common patterns and use cases for real-time stream processing 
• Describe the high-level architecture of Apache Kafka Streams
• Write real-time applications with the Kafka Streams API to filter, transform, enrich, aggregate, and join data streams
• Describe how ksqlDB combines the elastic, fault-tolerant, high-performance stream processing capabilities of Kafka
• Streams with the simplicity of a SQL-like syntax
• Author ksqlDB queries that showcase their balance of power and simplicity
• Test, secure, deploy, and monitor Kafka Streams applications and ksqlDB queries

Hands-on Training

Throughout the course, you will practice with hands-on lab exercises to reinforce stream processing concepts. Exercises include:

• Exploring the anatomy of a Kafka Streams Application
• Joining Two Streams
• Using the Kafka Streams Processor API
• Testing a Kafka Streams Application
• Using ksqlDB
• Using the ksqlDB REST API
• Scaling a Kafka Streams Application
• Securing a Kafka Streams Application
• Getting metrics from a Kafka Streams Application
• Using JConsole to monitor a Kafka Streams Application

Course Outline

Module 01: Introduction to Kafka Streams

• Gain a better understanding of the fundamentals of Apache Kafka
• Delve into how Apache Kafka uses the group management protocol to balance resources
• Give a description of some Stream Processing concepts

Module 02: Working with Kafka Streams

• Describe the anatomy of a Kafka Streams application
• Write a streams application employing components of the Kafka Streams DSL (stateless & stateful transformations, optimizations)

Module 03: Introduction to ksqlDB

• Examine end-to-end examples for using ksqlDB
• Interact with ksqlDB
• Integrate connectors with ksqlDB

Module 04: Using ksqlDB

• Perform data transformations with ksqlDB
• Exploring scalar and table functions in ksqlDB

Module 05: Stateful Transformation Using Kafka Streams and ksqlDB

• Review the concept of time in Apache Kafka Streams
• Use the stateful operations of windowing, aggregations, and joins with Kafka Streams and ksqlDB

Module 06: Advanced Concepts 

• Explore foreign key joins in Kafka Streams
• Creating user-defined functions with ksqlDB
• Use the Kafka Streams Processor API

Module 07: Testing, Monitoring and Troubleshooting

• Perform testing with Apache Kafka Streams
• Evaluate options available for monitoring with JMX, Confluent Control Center, and explain some ksqlDB-specific considerations
• Discuss common errors and troubleshooting approaches

Module 08: Deployment Strategies and Security Basics

• Discuss deployment strategies, when to use them: parallelism, capacity planning, elasticity, fault, ksqlDB-specific considerations
• Explore the security basics needed to begin to design your security architecture (Security overview, Access Control examples)