Name: Everything You Always Wanted to Know About Kafka’s Rebalance Protocol
Start: 2019-07-30T20:15:00Z
End: 2019-07-30T20:15:45.000Z
Location: BrightTALK
Rating: 5

Confluent is building the foundational platform for data in motion. Our cloud-native offering is designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organisation. With Confluent, organisations can create a central nervous system to innovate and win in a digital-first world.

What is Apache Kafka® and how does it work?

Apache Kafka® was built with the vision to become the central nervous system that makes real-time data available to all the applications that need to use it, with numerous use cases like stock trading and fraud detection, to transportation, data integration, and real-time analytics.

This 2 part series you will get an overview of what Kafka is, what it's used for, and the core concepts that enable it to power a highly scalable, available and resilient real-time event streaming platform. The series begins with an introduction to the shift toward real-time data streaming, and continues all the way through to best practices for developing applications with Apache Kafka® and how to integrate Kafka into your environment.

Whether you’re just getting started or have already built stream processing applications, you will find actionable insights in this series that will enable you to further derive business value from your data systems.

This training is comprised of the following topics::

1.Benefits of Stream Processing and Apache Kafka® Use Cases
2.Apache Kafka® Architecture & Fundamentals Explained
3.How Apache Kafka® Works
4. Integrating Apache Kafka® Into Your Environment
5. Confluent Cloud

Register now to learn Apache Kafka® from Confluent, the company founded by Kafka’s original developers.

Part 2: Fundamentals for Apache Kafka

Are you struggling to scale your data streaming use cases and break down organisational silos? Join Lyndon Hedderly and Burhan Nazir of Confluent as they share their expertise on deploying enterprise-wide data streaming and accelerating the speed to realising measurable business value derived from a data-streaming investment.

With years of experience enabling companies to deploy data streaming, Lyndon and Burhan have identified the missing factors in data streaming deployments, including a lack of general business visibility, an absence of allocated budget for Kafka or data streaming, and no senior-level or C-suite sponsorship.

During this webinar, you will learn five top tips on successfully implementing an enterprise-wide streaming platform. You will also discover how organisations are leveraging data streaming to scale their operations and break down silos.

Take advantage of this opportunity to learn from the experts in data streaming. Register now to join Lyndon and Burhan and take the first step towards realising the full potential of your data.

Five Tips for Enabling Enterprise-Wide Data Streaming

GenAI has the potential to revolutionize industries by improving efficiency, reducing costs, and providing better outcomes for individuals and businesses. This potential has come into being as a result of the availability of large amounts of data and the development of powerful computing systems that can process that data quickly. However, the current hype around AI is distracting businesses from its real value - the capacity to solve real-world problems and address practical challenges. As the AI hype dies down and productive tools begin to emerge, the integration between streaming data and AI will grow stronger.



Join Italo Nesi, Advisory Solutions Engineer at Confluent, for a webinar where he will discuss:



• Leading use cases for data streaming and GenAI
• How to tap into continuously enriched, trustworthy data streams to quickly scale and build real-time AI applications
• How to build governed, secured, and trusted AI
• How Confluent's depth of integration with Apache Flink makes it easier to build end-to-end AI pipelines that can handle data ingestion, processing, and analysis.

• The relationship between Machine Learning and Generative AI data products

Streaming & The Future: Artificial Intelligence as a Data Product

2024 allows retailers to leverage new technologies to provide customers with a more personalised, efficient, and seamless shopping experience and optimise business operations to drive profit. Making the right technology investments and data strategy choices will enable success in a challenging financial environment. Deploying a data streaming platform throughout your business allows a wide range of use cases that address some of the most challenging business objectives of 2024. 


Join the Confluent team for a webinar that will delve into the most impactful data streaming use cases, leaving you clear on the value these can deliver to your business and how to get started with deploying a data streaming platform. 


By attending, you will learn how a data streaming platform can enable the following:



•Last Mile Delivery: Discover how real-time data can enable retailers to optimise their last-mile delivery operations, reducing delivery times, improving efficiency, and providing a more personalised and convenient delivery experience.


•Customer Personalisation: Real-time data can enable retailers to provide a more personalised and relevant shopping experience for customers, improving customer satisfaction and loyalty and ultimately driving sales.


•Supply Chain and Logistics: A data streaming platform can enable retailers to optimise their supply chain and logistics operations, reducing costs, improving efficiency, and ensuring that products are delivered to customers on time.


•Dynamic Pricing: Use real-time data to enable retailers to implement dynamic pricing strategies responsive to changes in customer demand, competition, inventory levels, and customer behaviour and preferences.


•AI and Analytics: Discover how to train ML models with real-time data that is constantly streaming so your applications can act on every event with context and intelligence.

The Top Data Streaming Use Cases for Retail

Data mesh is a new approach to designing data architectures by embracing organisational and data-centric constructs, such as data management and governance. The idea is that data should be easily accessible and interconnected across the entire business.

In this talk, will cover:
 
- The basics of building a streaming data mesh with Kafka & how Confluent enables this. 
- The four principles of the data mesh: domain-driven decentralisation, data as a product, self-service data platform, and federated governance.
- The differences between working with event streams versus centralised approaches.
- How to onboard data from existing systems into a mesh, modelling the communication within the mesh
- How to deal with changes to your domain’s “public” data, give examples of global standards for governance
- The importance of taking a product-centric view of data sources and the data sets they share.

Apache Kafka®, Confluent, and the Data Mesh

This webinar explores the state of data streaming for the Financial Services Industry in 2023. Join this webinar to learn more about:

- What disruptive trends in financial services are driving the pace of innovation
- How data streaming provides the ability to integrate and correlate data in real-time at any scale.
- Which enterprise architectures are trending in the FinServ industry globally

The State of Data Streaming for Financial Services in 2023

Data Streaming with Apache Kafka and Apache Flink is one of the world's most relevant and talked about paradigms in technology. With the buzz around this technology growing, join Kai Waehner, Global Field CTO at Confluent, to hear his predictions for the 'Top Five Trends for Data Streaming in 2024'. In this webinar, Kai will discuss:

• Data Sharing: Decentralized data mesh architecture with independent data products building or buying cloud-native software components. Collaborating within and outside the organization with data sharing using streaming protocols like Kafka, APIs like REST/HTTP, and open standards like AsyncAPI

• Data Contracts: Policy enforcement for structure, integrity constraints, metadata, rules, and other requirements to ensure data governance and compliance across the end-to-end data streaming pipelines

• Serverless Stream Processing: Focusing on business value leveraging a fully managed, integrated, and secure infrastructure for data correlation to implement streaming ETL and application development.

• Multi-Cloud: Data bridge, migrations, or disaster recovery across cloud providers with reliable uni- or bidirectional data synchronization

• AI: Data fabric for machine learning infrastructures to train models, real-time model scoring, and integrating 3rd party services like GenAI LLMs or SaaS offerings

Top Five Data Streaming Trends for 2024

Apache Kafka® is a scalable streaming platform with built-in dynamic client scaling. The elastic scale-in/scale-out feature leverages Kafka’s “rebalance protocol” that was designed in the 0.9 release and improved ever since then. The original design aims for on-prem deployments of stateless clients. However, it does not always align with modern deployment tools like Kubernetes and stateful stream processing clients, like Kafka Streams. Those shortcomings lead to two major recent improvement proposals, namely static group membership and incremental rebalancing. 

This talk provides a deep dive into the details of the rebalance protocol, starting from its original design in version 0.9 up to the latest improvements and future work. 

We discuss internal technical details, pros and cons of the existing approaches, and explain how you configure your client correctly for your use case. Additionally, we discuss configuration tradeoffs for stateless, stateful, on-prem, and containerized deployments.

Everything You Always Wanted to Know About Kafka’s Rebalance Protocol

Apache Kafka

Confluent

Rebalance Protocol

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

Increasing expectations for good governance, effective risk management and complex demands for corporate compliance are presenting a growing challenge for organizations of all sizes. Join industry thought leaders as they provide you with practical advice on how to implement successful risk and compliance management strategies across your organization. Browse risk management resources in the form of interactive webinars and videos and ask questions of expert GRC professionals.

IT Governance, Risk and Compliance

The data center management community focuses on the holistic management and optimization of the data center. From technologies such as virtualization and cloud computing to data center design, colocation, energy efficiency and monitoring, the BrightTALK data center management community provides the most up-to-date and engaging content from industry experts to better your infrastructure and operations. Engage with a community of your peers and industry experts by asking questions, rating presentations and participating in polls during webinars, all while you gain insight that will help you transform your infrastructure into a next generation data center.

Data Center Management

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

Join thousands of engaged IT professionals in the application management community on BrightTALK. Interact with your peers in relevant webinars and videos on the latest trends and best practices for application lifecycle management, application performance management and application development.

Application Management

The IT project management community on BrightTALK includes thousands of IT project and portfolio management professionals. Find relevant webinars and videos on agile methodologies, scrum strategy, project management processes and more. Attend live webinars or view on demand content presented by recognized thought leaders in the IT project management industry.

IT Project Management

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Everything You Always Wanted to Know About Kafka’s Rebalance Protocol

Presented by

About this talk

More from this channel