More: https://current.confluent.io | The delivery service startup OPS (Otter Parcel Service) is efficiently managing its fulfillment process.
A team of skilled otters had written a small but mighty fleet of Kafka Streams topologies, and soon OPS was serving its first customers. All data streams ran smoothly, and operators and clients tracked parcel movements in real-time. Everyone was happy, and OPS was on the road to success.
But with the business growing, the otters started to notice the applications getting more and more unwieldy to operate by the day, and infrastructure costs kept increasing. The experienced otter team also knew the reason for that. Their streams state is growing larger and larger over time. The very first fulfilled delivery, which was completed six months back and idling away, is still in the state stores.
Join the otters in tackling this tech debt story to
- understand the implications and causes for the deterioration of operations
- find a solution to keep only the data that is still needed (unfulfilled deliveries)
- learn about different patterns to purge or evict entries from state stores and KTables, such as TTL-based cleanup to expire data
- review, evaluate, and compare implementations
Are you curious if the team can solve their problem and which strategy will prevail? See you at my session, fellow otter 🦦!
(Although this story is a work of fiction, the use cases and solutions presented are based on real-world examples.)
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe on YouTube: https://www.youtube.com/@ConfluentDeveloper
Community Slack: confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent