Guillem SoléÀngel Vilardell

Kafka and ksqlDB at The Hotels Network by Guillem Solé and Àngel Vilardell

The hotels Network decided to move from a batch processing analytics platform to a real time analytics platform based on a services architecture.

To accomplish this, we decided to look for kafka solutions (finally decided to go with Redpanda) and found we needed to add ksqlDB to our architecture.

We will share what drove us here, the key decisions, the mistakes we made, the goals we achieved... It might not be exactly your case, but sure you can benefit from some of our findings

Talk Questions

      
  • Question 202
    Why create this tables with a lot of data instead of have pre-coockeddata in a Solr or other caches systems?
  • Question 207
    Shouldn't application itself handle duplications instead of bus component? Because sometimes duplicates might make sense...
  • Question 201
    Did you evaluate other columnar databases before deciding to go with tinybird? If yes, why did you choose tinybird?
  • Question 205
    How did you manage migration of topics and consumer groups from Kafka to red panda? Did you do that without downtime?
  • Question 206
    How does a data correction event propagate through the system?
  • Question 203
    Why don't just use managed solution from aws (since you are already there) ?
  • Question 204
    What is the source of truth in your setup? Is it Kafka log or another storage which topics’ events are offloaded to?
  • Question 211
    Didn't you considered serverless architecture, for example Firehose for streaming and making a datalake with zipped buffers of events to S3, a lambda to get the buffers and store with batch inserts to aws timestream?
  • Question 212
    Any reason why you went for ksqlDB instead of building the topology using Kafka streams?
  • Question 213
    Doesn't consolidating the topics with ksqldb introduce a single point if failure un the data injection pipeline into tinybird?
  • Question 208
    Have you considered Flink (with Flink SQL) instead of ksqlDB? If yes, why did you go with ksqlDB?
  • Question 210
    What was the performance of ksqlDB like? Wasn't it too slow for some of the processes you showcased?