From Tremors to Saving Lives: A Serverless Kafka Architecture for Timely Earthquake Notifications by Vlad Onetiu
In our upcoming presentation, we'll explore a cutting-edge architectural solution for real-time SMS and email notifications, particularly geared towards responding to earthquake events. This system is designed to handle rapid data transmission, listening for event changes every second, making it ideal for real time critical alert scenarios. Central to our discussion will be the integration of Lambda functions and Confluent Kafka, coupled with advanced multithreading techniques and DynamoDB lock strategies.
A focal point of our presentation will be addressing the challenges and innovative solutions involved in integrating Confluent Kafka with Lambda functions to enable serverless operation of both producers and consumers. This is a key element in ensuring the quick and efficient distribution of notifications through parallel methods. Additionally, we will delve into the implementation of an automated scaling mechanism, which is vital for optimising the performance of the Serverless Notification ecosystem.
Our aim is to provide a comprehensive insight into how these technologies can be effectively combined to develop a robust and efficient system, capable of delivering critical real-time alerts for situations like earthquake occurrences, ultimately playing a crucial role in saving human lives.
Talk Questions
Questions moderation
All questions have to comply with our Code of conduct. So if you don't see your question right after sending it it's because either it has not been moderated yet or it's a question that does not comply with our CoC.
Question 828
Can you explain the role of topic roulet component? Kafka cluster automatically spread the data acroos different partitions. What logic is implemented in the topic roulet to change this behaivour?Question 825
We see SMS and email as destinations, but what about the network level notifications you mentioned at the beginning? How do you interact with that from AWS?Question 830
Do you test and monitor the amount of time (average, percentiles, …) needed for a notification to be sent from when the earthquake was detected?Question 824
Doesn’t serverless create latency problems with cold starts?Question 829
DDB supports optimistic locking out of the box, why implement a custom locking solution using another DDB table?Question 826
If you require such quick message responses, why do you use Lambdas given the cold start issue? Have you considered other solutions? Or do you maintain a minimum number of hot Lambdas always active? If so, what number do you set?Question 827
How much data in input do you process?Question 823
Why did you decide to use dynamodb instead rds for subscribers data which would seem to have a typical relational database structure?Question 831
How to make sure it will scale quickly as earth quakes can happen suddenly. Also to not lose data in the high peakQuestion 832
What kind of (load) testing do you have in place, if any?Question 833
What amount of notification recievers do you have?Question 834
In what way does the notification system (application) gets the warning about the earthquake in real time? Thanks.