Streaming data in Inter-Cloud (AWS-Azure) environment forming a Serverless-Hybrid cloud application.
I have been wanting to write a tech article on Medium for a while now, but never had free time in busy work-life. Thank God for Valentine's day I am free as ‘FORK’ ( TRUTH is often mistaken for jokes ). So here we go ….
After working for a few years in an AWS dominated cloud sector, I was building up a misconception that I would be inoculated to the hybrid cloud phenomenon at-least for some time. However, Azure’s increasing market share and demanding clients, who just want it all, have proven me otherwise. Hybrid cloud is no longer just desirable but rather a necessity.
So, one of the solutions to integrate data between AWS and Azure is using event streaming. AWS has Amazon Kinesis as its primary real-time data streaming tool. Having personally used this for almost more than a year, I can vouch for its incredible performance and ease of use. Also, NETFLIX uses this extensively, if it's good for them it’s good for me.
Microsoft Azure on its end has Azure Event Hub as a directly competing product for AWS Kinesis. Having almost as good data performance and data retention as Kinesis this is the aptest service to integrate with Amazon Kinesis.
Problem statement:-
Streaming data from Amazon Kinesis directly to Azure Event Hub in Real-Time. Using ‘serverless’ model and cloud computing to process and transfer events without having the need to manage any native application written on an on-premise server.
Prerequisites:-
Source kinesis streams in AWS
AWS Execution role that can be assumed by the lambda to read from the kinesis.
Azure event hub Connection string with SharedAccessKeyName and SharedAccessKey
Solution:-
https://www.lucidchart.com/documents/embeddedchart/cc0354e6-f19c-4c27-866f-0c17474474f0
- Firstly, fetch the connection string, SharedAccessKeyName, and SharedAccessKey from the Azure Event Hub. This will be needed to send data to Event Hub. Next, make sure the Event hub can receive data from the IP address you are running the program from. ( In our case AWS NAT gateway ) or keep it open for all if you like to live DANGEROUSLY !!!
2. Now, let's configure the serverless.yml in a fresh lambda project and add an entry for new lambda function to and to listen to your pre-built AWS Kinesis stream
3. In the new lambda function write up a function to retrieve the payloads from AWS Kinesis records array. After getting the payload, process it as per the business need.
4. Post-processing of the payload, use the @azure/event-processor-host library to create an EventHubClient with the connection String of your target EventHub.
5. Don’t forget to close the EventHubClient Connection lol.. ( although I did forget it)
6. If it doesn’t work get coffee, start DEBUGGING.
Now, once you do receive your data from Amazon Kinesis to Azure Event Hub we can do a bunch of cool things with it like saving it directly to Azure Storage, triggering further processing using Azure Functions or even having an on-premise native application listener.
But that’s a story for another single lonely bored Valentine's Day.
Have fun !!!
PS:- if even you don't have anything better to do and wanna try this, the following links may help
