WhaleBus Publishing - An example of Amazon SNS and Amazon SQS working together

So, lets talk a little bit about the Guts of WhaleBus...

The problem I'm solving at WhaleShark is to communicate things across different applications, as in when something happens on www.retailmenot.com, we may want to create some content on www.deals.com. This fits naturally with Amazon Simple Notification Services. 

Amazon SNS is simply a push mechanism. You have "Topics" that endpoints subscribe to. The endpoints can be of various types (HTTP post, HTTPS post, email with json payload, and Amazon Simple Queue Serviecs).  We'll be creating a topic per type of event.

To publish an event, consumers just call EventPublisher.Publish(T @event).  The event publisher will create a SNS topic if one doesn't exist, serialize the event, then publish it to Amazon. It's actually pretty simple.

I really like this approach, because at WhaleShark, we have a quite a few non .Net websites. We can still push up events pretty much any way we want (they're just protocol buffer serialized messages) to the proper SNS topic, and all subscribers will be notified - regardless of platform.

Back to .Net though... 

For subscription, we use WhaleBusHost. It's simply an executable that pulls from SQS queues. All instances of WhaleBus require a config property to be set for ApplicationName. This application name is used to create an SQS queue. All instances of WhaleBus with the same ApplicationName will share an SQS queue. This is pretty cool, because you can throw more instances easily for scalability.

After it creates the SQS queue, it then scans the application bin directory for all classes that implement IDomainEventHandler<>. It'll then pull out the type for the event, then have the SQS queue, subscribe to that Event's SNS Topic.

Let's make a pretty picture...

The green clouds are SNS topics, Orange ones are SQS queues for specific applications, the blue boxes are instances of WhaleBusHost.exe pulling from those queues.

Note that those green clouds can also send messages via HTTP posts, I just didn't show that in this diagram.  I really think this an awesome approach to a very real problem. At WhaleShark we want to be able to quickly acquire new websites, and allow our current operations staff to use the tools they're currently using to moderate content and activity on the new sites. Instead of ripping the guts out of each of the new sites, this way we can simply write a publishing and subscribing code that publishes events our administration website will care about. The subscription code for those events will already be written. Picture anyone?

Lets say WhaleShark buys the (I think) fictional website www.baconcoupons.com. Let's say it's a php site with a mysql database.   Well, when one of our awesome operation folks or one of or super secret automated tools finds a killer deal on bacon....


We'd definitely want to have that content to show on baconcoupons.com right?
Given a system kinda like this.

We've got a few options on how to make that happen.

Write some quick mapping code that maps from DealFound event to what is expected as a post to sharebacon.php and add it as a new HTTP endpoint for the DealFound SNS endpoint like so.

OR
Write an implementation of IDomainEventHandler that inserts directly into BaconCoupons.db

I don't think there is a right or wrong approach here. It totally depends on the logic that exists in sharebaconcoupon.php, and the team structure (did we get a bunch of new php superheros when we acquired baconcoupons.com?).    I'm guessing that we'd be more inclined to number 2, since any business logic that exists (checking for duplicates, etc), would be in .Net code, and could use libraries we've already written, but that doesn't mean that we can't (or won't) use the first approach.

Ok, I've gotta quit writing blogs and start writing code. Catch everyone later....