Monday, September 26, 2011

WhaleBus Publishing - An example of Amazon SNS and Amazon SQS working together

So, lets talk a little bit about the Guts of WhaleBus...

The problem I'm solving at WhaleShark is to communicate things across different applications, as in when something happens on, we may want to create some content on This fits naturally with Amazon Simple Notification Services. 

Amazon SNS is simply a push mechanism. You have "Topics" that endpoints subscribe to. The endpoints can be of various types (HTTP post, HTTPS post, email with json payload, and Amazon Simple Queue Serviecs).  We'll be creating a topic per type of event.

To publish an event, consumers just call EventPublisher.Publish(T @event).  The event publisher will create a SNS topic if one doesn't exist, serialize the event, then publish it to Amazon. It's actually pretty simple.

I really like this approach, because at WhaleShark, we have a quite a few non .Net websites. We can still push up events pretty much any way we want (they're just protocol buffer serialized messages) to the proper SNS topic, and all subscribers will be notified - regardless of platform.

Back to .Net though... 

For subscription, we use WhaleBusHost. It's simply an executable that pulls from SQS queues. All instances of WhaleBus require a config property to be set for ApplicationName. This application name is used to create an SQS queue. All instances of WhaleBus with the same ApplicationName will share an SQS queue. This is pretty cool, because you can throw more instances easily for scalability.

After it creates the SQS queue, it then scans the application bin directory for all classes that implement IDomainEventHandler<>. It'll then pull out the type for the event, then have the SQS queue, subscribe to that Event's SNS Topic.

Let's make a pretty picture...

The green clouds are SNS topics, Orange ones are SQS queues for specific applications, the blue boxes are instances of WhaleBusHost.exe pulling from those queues.

Note that those green clouds can also send messages via HTTP posts, I just didn't show that in this diagram.  I really think this an awesome approach to a very real problem. At WhaleShark we want to be able to quickly acquire new websites, and allow our current operations staff to use the tools they're currently using to moderate content and activity on the new sites. Instead of ripping the guts out of each of the new sites, this way we can simply write a publishing and subscribing code that publishes events our administration website will care about. The subscription code for those events will already be written. Picture anyone?

Lets say WhaleShark buys the (I think) fictional website Let's say it's a php site with a mysql database.   Well, when one of our awesome operation folks or one of or super secret automated tools finds a killer deal on bacon....

We'd definitely want to have that content to show on right?
Given a system kinda like this.

We've got a few options on how to make that happen.

Write some quick mapping code that maps from DealFound event to what is expected as a post to sharebacon.php and add it as a new HTTP endpoint for the DealFound SNS endpoint like so.

Write an implementation of IDomainEventHandler that inserts directly into BaconCoupons.db

I don't think there is a right or wrong approach here. It totally depends on the logic that exists in sharebaconcoupon.php, and the team structure (did we get a bunch of new php superheros when we acquired    I'm guessing that we'd be more inclined to number 2, since any business logic that exists (checking for duplicates, etc), would be in .Net code, and could use libraries we've already written, but that doesn't mean that we can't (or won't) use the first approach.

Ok, I've gotta quit writing blogs and start writing code. Catch everyone later....


  1. Any chances to read more info from you about secure virtual data room ? It is very actual these days and many companies are using these technologies now.

  2. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work... Quality Assurance tool

  3. Nice post. I learn something more challenging on different blogs everyday. It will always be stimulating to read content from other writers and practice a little something from their store. I?d prefer to use some with the content on my blog whether you don?t mind. Natually I?ll give you a link on your web blog. Thanks for sharing.
    hungry shark world apk

  4. The simple truth is that very few companies will ever be able to compete with Amazon. So why bother? how to drive traffic to my website

  5. Likewise, Amazon will move your book at various costs, they offer deals or limits yet don't stress despite everything you get a similar measure of cash in any case. So for a deal seeker Amazon is the place they will go.אינדקס אתרים

  6. Over the last couple of years Best silent mouse Amazon have been working on another way you can make money in partnership with them. It's called Fulfillment by Amazon.

  7. thank you for other splendid claim. in which else should everyone accumulate that sort of info in such a perfect support of writing? Click Here