Tuesday, September 20, 2011

Using Messaging in Real World, Legacy Apps (and intro to WhaleBus) - Amazon SNS and Amazon SQS

So, if you wanna pull down the code before you read my blog post... It's here...
https://github.com/elliottohara/WhaleBus

Also, If you're familiar with NServiceBus, and this code looks something like it, that's not an accident, I think Udi Dahan made a awesome product, and I've used it with great success on a few projects. I've just got a few different problems at the new gig, and figured I'd give my own little service bus a shot.  To be fair, it's pretty much the Ellemy.CQRS stuff I had been blogging about before moving here, but yeah, I spent the last week making it work a lot better to solve some problems we have here.

We own coupon websites - a lot of them. The business plan is pretty much, buy them up, make them a little better (hopefully), throw a little better marketing at them, and make LOTS OF money. The state of our current systems looks something like this.



Note that Deals2Buy.com, Deals.com, and Howzat where all greenfield applications. They're things our internal staff wrote from the ground up. They wrote them in a traditional N-Tier approach, they all share a Voldemort  document store that we've named "Madre".


We clearly have a lot of work to do. It's not cost effective to acquire a property, then train ops employees on how to use the admin features of each website. That's how things are being handled now. We've gotta change that..


Certainly one approach is to migrate all properties to "Madre". It would "kinda" work. However, to me, that's a LOT of work. At the end of the day, ops needs to be able to use one application to update ANY website that we own. That's it. There's no requirement that all apps share a single database, just that the apps can be updated from a single application and that our execs can see the money we're making.

Well…. That sounds like a great candidate for messaging huh?

Think about it. The business logic for most (if not all) use cases already exists on every website we'd ever acquire, right? I mean, when someone shares a coupon on some php site with a mysql data base, there's already code on that site that updates the data base… Otherwise, well, we probably wouldn't be buying it.


We already use Amazon Cloud services for lots of stuff… Simple Notification Service is PERFECT for this. It supports a few different types of endpoints, including HTTPS post. So, our admin interface pushes some message (a domain event) out to SNS, and we configure SNS to have an HTTPS endpoint to some page on our site that takes the post. Here's a pretty picture I just drew using google documents.



This way, the only thing new we need to write on RetailMeNot is some HTTP endpoint that translates the CouponSharedOnSite message to call whatever code ALREADY EXISTS on RetailMeNot. Pretty simple, and makes the acquisition of new properties MUCH much easier (something that makes our CEO happy - and having a happy CEO is a good thing - trust me).


Ok, so, that's awesome and all, but we've got a finance department, they like to say things like "Hey, we're 8% up in revenue over projection" and they wanna know if they're actually telling the truth. The picture above is awesome to get customers on RetailMeNot to see some coupon that one of our ops person created on the Howzat application, but what about when finance wants to know that someone actually clicked on one of our coupons?



Well, we're working on a really nice database, that for the purposes of this blog, we're gonna say is totally complete. All we need to do is update that database when someone clicks on a link that goes to a vendor. Note that we want this to happen for ALL of our properties, and also note that our CEO love buying new websites. Also note that this is how pretty much EVERY coupon site makes revenue so they all record these outclicks in some way. All we need to do is add code to whatever exists on those properties that records the outclick to send a message out to SNS, and we'll write an app that subscribes to these messages.
I'll draw you another picture, cause I think google drawings are cool.



Ok, cool pictures and everything right? But no one comes to my blog for pictures… Let's talk code.

WhaleBus

For now, I'm gonna just go over how to use my new little pet project, we'll dive deep into the code in some future blog.
First of all, all the messages we'll be sending are DomainEvents, not commands. This is because commands handling is (for the most part) application specific. If you don't understand this, well, read that link for DomainEvents. In a nutshell, DomainEvents communicate something that has ALREADY HAPPENED that other systems may care about, like "Customer Clicked On A Deal" or "Some deal just was just expired".


When an event happens, applications want to do things based on that event. Let's express that in code.

namespace WhaleBus
{
public interface IDomainEvent{}
public interface IDomainEventHandler<in TDomainEvent> where TDomainEvent : IDomainEvent
{
void Handle(TDomainEvent @event);
}
}
So, if we wanted to add a row to some db table when a customer clicked on a deal, we'd simply write the code in a class implementing IDomainEventHandler<CusomterClickedOnDeal>. I'll show some of this in a future blog, but for now, I'm gonna assume that you can figure that out.

I had two real use cases in this blog, one where we where publishing an event, and one to subscribe to it.


Publishing
Here's the code you'd use to publish events….

Configure.With()
.ProtoBufSerializer()
.StructureMapBuilder()
.EventsInAssemblyContaining<StoreAddedToSite>();
var publisher = EventPublisherFactory.CreateEventPublisher();

You'd probably do this during app startup and make your publisher a singleton since you only need one instance of him for the app. We can talk more about that later. This code tells WhaleBus to use GoogleProtocolBuffers for it's serialization mechanism (you can also use Json), and what assembly contains the events. If you've got events spread over multiple assemblies, just keep calling EventsAreInAssemblyContainingType... It'll add them for you. To publish code you simply call Publish<TEvent>(TEvent @event) on publisher. WhaleBus will do all sorts of cool stuff for you that will discuss in a future blog. 


Here's the code that's in the Example on github for a publisher.

using System;
using Events.Stores;
using WhaleBus.Implementations;
using WhaleBus.Infrastructure;
using WhaleBus.Infrastructure.Publisher;

namespace WhaleBus.PublisherConsole
{
    class Program
    {
        static void Main(string[] args)
        {
            Configure.With()
                .ProtoBufSerializer()
                .StructureMapBuilder(new PubSubSettingsFromFile(@"c:\specialsupersecret\elliott.ohara@gmail.com.txt"))
                .EventsInAssemblyContaining<StoreAddedToSite>();
                
            var publisher = EventPublisherFactory.CreateEventPublisher();

            while(true)
            {
                Console.WriteLine("Enter Store Name");
                var name = Console.ReadLine();
                Console.WriteLine("Enter Domain");
                var domain = Console.ReadLine();
                publisher.Publish(new StoreAddedToSite{Domain = domain, Name= name});
                Console.WriteLine("Published Event.");
            }
        }
    }
}

Subscribing 
Now lets show a subscriber. We handle subscribers a bit differently. For now, I've written a console app called WhaleBusHost. To use it, you simply reference it, and write a class that implements the IConfigureThisSubscriber class. Here's the one that's in the example on github.



using Events.Stores;
using WhaleBus;
using WhaleBus.Implementations;
using WhaleBus.Infrastructure;
using WhaleBusHost;

namespace AnEndpoint
{
    public class ConfigureMe : IConfigureThisSubscriber
    {
        public ConfigureMe()
        {
            Configure.With()
                .ProtoBufSerializer()
                .StructureMapBuilder(new PubSubSettingsFromFile(@"C:\SpecialSuperSecret\elliott.ohara@gmail.com.txt"))
                .EventsInAssemblyContaining<StoreAddedToSite>();
            

        }
    }
}

That's really it. WhaleBus wires up all your subscriptions for any event that has a handler in every assembly in the applications bin folder. So as you add new functionality (handlers), you don't have to manually do much of anything. Just run WhaleBusHost.exe.

We'll discuss how everything works on the next post.

4 comments:

  1. Nice post, thanks for sharing. One thought that automatically comes up is how do you publish events to a specific site? Do you configure a new publisher for each site? Do all sites get all messages and only accept theese where Domain is correct?

    I've never used SNS so i may be missing something.

    ReplyDelete
  2. Topics refer to type of event, so if, for instance we had an event called "DealDiscovered" and we had WhaleBusHosts running for many sites that had handlers for DealDiscovered, yes that message would be delivered to all of those WhaleBusHost instances.

    It would be the job of the application to act appropriately. Better yet, you could just subclass the event like so

    RetailMeNotDealDiscovered:DealDiscovered

    and only subscribe to the that event on the RetailMeNot WhaleBusHost instance.

    Does that make sense?

    ReplyDelete
  3. Yep, that absolutely makes sense. Would you then have a topic for RetailMeNotDealDiscovered, or just a DealDiscovered which checked the event name?

    I'm architecting a system where we have around 2k clients and a single server, where the server needs to push messages once or twice per day to all clients. One message is only meant for one client. Do you have any ideas on this? I could do a simple HTTP POST but i really like the benefits of a messaging solution. No replies are needed.

    ReplyDelete
  4. The topic would be RetailMeNotDealDiscovered. So only consumers of the concrete event would get the message.

    I think the problem your describing seems more along the line of command messages, not domain events. Although you probably could make it work, I specifically built WhaleBus to handle the publishing of events to many endpoints, so I don't think what I wrote is quite what you want.

    If you only want to publish events to a single endpoint SNS would certainly still work, I'd just publish the message and use some naming convention in the topic to tell it what client it's for. Then configure SQS endpoints for each client. If you download the AWS SDK, its actually really nice. Hit my up on twitter if you have any questions.

    ReplyDelete