Thursday, September 29, 2011

Looking for good software engineers in Austin, Tx

 Ok, so they put a link to my blog on the corporate site, and while I've been blowing up twitter about the fact that we're hiring, I figured I'd make one little blog post about the fact that we're hiring, what we're looking for and how AWESOME this job is. Email me at if you're interested, or hit

Why is WhaleShark awesome? 

Yeah, we hire interns too!

The deal industry is going nuts. It ain't no secret that the economy has seen better days, and consumers are trying how to buy more stuff with less money. So, yeah, we're killing it economically. How do I know? Well, every Tuesday, we have an all hands meeting, and they tell us actual numbers. It's really cool to work for a company where people start wondering what's wrong, when we only exceed forecast on revenue by 4 or 5%.

Our kitchen
The technology stack is pretty neat here. By "pretty neat", I mean AWESOME!  We use Voldemort for persistence. It's a distributed key/value store that's pretty nifty. We serialize everything using Google Protocol buffers (the value part of key/value). All queries (since you can't query a key/value store easily) are Solr. We also use the heck outta Amazon servies. We use Amazon S3 to store all sorts of data, SNS and SQS for pub/sub (see my WhaleBus post). We have massive amounts of data that we use Hadoop/Pig to process to get lots of cool analytics stuff. We've got sites in classic asp, 1.1, mvc, wordpress, LAMP - pretty much everything you can think of, and we make  all work together. In short, it's pretty freekin' cool. What's even cooler, is we'll keep getting more sites, the executive team is all about acquisition, so engineering has to stay agile. That's a good thing to me.

Yeah, that's Dublin baby!
Lunch is catered (at least) every Tuesday. The fridge is always stocked with frozen lunches, Blue Bell ice cream, bagels and waffles (for us morning guys), and lots of other goodies. And YES, to the right, that is a picture of Dublin Dr. Pepper on tap that you see. The office is, AMAZING. Glass conference rooms,  desks that promote pair programming and impromptu conversations, but still large enough that you can take a personal call and not let everyone know that your kid just got sent home for flicking a bugger on the teacher.  There's small couches all over the place so you can sit and talk about how awesome CQRS or why the Cowboys aren't gunna suck this year without the need to reserve some conference room. Oh yeah, there's usually beer in the fridge on Friday, and no one gives you crap if you drink it.

My Desk (and Pratik being awesome)
The culture at WhalesShark is another thing worth mentioning. I've been around the block a couple times.   Too many times I've seen a great company loose good engineers because of some silly political issues between product, engineering and executive staff. Cotter (our CEO) is amazing about NOT allowing that kinda thing, we have an open culture like I've never seen. Most (I think all) of our full time employees have options. So, we've all got a vested interest in the success, and we all work together at it.  We work hard, but have fun too. I'm the resident nerf dart expert, and as the Team Lead for the Internal systems team, it's not rare for Angie (one of the operations managers) to come shoot me with her nerf gun because one of the feeds imported a coupon wrong. We have a LOT of nerf gun wars actually.

Board Room
Conference room (and Jamie)
Elevator Lobby

 So what types of Engineers are we looking for?
  • Passionate about their craft, someone who codes because they love it, not because it's their carrier.
  • Is willing to step out of the comfort zone, and does. Someone who learns things because they want to.
  • Someone who understands the importance of testing (I'm a TDD guy, and that's what I want, but it's not a must)
  • Someone strong in OOP and Design (I'm a big DDD guy)
  • Someone who is interested or has experience in distributed systems (CQRS is a big plus in my book)
  • An awesome engineer - I WILL go through coding exercises with you, and if you think giant case statements are ok, then, move on :)
  • A github account with some examples of stuff you've done is a big plus.
Ok... sorry for the spam! Back to coding for me! 

Monday, September 26, 2011

WhaleBus Publishing - An example of Amazon SNS and Amazon SQS working together

So, lets talk a little bit about the Guts of WhaleBus...

The problem I'm solving at WhaleShark is to communicate things across different applications, as in when something happens on, we may want to create some content on This fits naturally with Amazon Simple Notification Services. 

Amazon SNS is simply a push mechanism. You have "Topics" that endpoints subscribe to. The endpoints can be of various types (HTTP post, HTTPS post, email with json payload, and Amazon Simple Queue Serviecs).  We'll be creating a topic per type of event.

To publish an event, consumers just call EventPublisher.Publish(T @event).  The event publisher will create a SNS topic if one doesn't exist, serialize the event, then publish it to Amazon. It's actually pretty simple.

I really like this approach, because at WhaleShark, we have a quite a few non .Net websites. We can still push up events pretty much any way we want (they're just protocol buffer serialized messages) to the proper SNS topic, and all subscribers will be notified - regardless of platform.

Back to .Net though... 

For subscription, we use WhaleBusHost. It's simply an executable that pulls from SQS queues. All instances of WhaleBus require a config property to be set for ApplicationName. This application name is used to create an SQS queue. All instances of WhaleBus with the same ApplicationName will share an SQS queue. This is pretty cool, because you can throw more instances easily for scalability.

After it creates the SQS queue, it then scans the application bin directory for all classes that implement IDomainEventHandler<>. It'll then pull out the type for the event, then have the SQS queue, subscribe to that Event's SNS Topic.

Let's make a pretty picture...

The green clouds are SNS topics, Orange ones are SQS queues for specific applications, the blue boxes are instances of WhaleBusHost.exe pulling from those queues.

Note that those green clouds can also send messages via HTTP posts, I just didn't show that in this diagram.  I really think this an awesome approach to a very real problem. At WhaleShark we want to be able to quickly acquire new websites, and allow our current operations staff to use the tools they're currently using to moderate content and activity on the new sites. Instead of ripping the guts out of each of the new sites, this way we can simply write a publishing and subscribing code that publishes events our administration website will care about. The subscription code for those events will already be written. Picture anyone?

Lets say WhaleShark buys the (I think) fictional website Let's say it's a php site with a mysql database.   Well, when one of our awesome operation folks or one of or super secret automated tools finds a killer deal on bacon....

We'd definitely want to have that content to show on right?
Given a system kinda like this.

We've got a few options on how to make that happen.

Write some quick mapping code that maps from DealFound event to what is expected as a post to sharebacon.php and add it as a new HTTP endpoint for the DealFound SNS endpoint like so.

Write an implementation of IDomainEventHandler that inserts directly into BaconCoupons.db

I don't think there is a right or wrong approach here. It totally depends on the logic that exists in sharebaconcoupon.php, and the team structure (did we get a bunch of new php superheros when we acquired    I'm guessing that we'd be more inclined to number 2, since any business logic that exists (checking for duplicates, etc), would be in .Net code, and could use libraries we've already written, but that doesn't mean that we can't (or won't) use the first approach.

Ok, I've gotta quit writing blogs and start writing code. Catch everyone later....

Tuesday, September 20, 2011

Using Messaging in Real World, Legacy Apps (and intro to WhaleBus) - Amazon SNS and Amazon SQS

So, if you wanna pull down the code before you read my blog post... It's here...

Also, If you're familiar with NServiceBus, and this code looks something like it, that's not an accident, I think Udi Dahan made a awesome product, and I've used it with great success on a few projects. I've just got a few different problems at the new gig, and figured I'd give my own little service bus a shot.  To be fair, it's pretty much the Ellemy.CQRS stuff I had been blogging about before moving here, but yeah, I spent the last week making it work a lot better to solve some problems we have here.

We own coupon websites - a lot of them. The business plan is pretty much, buy them up, make them a little better (hopefully), throw a little better marketing at them, and make LOTS OF money. The state of our current systems looks something like this.

Note that,, and Howzat where all greenfield applications. They're things our internal staff wrote from the ground up. They wrote them in a traditional N-Tier approach, they all share a Voldemort  document store that we've named "Madre".

We clearly have a lot of work to do. It's not cost effective to acquire a property, then train ops employees on how to use the admin features of each website. That's how things are being handled now. We've gotta change that..

Certainly one approach is to migrate all properties to "Madre". It would "kinda" work. However, to me, that's a LOT of work. At the end of the day, ops needs to be able to use one application to update ANY website that we own. That's it. There's no requirement that all apps share a single database, just that the apps can be updated from a single application and that our execs can see the money we're making.

Well…. That sounds like a great candidate for messaging huh?

Think about it. The business logic for most (if not all) use cases already exists on every website we'd ever acquire, right? I mean, when someone shares a coupon on some php site with a mysql data base, there's already code on that site that updates the data base… Otherwise, well, we probably wouldn't be buying it.

We already use Amazon Cloud services for lots of stuff… Simple Notification Service is PERFECT for this. It supports a few different types of endpoints, including HTTPS post. So, our admin interface pushes some message (a domain event) out to SNS, and we configure SNS to have an HTTPS endpoint to some page on our site that takes the post. Here's a pretty picture I just drew using google documents.

This way, the only thing new we need to write on RetailMeNot is some HTTP endpoint that translates the CouponSharedOnSite message to call whatever code ALREADY EXISTS on RetailMeNot. Pretty simple, and makes the acquisition of new properties MUCH much easier (something that makes our CEO happy - and having a happy CEO is a good thing - trust me).

Ok, so, that's awesome and all, but we've got a finance department, they like to say things like "Hey, we're 8% up in revenue over projection" and they wanna know if they're actually telling the truth. The picture above is awesome to get customers on RetailMeNot to see some coupon that one of our ops person created on the Howzat application, but what about when finance wants to know that someone actually clicked on one of our coupons?

Well, we're working on a really nice database, that for the purposes of this blog, we're gonna say is totally complete. All we need to do is update that database when someone clicks on a link that goes to a vendor. Note that we want this to happen for ALL of our properties, and also note that our CEO love buying new websites. Also note that this is how pretty much EVERY coupon site makes revenue so they all record these outclicks in some way. All we need to do is add code to whatever exists on those properties that records the outclick to send a message out to SNS, and we'll write an app that subscribes to these messages.
I'll draw you another picture, cause I think google drawings are cool.

Ok, cool pictures and everything right? But no one comes to my blog for pictures… Let's talk code.


For now, I'm gonna just go over how to use my new little pet project, we'll dive deep into the code in some future blog.
First of all, all the messages we'll be sending are DomainEvents, not commands. This is because commands handling is (for the most part) application specific. If you don't understand this, well, read that link for DomainEvents. In a nutshell, DomainEvents communicate something that has ALREADY HAPPENED that other systems may care about, like "Customer Clicked On A Deal" or "Some deal just was just expired".

When an event happens, applications want to do things based on that event. Let's express that in code.

namespace WhaleBus
public interface IDomainEvent{}
public interface IDomainEventHandler<in TDomainEvent> where TDomainEvent : IDomainEvent
void Handle(TDomainEvent @event);
So, if we wanted to add a row to some db table when a customer clicked on a deal, we'd simply write the code in a class implementing IDomainEventHandler<CusomterClickedOnDeal>. I'll show some of this in a future blog, but for now, I'm gonna assume that you can figure that out.

I had two real use cases in this blog, one where we where publishing an event, and one to subscribe to it.

Here's the code you'd use to publish events….

var publisher = EventPublisherFactory.CreateEventPublisher();

You'd probably do this during app startup and make your publisher a singleton since you only need one instance of him for the app. We can talk more about that later. This code tells WhaleBus to use GoogleProtocolBuffers for it's serialization mechanism (you can also use Json), and what assembly contains the events. If you've got events spread over multiple assemblies, just keep calling EventsAreInAssemblyContainingType... It'll add them for you. To publish code you simply call Publish<TEvent>(TEvent @event) on publisher. WhaleBus will do all sorts of cool stuff for you that will discuss in a future blog. 

Here's the code that's in the Example on github for a publisher.

using System;
using Events.Stores;
using WhaleBus.Implementations;
using WhaleBus.Infrastructure;
using WhaleBus.Infrastructure.Publisher;

namespace WhaleBus.PublisherConsole
    class Program
        static void Main(string[] args)
                .StructureMapBuilder(new PubSubSettingsFromFile(@"c:\specialsupersecret\"))
            var publisher = EventPublisherFactory.CreateEventPublisher();

                Console.WriteLine("Enter Store Name");
                var name = Console.ReadLine();
                Console.WriteLine("Enter Domain");
                var domain = Console.ReadLine();
                publisher.Publish(new StoreAddedToSite{Domain = domain, Name= name});
                Console.WriteLine("Published Event.");

Now lets show a subscriber. We handle subscribers a bit differently. For now, I've written a console app called WhaleBusHost. To use it, you simply reference it, and write a class that implements the IConfigureThisSubscriber class. Here's the one that's in the example on github.

using Events.Stores;
using WhaleBus;
using WhaleBus.Implementations;
using WhaleBus.Infrastructure;
using WhaleBusHost;

namespace AnEndpoint
    public class ConfigureMe : IConfigureThisSubscriber
        public ConfigureMe()
                .StructureMapBuilder(new PubSubSettingsFromFile(@"C:\SpecialSuperSecret\"))


That's really it. WhaleBus wires up all your subscriptions for any event that has a handler in every assembly in the applications bin folder. So as you add new functionality (handlers), you don't have to manually do much of anything. Just run WhaleBusHost.exe.

We'll discuss how everything works on the next post.