Tuesday, February 15, 2011

Playing around in the Cloud, Some exploratory testing before I can write a Amazon SQS subscriber

So, in my quest to send everything to the cloud, I wanna build a subscriber that uses Amazon SQS to get messages.  What is Amazon SQS you ask? Here’s what they say….

Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed components of their applications that perform different tasks, without losing messages or requiring each component to be always available. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services.
Amazon SQS works by exposing Amazon’s web-scale messaging infrastructure as a web service. Any computer on the Internet can add or read messages without any installed software or special firewall configurations. Components of applications using Amazon SQS can run independently, and do not need to be on the same network, developed with the same technologies, or running at the same time.

Yeah, it’s just a Queuing service. I’m gunna link my Amazon SNS Event publisher up with SQS and be all cool Amazonish. Unfortunately,  SQS doesn’t have a cute little console like SNS did, so I’m gunna spend a little time doing some exploratory testing before I go wire all this up to Ellemy.CQRS. I really want my Amazon stuff to be easy and friendly to use with Ellemy.CQRS. I figure that if I get all my hacking out of the way with this testing, I’ll have my head wrapped around everything by the time I’m ready to write my subscriber.

Ok, lets get to it.

First of all, we’ve gotta create a Queue. The Amazon Documentation says it’s pretty simple. Let’s give that a spin, but I’ll use NUnit as my runner (instead of a Console app).

I wrote this.

   1:  [TestFixture]
   2:      public class check_message_tests
   3:      {
   4:          private AmazonSQSClient _client;
   5:          [SetUp]
   6:          public void create_client()
   7:          {
   8:              _client = new AmazonSQSClient("MyAccountId", "MySecret");
   9:          }
  10:          [Test]
  11:          public void create_queue()
  12:          {
  13:              var request = new CreateQueueRequest().WithQueueName("Test_for_blog");
  14:              var response = _client.CreateQueue(request);
  15:              Console.WriteLine(response.CreateQueueResult.QueueUrl);
  17:          }
  18:  }

Here’s the results….


Sweet… I guess that means I created a Queue!

Let’s go check from Amazon what Queues I have… That’s simple enough. Here’s what I did.

   1:  [Test]
   2:          public void list_queues()
   3:          {
   4:              var response = _client.ListQueues(new ListQueuesRequest());
   5:              foreach (var result in response.ListQueuesResult.QueueUrl)
   6:              {
   7:                  Console.WriteLine(result);
   8:              }
   9:          }


I won’t do another screen capture…  Just take my word for it, the test passed and I saw that Queue show up in the console. This is nice, because I can write a little utility that’ll check if the requested Queue exists for my subscriber, if not, we’ll use the code from that first test to create one.

Lets go send and receive a message it, just to get familiar with this stuff. I wrote this.

   1:  [Test]
   2:          public void send_and_receive_messages()
   3:          {
   4:              var url = "https://queue.amazonaws.com/xxxxxxxxx/Test_for_blog";
   6:              Console.WriteLine("Sending Message");
   7:              var sendMessageRequest = new SendMessageRequest()
   8:                  .WithQueueUrl(url)
   9:                  .WithMessageBody("Hello from the cloud");
  11:              var sendResult = _client.SendMessage(sendMessageRequest);
  12:              Console.WriteLine(sendResult.ToXML());
  14:              Console.WriteLine("Receiving Message");
  15:              var request =
  16:                  new ReceiveMessageRequest().
  17:                  WithQueueUrl(url);
  19:              var result = _client.ReceiveMessage(request);
  20:              foreach (var message in result.ReceiveMessageResult.Message)
  21:              {
  22:                  Console.WriteLine(message.Body);
  23:              }
  24:          }

Ok, so after screwing around for a while, I had some help from a buddy, on what I needed to do. Basically, if we’re gunna allow SNS to Send messages to SQS, we have to send a Queue Attribute up to SQS. I did it like this (this is a “trust me here” kinda thing).


   1:  [Test]
   2:          public void set_permissions()
   3:          {
   4:              var setQueueAttributeRequest = new SetQueueAttributesRequest()
   5:                  .WithQueueUrl(_queueUrl)
   6:                  .WithAttribute(new Attribute { Name = "Policy", Value = AllowSnsAttribute() });
   7:              var result = _client.SetQueueAttributes(setQueueAttributeRequest);
   8:              Console.WriteLine(result.ToXML());
  10:              var getQueueAttributesResponse = _client.GetQueueAttributes(
  11:                  new GetQueueAttributesRequest().WithAttributeName("Policy").WithQueueUrl(_queueUrl));
  12:              foreach (var attribute in getQueueAttributesResponse.GetQueueAttributesResult.Attribute)
  13:              {
  14:                  Console.WriteLine("{0} : {1}",attribute.Name,attribute.Value);
  15:              }
  16:          }
  17:          private string AllowSnsAttribute()
  18:          {
  19:              return @"{""Version"": ""2008-10-17"",""Statement"": [{""Resource"": ""arn:aws:sqs:us-east-1:XXXXX:Test_for_blog"", ""Effect"": ""Allow"", ""Sid"": ""1"", ""Action"": ""sqs:*"", ""Condition"": {""StringEquals"": {""aws:SourceArn"": ""arn:aws:sns:us-east-1:XXXXX:EventMessage""}}, ""Principal"": {""AWS"": ""*""}}]}";
  20:          }

Yeah, so it’s pretty nifty actually, but for now, just trust me that we HAVE TO do that to make stuff work. I’ll save this code, cause I’ll be using it later when I start automagically wiring stuff up.

Next up I think I wanna publish from Amazon SNS and receive the resulting message. To do that, I’m gunna need to subscribe to a Topic ARN. Here’s what I did.

   1:   [Test]
   2:          public void subscribe_to_topic()
   3:          {
   4:              var getArnRequest = new GetQueueAttributesRequest().WithQueueUrl(_queueUrl).WithAttributeName("QueueArn");
   5:              var clientArn = _client.GetQueueAttributes(getArnRequest).GetQueueAttributesResult.Attribute[0].Value;
   7:              var sns = Amazon.AWSClientFactory.CreateAmazonSNSClient(awsKeyId,
   8:                                                                      secret);
  10:              var subscriptionRequest = new SubscribeRequest()
  11:                  .WithEndpoint(clientArn)
  12:                  .WithProtocol("sqs")
  13:                  .WithTopicArn("arn:aws:sns:us-east-1:xxxxx:EventMessage");
  15:              var response = sns.Subscribe(subscriptionRequest);
  16:              Console.WriteLine(response.SubscribeResult.SubscriptionArn);
  17:          }

Works like a charm, cause when I go to my AWS console, I see the new subscription. Cool!

Next up, we’re gunna manually publish a message via the AWS console, and rip out the code from that send and receive test (just the receive part) and make sure that the message we published shows. First I fire up the AWS Console and  hit the little publish to topic button. It looks like this….


I’ll type in some mumbo jumbo and run a test that looks like this.


   1:   [Test]
   2:          public void receive_message()
   3:          {
   4:              var request =
   5:                new ReceiveMessageRequest().
   6:                WithQueueUrl(_queueUrl);
   7:              var result = _client.ReceiveMessage(request);
   8:              foreach (var message in result.ReceiveMessageResult.Message)
   9:              {
  10:                  Console.WriteLine(message.Body);
  11:              }
  12:          }


And the results?


Wow! Done… So, now I’ve proven that I can indeed get all this technology to work together… Next up, is doing it!

Stay tuned!

1 comment: