Screen scaping with node.js and node.io

I'm doing an integration with a 3rd party for one of my side projects, and it's funny because they have api calls to get the information I want, given that I know the business id. Of course they don't offer that via their api, so I have to go through the front end interface to grab it.

I wanted something that I can use with jQuery selectors, and I came across a very cool node js plugin called node.io. The plugin is a screen scaper records the source and lets you query using jQuery like selectors.

Let's have a look at the code shall we?

File: server.js

 var http = require('http'); 

 var URL = require('url'); 

 var nodeio = require('node.io'); 

 var port = 8081; 

 var gigParkSearchJobOptions = {timeout : 10}; 

 var gigParkSearchJob = { 

   input: false, 

   run: function (keyword) { 

        var phone = this.options.args[0]; 

     this.getHtml("http://www.gigpark.com/?search=" + phone + "&commit=Search&city", function (err, $) { 

       var results = $('h3 .search-results b').text.toLowerCase(); 

       this.emit('Results found: ' + results); 

     }); 

   } 

 }; 

 /** 

  * Main http serve method. 

  */ 

 function serve() { 

      var serv = http.createServer(function(req, res) { 

           var url = URL.parse(req.url); 

           if (req.method == 'GET' && url.pathname == '/gigpark_search') { 

                res.statusCode = 200; 

                //query looks like this: 'q=1234567890, so we split it. 

                var phoneNum = url.query.split('=')[1]; 

                console.log('query is: ' + phoneNum); 

                nodeio.start(new nodeio.Job(gigParkSearchJobOptions, gigParkSearchJob), [phoneNum], function(err, output) {  

                     console.log('result is: ' + output); 

                     res.write(JSON.stringify(output)); 

                     res.end(); 

                     }, true); 

           } else { 

                res.statusCode = 404; 

                res.end("Sorry, page not found."); 

           } 

      }); 

      serv.listen(port); 

 } 

 console.log("Starting server..."); 

 serve();  

Basically this code create an http server with a path of /gigpark_search which takes a query parameter. We then parse the parameter value, and then send it in as a paramter to a node.io job. It then fetches the search page and parses the html and gives me the value of var results = $('h3 .search-results b').text, which is an integer.

So if one makes a query with this url: http://localhost:8081/gigpark_search?q=7788911331

One would see a response like this: ["Results found: 1"].

Of course, my next step is to make it into a JSON response, but for illustration I just wanted to put some result in.

If you're thinking of using grailsLinkGenerator in Bootstrap.groovy

In Grails 2, the creators have made it easier for us developer to create the urls in our service/taglib classes.

One would use it like this:

 class MyService { 

      LinkGenerator grailsLinkGenerator 

     ... 

 }  

However, after some investigation, it seems that at the point where Bootstrap.groovy is run, it doesn't have access  to the contextual information such as the host name and context path. So even though you can inject this class to Bootstrap.groovy, it doesn't provide us with much contextual information.

Getting the quartz2 plugin to work with Grails

My app needs to run a job periodically, and I got the quartz2 plugin https://github.com/9ci/grails-quartz2 with one caveat.

In the tutorials page, it said to externalize the settings to something like grails.config.locations = [ "file:Quartz-config.groovy"] in your config.groovy.


Seems like a sensible idea...but for the life of me I couldn't get it to fetch the configs from an externalized source as described here: http://grails.org/doc/latest/guide/single.html#configExternalized

In the end, I gave up and just put my quartz config in Config.groovy. Probably not the ideal place to be but hopefully I'll figure it out and update you folks!

A thought on testing Grails controllers

I'm writing tests for my grails app, and I found a handy shortcut when you are testing your controllers and don't want to populate all of the domain object under test's fields. You can simply set the validate field to be false on a domain object:

1:       void testEditExistingBusiness() {  
2:            def user = new User(username : 'tommy')  
3:            user.save(validate: false)  
4:            def business = new Business(id : 1, user : user)  
5:            business.save(validate: false)  
6:            def securityMock = createSpringSecurityMock(user)  
7:            controller.springSecurityService = securityMock  
8:            params.id = business.id  
9:            def model = controller.edit()  
10:            assert model.businessInstance == business  
11:       }  

That way you don't have to populate all of the User's and Business's required fields, which in this case is perfectly fine for my test.

How-to: Set up MongoHQ in Grails

So I'm using MongoHQ for my running my integration tests, and I figured out how to map the connection string to the grails config.

Consider this MongoHQ config that you got when you signed up:


mongodb://cloudbees:password@flame.mongohq.com:27029/


The corresponding config would look like this:

 
1:     test {   
2:        dataSource {   
3:          dbCreate = "update"   
4:          url = "jdbc:h2:mem:testDb;MVCC=TRUE"   
5:        }   
6:        grails {   
7:          mongo {   
8:             host = "flame.mongohq.com"   
9:             port = 27029   
10:             username = "cloudbees"   
11:             password = "password"   
12:             databaseName = ""   
13:             options {   
14:               autoConnectRetry = true   
15:               connectTimeout = 300   
16:             }   
17:          }   
18:        }   
19:     }   

 

But wait! The initial password they gave you needed to be changed in order for it to work (at least for me anyways). To change your password, login to MongoHQ and change it.



Hopefully that helps someone!

Using Groovy's mockFor closure

I'm writing my Grails application and have come to a point where I realized that I need to start writing some tests before things get real ugly. I think as a software developer tests are invaluable to any project and I'm a firm believer in tests. With that I would like to share with you my experience and the gotchas of the mockFor closure.

Groovy 2.x provides us with mixins in our tests so that we don't have to extend other test classes (something that I also implemented at work) with the main goal being "favor composition over inheritance".  I like this approach. However, that means that magical methods can come out from anywhere, and if there's a problem, you have to dig deeper to find the answer. Case in point, consider this test code that I have written:

      def void setUp() { 

           def control = mockFor(SpringSecurityService) 

           control.demand.getPrincipal { return new User() } 

           controller.springSecurityService = control.createMock() 

      }  

The code seems simple enough -- we create a mock for the SpringSecurityService, and then when a call to getPrincipal() is invoked, it would return a new user.

However, when I run it, this is what I get:

 | Failure: testIndex(com.tweetcommendation.BusinessControllerTests) 

 | groovy.lang.MissingMethodException: No signature of method: com.tweetcommendation.Business.findByUser() is applicable for argument types: (com.tweetcommendation.BusinessControllerTests$_setUp_closure1) values: [com.tweetcommendation.BusinessControllerTests$_setUp_closure1@3ee545fa] 

      at org.grails.datastore.gorm.finders.DynamicFinder.createFinderInvocation(DynamicFinder.java:264) 

      at org.grails.datastore.gorm.finders.DynamicFinder.invoke(DynamicFinder.java:150) 

      at org.grails.datastore.gorm.finders.DynamicFinder.invoke(DynamicFinder.java:352) 

      at org.grails.datastore.gorm.GormStaticApi.methodMissing(GormStaticApi.groovy:108) 

      at com.tweetcommendation.BusinessController.index(BusinessController.groovy:14) 

      at com.tweetcommendation.BusinessControllerTests.testIndex(BusinessControllerTests.groovy:25)  

Okay, so what happened? I read through the docs, and then realized one thing: if the mocking method doesn't have any input parameters, then you still need to add the -> param.

So, the code should look like this instead:

      def void setUp() { 

           def control = mockFor(SpringSecurityService) 

           control.demand.getPrincipal { -> return new User() } 

           controller.springSecurityService = control.createMock() 

      }  

Grails - directing user to different auth pages

I'm using the Spring security plugin (which uses Spring Security 3.x), and figured out how one would show different auth pages based on the success redirection url:

           SavedRequest savedRequest = 
                     new HttpSessionRequestCache().getRequest(request, response); 
           String requestUrl = savedRequest.getRequestURL(); 
           String view = requestUrl.contains("business") ? 'auth-business' : 'auth' 

Here we are getting the SaveRequest object from the HttpSessionRequestCache(), which is a little different from how you would do it in Spring Security 2.x.

Enjoy!

Thoughts on Grails and RabbitMQ on Ubunbu

So I'm building the email notification portion of my app, and I decided to decouple the system by using the AMPQ protocol. I specifically chose RabbitMQ because I have heard good things about it, and cloud foundry (which is hosting my app) also has good support for it.

The problem:
I proceeded to go on the rabbitMQ site to find the latest version of the software. They have the download as a .deb package which I installed. The install took 45 minutes, and right there and then I should've known that something was wrong. Anyways I didn't think too much of it at that point.

Fast forward to yesterday, when I was hooking in the Grails RabbitMQ plugin with my code. From what it seems it was dead simple to send a message:

           if (isSaved) { 
                rabbitSend 'newsubscriber', '', subscriber.id 
                render new StatusResponse(status : 'success') as JSON 
           } 

Here we are calling rabbitSend (provided by the plugin), which makes an sync call to put a message in the 'newsubscriber' topic (which I configured to be a fanout topic). Seems dead simply right? Well, there were a couple problems I found.

1.) First, I checked the server logs, and immediately I saw an error with the grails plugin having problems trying to convert the message. I eventually tracked it down, and it seems that the latest version of the plugin (1.0.0-RC1) had an issue where other people experienced. To resolve that I just fetched the 0.3.3 version and used that instead. All was well (on the deployed app). That took a few hours to track down.

2.) The second problem ... I wasn't able to track down until today. For some reason, when I run the app locally, the messages were not being sent. I logged into the web console and posted messages through their console, and everything worked. So I thought and I thought, and came to the conclusion that maybe the versions of the rabbitmq that is running are different. So I went and downloaded the cloud foundry plugin which had an interface to see the deployed apps. Aha, I thought, it was the version difference...cloud foundry is running 2.4.0, and I was running 2.8.2. So I un installed it and proceeded back to the website to fetch the 2.4.0 deb package. I then tried to install package...but it just stood there...and I waited..and waited...and went for lunch...and waited..

Finally, I realized that something could be wrong with fetching those deb packages from their site, so I instead installed using the command line as they have suggested here: http://www.rabbitmq.com/install-debian.html

Install took 30 seconds, and the server was up. I tried to run my test again, and viola this time it worked!


*I'm using Ubunbtu 12.01 and updated the latest version of the package manager recently, so I'm not sure if that had anything to do with it.

Deploying to Cloud Foundry when you app uses more than 512 MB

So I was struggling a little today with trying to deploy my Grails application to Cloud Foundry. Initially I was using the Grails cloud-foundry plugin to push my app. However, I kept getting errors about exceeding the memory limit when the server tries to start up. I almost gave up on it, but then I discovered the VMC tool provided by cloud foundry themselves, which when you upload via that tool you can set the memory size limit:

Would you like to deploy from the current directory? [Yn]: y 
 Detected a Java SpringSource Grails Application, is this correct? [Yn]: y 
 Application Deployed URL [tweetcommendation.cloudfoundry.com]:  
 Memory reservation (128M, 256M, 512M, 1G, 2G) [512M]: 2G 
 How many instances? [1]: 1 

So, lesson of the day...don't rely on the Grails cloud foundry plugin...instead use the vmc tool!

Watch out for Groovy closures called from a subclass calling method in the superclass!

Okay I'll admit the title of the post is a little long, but it is really hard to describe it. I ran into this problem today and am posting about it, just in case some other poor chap comes across it.

It might be best to see the code first:

class SubClass extends SuperClass { 
   public static void main(String[] args) { 
     new SubClass().start() 
   } 
   public void start() { 
     closureFromSubClass() 
   } 
   private closureFromSubClass = { 
     closure("hello") 
   } 
 } 
 class SuperClass { 
   def closure = { str -> 
     method(str) 
   } 
   private void method(str) { 
     println (str) 
   } 
 } 

So what do you think is the output when I run this script? You would think that it'll print `hello`, right?

Well, I thought that as well, since there are no compile errors that show up in Eclipse...but instead, this is what I got:


Caught: groovy.lang.MissingMethodException: No signature of method: SubClass.method() is applicable for argument types: (java.lang.String) values: [hello] 
 Possible solutions: getAt(java.lang.String), with(groovy.lang.Closure) 
 groovy.lang.MissingMethodException: No signature of method: SubClass.method() is applicable for argument types: (java.lang.String) values: [hello] 
 Possible solutions: getAt(java.lang.String), with(groovy.lang.Closure) 
      at SuperClass$_closure1.doCall(Test.groovy:20) 
      at SuperClass.invokeMethod(Test.groovy) 
      at SubClass$_closure1.doCall(Test.groovy:13) 
      at SubClass$_closure1.doCall(Test.groovy) 
      at SubClass.start(Test.groovy:9) 
      at SubClass$start.call(Unknown Source) 
      at SubClass.main(Test.groovy:5) 

Well, it turns out that the private identifier from the method method() was the culprit. Once I removed it private access then it works. So next time when you see Groovy complaining and you are calling closures within closures from a different class, check to make sure your access modifiers are right!

Looking into the H2 database for a Grails app

Just wanted to point out that if you want to see you data in a development Grail environment, point your browser to:

http://localhost:8080//dbconsole

Before saving your entities...

Grails uses the ActiveRecord paradigm borrowed from Rails, so one can do something like:

new UserRole(user: user, role: role).save(flush: false, insert: true)  

Unlike Rails though, Grails uses Hibernate in the background, and unfortunately you can get yourself into a heap of Hibernate nastyness (like what happened to me) if your User object above isn't a persistent entity (ie. has no ID assigned to it). I found that the hard way after debugging deep into the Hibernate code yesterday =(

I'm back!!!

I'm back again! After a long hibernation with a paid project out of the way, I can finally work some new things that I've always wanted to try. In the next couple of weeks, I am going to chronicle my crusade with the Grails framework and document any challenges that came my way.

The idea is Tweetcommendation, where businesses can post a link for their customers to click on, which gets redirected on to Tweetcommednation. From there we will pre-populate the templates that users can select which will post to their twitter with a back link to the business that they want to give thanks to.