How to keep session in HttpBuilder with cookies

In my real-world scenario I have a REST service for AJAX purposes. It renders data series for graphs. I want to test it with groovy's excellent HttpBuilder. There is a problem though - these requests are only available for already logged in users.

In this post I present a complete solution to maintain a session state between HttpBuilder's requests.

Session in HttpBuilder

First of all a quick reminder about session. Session is a simulation of state for HTTP requests, which are stateless by its nature. Once you log in you receive a unique cookie (one or more) that identifies you for sequential requests. Every time you send request you send this cookie along. This way server recognizes you and matches you to your session, which is kept on server. Cookie gets invlid once you log out or it times out, for example after 20 minutes of inactivity. Next time you visit a page you get a new, unique cookie.

In order to keep session alive in HttpBuilder I need to:

  1. log in to my Grails application
  2. receive a JSESSIONID cookie in response
  3. store that cookie and send it along with every subsenquential request

I've created RestConnector class that wraps up HttpBuilder. It's main improvement is that it keeps received cookie in a list.

package eu.spoonman.connectors.RestConnector

import groovyx.net.http.Method
import groovyx.net.http.ContentType
import groovyx.net.http.HTTPBuilder
import groovyx.net.http.HttpResponseDecorator

class RestConnector {
private String baseUrl
private HTTPBuilder httpBuilder
private List<String> cookies

RestConnector(String url) {
this.baseUrl = url
this.httpBuilder = initializeHttpBuilder()
this.cookies = []
}

public def request(Method method, ContentType contentType, String url, Map<String, Serializable> params) {
debug("Send $method request to ${this.baseUrl}$url: $params")
httpBuilder.request(method, contentType) { request ->
uri.path = url
uri.query = params
headers['Cookie'] = cookies.join(';')
}
}

private HTTPBuilder initializeHttpBuilder() {
def httpBuilder = new HTTPBuilder(baseUrl)

httpBuilder.handler.success = { HttpResponseDecorator resp, reader ->
resp.getHeaders('Set-Cookie').each {
//[Set-Cookie: JSESSIONID=E68D4799D4D6282F0348FDB7E8B88AE9; Path=/frontoffice/; HttpOnly]
String cookie = it.value.split(';')[0]
debug("Adding cookie to collection: $cookie")
cookies.add(cookie)
}
debug("Response: ${reader}")
return reader
}
return httpBuilder
}


private debug(String message) {
System.out.println(message) //for Gradle
}
}

A few things to notice in a class above. Constructor sets base URL and creates HttpBuilder instance that can be reused. Next, there is a handler on successful request that checks if I receive any cookie. It adds received cookies to list. Finally, there is a request method that calls HttpBuilder#request but it adds cookies to HTTP headers so server can recognize me as a logged in user.

Sending cookies with every request is a core component in here. It simulates browser's behavior and maintains session.

How to use it?

I will show you how to use this utility class it in Spock test below. It is fairly simple.

First I login to my application and I ensure that I receive a cookie in return, which is equivalent to being logged in. Then I send a request with that cookie sent in HTTP header. This is a Spock test that implements it:

package eu.spoonman.specs.rest

import eu.spoonman.connectors.RestConnector.RestConnector
import groovyx.net.http.ContentType
import groovyx.net.http.Method
import spock.lang.Shared
import spock.lang.Specification
import spock.lang.Stepwise

@Stepwise
class RestChartSpec extends Specification {
@Shared
RestConnector restConnector

def setupSpec() {
restConnector = new RestConnector('http://localhost:8080')
}

def "should login as test"() {
given:
Map params = [j_username: 'test', j_password: 'test']
when:
restConnector.request(Method.POST, ContentType.ANY, '/frontoffice/j_spring_security_check', params)
then:
!(restConnector.cookies.empty)
}

def "should allow access to chart data series"() {
given:
Map params = [days: 14]
when:
Map result = restConnector.request(Method.POST, ContentType.JSON, "frontoffice/chart/series", params)
then:
result != null
result.series.size() > 0
}
}

I create a new RestConnector instance in setupSpec with my application's base URL. Please notice that it has @Shared annotation so it's shared between tests.

@Stepwise is crucial annotation for this specification. It means that Spock executes tests exactly in order they're defined. I need to ensure that login is executed first. I also need to assert that I receive a cookie and list is not empty. I could move this step into setupSpec method too, but I prefer it to be a first test in a specification.

Second test is always executed after login thus it sends cookies within request headers. This is exactly what I wanted to achieve.

How to keep session in HttpBuilder with cookies

In my real-world scenario I have a REST service for AJAX purposes. It renders data series for graphs. I want to test it with groovy's excellent HttpBuilder. There is a problem though - these requests are only available for already logged in users.

In this post I present a complete solution to maintain a session state between HttpBuilder's requests.

Session in HttpBuilder

First of all a quick reminder about session. Session is a simulation of state for HTTP requests, which are stateless by its nature. Once you log in you receive a unique cookie (one or more) that identifies you for sequential requests. Every time you send request you send this cookie along. This way server recognizes you and matches you to your session, which is kept on server. Cookie gets invlid once you log out or it times out, for example after 20 minutes of inactivity. Next time you visit a page you get a new, unique cookie.

In order to keep session alive in HttpBuilder I need to:

  1. log in to my Grails application
  2. receive a JSESSIONID cookie in response
  3. store that cookie and send it along with every subsenquential request

I've created RestConnector class that wraps up HttpBuilder. It's main improvement is that it keeps received cookie in a list.

package eu.spoonman.connectors.RestConnector

import groovyx.net.http.Method
import groovyx.net.http.ContentType
import groovyx.net.http.HTTPBuilder
import groovyx.net.http.HttpResponseDecorator

class RestConnector {
private String baseUrl
private HTTPBuilder httpBuilder
private List<String> cookies

RestConnector(String url) {
this.baseUrl = url
this.httpBuilder = initializeHttpBuilder()
this.cookies = []
}

public def request(Method method, ContentType contentType, String url, Map<String, Serializable> params) {
debug("Send $method request to ${this.baseUrl}$url: $params")
httpBuilder.request(method, contentType) { request ->
uri.path = url
uri.query = params
headers['Cookie'] = cookies.join(';')
}
}

private HTTPBuilder initializeHttpBuilder() {
def httpBuilder = new HTTPBuilder(baseUrl)

httpBuilder.handler.success = { HttpResponseDecorator resp, reader ->
resp.getHeaders('Set-Cookie').each {
//[Set-Cookie: JSESSIONID=E68D4799D4D6282F0348FDB7E8B88AE9; Path=/frontoffice/; HttpOnly]
String cookie = it.value.split(';')[0]
debug("Adding cookie to collection: $cookie")
cookies.add(cookie)
}
debug("Response: ${reader}")
return reader
}
return httpBuilder
}


private debug(String message) {
System.out.println(message) //for Gradle
}
}

A few things to notice in a class above. Constructor sets base URL and creates HttpBuilder instance that can be reused. Next, there is a handler on successful request that checks if I receive any cookie. It adds received cookies to list. Finally, there is a request method that calls HttpBuilder#request but it adds cookies to HTTP headers so server can recognize me as a logged in user.

Sending cookies with every request is a core component in here. It simulates browser's behavior and maintains session.

How to use it?

I will show you how to use this utility class it in Spock test below. It is fairly simple.

First I login to my application and I ensure that I receive a cookie in return, which is equivalent to being logged in. Then I send a request with that cookie sent in HTTP header. This is a Spock test that implements it:

package eu.spoonman.specs.rest

import eu.spoonman.connectors.RestConnector.RestConnector
import groovyx.net.http.ContentType
import groovyx.net.http.Method
import spock.lang.Shared
import spock.lang.Specification
import spock.lang.Stepwise

@Stepwise
class RestChartSpec extends Specification {
@Shared
RestConnector restConnector

def setupSpec() {
restConnector = new RestConnector('http://localhost:8080')
}

def "should login as test"() {
given:
Map params = [j_username: 'test', j_password: 'test']
when:
restConnector.request(Method.POST, ContentType.ANY, '/frontoffice/j_spring_security_check', params)
then:
!(restConnector.cookies.empty)
}

def "should allow access to chart data series"() {
given:
Map params = [days: 14]
when:
Map result = restConnector.request(Method.POST, ContentType.JSON, "frontoffice/chart/series", params)
then:
result != null
result.series.size() > 0
}
}

I create a new RestConnector instance in setupSpec with my application's base URL. Please notice that it has @Shared annotation so it's shared between tests.

@Stepwise is crucial annotation for this specification. It means that Spock executes tests exactly in order they're defined. I need to ensure that login is executed first. I also need to assert that I receive a cookie and list is not empty. I could move this step into setupSpec method too, but I prefer it to be a first test in a specification.

Second test is always executed after login thus it sends cookies within request headers. This is exactly what I wanted to achieve.

Grails render as JSON catch

One of a reasons your controller doesn't render a proper response in JSON format might be wrong package name that you use. It is easy to overlook. Import are on top of a file, you look at your code and everything seems to be fine. Except response is still not in JSON format.

Consider this simple controller:

class RestJsonCatchController {
def grailsJson() {
render([first: 'foo', second: 5] as grails.converters.JSON)
}

def netSfJson() {
render([first: 'foo', second: 5] as net.sf.json.JSON)
}
}

And now, with finger crossed... We have a winner!

$ curl localhost:8080/example/restJsonCatch/grailsJson
{"first":"foo","second":5}
$ curl localhost:8080/example/restJsonCatch/netSfJson
{first=foo, second=5}

As you can see only grails.converters.JSON converts your response to JSON format. There is no such converter for net.sf.json.JSON, so Grails has no converter to apply and it renders Map normally.

Conclusion: always carefully look at your imports if you're working with JSON in Grails!

Edit: Burt suggested that this is a bug. I've submitted JIRA issue here: GRAILS-9622 render as class that is not a codec should throw exception

Inconsistent Dependency Injection to domains with Grails

I've encountered strange behavior with a domain class in my project: services that should be injected were null. I've became suspicious as why is that? Services are injected properly in other domain classes so why this one is different?

Constructors experiment

I've created an experiment. I've created empty LibraryService that should be injected and Book domain class like this:

Book has 4 explicit constructors. I want to check which constructor is injecting dependecies. This is my method that constructs Book objects and I called it in controller:

class BookController {
def index() {
constructAndExamineBooks()
}

static constructAndExamineBooks() {
println("Started constructAndExamineBooks")
Book book1 = new Book().logInjectedService()
Book book2 = new Book("foo").logInjectedService()
Book book3 = new Book("foo", 'bar').logInjectedService()
Book book4 = new Book("foo", 'bar', 100).logInjectedService()
Book book5 = new Book(author: "foo", title: 'bar')
println("Finished constructor Book(Map params)")
book5.logInjectedService()
}
}

Analysis

Output looks like this:

What do we see?

  1. Empty constructor injects dependencies.
  2. Constructor that invokes empty constructor explicitly injects dependencies.
  3. Constructor that invokes parent's constructor explicitly does not inject dependencies.
  4. Constructor without any explicit call declared does not call empty constructor thus it does not inject dependencies.
  5. Constructor provied by Grails with a map as a parameter invokes empty constructor and injects dependencies.

Conclusion

Always explicitily invoke empty constructor in your Grail domain classes to ensure Dependency Injection! I didn't know until today either!

How to use mocks in controller tests

Even since I started to write tests for my Grails application I couldn't find many articles on using mocks. Everyone is talking about tests and TDD but if you search for it there isn't many articles.

Today I want to share with you a test with mocks for a simple and complete scenario. I have a simple application that can fetch Twitter tweets and present it to user. I use REST service and I use GET to fetch tweets by id like this: http://api.twitter.com/1/statuses/show/236024636775735296.json. You can copy and paste it into your browser to see a result.

My application uses Grails 2.1 with spock-0.6 for tests. I have TwitterReaderService that fetches tweets by id, then I parse a response into my Tweet class.

TwitterController plays main part here. Users call show action along with id of a tweet. This action is my subject under test. I've implemented some basic functionality. It's easier to focus on it while writing tests.

Let's start writing a test from scratch. Most important thing here is that I use mock for my TwitterReaderService. I do not construct new TwitterReaderService(), because in this test I test only TwitterController. I am not interested in injected service. I know how this service is supposed to work and I am not interested in internals. So before every test I inject a twitterReaderServiceMock into controller:

Now it's time to think what scenarios I need to test. This line from TwitterReaderService is the most important:

You must think of this method like a black box right now. You know nothing of internals from controller's point of view. You're only interested what can be returned for you:

  • a TwitterError can be thrown
  • null can be returned
  • Tweet instance can be returned

This list is your test blueprint. Now answer a simple question for each element: "What do I want my controller to do in this situation?" and you have plan test:

  • show action should redirect to index if TwitterError is thrown and inform about error
  • show action should redirect to index and inform if tweet is not found
  • show action should show found tweet

That was easy and straightforward! And now is the best part: we use twitterReaderServiceMock to mock each of these three scenarios!

In Spock there is a good documentation about interaction with mocks. You declare what methods are called, how many times, what parameters are given and what should be returned. Remember a black box? Mock is your black box with detailed instruction, e.g.: I expect you that if receive exactly one call to readTweet with parameter '1' then you should throw me a TwitterError. Rephrase this sentence out loud and look at this:

This is a valid interaction definition on mock! It's that easy! Here is a complete test that fails for now:

You may notice 0 * _._ notation. It says: I don't want any other mocks or any other methods called. Fail this test if something is called! It's a good practice to ensure that there are no more interactions than you want.

Ok, now I need to implement controller logic to handle TwitterError.

My tests passes! We have two scenarios left. Rule stays the same: TwitterReaderService returns something and we test against it. So this line is the heart of each test, change only returned values after >>:

Here is a complete test for three scenarios and controller that passes it.

The most important thing here is that we've tested controller-service interaction without logic implementation in service! That's why mock technique is so useful. It decouples your dependencies and let you focus on exactly one subject under test. Happy testing!

Hibernate Envers with Grails 2.1.0

Our client requires that every entity in his Grails application has to be audited with detailed information: who and when did exactly what? It is a perfect opportunity to use Hibernate Envers! It turned out not as straightforward as I thought, so I want to share my experience with you.

Introduction

I use latest release of Grails with version 2.1.0. I've created a sample application on github for you as an example - SpOOnman/enversdemo. All files and techniques mentioned here can be found inside it. If you have other experiences with Grails and Envers working together, please share in comments.

Grails Envers plugin

First I've searched for a Grails plugin. I've found one outdated on google code and a (more modern) on github. This plugin is created by Lucas Ward and it has great introductionary blog post by author. This is a great place to start.

Unfortunately Lucas Ward's plugin supports Grails up 1.3.7 and it comes with outdated Envers version. I've searched through forks and I've found most up-to-date fork by Matija Folnovic: https://github.com/mfolnovic/grails-envers-plugin. It supports Grails 2.1 out of the box. Edit: Matija pointed out that credit should go to Jay Hogan and Damir Murat for their work on upgrading Envers plugin to Grails 2.

Matija's plugin is not packaged nor published so you have to package it by yourself:

Copy grails-envers-0.3-SNAPSHOT.zip (and rename to envers-0.3-SNAPSHOT.zip) to your application's lib diretory or publish to your company maven repository if you have one. Last step is to include a plugin in your BuildConfig.groovy:

Audit with Envers!

It's really easy to audit all entities with Envers. All you have to do is to use @Audit annotation on entities. SpOOnman/enversdemo audits Hotels and Bookings:

Envers plugin comes with some handy methods. For example Hotel.findAllRevisions() returns list of previous revisions. Once again I recommend a great blog post by Lucas Ward with many examples.

Grails and transactions

Grails has some glitches working with Envers. Envers is closely tied to database transactions. This means that it runs its listeners at the transaction end, when session is flushed. This is not always the case in Grails and it's the cause of some problems. You need to make some small modifications to your application to be sure that Envers works fine.

  • controllers - you need to annotate your controllers (or separate controller methods) with @Transactional and for every save you need to flush a session manually, e.g. save(flush: true). Otherwise Envers may not be notified.
  • services in Grails are transactional by default. However you mey explicitily set them with static transactional = true to be sure that this is not altered. I recommend to use session flushing on every save as well.
  • BootStrap.groovy script is not transactional. If you want Envers to audit your changes in this script you need to wrap your inserts with transactions. You can achieve it by using withTransaction closure. You need session flushing too. Take a look at SpOOnman/enversdemo BootStrap.groovy for an example.
  • integration tests wrap every test in a separate transaction that is rolled back at the end of a test, hence Envers won't work here. You need to disable wrapped transactions with static transactional = false. Remember that your changes are commited to database. To fulfill a contract of an empty database for every test you need to take care of cleaning up a database manually. Here is my sample integration test.

Add User to revision entities

By default Envers comes with a DefaultRevisionEntity class and creates its instance for every change on audited instance. It keeps information about entity revision, date and modification type. It lacks information about a user that made a change. I want to extend it.

There is already a code inside a plugin (here and here) but it's not packaged with a plugin itself. I guess it's because it adds dependency to Spring Security. This not a problem for me since I already use Spring Security plugin already in my application. I decided to add missing classes inside my application.

First I need a domain class - UserRevisionEntity. New instance of this class is created for every revision saved by Envers. It is a simple domain class with some extra annotations:

Notice that I've made currentUser field nullable. There may be some actions that doesn't require user to be logged in. Also, there is no user while executing BootStrap.groovy script.

Annotations @RevisionNumber and @RevisionTimestamp are required by Envers as described in documentation. @RevisionEntity annotation configures Envers to use non-default listener - SpringSecurityRevisionListener. It looks simple:

SpringSecurityRevisionListener's newRevision method fills each UserRevisionEntity instances with currently logged User.

Configuration

Envers offers some configuration options listed on documentation page. Options can be set in persistence.xml. However in Grails there is no persistence.xml. You can add it (and Grails supports it), but I've found other way to configure properties. I use System.setProperties to configure Envers and I place it in grails-app/conf/spring/resources.groovy. This is an example to change Envers tables' prefix and suffix:

Testing

Testing Envers makes sense only with Grails integration tests. As I've mentioned earlier you must disable transactions that wrap your tests in order for Envers listeners to work. Also make sure you use withTransaction, session flushing and you've marked your controllers as @Transactional. There rules all stand in integration tests too. Example integration test can look like this:

Summary

Hibernate Envers does a great job auditing my application. It is easy to use, despite all the small glitches I've mentioned here. Special thanks to plugin authors that package Envers library in a plugin that can be used out of the box. I really recommend it! It would be great to hear all your thoughts on Envers and Grails working together so I can improve this post for others too.

Update 12.04.2013

Lucas Ward has posted updaate entry on Grails Hibernate Envers plugin here: http://www.lucasward.net/2013/04/grails-envers-plugin-update.html. Thanks!

Private fields and methods are not private in groovy

I used to code in Java before I met groovy. Like most of you, groovy attracted me with many enhancements. This was to my surprise to discover that method visibility in groovy is handled different than Java!

Consider this example:

class Person {
private String name
public String surname

private Person() {}

private String signature() { "${name?.substring(0, 1)}. $surname" }

public String toString() { "I am $name $surname" }
}

How is this class interpreted with Java?

  1. Person has private constructor that cannot be accessed
  2. Field "name" is private and cannot be accessed
  3. Method signature() is private and cannot be accessed

Let's see how groovy interpretes Person:

public static void main(String[] args) {
def person = new Person() // constructor is private - compilation error in Java
println(person.toString())

person.@name = 'Mike' // access name field directly - compilation error in Java
println(person.toString())

person.name = 'John' // there is a setter generated by groovy
println(person.toString())

person.@surname = 'Foo' // access surname field directly
println(person.toString())

person.surname = 'Bar' // access auto-generated setter
println(person.toString())

println(person.signature()) // call private method - compilation error in Java
}

I was really astonished by its output:

I am null null
I am Mike null
I am John null
I am John Foo
I am John Bar
J. Bar

As you can see, groovy does not follow visibility directives at all! It treats them as non-existing. Code compiles and executes fine. It's contrary to Java. In Java this code has several errors, pointed out in comments.

I've searched a bit on this topic and it seems that this behaviour is known since version 1.1 and there is a bug report on that: http://jira.codehaus.org/browse/GROOVY-1875. It is not resolved even with groovy 2 release. As Tim Yates mentioned in this Stackoverflow question: "It's not clear if it is a bug or by design". Groovy treats visibility keywords as a hint for a programmer.

I need to keep that lesson in mind next time I want to make some field or method private!