DDevelopment & Design

Confitura 2013 afterthoughts

Confitura, the biggest free-of-charge Java conference in Europe, took place on the 6th of July in Warsaw. TouK’s presence was heavy, with 5 separate talks, all chosen in call for papers, no sponsored bullshit. We were sponsoring deck chairs during the…Confitura, the biggest free-of-charge Java conference in Europe, took place on the 6th of July in Warsaw. TouK’s presence was heavy, with 5 separate talks, all chosen in call for papers, no sponsored bullshit. We were sponsoring deck chairs during the…
Confitura, the biggest free-of-charge Java conference in Europe, took place on the 6th of July in Warsaw. TouK’s presence was heavy, with 5 separate talks, all chosen in call for papers, no sponsored bullshit. We were sponsoring deck chairs during the conference, and beer on the after-party, though. Both were quite popular. Oh, and we had Bartek Zdanowski with TouK Lab, doing funny things with lights, Jenkins, and raspberry-pi.

Last year, together with Tomasz Przybysz, I had a talk about Groovy and Grails. This year I had the pleasure of presenting revised Test Driven Traps.

Now, this wasn’t the same talk I had in Tallinn. In fact, I rewrote most of it, with three major changes.

While at first I was focused on three goals of TDD (security, design, feedback), I’ve noticed that I underestimate the value of communication. The funny thing is, I do use TDD in that manner frequently, it just didn’t appear to me as very important. Which, as an afterthought, was a big mistake.

Perhaps the reason for it, is how people distinguish between TDD, BDD (Behaviour Driven Development) and ATDD (Acceptance Test-Driven Development). I use Spock most of the time, and Spock mixes all three approaches very well. To be quite honest, I’ve been mixing all three of them in every testing tool I ever had. Whether my tests looks more like BDD or ATDD (that is, had more emphasis on communication and specification) depends on the nature of the project, and the nature of my client. Technology is not a problem here. It just used to be more awkward to write down user stories as a tests, in languages which were not as flexible as Groovy.

I had a lot of success using tests as a communication tool. Both internally, with developers, including myself while returning to an old code base, and with external clients. But I also had some failures.

A few months ago, while doing a mobile banking project, I had a session with a domain expert. We were trying to get to the bottom of fees and pricing in a specific payment scenario. While we were siting and talking, I wrote down a simple specification

static Closure fixed = {BigDecimal value -> new FixedFeeAddedCalculator(value: value)}
static Closure percentage = {BigDecimal value -> new PercentageFeeCalculator(value: value)}

def "should calculate fees"() {
    given:
        biller.feeCalculator = billerFeeType.call(billerFee)
        biller.feesForChannels = [new FeePerChannel(channel: Channel.USSD, feeCalculator: channelFeeType.call(feeForChannel))]
        biller.feesForPrefixes = [new FeePerPrefix(msisdnPrefix: "123", feeCalculator: prefixFeeType.call(feeForPrefix))]
    when:
        BillerFees billerFees = biller.calculateFees(productPrice, "1234", Channel.USSD)
    then:
        billerFees.total + productPrice == totalMoneyPaid
        billerFees.feeForChannel == feeForChannelPaid
        billerFees.feeForPrefix == feeForPrefixPaid
    where:
        productPrice | feeForChannel | channelFeeType | feeForPrefix | prefixFeeType | billerFee | billerFeeType || totalMoneyPaid | feeForChannelPaid | feeForPrefixPaid | feeForBillerPaid
        100          | 7             | fixed          | 3            | fixed         | 10        | fixed         || 120            | 7                 | 3                | 10
        100          | 7             | percentage     | 3            | percentage    | 10        | percentage    || 120            | 7                 | 3                | 10
        123.4        | 10            | percentage     | 10           | percentage    | 10        | percentage    || 160.42         | 12.34             | 12.34            | 12.34
        123.45       | 10            | percentage     | 10           | percentage    | 10        | percentage    || 160.47         | 12.34             | 12.34            | 12.34
        123.45       | 10.05         | percentage     | 10.05        | percentage    | 10.05     | percentage    || 160.68         | 12.41             | 12.41            | 12.41
        100.99       | 0.99          | percentage     | 9.99         | percentage    | 0.09      | percentage    || 112.17         | 1                 | 10.09            | 1
        100.99       | 0             | percentage     | 0            | fixed         | 0.01      | percentage    || 101.00         | 0                 | 0                | 0.01
        10           | 0             | fixed          | 0            | fixed         | 100       | fixed         || 110.00         | 0                 | 0                | 100
}

Those static fields/closures are a little hack, so that I’m able to use something more readable in the ‘where’ clause, than a ‘new’ with the name of the class. After we confirmed, that it looks OK, but we don’t really know whether this is accurate, I just copied the ‘where’ part with a little description, and sent it to the bank for confirmation. It looked pretty much like what the bank likes to send with their requirements. The expert was happy with this, I was happy with this, we’ve had that working for us before.

No word about fees and pricing has been heard from them since. And that’s about six months now.
Which shows, that no matter how good your tools are, no matter how smart domain experts are, if you have a fundamental problem with communication with your client, you are doomed anyway. And working through a proxy with clients from another continent, all I have are problems in communication. The technology is simple and easy, compared to that.

But I digress, I do encourage you to think of tests as a communication tool. It’s a blast when it works. And it does help a lot with keeping our domain knowledge up to date. Something requirement documents fail miserably to do.

The thing I do not know, is whether it is better to keep specifications and user stories separate from the rest of your tests or not. I’ve tried both approaches, but I don’t have any conclusion yet.

I also added a few simple thought on domain setup, to my presentation. Most systems need some domain objects before even unit tests will run. You can create that step by step, calling each constructor in every test, but this soon becomes a DRY violation, and a pain in the ass, as the domain gets more complex. Not to mention refactoring problems. A simple factory helps, but doesn’t change that much, so in my experience, the best way it to have your aggregate roots built in a single class, preferably using the same builders you use in production code (if you use any). This kind of setup definitely simplifies writing new tests, and is trivial to refactor.

But that’s most useful for unit tests. For integration tests, there are three approaches I’m aware of. The first one is the ‘go crazy, copy data from production’. Fortunately, this is possible only if you join an existing project, that had no tests before, so I’ll just skip that, as I am inherently incompatible with such situations. The other two are more practical.

You can have an empty database (you can even create it on the fly), where every test has to provide the data (domain objects) required for itself, or you can have a prepopulated database, with a precise set of data, representing some sample domain objects, and have that reflected in your test code in form of static finals. In both approaches, a rollback (for transactional Dbs) or a manual clean-up (for everything else) keeps the database at the initial state.

Everybody seems to prefer an empty database. But a prepopulated database has two great advantages: it usually simplifies test setups, and may have big impact on test performance. Just a few weeks ago I switched one of our projects to use the prepopulated database, and saved 15 seconds of test time.
15 seconds may not look like much, but our integration tests run now in 1m30s. I’ve saved another 2 seconds, by removing an ugly sleep in one test, and I can save a few more seconds, by just not being wasteful.

And that brings me to the next change in my presentation. I’ve pointed out the importance of performance in TDD. You have 30 seconds for unit tests, before you loose the patience of a developer. Make your tests run more than 3 minutes, and devs will only run them before push to the repo. Make them even longer, and nobody’s gonna care about tests anymore.

This is a single reason, why I thing GORM (Grails object-relational mapping) is not good enough for bigger projects. Grails use Active Record pattern, with a nifty mocking of database access methods via @Mock annotation, for unit tests. But with a complex domain and large aggregate roots, you get a heavy performance hit because of those annotations, and you end up with unit tests having half the speed of integration tests.

And that’s pretty bad. I have ~900 unit tests running in about 2.5 minutes, on i7. And most of that is due to @Mock annotations. I could bring it down to less than 30 seconds, if I could mock those bastard database calls the normal way, but with active record, I would either have to do meta-programming, or go into mixins and categories. The first option is error-prone, the second very cumbersome. Both are probably inefficient. Comparing the amount of work required to make those unit tests fast, with mocking a simple repository interface, puts all the other benefits of Grails active record into question. My conclusion is to not use GORM, if I’m going for anything more than 300 man-days. Or not use Grails altogether.

Those conclusions are kind of obvious, I suppose. The reason why I don’t see them used too often, is because people do not treat tests as a normal object (or functional) oriented code. That at least was my excuse for years and years, till I finally found out, this approach ends up in unmaintainable test code, that keeps you production code from ever being refactored as well.

And that’s pretty much all of my presentation changes.

Confitura 2013 was a fantastic experience. I am usually scared shitless, when giving a talk at a conference. Somehow, maybe because I know a lot of people on the audience personally, I’m less scared in here. During the first half of this year, I gave 6 talks, led two workshops, helped organize Warsaw Java User Group, and wrote a book. Confitura was a great finish for that marathon. Now I need a summer break.

Solution Architect @ TouK

— Previous article

Confitura – byliśmy, wróciliśmy, wspominamy

Next article —

<!--:pl-->Commitment in Scrum<!--:--><!--:en-->Commitment in Scrum<!--:-->

You May Also Like
DDevelopment & Design

Integration testing custom validation constraints in Jersey 2

I recently joined a team trying to switch a monolithic legacy system into set of RESTful services in Java. They decided to use latest 2.x version of Jersey as a REST container which was not a first choice for me, since I’m not a big fan of JSR-* specs. But now I must admit that JAX-RS 2.x is doing things right: requires almost zero boilerplate code, support auto-discovery of features and prefers convention over configuration like other modern frameworks. Since the spec is still young, it’s hard to find good tutorials and kick-off projects with some working code. I created jersey2-starter project on GitHub which can be used as starting point for your own production-ready RESTful service. In this post I’d like to cover how to implement and integration test your own validation constraints of REST resources.

Custom constraints

One of the issues which bothers me when coding REST in Java is littering your class model with annotations. Suppose you want to build a simple Todo list REST service, when using Jackson, validation and Spring Data, you can easily end up with this as your entity class:

@Document
public class Todo {
    private Long id;
    @NotNull
    private String description;
    @NotNull
    private Boolean completed;
    @NotNull
    private DateTime dueDate;

    @JsonCreator
    public Todo(@JsonProperty("description") String description, @JsonProperty("dueDate") DateTime dueDate) {
        this.description = description;
        this.dueDate = dueDate;
        this.completed = false;
    }
    // getters and setters
}

Your domain model is now effectively blured by messy annotations almost everywhere. Let’s see what we can do with validation constraints (@NotNulls). Some may say that you could introduce some DTO layer with own validation rules, but it conflicts for me with pure REST API design, which stands that you operate on resources which should map to your domain classes. On the other hand - what does it mean that Todo object is valid? When you create a Todo you should provide a description and due date, but what when you’re updating? You should be able to change any of description, due date (postponing) and completion flag (marking as done) - but you should provide at least one of these as valid modification. So my idea is to introduce custom validation constraints, different ones for creation and modification: 

@Target({TYPE, PARAMETER})
@Retention(RUNTIME)
@Constraint(validatedBy = ValidForCreation.Validator.class)
public @interface ValidForCreation {
    //...
    class Validator implements ConstraintValidator<ValidForCreation, Todo> {
    /...
        @Override
        public boolean isValid(Todo todo, ConstraintValidatorContext constraintValidatorContext) {
            return todo != null
                && todo.getId() == null
                && todo.getDescription() != null
                && todo.getDueDate() != null;
        }
    }
}

@Target({TYPE, PARAMETER})
@Retention(RUNTIME)
@Constraint(validatedBy = ValidForModification.Validator.class)
public @interface ValidForModification {
    //...
    class Validator implements ConstraintValidator<ValidForModification, Todo> {
    /...
        @Override
        public boolean isValid(Todo todo, ConstraintValidatorContext constraintValidatorContext) {
            return todo != null
                && todo.getId() == null
                && (todo.getDescription() != null || todo.getDueDate() != null || todo.isCompleted() != null);
        }
    }
}

And now you can move validation annotations to the definition of a REST endpoint:

@POST
@Consumes(APPLICATION_JSON)
public Response create(@ValidForCreation Todo todo) {...}

@PUT
@Consumes(APPLICATION_JSON)
public Response update(@ValidForModification Todo todo) {...}

And now you can remove those NotNulls from your model.

Integration testing

There are in general two approaches to integration testing:

  • test is being run on separate JVM than the app, which is deployed on some other integration environment
  • test deploys the application programmatically in the setup block.

Both of these have their pros and cons, but for small enough servoces, I personally prefer the second approach. It’s much easier to setup and you have only one JVM started, which makes debugging really easy. You can use a generic framework like Arquillian for starting your application in a container environment, but I prefer simple solutions and just use emdedded Jetty. To make test setup 100% production equivalent, I’m creating full Jetty’s WebAppContext and have to resolve all runtime dependencies for Jersey auto-discovery to work. This can be simply achieved with Maven resolved from Shrinkwrap - an Arquillian subproject: 

    WebAppContext webAppContext = new WebAppContext();
    webAppContext.setResourceBase("src/main/webapp");
    webAppContext.setContextPath("/");
    File[] mavenLibs = Maven.resolver().loadPomFromFile("pom.xml")
                .importCompileAndRuntimeDependencies()
                .resolve().withTransitivity().asFile();
    for (File file: mavenLibs) {
        webAppContext.getMetaData().addWebInfJar(new FileResource(file.toURI()));
    }
    webAppContext.getMetaData().addContainerResource(new FileResource(new File("./target/classes").toURI()));

    webAppContext.setConfigurations(new Configuration[] {
        new AnnotationConfiguration(),
        new WebXmlConfiguration(),
        new WebInfConfiguration()
    });
    server.setHandler(webAppContext);

(this Stackoverflow thread inspired me a lot here)

Now it’s time for the last part of the post: parametrizing our integration tests. Since we want to test validation constraints, there are many edge paths to check (and make your code coverage close to 100%). Writing one test per each case could be a bad idea. Among the many solutions for JUnit I’m most convinced to the Junit Params by Pragmatists team. It’s really simple and have nice concept of JQuery-like helper for creating providers. Here is my tests code (I’m also using builder pattern here to create various kinds of Todos):

@Test
@Parameters(method = "provideInvalidTodosForCreation")
public void shouldRejectInvalidTodoWhenCreate(Todo todo) {
    Response response = createTarget().request().post(Entity.json(todo));

    assertThat(response.getStatus()).isEqualTo(BAD_REQUEST.getStatusCode());
}

private static Object[] provideInvalidTodosForCreation() {
    return $(
        new TodoBuilder().withDescription("test").build(),
        new TodoBuilder().withDueDate(DateTime.now()).build(),
        new TodoBuilder().withId(123L).build(),
        new TodoBuilder().build()
    );
}

OK, enough of reading, feel free to clone the project and start writing your REST services!

I recently joined a team trying to switch a monolithic legacy system into set of RESTful services in Java. They decided to use latest 2.x version of Jersey as a REST container which was not a first choice for me, since I’m not a big fan of JSR-* specs. But now I must admit that JAX-RS 2.x is doing things right: requires almost zero boilerplate code, support auto-discovery of features and prefers convention over configuration like other modern frameworks. Since the spec is still young, it’s hard to find good tutorials and kick-off projects with some working code. I created jersey2-starter project on GitHub which can be used as starting point for your own production-ready RESTful service. In this post I’d like to cover how to implement and integration test your own validation constraints of REST resources.

Custom constraints

One of the issues which bothers me when coding REST in Java is littering your class model with annotations. Suppose you want to build a simple Todo list REST service, when using Jackson, validation and Spring Data, you can easily end up with this as your entity class:

@Document
public class Todo {
    private Long id;
    @NotNull
    private String description;
    @NotNull
    private Boolean completed;
    @NotNull
    private DateTime dueDate;

    @JsonCreator
    public Todo(@JsonProperty("description") String description, @JsonProperty("dueDate") DateTime dueDate) {
        this.description = description;
        this.dueDate = dueDate;
        this.completed = false;
    }
    // getters and setters
}

Your domain model is now effectively blured by messy annotations almost everywhere. Let’s see what we can do with validation constraints (@NotNulls). Some may say that you could introduce some DTO layer with own validation rules, but it conflicts for me with pure REST API design, which stands that you operate on resources which should map to your domain classes. On the other hand - what does it mean that Todo object is valid? When you create a Todo you should provide a description and due date, but what when you’re updating? You should be able to change any of description, due date (postponing) and completion flag (marking as done) - but you should provide at least one of these as valid modification. So my idea is to introduce custom validation constraints, different ones for creation and modification: 

@Target({TYPE, PARAMETER})
@Retention(RUNTIME)
@Constraint(validatedBy = ValidForCreation.Validator.class)
public @interface ValidForCreation {
    //...
    class Validator implements ConstraintValidator<ValidForCreation, Todo> {
    /...
        @Override
        public boolean isValid(Todo todo, ConstraintValidatorContext constraintValidatorContext) {
            return todo != null
                && todo.getId() == null
                && todo.getDescription() != null
                && todo.getDueDate() != null;
        }
    }
}

@Target({TYPE, PARAMETER})
@Retention(RUNTIME)
@Constraint(validatedBy = ValidForModification.Validator.class)
public @interface ValidForModification {
    //...
    class Validator implements ConstraintValidator<ValidForModification, Todo> {
    /...
        @Override
        public boolean isValid(Todo todo, ConstraintValidatorContext constraintValidatorContext) {
            return todo != null
                && todo.getId() == null
                && (todo.getDescription() != null || todo.getDueDate() != null || todo.isCompleted() != null);
        }
    }
}

And now you can move validation annotations to the definition of a REST endpoint:

@POST
@Consumes(APPLICATION_JSON)
public Response create(@ValidForCreation Todo todo) {...}

@PUT
@Consumes(APPLICATION_JSON)
public Response update(@ValidForModification Todo todo) {...}

And now you can remove those NotNulls from your model.

Integration testing

There are in general two approaches to integration testing:

  • test is being run on separate JVM than the app, which is deployed on some other integration environment
  • test deploys the application programmatically in the setup block.

Both of these have their pros and cons, but for small enough servoces, I personally prefer the second approach. It’s much easier to setup and you have only one JVM started, which makes debugging really easy. You can use a generic framework like Arquillian for starting your application in a container environment, but I prefer simple solutions and just use emdedded Jetty. To make test setup 100% production equivalent, I’m creating full Jetty’s WebAppContext and have to resolve all runtime dependencies for Jersey auto-discovery to work. This can be simply achieved with Maven resolved from Shrinkwrap - an Arquillian subproject: 

    WebAppContext webAppContext = new WebAppContext();
    webAppContext.setResourceBase("src/main/webapp");
    webAppContext.setContextPath("/");
    File[] mavenLibs = Maven.resolver().loadPomFromFile("pom.xml")
                .importCompileAndRuntimeDependencies()
                .resolve().withTransitivity().asFile();
    for (File file: mavenLibs) {
        webAppContext.getMetaData().addWebInfJar(new FileResource(file.toURI()));
    }
    webAppContext.getMetaData().addContainerResource(new FileResource(new File("./target/classes").toURI()));

    webAppContext.setConfigurations(new Configuration[] {
        new AnnotationConfiguration(),
        new WebXmlConfiguration(),
        new WebInfConfiguration()
    });
    server.setHandler(webAppContext);

(this Stackoverflow thread inspired me a lot here)

Now it’s time for the last part of the post: parametrizing our integration tests. Since we want to test validation constraints, there are many edge paths to check (and make your code coverage close to 100%). Writing one test per each case could be a bad idea. Among the many solutions for JUnit I’m most convinced to the Junit Params by Pragmatists team. It’s really simple and have nice concept of JQuery-like helper for creating providers. Here is my tests code (I’m also using builder pattern here to create various kinds of Todos):

@Test
@Parameters(method = "provideInvalidTodosForCreation")
public void shouldRejectInvalidTodoWhenCreate(Todo todo) {
    Response response = createTarget().request().post(Entity.json(todo));

    assertThat(response.getStatus()).isEqualTo(BAD_REQUEST.getStatusCode());
}

private static Object[] provideInvalidTodosForCreation() {
    return $(
        new TodoBuilder().withDescription("test").build(),
        new TodoBuilder().withDueDate(DateTime.now()).build(),
        new TodoBuilder().withId(123L).build(),
        new TodoBuilder().build()
    );
}

OK, enough of reading, feel free to clone the project and start writing your REST services!

DDevelopment & Design

Control your bandwidth using ntop

I was looking for tool which could help me check who is using my bandwidth. Here are requirements which I want from this kind of tool:local hosts bandwidth distribution - it is helpful when you are loosing your bandwidth and don't know who abuse it in ...