Clojure web development – state of the art

It’s now more than a year that I’m getting familiar with Clojure and the more I dive into it, the more it becomes the language. Once you defeat the “parentheses fear”, everything else just makes the difference: tooling, community, good engineering practices. So it’s now time for me to convince others. In this post I’ll try to walktrough a simple web application from scratch to show key tools and libraries used to develop with Clojure in late 2015.

Note for Clojurians: This material is rather elementary and may be useful for you if you already know Clojure a bit but never did anything bigger than hello world application.

Note for Java developers: This material shows how to replace Spring, Angular, grunt, live-reload with a bunch of Clojure tools and libraries and a bit of code.

The repo with final code and individual steps is here.

Bootstrap

I think all agreed that component is the industry standard for managing lifecycle of Clojure applications. If you are a Java developer you may think of it as a Spring (DI) replacement - you declare dependencies between “components” which are resolved on “system” startup. So you just say “my component needs a repository/database pool” and component library “injects” it for you.

To keep things simple I like to start with duct web app template. It’s a nice starter component application following the 12-factor philosophy. So let’s start with it:

lein new duct clojure-web-app +example

The +example parameter tells duct to create an example endpoint with HTTP routes - this would be helpful. To finish bootstraping run lein setup inside clojure-web-app directory.

Ok, let’s dive into the code. Component and injection related code should be in system.clj file:

(defn new-system [config]
  (let [config (meta-merge base-config config)]
    (-> (component/system-map
         :app  (handler-component (:app config))
         :http (jetty-server (:http config))
         :example (endpoint-component example-endpoint))
        (component/system-using
         {:http [:app]
          :app  [:example]
          :example []}))))

In the first section you instantiate components without dependencies, which are resolved in the second section. So in this example, “http” component (server) requires “app” (application abstraction), which in turn is injected with “example” (actual routes). If your component needs others, you just can get then by names (precisely: by Clojure keywords).

To start the system you must fire a REPL - interactive environment running within context of your application:

lein repl

After seeing prompt type (go). Application should start, you can visit http://localhost:3000 to see some example page.

A huge benefit of using component approach is that you get fully reloadable application. When you change literally anything - configuration, endpoints, implementation, you can just type (reset) in REPL and your application is up-to-date with the code. It’s a feature of the language, no JRebel, Spring-reloaded needed.

Adding REST endpoint

Ok, in the next step let’s add some basic REST endpoint returning JSON. We need to add 2 dependencies in project.clj file:

:dependencies
 ...
  [ring/ring-json "0.3.1"]
  [cheshire "5.1.1"]

Ring-json adds support for JSON for your routes (in ring it’s called middleware) and cheshire is Clojure JSON parser (like Jackson in Java). Modifying project dependencies if one of the few tasks that require restarting the REPL, so hit CTRL-C and type lein repl again.

To configure JSON middleware we have to add wrap-json-body and wrap-json-response just before wrap-defaults in system.clj:

(:require 
 ...
 [ring.middleware.json :refer [wrap-json-body wrap-json-response]])

(def base-config
   {:app {:middleware [[wrap-not-found :not-found]
                      [wrap-json-body {:keywords? true}]
                      [wrap-json-response]
                      [wrap-defaults :defaults]]

And finally, in endpoint/example.clj we must add some route with JSON response:

(:require 
 ...
 [ring.util.response :refer [response]]))

(defn example-endpoint [config]
  (routes
    (GET "/hello" [] (response {:hello "world"}))
    ...

Reload app with (reset) in REPL and test new route with curl:

curl -v http://localhost:3000/hello

< HTTP/1.1 200 OK
< Date: Tue, 15 Sep 2015 21:17:37 GMT
< Content-Type: application/json; charset=utf-8
< Set-Cookie: ring-session=37c337fb-6bbc-4e65-a060-1997718d03e0;Path=/;HttpOnly
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Content-Length: 151
* Server Jetty(9.2.10.v20150310) is not blacklisted
< Server: Jetty(9.2.10.v20150310)
<
* Connection #0 to host localhost left intact
{"hello": "world"}

It works! In case of any problems you can find working version in this commit.

Adding frontend with figwheel

Coding backend in Clojure is great, but what about the frontend? As you may already know, Clojure could be compiled not only to JVM bytecode, but also to Javascript. This may sound familiar if you used e.g. Coffeescript. But ClojureScript philosophy is not only to provide some syntax sugar, but improve your development cycle with great tooling and fully interactive development. Let’s see how to achieve it.

The best way to introduce ClojureScript to a project is figweel. First let’s add fighweel plugin and configuration to project.clj:

:plugins
   ...
   [lein-figwheel "0.3.9"]

And cljsbuild configuration:

:cljsbuild
    {:builds [{:id "dev"
               :source-paths ["src-cljs"]
               :figwheel true
               :compiler {:main       "clojure-web-app.core"
                          :asset-path "js/out"
                          :output-to  "resources/public/js/clojure-web-app.js"
                          :output-dir "resources/public/js/out"}}]}

In short this tells ClojureScript compiler to take sources from src-cljs with figweel support and put resulting JavaScript into resources/public/js/clojure-web-app.js file. So we need to include this file in a simple HTML page:

<!DOCTYPE html>
<html>
<head>
</head>
<body>
  <div id="main">
  </div>
  <script src="js/clojure-web-app.js" type="text/javascript"></script>
</body>
</html>

To serve this static file we need to change some defaults and add corresponding route. In system.clj change api-defaults to site-defaults both in require section and base-config function. In example.clj add following route:

(GET "/" [] (io/resource "public/index.html"))

Again (reset) in REPL window should reload everything.

But where is our ClojureScript source file? Let’s create file core.cljs in src-cljs/clojure-web-app directory:

(ns ^:figwheel-always clojure-web-app.core)

(enable-console-print!)

(println "hello from clojurescript")

Open another terminal and run lein figwheel. It should compile ClojureScript and print ‘Prompt will show when figwheel connects to your application’. Open http://localhost:3000. Fighweel window should prompt:

To quit, type: :cljs/quit
cljs.user=>

Type (js/alert "hello"). Boom! If everything worked you should see and alert in your browser. Open developers console in your browser. You should see hello from clojurescript printed on the console. Change it in core.cljs to (println "fighweel rocks") and save the file. Without reloading the page your should see updated message. Fighweel rocks! Again, in case of any problems, refer to this commit.

UPDATE: In the latest duct release, there is an +cljs option which make it possible to use reloaded repl and fighweel in a single REPL. Highly recommended!

In the next post I’ll show how to fetch data from MongoDB, serve it with REST to the browser and write ReactJs/Om components to render it. Stay tuned!

New HTTP Logger Grails plugin

I've wrote a new Grails plugin - httplogger. It logs:

  • request information (url, headers, cookies, method, body),
  • grails dispatch information (controller, action, parameters),
  • response information (elapsed time and body).

It is mostly useful for logging your REST traffic. Full HTTP web pages can be huge to log and generally waste your space. I suggest to map all of your REST controllers with the same path in UrlMappings, e.g. /rest/ and configure this plugin with this path.

Here is some simple output just to give you a taste of it.

17:16:00,331 INFO  filters.LogRawRequestInfoFilter  - << #1 GET http://localhost:8080/riddle/rest/index?username=admin&search=foo
17:16:00,340 INFO filters.LogRawRequestInfoFilter - << #1 headers Cookie: 'JSESSIONID=DF4EA5725AC4A4990281BD96963739B0; splashShown1.6=1', Accept-Language: 'en-US,en;q=0.8,pl;q=0.6', X-MyHeader: 'null'
17:16:00,342 INFO filters.LogGrailsUrlsInfoFilter - << #1 dispatched to rest/index with parsed params [username:[admin], search:[foo]].
17:16:00,731 INFO filters.LogOutputResponseFilter - >> #1 returned 200, took 405 ms.
17:16:00,745 INFO filters.LogOutputResponseFilter - >> #1 responded with '{count:0}'
17:18:55,799 INFO  filters.LogRawRequestInfoFilter  - << #2 POST http://localhost:8080/riddle/rest/login
17:18:55,799 INFO filters.LogRawRequestInfoFilter - << #2 headers Cookie: 'JSESSIONID=DF4EA5725AC4A4990281BD96963739B0; splashShown1.6=1', Accept-Language: 'en-US,en;q=0.8,pl;q=0.6', X-MyHeader: 'null'
17:18:55,800 INFO filters.LogRawRequestInfoFilter - << #2 body: 'username=admin&password=password'
17:18:55,801 INFO filters.LogOutputResponseFilter - >> #2 returned 404, took 3 ms.
17:18:55,802 INFO filters.LogOutputResponseFilter - >> #2 responded with ''

Official plugin information can be found on Grails plugins website here: http://grails.org/plugins/httplogger or you can browse code on github: TouK/grails-httplogger.

New HTTP Logger Grails plugin

I've wrote a new Grails plugin - httplogger. It logs:

  • request information (url, headers, cookies, method, body),
  • grails dispatch information (controller, action, parameters),
  • response information (elapsed time and body).

It is mostly useful for logging your REST traffic. Full HTTP web pages can be huge to log and generally waste your space. I suggest to map all of your REST controllers with the same path in UrlMappings, e.g. /rest/ and configure this plugin with this path.

Here is some simple output just to give you a taste of it.

17:16:00,331 INFO  filters.LogRawRequestInfoFilter  - << #1 GET http://localhost:8080/riddle/rest/index?username=admin&search=foo
17:16:00,340 INFO filters.LogRawRequestInfoFilter - << #1 headers Cookie: 'JSESSIONID=DF4EA5725AC4A4990281BD96963739B0; splashShown1.6=1', Accept-Language: 'en-US,en;q=0.8,pl;q=0.6', X-MyHeader: 'null'
17:16:00,342 INFO filters.LogGrailsUrlsInfoFilter - << #1 dispatched to rest/index with parsed params [username:[admin], search:[foo]].
17:16:00,731 INFO filters.LogOutputResponseFilter - >> #1 returned 200, took 405 ms.
17:16:00,745 INFO filters.LogOutputResponseFilter - >> #1 responded with '{count:0}'
17:18:55,799 INFO  filters.LogRawRequestInfoFilter  - << #2 POST http://localhost:8080/riddle/rest/login
17:18:55,799 INFO filters.LogRawRequestInfoFilter - << #2 headers Cookie: 'JSESSIONID=DF4EA5725AC4A4990281BD96963739B0; splashShown1.6=1', Accept-Language: 'en-US,en;q=0.8,pl;q=0.6', X-MyHeader: 'null'
17:18:55,800 INFO filters.LogRawRequestInfoFilter - << #2 body: 'username=admin&password=password'
17:18:55,801 INFO filters.LogOutputResponseFilter - >> #2 returned 404, took 3 ms.
17:18:55,802 INFO filters.LogOutputResponseFilter - >> #2 responded with ''

Official plugin information can be found on Grails plugins website here: http://grails.org/plugins/httplogger or you can browse code on github: TouK/grails-httplogger.

How to keep session in HttpBuilder with cookies

In my real-world scenario I have a REST service for AJAX purposes. It renders data series for graphs. I want to test it with groovy's excellent HttpBuilder. There is a problem though - these requests are only available for already logged in users.

In this post I present a complete solution to maintain a session state between HttpBuilder's requests.

Session in HttpBuilder

First of all a quick reminder about session. Session is a simulation of state for HTTP requests, which are stateless by its nature. Once you log in you receive a unique cookie (one or more) that identifies you for sequential requests. Every time you send request you send this cookie along. This way server recognizes you and matches you to your session, which is kept on server. Cookie gets invlid once you log out or it times out, for example after 20 minutes of inactivity. Next time you visit a page you get a new, unique cookie.

In order to keep session alive in HttpBuilder I need to:

  1. log in to my Grails application
  2. receive a JSESSIONID cookie in response
  3. store that cookie and send it along with every subsenquential request

I've created RestConnector class that wraps up HttpBuilder. It's main improvement is that it keeps received cookie in a list.

package eu.spoonman.connectors.RestConnector

import groovyx.net.http.Method
import groovyx.net.http.ContentType
import groovyx.net.http.HTTPBuilder
import groovyx.net.http.HttpResponseDecorator

class RestConnector {
private String baseUrl
private HTTPBuilder httpBuilder
private List<String> cookies

RestConnector(String url) {
this.baseUrl = url
this.httpBuilder = initializeHttpBuilder()
this.cookies = []
}

public def request(Method method, ContentType contentType, String url, Map<String, Serializable> params) {
debug("Send $method request to ${this.baseUrl}$url: $params")
httpBuilder.request(method, contentType) { request ->
uri.path = url
uri.query = params
headers['Cookie'] = cookies.join(';')
}
}

private HTTPBuilder initializeHttpBuilder() {
def httpBuilder = new HTTPBuilder(baseUrl)

httpBuilder.handler.success = { HttpResponseDecorator resp, reader ->
resp.getHeaders('Set-Cookie').each {
//[Set-Cookie: JSESSIONID=E68D4799D4D6282F0348FDB7E8B88AE9; Path=/frontoffice/; HttpOnly]
String cookie = it.value.split(';')[0]
debug("Adding cookie to collection: $cookie")
cookies.add(cookie)
}
debug("Response: ${reader}")
return reader
}
return httpBuilder
}


private debug(String message) {
System.out.println(message) //for Gradle
}
}

A few things to notice in a class above. Constructor sets base URL and creates HttpBuilder instance that can be reused. Next, there is a handler on successful request that checks if I receive any cookie. It adds received cookies to list. Finally, there is a request method that calls HttpBuilder#request but it adds cookies to HTTP headers so server can recognize me as a logged in user.

Sending cookies with every request is a core component in here. It simulates browser's behavior and maintains session.

How to use it?

I will show you how to use this utility class it in Spock test below. It is fairly simple.

First I login to my application and I ensure that I receive a cookie in return, which is equivalent to being logged in. Then I send a request with that cookie sent in HTTP header. This is a Spock test that implements it:

package eu.spoonman.specs.rest

import eu.spoonman.connectors.RestConnector.RestConnector
import groovyx.net.http.ContentType
import groovyx.net.http.Method
import spock.lang.Shared
import spock.lang.Specification
import spock.lang.Stepwise

@Stepwise
class RestChartSpec extends Specification {
@Shared
RestConnector restConnector

def setupSpec() {
restConnector = new RestConnector('http://localhost:8080')
}

def "should login as test"() {
given:
Map params = [j_username: 'test', j_password: 'test']
when:
restConnector.request(Method.POST, ContentType.ANY, '/frontoffice/j_spring_security_check', params)
then:
!(restConnector.cookies.empty)
}

def "should allow access to chart data series"() {
given:
Map params = [days: 14]
when:
Map result = restConnector.request(Method.POST, ContentType.JSON, "frontoffice/chart/series", params)
then:
result != null
result.series.size() > 0
}
}

I create a new RestConnector instance in setupSpec with my application's base URL. Please notice that it has @Shared annotation so it's shared between tests.

@Stepwise is crucial annotation for this specification. It means that Spock executes tests exactly in order they're defined. I need to ensure that login is executed first. I also need to assert that I receive a cookie and list is not empty. I could move this step into setupSpec method too, but I prefer it to be a first test in a specification.

Second test is always executed after login thus it sends cookies within request headers. This is exactly what I wanted to achieve.

How to keep session in HttpBuilder with cookies

In my real-world scenario I have a REST service for AJAX purposes. It renders data series for graphs. I want to test it with groovy's excellent HttpBuilder. There is a problem though - these requests are only available for already logged in users.

In this post I present a complete solution to maintain a session state between HttpBuilder's requests.

Session in HttpBuilder

First of all a quick reminder about session. Session is a simulation of state for HTTP requests, which are stateless by its nature. Once you log in you receive a unique cookie (one or more) that identifies you for sequential requests. Every time you send request you send this cookie along. This way server recognizes you and matches you to your session, which is kept on server. Cookie gets invlid once you log out or it times out, for example after 20 minutes of inactivity. Next time you visit a page you get a new, unique cookie.

In order to keep session alive in HttpBuilder I need to:

  1. log in to my Grails application
  2. receive a JSESSIONID cookie in response
  3. store that cookie and send it along with every subsenquential request

I've created RestConnector class that wraps up HttpBuilder. It's main improvement is that it keeps received cookie in a list.

package eu.spoonman.connectors.RestConnector

import groovyx.net.http.Method
import groovyx.net.http.ContentType
import groovyx.net.http.HTTPBuilder
import groovyx.net.http.HttpResponseDecorator

class RestConnector {
private String baseUrl
private HTTPBuilder httpBuilder
private List<String> cookies

RestConnector(String url) {
this.baseUrl = url
this.httpBuilder = initializeHttpBuilder()
this.cookies = []
}

public def request(Method method, ContentType contentType, String url, Map<String, Serializable> params) {
debug("Send $method request to ${this.baseUrl}$url: $params")
httpBuilder.request(method, contentType) { request ->
uri.path = url
uri.query = params
headers['Cookie'] = cookies.join(';')
}
}

private HTTPBuilder initializeHttpBuilder() {
def httpBuilder = new HTTPBuilder(baseUrl)

httpBuilder.handler.success = { HttpResponseDecorator resp, reader ->
resp.getHeaders('Set-Cookie').each {
//[Set-Cookie: JSESSIONID=E68D4799D4D6282F0348FDB7E8B88AE9; Path=/frontoffice/; HttpOnly]
String cookie = it.value.split(';')[0]
debug("Adding cookie to collection: $cookie")
cookies.add(cookie)
}
debug("Response: ${reader}")
return reader
}
return httpBuilder
}


private debug(String message) {
System.out.println(message) //for Gradle
}
}

A few things to notice in a class above. Constructor sets base URL and creates HttpBuilder instance that can be reused. Next, there is a handler on successful request that checks if I receive any cookie. It adds received cookies to list. Finally, there is a request method that calls HttpBuilder#request but it adds cookies to HTTP headers so server can recognize me as a logged in user.

Sending cookies with every request is a core component in here. It simulates browser's behavior and maintains session.

How to use it?

I will show you how to use this utility class it in Spock test below. It is fairly simple.

First I login to my application and I ensure that I receive a cookie in return, which is equivalent to being logged in. Then I send a request with that cookie sent in HTTP header. This is a Spock test that implements it:

package eu.spoonman.specs.rest

import eu.spoonman.connectors.RestConnector.RestConnector
import groovyx.net.http.ContentType
import groovyx.net.http.Method
import spock.lang.Shared
import spock.lang.Specification
import spock.lang.Stepwise

@Stepwise
class RestChartSpec extends Specification {
@Shared
RestConnector restConnector

def setupSpec() {
restConnector = new RestConnector('http://localhost:8080')
}

def "should login as test"() {
given:
Map params = [j_username: 'test', j_password: 'test']
when:
restConnector.request(Method.POST, ContentType.ANY, '/frontoffice/j_spring_security_check', params)
then:
!(restConnector.cookies.empty)
}

def "should allow access to chart data series"() {
given:
Map params = [days: 14]
when:
Map result = restConnector.request(Method.POST, ContentType.JSON, "frontoffice/chart/series", params)
then:
result != null
result.series.size() > 0
}
}

I create a new RestConnector instance in setupSpec with my application's base URL. Please notice that it has @Shared annotation so it's shared between tests.

@Stepwise is crucial annotation for this specification. It means that Spock executes tests exactly in order they're defined. I need to ensure that login is executed first. I also need to assert that I receive a cookie and list is not empty. I could move this step into setupSpec method too, but I prefer it to be a first test in a specification.

Second test is always executed after login thus it sends cookies within request headers. This is exactly what I wanted to achieve.

Grails render as JSON catch

One of a reasons your controller doesn't render a proper response in JSON format might be wrong package name that you use. It is easy to overlook. Import are on top of a file, you look at your code and everything seems to be fine. Except response is still not in JSON format.

Consider this simple controller:

class RestJsonCatchController {
def grailsJson() {
render([first: 'foo', second: 5] as grails.converters.JSON)
}

def netSfJson() {
render([first: 'foo', second: 5] as net.sf.json.JSON)
}
}

And now, with finger crossed... We have a winner!

$ curl localhost:8080/example/restJsonCatch/grailsJson
{"first":"foo","second":5}
$ curl localhost:8080/example/restJsonCatch/netSfJson
{first=foo, second=5}

As you can see only grails.converters.JSON converts your response to JSON format. There is no such converter for net.sf.json.JSON, so Grails has no converter to apply and it renders Map normally.

Conclusion: always carefully look at your imports if you're working with JSON in Grails!

Edit: Burt suggested that this is a bug. I've submitted JIRA issue here: GRAILS-9622 render as class that is not a codec should throw exception