Announcing Krush – idiomatic persistence layer for Kotlin, based on Exposed

We’ve released a persistence library for Kotlin, you can find it on our Github. It’s a JPA-to-Exposed SQL DSL generator.

TL;DR

We’ve released a persistence library for Kotlin, you can find it on our Github. It’s a JPA-to-Exposed SQL DSL generator.

The state of persistence in Kotlin

One of the key decisions that helped Kotlin gain massive popularity was to reuse Java ecosystem instead of inventing it’s own. This means that you can safely use Kotlin as a primary language for a project developed using any popular Java stack like Spring Boot and built with Java build tool like Maven. What this also means is that natural choice for persistence layer in Kotlin is Spring Data with JPA 3 with Hibernate as an implementation.

However, JPA, which highly relies on mutable objects and dirty checking, may not look like pure Kotlin, which tries to embrace functional programming and immutability. The official Spring JPA guide for Kotlin uses mutable classes and properties which is not really idiomatic for Kotlin where you want to use immutable data classes whenever it’s possible.

There are some other options, which can be used safely with Kotlin and data classes, like Spring Data JDBC — interesting approach based on pure JDBC, embracing DDD and aggregate root concepts or Micronaut Data JDBC — if you’re not tied to Spring ecosystem. But they’re both relatively new, not mature yet and miss another idiomatic Kotlin feature — a DSL for making SQL queries.

DSL for SQL queries

Another thing that made Kotlin really powerful and popular is its ability to construct Domain Specific Languages using features like property reference, operators, infix and extension functions. For example, for Android development there is excellent anko library for constructing complex view layouts for Android apps. In Spring/JPA the default approach to SQL queries are query methods, where you use special naming convention of methods in repository interfaces. The method names are parsed at runtime to provide required SQL queries and mapping. The naming convention is supported by IntelliJ Idea and other IDEs and works well in simple cases, but may be not flexible enough when you want complex queries e.g. with some conditions based on dynamic filters. If you want to use a true, type-safe, composable and idiomatic Kotlin SQL DSL, you can try to use other libraries, like Requery or Exposed.

Requery

Requery is a lightweight persistence library for Java and Kotlin with RxJava and Java 8 streams support. It uses annotations (both custom and JPA) to process your entities and generate some infrastructure code called “model”.

So given a Book interface:

@Entity
@Table(name = "books")

interface Book : Persistable {
    @get:Key @get:Generated
    val id: Long

    val isbn: String
    val author: String
    val title: String

    val publishDate: LocalDate
}

 

You can instantiate and persist it by using generated BookEntity class:

//given
val book = BookEntity().apply {
    setIsbn("1449373321")
    setPublishDate(LocalDate.of(2017, Month.APRIL, 11))
    setTitle("Designing Data-Intensive Applications")
    setAutor("Martin Kleppmann")
}

// when
val persistedBook = dataStore.insert(book)

And the use SQL DSL to fetch data and map the results back to entities:

// then
val books = dataStore.select(Book::class).where(Book::id eq  book.id).get().toList()

assertThat(books).containsExactly(persistedBook)

This was really close to our needs! We like the idea of having annotations on the entities combined with the rich SQL DSL. Also the RxJava bindings and lazy Kotlin sequences support looks promising. On the other side, there are few minor issues related to immutable classes support:

  • immutable interface approach needs to be backed up with this generated, mutablexxxEntity class
  • there are some restrictions: e.g. you cannot use them to map relations to other entities (just foreign keys by ids)
  • @Generated also doesn’t work for ids in data classes.

You can check example project using Requery in requery branch of krush-example project on GitHub.

Exposed

Another approach which given you rich SQL DSL support is Exposed — a Kotlin-only persistence layer maintained by the JetBrains team. It comes in two flavors: active-record DAO and lightweight SQL DSL. As we are not the fans of active records, we tried the SQL DSL flavor. It works by creating additional mapping code using Kotlin objects and extension functions:

object  BookTable : Table("books") {
    val id: Column<Long> = long("id").promaryKey().autoIncrement()
    val isbn: Column<String> = varchar("isbn". 255)
    val autor: Column<String> = varchar("author". 255)
    val title: Column<String> = varchar("title". 255)
    val publishDate: Column<LocalDate> = date("publishDate")
}

Then you can refer to these Column properties to create type-safe queries and map results using Kotlin collections API:

val titles: List<String> = BookTable
        .select { BookTable.author like "Martin K%" }
        .map { it[BookTable.title] }

As you can see, Exposed is not a full-blown ORM — there is no direct mapping to/from your domain classes into these Table objects, but it’s not hard to write simple mapping functions for that. You can also benefit from Kotlin null-types support and write bindings for your own types by using Kotlin’s extension functions. We wrote some time ago this article about our approach to using Exposed in our projects.

Krush

We really like the Kotlin-first feeling combined with great flexibility of Exposed, but at some time we were tired of writing these table mappings manually. We thought that it would be nice to generate them from JPA-compatible annotations, in similar way it’s done in Requery. This ended with building a library called Krush, which we’re announcing today ;)

Krush consist of two components:

  • annotation-processor which generates Exposed mappings by reading (a subset of) standard JPA annotations found on entity classes
  • utility functions for persisting entities and mapping from/to Exposed objects

So given this entity:

@Entity
@Table(name = "books")

data class Book(
    @Id @GeneratedValue
    val id: Long? = null,

    val isbn: String,
    val author: String,
    val title: String,
    val publishDate: LocalDate
)

Krush will generate BookTable object which allows to persist it like this:

//given
val book = Book(
        isbn = "1449373321", publishDate = LocalDate.of(2017, Month.APRIL, 11),
        title = "Designing Data-Intensive Applications", author = "Martin Kleppmann"
)
val persistedBook = BookTable.insert(book)
assertThat(persistedBook.id).isNotNull()

And write queries using type-safe DSL just like you were using plain Exposed:

val bookId = book.id ?: throw IllegalargumentException( )
val fetchedBook = BookTable.select { BookTable.id eq bookId }.singleOrNull()?.toBook( )
assertThat(fetchedBook).isEqualTo(book)

val selectedBooks = BookTable
        .select { BookTable.author like "Martin Kx" }
        .toBookList()

assertThat(selectedBooks).containsOnly(book)

That’s it! You can find more details and supported features in the README of Krush repository or in some example projects.

Enjoy! Looking for feedback from the community!

You May Also Like

Super Confitura Man

How Super Confitura Man came to be :)

Recently at TouK we had a one-day hackathon. There was no main theme for it, you just could post a project idea, gather people around it and hack on that idea for a whole day - drinks and pizza included.

My main idea was to create something that could be fun to build and be useful somehow to others. I’d figured out that since Confitura was just around a corner I could make a game, that would be playable at TouK’s booth at the conference venue. This idea seemed good enough to attract Rafał Nowak @RNowak3 and Marcin Jasion @marcinjasion - two TouK employees, that with me formed a team for the hackathon.

Confitura 01

The initial plan was to develop a simple mario-style game, with preceduraly generated levels, random collectible items and enemies. One of the ideas was to introduce Confitura Man as the main character, but due to time constraints, this fall through. We’ve decided to just choose a random available sprite for a character - hence the onion man :)

Confitura 02

How the game is played?

Since we wanted to have a scoreboard and have unique users, we’ve printed out QR codes. A person that would like to play the game could pick up a QR code, show it against a camera attached to the play booth. The start page scanned the QR code and launched the game with username read from paper code.

The rest of the game was playable with gamepad or keyboard.

Confitura game screen

Technicalities

Writing a game takes a lot of time and effort. We wanted to deliver, so we’ve decided to spend some time in the days before the hackathon just to bootstrap the technology stack of our enterprise.

We’ve decided that the game would be written in some Javascript based engine, with Google Chrome as a web platform. There are a lot of HTML5 game engines - list of html5 game engines and you could easily create a game with each and every of them. We’ve decided to use Phaser IO which handles a lot of difficult, game-related stuff on its own. So, we didn’t have to worry about physics, loading and storing assets, animations, object collisions, controls input/output. Go see for yourself, it is really nice and easy to use.

Scoreboard would be a rip-off from JIRA Survivor with stats being served from some web server app. To make things harder, the backend server was written in Clojure. With no experience in that language in the team, it was a bit risky, but the tasks of the server were trivial, so if all that clojure effort failed, it could be rewritten in something we know.

Statistics

During the whole Confitura day there were 69 unique players (69 QR codes were used), and 1237 games were played. The final score looked like this:

  1. Barister Lingerie 158 - 1450 points
  2. Boilerdang Custardbath 386 - 1060 points
  3. Benadryl Clarytin 306 - 870 points

And the obligatory scoreboard screenshot:

Confitura 03

Obstacles

The game, being created in just one day, had to have problems :) It wasn’t play tested enough, there were some rough edges. During the day we had to make a few fixes:

  • the server did not respect the highest score by specific user, it was just overwritting a user’s score with it’s latest one,
  • there was one feature not supported on keyboard, that was available on gamepad - turbo button
  • server was opening a database connection each time it got a request, so after around 5 minutes it would exhaust open file limit for MongoDB (backend database), this was easily fixed - thou the fix is a bit hackish :)

These were easily identified and fixed. Unfortunately there were issues that we were unable to fix while the event was on:

  • google chrome kept asking for the permission to use webcam - this was very annoying, and all the info found on the web did not work - StackOverflow thread
  • it was hard to start the game with QR code - either the codes were too small, or the lighting around that area was inappropriate - I think this issue could be fixed by printing larger codes,

Technology evaluation

All in all we were pretty happy with the chosen stack. Phaser was easy to use and left us with just the fun parts of the game creation process. Finding the right graphics with appropriate licensing was rather hard. We didn’t have enough time to polish all the visual aspects of the game before Confitura.

Writing a server in clojure was the most challenging part, with all the new syntax and new libraries. There were tasks, trivial in java/scala, but hard in Clojure - at least for a whimpy beginners :) Nevertheless Clojure seems like a really handy tool and I’d like to dive deeper into its ecosystem.

Source code

All of the sources for the game can be found here TouK/confitura-man.

The repository is split into two parts:

  • game - HTML5 game
  • server - clojure based backend server

To run the server you need to have a local MongoDB installation. Than in server’s directory run: $ lein ring server-headless This will start a server on http://localhost:3000

To run the game you need to install dependencies with bower and than run $ grunt from game’s directory.

To launch the QR reading part of the game, you enter http://localhost:9000/start.html. After scanning the code you’ll be redirected to http://localhost:9000/index.html - and the game starts.

Conclusion

Summing up, it was a great experience creating the game. It was fun to watch people playing the game. And even with all those glitches and stupid graphics, there were people vigorously playing it, which was awesome.

Thanks to Rafał and Michał for great coding experience, and thanks to all the players of our stupid little game. If you’d like to ask me about anything - feel free to contact me by mail or twitter @zygm0nt

Recently at TouK we had a one-day hackathon. There was no main theme for it, you just could post a project idea, gather people around it and hack on that idea for a whole day - drinks and pizza included.

My main idea was to create something that could be fun to build and be useful somehow to others. I’d figured out that since Confitura was just around a corner I could make a game, that would be playable at TouK’s booth at the conference venue. This idea seemed good enough to attract >Conclusion

Multi module Gradle project with IDE support

This article is a short how-to about multi-module project setup with usage of the Gradle automation build tool.

Here's how Rich Seller, a StackOverflow user, describes Gradle:
Gradle promises to hit the sweet spot between Ant and Maven. It uses Ivy's approach for dependency resolution. It allows for convention over configuration but also includes Ant tasks as first class citizens. It also wisely allows you to use existing Maven/Ivy repositories.
So why would one use yet another JVM build tool such as Gradle? The answer is simple: to avoid frustration involved by Ant or Maven.

Short story

I was fooling around with some fresh proof of concept and needed a build tool. I'm pretty familiar with Maven so created project from an artifact, and opened the build file, pom.xml for further tuning.
I had been using Grails with its own build system (similar to Gradle, btw) already for some time up then, so after quite a time without Maven, I looked on the pom.xml and found it to be really repulsive.

Once again I felt clearly: XML is not for humans.

After quick googling I found Gradle. It was still in beta (0.8 version) back then, but it's configured with Groovy DSL and that's what a human likes :)

Where are we

In the time Ant can be met but among IT guerrillas, Maven is still on top and couple of others like for example Ivy conquer for the best position, Gradle smoothly went into its mature age. It's now available in 1.3 version, released at 20th of November 2012. I'm glad to recommend it to anyone looking for relief from XML configured tools, or for anyone just looking for simple, elastic and powerful build tool.

Lets build

I have already written about basic project structure so I skip this one, reminding only the basic project structure:
<project root>

├── build.gradle
└── src
├── main
│ ├── java
│ └── groovy

└── test
├── java
└── groovy
Have I just referred myself for the 1st time? Achievement unlocked! ;)

Gradle as most build tools is run from a command line with parameters. The main parameter for Gradle is a 'task name', for example we can run a command: gradle build.
There is no 'create project' task, so the directory structure has to be created by hand. This isn't a hassle though.
Java and groovy sub-folders aren't always mandatory. They depend on what compile plugin is used.

Parent project

Consider an example project 'the-app' of three modules, let say:
  1. database communication layer
  2. domain model and services layer
  3. web presentation layer
Our project directory tree will look like:
the-app

├── dao-layer
│ └── src

├── domain-model
│ └── src

├── web-frontend
│ └── src

├── build.gradle
└── settings.gradle
the-app itself has no src sub-folder as its purpose is only to contain sub-projects and build configuration. If needed it could've been provided with own src though.

To glue modules we need to fill settings.gradle file under the-app directory with a single line of content specifying module names:
include 'dao-layer', 'domain-model', 'web-frontend'
Now the gradle projects command can be executed to obtain such a result:
:projects

------------------------------------------------------------
Root project
------------------------------------------------------------

Root project 'the-app'
+--- Project ':dao-layer'
+--- Project ':domain-model'
\--- Project ':web-frontend'
...so we know that Gradle noticed the modules. However gradle build command won't run successful yet because build.gradle file is still empty.

Sub project

As in Maven we can create separate build config file per each module. Let say we starting from DAO layer.
Thus we create a new file the-app/dao-layer/build.gradle with a line of basic build info (notice the new build.gradle was created under sub-project directory):
apply plugin: 'java'
This single line of config for any of modules is enough to execute gradle build command under the-app directory with following result:
:dao-layer:compileJava
:dao-layer:processResources UP-TO-DATE
:dao-layer:classes
:dao-layer:jar
:dao-layer:assemble
:dao-layer:compileTestJava UP-TO-DATE
:dao-layer:processTestResources UP-TO-DATE
:dao-layer:testClasses UP-TO-DATE
:dao-layer:test
:dao-layer:check
:dao-layer:build

BUILD SUCCESSFUL

Total time: 3.256 secs
To use Groovy plugin slightly more configuration is needed:
apply plugin: 'groovy'

repositories {
mavenLocal()
mavenCentral()
}

dependencies {
groovy 'org.codehaus.groovy:groovy-all:2.0.5'
}
At lines 3 to 6 Maven repositories are set. At line 9 dependency with groovy library version is specified. Of course plugin as 'java', 'groovy' and many more can be mixed each other.

If we have settings.gradle file and a build.gradle file for each module, there is no need for parent the-app/build.gradle file at all. Sure that's true but we can go another, better way.

One file to rule them all

Instead of creating many build.gradle config files, one per each module, we can use only the parent's one and make it a bit more juicy. So let us move the the-app/dao-layer/build.gradle a level up to the-app/build-gradle and fill it with new statements to achieve full project configuration:
def langLevel = 1.7

allprojects {

apply plugin: 'idea'

group = 'com.tamashumi'
version = '0.1'
}

subprojects {

apply plugin: 'groovy'

sourceCompatibility = langLevel
targetCompatibility = langLevel

repositories {
mavenLocal()
mavenCentral()
}

dependencies {
groovy 'org.codehaus.groovy:groovy-all:2.0.5'
testCompile 'org.spockframework:spock-core:0.7-groovy-2.0'
}
}

project(':dao-layer') {

dependencies {
compile 'org.hibernate:hibernate-core:4.1.7.Final'
}
}

project(':domain-model') {

dependencies {
compile project(':dao-layer')
}
}

project(':web-frontend') {

apply plugin: 'war'

dependencies {
compile project(':domain-model')
compile 'org.springframework:spring-webmvc:3.1.2.RELEASE'
}
}

idea {
project {
jdkName = langLevel
languageLevel = langLevel
}
}
At the beginning simple variable langLevel is declared. It's worth knowing that we can use almost any Groovy code inside build.gradle file, statements like for example if conditions, for/while loops, closures, switch-case, etc... Quite an advantage over inflexible XML, isn't it?

Next the allProjects block. Any configuration placed in it will influence - what a surprise - all projects, so the parent itself and sub-projects (modules). Inside of the block we have the IDE (Intellij Idea) plugin applied which I wrote more about in previous article (look under "IDE Integration" heading). Enough to say that with this plugin applied here, command gradle idea will generate Idea's project files with modules structure and dependencies. This works really well and plugins for other IDEs are available too.
Remaining two lines at this block define group and version for the project, similar as this is done by Maven.

After that subProjects block appears. It's related to all modules but not the parent project. So here the Groovy language plugin is applied, as all modules are assumed to be written in Groovy.
Below source and target language level are set.
After that come references to standard Maven repositories.
At the end of the block dependencies to groovy version and test library - Spock framework.

Following blocks, project(':module-name'), are responsible for each module configuration. They may be omitted unless allProjects or subProjects configure what's necessary for a specific module. In the example per module configuration goes as follow:
  • Dao-layer module has dependency to an ORM library - Hibernate
  • Domain-model module relies on dao-layer as a dependency. Keyword project is used here again for a reference to other module.
  • Web-frontend applies 'war' plugin which build this module into java web archive. Besides it referes to domain-model module and also use Spring MVC framework dependency.

At the end in idea block is basic info for IDE plugin. Those are parameters corresponding to the Idea's project general settings visible on the following screen shot.


jdkName should match the IDE's SDK name otherwise it has to be set manually under IDE on each Idea's project files (re)generation with gradle idea command.

Is that it?

In the matter of simplicity - yes. That's enough to automate modular application build with custom configuration per module. Not a rocket science, huh? Think about Maven's XML. It would take more effort to setup the same and still achieve less expressible configuration quite far from user-friendly.

Check the online user guide for a lot of configuration possibilities or better download Gradle and see the sample projects.
As a tasty bait take a look for this short choice of available plugins:
  • java
  • groovy
  • scala
  • cpp
  • eclipse
  • netbeans
  • ida
  • maven
  • osgi
  • war
  • ear
  • sonar
  • project-report
  • signing
and more, 3rd party plugins...