DDevelopment & Design

Use XMLs not groovy scripts with db migration plugin!

I have 48 domain classes in my Grails 2.1 project and I use Grails Database Migration Plugin 1.2 for a database management. Recently I’ve noticed that it becomes terribly slow when running application, even if there are no changes to be applied. I swit…

I have 48 domain classes in my Grails 2.1 project and I use Grails Database Migration Plugin 1.2 for a database management. Recently I’ve noticed that it becomes terribly slow when running application, even if there are no changes to be applied.

I switched do debug logging level for liquibase package and I found that it takes about 15 seconds to parse changelog.groovy and 20 files that were included in it!

Prepare benchmark

I couldn’t belive it so I’ve created two new clean changelogs:

$ grails dbm-generate-changelog changelog.groovy
$ grails dbm-generate-changelog changelog.xml

Both of these changelogs contain 229 change sets. It is enough that you can benchmark parsers for them. Two parsers in question are:

  • grails.plugin.databasemigration.GrailsChangeLogParser
  • liquibase.parser.core.xml.XMLChangeLogSAXParser

I need to modify a line in my Config.groovy and switch changelog.groovy with changelog.xml for a second test:

grails.plugin.databasemigration.updateOnStart = true
grails.plugin.databasemigration.updateOnStartFileNames = ["changelog.groovy"]
// grails.plugin.databasemigration.updateOnStartFileNames = ["changelog.xml"] grails.plugin.databasemigration.updateOnStart = true
grails.plugin.databasemigration.updateOnStartFileNames = ["changelog-all.groovy"]
// grails.plugin.databasemigration.updateOnStartFileNames = ["changelog-all.xml"]

Profile with JProfiler

I want to profile execution time. I use JProfiler from ej-technologies to measure execution times. Please notice that I don’t want to benchmark SQL queries performed by liquibase. I am only focused on parse method of these two classes.

Here’s how I set up JProfiler:

I switch to CPU Views – Method statistics and I click “Record”. Here are results for both parsers:

Results for changelog.groovy

Results for changelog.xml

Analysis

My assumptions were correct: 8 339 ms vs 139 ms. Parsing XML is 60 times faster! I want to jump and sing: “I switch to XML now!”, but I have some concerns. I have a production database that I need to be compatible with. And I should rewrite my all groovy changelog files by hand. So it’s not so trivial and it’s a time consuming and error prone task.

So as much as I want to switch to XML now, I won’t. But if you start your adventure with database migration plugin today I have an advice for you: use XML if you start from scratch.

For now I’ve just submitted a new JIRA issue – GrailsChangeLogParser – parse method is very slow and I hope it can be greatly improved.

— Previous article

Multi module Gradle project with IDE support

Next article —

After WHUG meeting

You May Also Like
DDevelopment & Design

Need to make a quick json fixes – JSONPath for rescue

From time to time I have a need to do some fixes in my json data. In a world of flat files I do this with grep/sed/awk tool chain. How to handle it for JSON? Searching for a solution I came across the JSONPath. It quite mature tool (from 2007) but I haven't hear about it so I decided to share my experience with others.

First of all you can try it without pain online: http://jsonpath.curiousconcept.com/. Full syntax is described at http://goessner.net/articles/JsonPath/



But also you can download python binding and run it from command line:
$ sudo apt-get install python-jsonpath-rw
$ sudo apt-get install python-setuptools
$ sudo easy_install -U jsonpath

After that you can use inside python or with simple cli wrapper:
#!/usr/bin/python
import sys, json, jsonpath

path = sys.argv[1]

result = jsonpath.jsonpath(json.load(sys.stdin), path)
print json.dumps(result, indent=2)

… you can use it in your shell e.g. for json:
{
"store": {
"book": [
      {
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
      },
      {
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
      },
      {
"category": "fiction",
"author": "Herman Melville",
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"price": 8.99
      },
      {
"category": "fiction",
"author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
      }
    ],
"bicycle": {
"color": "red",
"price": 19.95
    }
  }
}

You can print only book nodes with price lower than 10 by:
$ jsonpath '$..book[?(@.price

Result:
[
  {
"category": "reference",
"price": 8.95,
"title": "Sayings of the Century",
"author": "Nigel Rees"
  },
  {
"category": "fiction",
"price": 8.99,
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"author": "Herman Melville"
  }
]

Have a nice JSON hacking!From time to time I have a need to do some fixes in my json data. In a world of flat files I do this with grep/sed/awk tool chain. How to handle it for JSON? Searching for a solution I came across the JSONPath. It quite mature tool (from 2007) but I haven't hear about it so I decided to share my experience with others.