Author: Marcin Cylke

SoapUI ext libs and its weirdness

Suppose you want to add some additional jars to your SoapUI installation. It all should work ok if you put them in bin/ext directory. It is scanned at startup, and jars found there are automatically added to classpath. However if you want to add some JDBC drivers, and happen to be using SoapUI version higher than 3.5.1 it is a bit more tricky. You may face this NoClassDefFoundError: 
An error occured [oracle/jdbc/Driver], see error log for details
java.lang.NoClassDefFoundError: oracle/jdbc/Driver
If so, try registering your drivers with registerJdbcDriver function, like I did in this snippet of code:   What a crappy thing!

What is NoSQL good for?

... or how I ended up writing a CouchDB proof of concept app? Once upon a time I set out on a journey to discover the NoSQL land. I've decided that doing simple queries wouldn't be interesting enough. That's why I've chose to create an app that would be based on some NoSQL database. The main idea was to create an app, that would dynamically update itself with geographic data flowing in. Since there are myriads of geo-data that are available on the internet, you can pick your favorite one and load them into your SQL database of choice. In my case the primary source of data was a proprietary database, or more specifically - one table in it continuously updated with new data. To make that data visible on my map I needed to:
  • buffer the huge amount of those records - so as not to overhoul other services with large traffic, and not to flood the frontend
  • convert then to my representation
  • display them - have presentation layer in a browser - since browser-based frontend was the easiest and fastest to develop
The idea of the front-end HTML page was to show new points on the map. From the moment of opening the page records that appear in database table should be shown interactively on the screen.

Toys used

For the first step I chose to use RabbitMQ broker. A queue on the broker would receive messages - one message per database table's row. Then I'd use some simple groovy middle ware to convert the data to appropriate format and put it onto another db - this time db specific to my app. You may ask why incorporate another database. It would be good for separating environments - assuming the original data contains some vulnerable content that should be anatomised, or we just don't feel comfortable exposing the whole database of some XYZ-system just to have access to its one table. Since for my presentation layer I chose HTML+JS without any application server-based back-end I've decided on CouchDB . This seemed like a perfect match for this scenario. Why? - ease of use, REST API, with JSON responses - just great for interacting with my simple front-end. The flow of things was as shown on the image below: diagram

Avro - for the beginning

As you can see, I've chosen JSON as my data-format. I've been considering Apache Avro in the first place but using it was a real pain in the ass. Avro itself is used in Apache Hadoop as a serialization layer, so it would seem OK, but it has virtually no documentation. But once you tear through the unintuitive interface and manage to handle all those unthinkable exceptions you get a few pros for this library. It's great in that it does not require code generation - I like it being made on the fly. It also offers sending data in binary format, which was not necessary, but never the less is a nice feature. What I certainly didn't like about it was its orientation on the files rather than chunks of data - so it was not so obvious how should I send data through the wire. Than I found out it can produce JSON output, which would work for me, except the output could not have been parsed by other JSON libraries :) (I've asked on stackoverflow about that, but with no luck). If my whining haven't put you back and still would like to see how to use Avro, try this unit test in project's GitHub repo: AvroSimpleTest.groovy

Svenson

I've dropped Avro in favour of a simple JSON lib called (Svenson and that was painless. The only thing I was forced to do was create my model class in Java - the rest of the project is written in Groovy. I've no idea why was that necessary, and didn't want to look into it.

RabbitMQ

Further on the way is RabbitMQ, to which records are filled by a feeding middle-ware written in Groovy. Since I use ActiveMQ on a day-to-day basis, I've decided to try something new. This broker is a really nice piece of software. Being written in Erlang makes it really fast. What's more it has some extensive capabilities and is easy to approach for anyone similar with messaging (JMS and friends). For such a lightweight product it is really powerful - implements AMQP!

CouchDB

From the broker's queue messages are again fetched by a middle-ware just to be put into CouchDB view. This database is also written in Erlang. It's very reliable, however the way it handles refreshing view isn't the most pleasant one - performance-wise. Word of advice - if you're on Debian derivative, be cautious with apt-repository version. It's rather _ancient_. Also remember to add allow_jsonp = true to you config file /opt/couchbase/etc/couchdb/local.ini. It's not enabled by default, and not having this set would result with empty responses from the CouchDB server. The problem here is, that the browser doesn't allow quering a web server with hostname other than the one the script originates. More on this case here. Seems like my problem could be overcame by changing url in index.html and hostname couchdb listens on to the same address. I've also created a view, that would expose an event by key: view code

Presenting the dots

As a back-end I've done some JQuery based AJAX calls - nothing too fancy. All things necessary for presentation layer are in this file.

Things to consider

Please bear in mind that this whole application is rather a playground, not a full-fledged project!! After creating all the parts I have some doubts about some architectural decisions I made. I don't think the security have been taken into account seriously enough. Also scalability was never an issue ;-) If you have some thoughts about any of the aspects mentioned in this post, please feel free to comment or contact me directly :) And also you may try the application by yourself - it's on the GitHub.

OVal – validate your models quickly and effortlessly!

Some time ago one of the projects at work required me to validate some Java POJOs. Theses were my model classes and I've been creating them from incoming WebService requests. One would say that XSD would be sufficient for the task, for parts of this validations - sure, it would. But there were some advanced rules XSD would not handle, or would render the schema document very complicated. Rules I needed to express were like:
  • person's first_name and last_name should be of appropriate length - between 2 and 20, and additionally one could pass a zero-length string just to remove the previous value
  • state field should consist only defined values - as in dictionary value - this one would be completable with XSD's enumerations, but would require often changing schema files and redistributing them to interested parties :(
The library I've decided to use for this task is OVal and it came out really nice! Read on to find out the details! Oval is quite mature library that allows POJO validation, but is not JSR303 (bean validation) implementation. It has converters that enable it to understand those annotations, but I'm not sure about the compatibility. I've tried only a subset of the available checks, among which were:
  • NotNull
  • NotEmpty
  • Length
There are many more, and their attributes give interesting ways to configure the validation process. But using them was rather easy and did not require to much brainstorming. What I really needed were custom checks. And in this area OVal shows it's strength. Implementing a check is really easy. I needed an annotation that would check a field against some values in a dictionary. If field's value was in the given set, than the validation would succeed, if not, an exception would be thrown. To accomplish this task it is required to implement two classes: annotation class and check class - called by the validation engine on a given field. Let's start with our new annotation:   In the above snippet I've defined a check-annotation, that would be used like this:   You can pass file - containing dictionary values for this field. There is also message field in the annotation which is an error message returned by the validation engine of failed check - pretty handy. And can be expressed in .properties file as:   Placeholder, like context, will be replaced with correct values supplied by the validation engine. Annotating a field is not enough. It is also needed to create a validator for this kind of check. The name of the class is already defined in DictionaryValue annotation, it is called DictionaryValueCheck and I've done this check this way:   What this basically does is:
  1. when file is set - read dictionary content from the file into map
  2. upon check request just lookup value in dictionary parsed from the input file
And that's it! For me Oval is really great tool. With it at ones disposal it is extremely easy to create any imaginable validation you need. This library is really easy to use and offers lots of handy features. But perhaps I'm reinventing the wheel and all this can be done easily with some other library? Share Your opinion!

Geecon 2011 – day 2

And now for part 2 of my visit to Geecon 2011!

1. Jim Webber "Revisiting SOA for the 21st century"

Now this was awesome! Jim Webber, a former ThoughtWorks employee, now Neo4j evangelist (in Neotechnology) described his views on how SOA should look - according to him. This was presented previously, on other occasions as his "Guerilla SOA" talk - generally he advocated for REST based services, loose contracts (stating that WSDLs are too verbose and code generation is evil).

Jim mentioned Martin Fowler's article on integration databases but I couldn't find it anywhere - thou the topic looks interesting. He also recommended BDD and exposing tests on the web for the end user to use them as early as possible.

One big point he made his case with was not relying on enterprise software. Simple tools can do much better job. He compared implementing Web Services security (Secured SOAP over HTTP over TCP IP) to REST based service accessed through HTTPS - basic and easily testable with tools like curl.

Great talk. One of the best!

2. Staffan Noteberg "Regex - the future programming"

I must confess, that this did not go too well. The whole talk was well prepared and laid out but it lacked depth. It was pretty basic introduction to regex. From the presentation's subject I was rather prepared for some novel uses of regex - like for example: showing how to filter big volume of data with simple regex or sth.

But the talk was fun, Staffan is a good speaker. He is also an author of pomodoro technique book - I intend to read sth abut this technique and this may be a nice start

3. Bartosz Kowalewski "Is OSGI ready for wide adoption?"

If it comes to titles I tend to rely on them pretty heavily, however strange it may seem. This time I also did - and the whole talk did not give me a definitive answer to the stated question.

Sure, the presentation was informative, but it described some OSGI specific, quite low level stuff. Of course, if you want to use OSGI - even by leveraging application server with OSGI under the hood - you should know a fair bit about the technology itself. Even thou the AS does a good job of hiding OSGI container specifics from the developer, in case of problems it's better to be well informed. All in all - the talk gave too little information for me.

4. Vaclav Pech "Pick low hanging fruit"

"Parallelism is not hard, multithreading is" - this was the key sentence of the presentation. The speaker showed how to introduce concurrency into normal java/groovy code by sprinkling it with concurrency powder. Easy enough! With GPars library he showed:

  • running processing tasks with thread pools
  • testing concurrent code
  • Fork/join Thread Pool - multiple thread queues (note to self: fork/join is good for hierarchical problems)
  • low-hanging fruits:
    • async calculations
    • fork/join
    • dataflow
    • parallel collection processing
  • Actors are great - use GPars or Akka, is sufficient to use @ActiveMethod and @ActiveObject annotations and Actors are usable in OO-world

Good talk, well received!

5. Anton Arhipov "Bytecode for discriminating developers"

Technical introduction to the world of bytecode, jvm specification details. I've drifted away to some other topics - really - can't recall what this was all about.

6. Andreas Almiray "Polyglot Programming"

This was a nice talk covering Groovy, Scala and Closure. The whole point of it was to show how cool it is to play with emerging JVM languages. They are not only fun but also useful. What's more, they bring freshness to java world, injecting it with some new paradigms and methodologies. It is easier to incorporate new ideas into younger JVM languages than to the mature Java.

7. Jim Webber "A pragmatic introduction to Neo4j"

And Jim Webber again, this time with some Neo4j evangelism. First came some taxonomy information on NoSQL databases (Not Only SQL) as a whole - than some specific examples of problems solvable with graph databases - and Neo4j is a graph database.

Main points of Jim's talk were:

  • sharding a database is important for scalability
  • series data - should be OK to use Neo4j as their storage

Conclusion

These were all the sessions I attended. On Saturday there was a Hacker-garden, but neither I had time nor will to stay - the topics were very interesting and I'd definitely like to experience such an event, but after 2 days of continuous talks I was rather tired.

To sum up, 2011's Geecon was a great experience, with lots of interesting talks and lots of new inspirations. Keep up the good work guys!

Geecon 2011 – day 1

Last week's Java conference - Geecon was very interesting. It was well prepared, and gave an insight into the current Java related trends - concurrency, DSLs, polyglot programming. But not only that - there were also some pretty different talks from excellent speakers.

The whole event took 4 days:

  • University day (wednesday)
  • 2 regular conference days (Thursday + Friday)
  • hacker garden (Sunday)

I decided to attend only on Thursday and Friday - no time for more. Here is what interesting happened during those days.

Day 1

The morning got me unprepared. After hard enough, after work travel to Krakow on Wednesday, I wasn't in the best shape. However after arriving at the venue, being greeted with breakfast and refreshments I looked at the rest of the day with real hope.

Since the schedule was tight - three parallel tracks of lecture, I had to choose, so bare in mind, that is my account of what I've seen and heard. Others may, of course, differ.

1. Danny Coward "Java SE: The Road Ahead"

Danny, being on Oracle (considering being also former Sun's employee a plus) payroll, gave an insightful talk on new things to came in Java 7. He drew rather serious plans for Java 8. According to Danny, the main trends in today's Java ecosystem are:

  • parallel programming
  • language dynamics

and he probably is right :) The great things to come with new versions of Java are:

  • closures in Java (finally!)
  • extending interfaces
  • map, filter - functions for collections
  • lambda expressions - thou in Java 8

The talk itself was a nice keynote, but I doubt the road map for Java will be met in its full extent - the goals aren't that small.

2. Juergen Hoeller "Enterprise Java in 2011"

Spring Source as one of the sponsors sent Juergen to evangelize about the world of enterprise and Java's place in it :) He emphasized different kinds of deployment: WAR, cloud deployment - and the latter's rise of importance.

He pointed out how outdated current application servers are - the usually lag ~3 years behind the main trends and developers' needs - good point! He proposed looking under the hood of now-popular cloud environments: Google App Engine or Amazon Elastic Cloud to look for schemas in them, etc - I intend to listen to his advice.

All in all this guy gave a great talk covering wide spectrum of technologies and not focusing on technical stuff too much.

3. Heinz Kabutz "Reflection madness"

Despite living on a Greek island, this guy showed also how to whack ones mind with Java Reflection API. Pure magic! Some highlights of his talk were:

  • how to get 42 + 1 = 44
  • get the size of an object
  • get method caller's id
  • add enum values dynamically

With all this examples he pointed that using SecurityManager will prevent such nasty coding practices.

Since he is an editor of Javaspecialists.eu newsletter, all the answers to problems presented in his talk (and many many more) can be found there.

Well done, not to useful for me, nevertheless - interesting.

4. Michael Figuiere, Cyrille Le Clere "NoSQL & Datagrid from developer perspective"

I don't know what to think of this talk. It consisted an introduction to NoSQL databases but also a bit of problem's description that can be encountered when dealing with them. Notable thoughts were on:

  • creating a sharding ready data structure
  • denormalization as a useful process for NoSQL DBs
  • NoSQL usually means no transactions

5. Hamlet D'Arcy "New Ideas for old code"

Since a lot of developers (all?) have to deal with legacy code - one way or another, this talk was a must!

. The speaker shared some ideas on how to work with such code and remain sane. The talk was vivid, interesting and entertaining, well, and the notable thought? Here they are:

  • 51% rule - if you're not committing 51% of your time/your tasks into fixing your situation than the whole battle is already lost,
  • read some good stuff!:
  • use static analysis - Find Bugz, pmc
  • query-command - a method should be a query or a command:
    • query - returns sth
    • command - change the state of an object
  • he also proposed scratch refactoring
    1. set a timer
    2. tag your code
    3. refactor without tests
    4. step back and analyze
    5. is it better? if not, revert

This was nice! - it assumed arriving at a project with no ( or little) tests.

6. Aslan Knutsen "Arquillian"

The last talk of that day was about some new library from JBoss that would allow to test your components with unit tests - test them in a destination container. The whole point of this library is to run the specific fragment of code as if it was build and deployed to some application server (let's say JBoss AS ;-) ). To be honest, I can't find much application for that - thou I'm not doing any serious work in JEE world.

Party

And the day ended. But there was sth else to do after the official part - party time! It took place at Klub Pauza on Floriańska street. It was a rather nice social event.

... to be continued - stay tuned for part 2

JCE keystore and untrusted sites

Recently at work I was in need of connecting to a web service exposed via HTTPS. I've been doing this from inside Servicemix 3.3.1, which may seem a bit inhibiting, but that was a requirement. Nevertheless I've been trying my luck with the included servicemix-http-2008.01 component. I've created a simple Service Unit using that component and made connection attempt. Unfortunately I've encountered issues with the SSL conversation negotiation. I had to dig deeper into the servicemix-http code to find out these had something to do with my JCE keystore. Read more to find out what happened! Ok, so I had my xbean.xml for http component looking like this:   As you can see this is a proxy adapter to some outside service exposed via secured HTTP protocol. Since it's HTTPS I've specified some SSL parameters. It was sufficient in my case to just pass the keystore file and it's password. I've created my keystore.jks file in smx_home/conf with password servicemix in the following manner:   You can see what's in this file with this command:   At this point I thought, that having a configured keystore and my component would suffice. Wrong! As soon as I've tried to connect to the external service I got an exception:   Hmmm.. this looks pretty nasty, but it's not that bad. As one can read here, it's associated with the other site's having an untrusted (unsigned) certificate. Assuming you actually trust the other end of the communication and this situation is ok for you, you should add the servers certificate to your keystore. The previously mentioned link contained a little java class that would do just that. You can find it here (original code) InstallCert.java or you can look into my slightly changed version here at github. You should call it as follows, assuming that file keystore.jks is in the current directory:   What you'll probably see, when you execute this app is this:   Please note that there is a prompt (Enter certificate to add to trusted keystore...) in which you can enter the certificate number you wish to add to your keystore. After all those steps my request got through and I could happily query HTTPS service as long as I wanted to! Great!

Possible problems

In my search for this problem's solution I've encountered this kind of exception:   A little googling led me to this StackOverflow question. It seems that you cannot have multiple keys with different passwords in the same keystore and use KeyManagerFactory class. Oh well... .

Ending

To sum up, the solution given works, but in my opinion using the InstallCert.java app is rather dirty. I've been wondering, do you know other ways of doing that thing?

Advisory Messages to the rescue

The most crucial part of software development is testing. It should ensure us, that our code is correct, works according to given specs, etc. There are many kinds of tests: unit tests, integration, functional. In general you should try to test the smallest possible subset of your code and be able to check the state of the objects after the test. This seems as rather easy task, but what if you have an integration end-to-end test to perform? In most cases asserting state in integration test is rather hard due to multiple systems interoperability. Let's focus on a specific situation. What I needed to do the other day was write some integration test for Jms based system. The processing pipeline is easy:
  • fetch object from DB
  • process it
  • publish on JMS
some other system (X-system) polls JMS:
  • if message is found
  • fetch it (message disappears from the JMS queue)
  • do sth with it
  • Looks simple but since I didn't have any sane access to the X-system I wanted to be sure that my object was actually put into the queue. It was not acceptable to subscribe to the queue and fetch that object in my test - it would dusrupt the flow of the whole process. Fortunately I've been using ActiveMQ and since it offers a thing called Advisory Messages I've decided to use just them. What are advisory messages? They are a set of administrative messages that are generated on a specific event, like message consumption, message delivery, topic destruction, and many more. Each type of message is delivered to a separate topic - prefixed with ActiveMQ.Advisory. Since generation of such messages may be an overhead in production systems these features are turned off by default. You need to enable specific type of advisory message for a specific jms destination. You can do this with ths configuration change to activemq.xml   As you can see, I've specified which advisories I want enabled. The full list of available advisories can be found here. Since I wanted to read messages from that topic I've added the following configuration to my spring context - there is one destination bean for inserting messages and one bean for advisory topic.   Thanks to this configuration I've been able to check that my message was actually delivered to the queue. There've been no need to worry about race conditions in consuming the message from original queue - if the X-system read the message, I'd be unable to determine if it has ever been in JMS at all. What's not so nice about that:
    • advisory messages can be thought of as counters rather than debugging information
    • they don't contain any data that would allow us to match advisory message to the original message - thou you could correlate by timestamp
    All in all, it's a good tool to have! But perhaps you have some other thoughts on this subject? How do you test JMS?

How to run multiple guest OS in QEMU?


This weekend I've been fiddling with QEMU. I've installed OpenBSD on a single image and wanted to have two instances of it communicating via network. Installing the system was easy, but the networking setup was quite a pain. See how I did that... To make QEMU instances communicate with each other I needed to plug them to a "network". That's why I've created a bridge to which Virtual Instances would connect to. I've used the following script: Then I just needed to start Qemu with this command line: Since I've set up bridge for Qemu instances, I've plugged TAP interfaces into it. That's why I've needed to specify this in my qemu exec line. I've also added macaddress setting since both my instances were getting the same one. And that's all! It works like a charm. Now on to some harder things!