Here’s the video from my presentation and the discussion about
Agile Skills Project and our experiments in motivating developers at my company, that I had a chance to show at Agile Warsaw. Do not ask my why the camera is 100% time focused on the wall, I have no freaking idea :) The voices are there, and that matters. You can either watch it on Parleys: http://parleys.com/d/2019 or right here. Be warned: it’s in Polish.
I've been reading through storm-users Google Group recently. This resolution was heavily inspired by Adam Kawa's post "Football zero, Apache Pig hero". Since I've encountered a lot of insightful and very interesting information I've decided to describe some of those in this post.
nimbus will work in HA mode - There's a pull request open for it already... but some recent work (distributing topology files via Bittorrent) will greatly simplify the implementation. Once the Bittorrent work is done we'll look at reworking the HA pull request. (storm’s pull request)
pig on storm - Pig on Trident would be a cool and welcome project. Join and groupBy have very clear semantics there, as those concepts exist directly in Trident. The extensions needed to Pig are the concept of incremental, persistent state across batches (mirroring those concepts in Trident). You can read a complete proposal.
implementing topologies in pure python with petrel looks like this:
class Bolt(storm.BasicBolt): def initialize(self, conf, context): ''' This method executed only once ''' storm.log('initializing bolt') def process(self, tup): ''' This method executed every time a new tuple arrived ''' msg = tup.values[0] storm.log('Got tuple %s' %msg) if __name__ == "__main__": Bolt().run()
Fliptop is happy with storm - see their presentation here
topology metrics in 0.9.0: The new metrics feature allows you to collect arbitrarily custom metrics over fixed windows. Those metrics are exported to a metrics stream that you can consume by implementing IMetricsConsumer and configure with Config.java#L473. Use TopologyContext#registerMetric to register new metrics.
storm vs flume - some users' point of view: I use Storm and Flume and find that they are better at different things - it really depends on your use case as to which one is better suited. First and foremost, they were originally designed to do different things: Flume is a reliable service for collecting, aggregating, and moving large amounts of data from source to destination (e.g. log data from many web servers to HDFS). Storm is more for real-time computation (e.g. streaming analytics) where you analyse data in flight and don't necessarily land it anywhere. Having said that, Storm is also fault-tolerant and can write to external data stores (e.g. HBase) and you can do real-time computation in Flume (using interceptors)
That's all for this day - however, I'll keep on reading through storm-users, so watch this space for more info on storm development.
I've been reading through storm-users Google Group recently. This resolution was heavily inspired by Adam Kawa's post "Football zero, Apache Pig hero". Since I've encountered a lot of insightful and very interesting information I've decided to describe some of those in this post.
nimbus will work in HA mode - There's a pull request open for it already... but some recent work (distributing topology files via Bittorrent) will greatly simplify the implementation. Once the Bittorrent work is done we'll look at reworking the HA pull request. (storm’s pull request)
pig on storm - Pig on Trident would be a cool and welcome project. Join and groupBy have very clear semantics there, as those concepts exist directly in Trident. The extensions needed to Pig are the concept of incremental, persistent state across batches (mirroring those concepts in Trident). You can read a complete proposal.
implementing topologies in pure python with petrel looks like this:
class Bolt(storm.BasicBolt): def initialize(self, conf, context): ''' This method executed only once ''' storm.log('initializing bolt') def process(self, tup): ''' This method executed every time a new tuple arrived ''' msg = tup.values[0] storm.log('Got tuple %s' %msg) if __name__ == "__main__": Bolt().run()
Fliptop is happy with storm - see their presentation here
topology metrics in 0.9.0: The new metrics feature allows you to collect arbitrarily custom metrics over fixed windows. Those metrics are exported to a metrics stream that you can consume by implementing IMetricsConsumer and configure with Config.java#L473. Use TopologyContext#registerMetric to register new metrics.
storm vs flume - some users' point of view: I use Storm and Flume and find that they are better at different things - it really depends on your use case as to which one is better suited. First and foremost, they were originally designed to do different things: Flume is a reliable service for collecting, aggregating, and moving large amounts of data from source to destination (e.g. log data from many web servers to HDFS). Storm is more for real-time computation (e.g. streaming analytics) where you analyse data in flight and don't necessarily land it anywhere. Having said that, Storm is also fault-tolerant and can write to external data stores (e.g. HBase) and you can do real-time computation in Flume (using interceptors)
That's all for this day - however, I'll keep on reading through storm-users, so watch this space for more info on storm development.
An application developed for one of my recent projects at TouK involved multiple servers. There was a requirement to ensure failover for the system’s components. Since I had already a few separate components I didn’t want to add more of that, and since there already was a Zookeeper ensemble running - required by one of the services, I’ve decided to go that way with my solution.
Just a crude distributed synchronization framework. However, it implements Paxos-style algorithms (http://en.wikipedia.org/wiki/Paxos_(computer_science)) to ensure no split-brain scenarios would occur. This is quite an important feature, since I don’t have to care about that kind of problems while using this app. You just need to create an ensemble of a couple of its instances - to ensure high availability. It is basically a virtual filesystem, with files, directories and stuff. One could ask why another filesystem? Well this one is a rather special one, especially for distributed systems. The reason why creating all the locking algorithms on top of Zookeeper is easy is its Ephemeral Nodes - which are just files that exist as long as connection for them exists. After it disconnects - such file disappears.
With such paradigms in place it’s fairly easy to create some high level algorithms for synchronization.
Having that in place, it can safely integrate multiple services ensuring loose coupling in a distributed way.
With all the base services for Zookeeper started, it seems there is nothing else, than just connect to it and start implementing necessary algorithms. Unfortunately, the API is quite basic and offers files and directories abstractions with the addition of different node type (file types) - ephemeral and sequence. It is also possible to watch a node for changes.
Creating connections is tedious - and there is lots of things to take care of. Handling an established connection is hard - when establishing connection to ensemble, it’s necessary to negotiate a session also. During the whole process a number of exceptions can occur - these are “recoverable” exceptions, that can be gracefully handled and not break the connection.
class="c8"><span>So, Zookeeper API is hard.</span></p><p class="c1"><span></span></p><p class="c8"><span>Even if one is proficient with that API, then there come recipes. The reason for using Zookeeper is to be able to implement some more sophisticated algorithms on top of it. Unfortunately those aren’t trivial and it is again quite hard to implement them without bugs.</span>
And since distributed systems are hard, why would anyone want another difficult to handle tool?
class="c8"><span>Happily, guys from Netflix implemented a nice abstraction for dealing with Zookeeper internals. They called it Curator and use it extensively in the company’s environment. Curator offers consistent API for Zookeeper’s functionality. It even implements a couple of recipes for distributed systems.</span>
class="c8"><span>The basic use of Zookeeper is as a distributed configuration repository. For this scenario I only need read/write capabilities, to be able to write and read files from the Zookeeper filesystem. This code snippet writes a sample json to a file on ZK filesystem.</span>
<a href="#"
name="0"></a>
EnsurePath ensurePath = new EnsurePath(markerPath);
ensurePath.ensure(client.getZookeeperClient());
String json = “...”;
if (client.checkExists().forPath(statusFile(core)) != null)
client.setData().forPath(statusFile(core), json.getBytes());
else
client.create().forPath(statusFile(core), json.getBytes());
Having multiple systems there may be a need of using an exclusive lock for some resource, or perhaps some big system requires it’s components to synchronize based on locks. This “recipe” is an ideal match for those situations.
ref="#"
name="b0329bbbf14b79ffaba1139881914aea887ef6a3"></a>
lock = new InterProcessSemaphoreMutex(client, lockPath);
lock.acquire(5, TimeUnit.MINUTES);
… do sth …
lock.release();
<p
class="c8"><span>This is quite an interesting use case. With many small services on different servers it is not wise to exchange ip addresses and ports between them. When some of those services may go down, while other will try to replace them - the task gets even harder. </span>
That’s why, with Zookeeper in place, it can be utilised as a registry of existing services.
If a service starts, it registers into the ServiceRegistry, offering basic information, like it’s purpose, role, address, and port.
Services that want to use a specific kind of service request an access to some instance. This way of configuring easily decouples services from their configuration.
Basically this scenario needs ? steps:
<span>1. Service starts and registers its presence (</span><span class="c5"><a class="c0"
href="https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44">https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44</a></span><span>)</span><span>:</span>
ServiceDiscovery discovery = getDiscovery();
discovery.start();
ServiceInstance si = getInstance();
log.info(si);
discovery.registerService(si);
2. Another service - on another host or in another JVM on the same machine tries to discover who is implementing the service (https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerFinder.java#L50):
name="3"></a>
instances = discovery.queryForInstances(serviceName);
The whole concept here is ridiculously simple - the service advertising its presence just stores a file with its whereabouts. The service that is looking for service providers just look into specific directory and read stored definitions.
In my example, the structure advertised by services looks like this (+ some getters and constructor - the rest is here: https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/model/WorkerMetadata.java):
public final class WorkerMetadata {
private final UUID workerId;
private final String listenAddress;
private final int listenPort;
}
<p
class="c8"><span>The above recipes are available in Curator library (</span><span class="c5"><a class="c0"
href="http://curator.incubator.apache.org/">http://curator.incubator.apache.org/</a></span><span>). Recipes’
usage examples are in my github repo at </span><span class="c5"><a class="c0"
href="https://github.com/zygm0nt/curator-playground">https://github.com/zygm0nt/curator-playground</a></span>
class="c8"><span>If you’re in need of a reliable platform for exchanging data and managing synchronization, and you need to do it in a distributed fashion - just choose Zookeeper. Then add Curator for the ease of using it. Enjoy!</span>
An application developed for one of my recent projects at TouK involved multiple servers. There was a requirement to ensure failover for the system’s components. Since I had already a few separate components I didn’t want to add more of that, and since there already was a Zookeeper ensemble running - required by one of the services, I’ve decided to go that way with my solution.
Just a crude distributed synchronization framework. However, it implements Paxos-style algorithms (http://en.wikipedia.org/wiki/Paxos_(computer_science)) to ensure no split-brain scenarios would occur. This is quite an important feature, since I don’t have to care about that kind of problems while using this app. You just need to create an ensemble of a couple of its instances - to ensure high availability. It is basically a virtual filesystem, with files, directories and stuff. One could ask why another filesystem? Well this one is a rather special one, especially for distributed systems. The reason why creating all the locking algorithms on top of Zookeeper is easy is its Ephemeral Nodes - which are just files that exist as long as connection for them exists. After it disconnects - such file disappears.
With such paradigms in place it’s fairly easy to create some high level algorithms for synchronization.
Having that in place, it can safely integrate multiple services ensuring loose coupling in a distributed way.
With all the base services for Zookeeper started, it seems there is nothing else, than just connect to it and start implementing necessary algorithms. Unfortunately, the API is quite basic and offers files and directories abstractions with the addition of different node type (file types) - ephemeral and sequence. It is also possible to watch a node for changes.
Creating connections is tedious - and there is lots of things to take care of. Handling an established connection is hard - when establishing connection to ensemble, it’s necessary to negotiate a session also. During the whole process a number of exceptions can occur - these are “recoverable” exceptions, that can be gracefully handled and not break the connection.
class="c8"><span>So, Zookeeper API is hard.</span></p><p class="c1"><span></span></p><p class="c8"><span>Even if one is proficient with that API, then there come recipes. The reason for using Zookeeper is to be able to implement some more sophisticated algorithms on top of it. Unfortunately those aren’t trivial and it is again quite hard to implement them without bugs.</span>
And since distributed systems are hard, why would anyone want another difficult to handle tool?
class="c8"><span>Happily, guys from Netflix implemented a nice abstraction for dealing with Zookeeper internals. They called it Curator and use it extensively in the company’s environment. Curator offers consistent API for Zookeeper’s functionality. It even implements a couple of recipes for distributed systems.</span>
class="c8"><span>The basic use of Zookeeper is as a distributed configuration repository. For this scenario I only need read/write capabilities, to be able to write and read files from the Zookeeper filesystem. This code snippet writes a sample json to a file on ZK filesystem.</span>
<a href="#"
name="0"></a>
EnsurePath ensurePath = new EnsurePath(markerPath);
ensurePath.ensure(client.getZookeeperClient());
String json = “...”;
if (client.checkExists().forPath(statusFile(core)) != null)
client.setData().forPath(statusFile(core), json.getBytes());
else
client.create().forPath(statusFile(core), json.getBytes());
Having multiple systems there may be a need of using an exclusive lock for some resource, or perhaps some big system requires it’s components to synchronize based on locks. This “recipe” is an ideal match for those situations.
ref="#"
name="b0329bbbf14b79ffaba1139881914aea887ef6a3"></a>
lock = new InterProcessSemaphoreMutex(client, lockPath);
lock.acquire(5, TimeUnit.MINUTES);
… do sth …
lock.release();
<p
class="c8"><span>This is quite an interesting use case. With many small services on different servers it is not wise to exchange ip addresses and ports between them. When some of those services may go down, while other will try to replace them - the task gets even harder. </span>
That’s why, with Zookeeper in place, it can be utilised as a registry of existing services.
If a service starts, it registers into the ServiceRegistry, offering basic information, like it’s purpose, role, address, and port.
Services that want to use a specific kind of service request an access to some instance. This way of configuring easily decouples services from their configuration.
Basically this scenario needs ? steps:
<span>1. Service starts and registers its presence (</span><span class="c5"><a class="c0"
href="https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44">https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44</a></span><span>)</span><span>:</span>
ServiceDiscovery discovery = getDiscovery();
discovery.start();
ServiceInstance si = getInstance();
log.info(si);
discovery.registerService(si);
2. Another service - on another host or in another JVM on the same machine tries to discover who is implementing the service (https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerFinder.java#L50):
name="3"></a>
instances = discovery.queryForInstances(serviceName);
The whole concept here is ridiculously simple - the service advertising its presence just stores a file with its whereabouts. The service that is looking for service providers just look into specific directory and read stored definitions.
In my example, the structure advertised by services looks like this (+ some getters and constructor - the rest is here: https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/model/WorkerMetadata.java):
public final class WorkerMetadata {
private final UUID workerId;
private final String listenAddress;
private final int listenPort;
}
<p
class="c8"><span>The above recipes are available in Curator library (</span><span class="c5"><a class="c0"
href="http://curator.incubator.apache.org/">http://curator.incubator.apache.org/</a></span><span>). Recipes’
usage examples are in my github repo at </span><span class="c5"><a class="c0"
href="https://github.com/zygm0nt/curator-playground">https://github.com/zygm0nt/curator-playground</a></span>
class="c8"><span>If you’re in need of a reliable platform for exchanging data and managing synchronization, and you need to do it in a distributed fashion - just choose Zookeeper. Then add Curator for the ease of using it. Enjoy!</span>
I've encountered strange behavior with a domain class in my project: services that should be injected were null. I've became suspicious as why is that? Services are injected properly in other domain classes so why this one is different?
I've created an experiment. I've created empty LibraryService that should be injected and Book domain class like this:
class Book {
def libraryService
String author
String title
int pageCount
Book() {
println("Finished constructor Book()")
}
Book(String author) {
this()
this.@author = author
println("Finished constructor Book(String author)")
}
Book(String author, String title) {
super()
this.@author = author
this.@title = title
println("Finished constructor Book(String author, String title)")
}
Book(String author, String title, int pageCount) {
this.@author = author
this.@title = title
this.@pageCount = pageCount
println("Finished constructor Book(String author, String title, int pageCount)")
}
void logInjectedService() {
println(" Service libraryService is injected? -> $libraryService")
}
}class LibraryService {
def serviceMethod() {
}
}Book has 4 explicit constructors. I want to check which constructor is injecting dependecies. This is my method that constructs Book objects and I called it in controller:
class BookController {
def index() {
constructAndExamineBooks()
}
static constructAndExamineBooks() {
println("Started constructAndExamineBooks")
Book book1 = new Book().logInjectedService()
Book book2 = new Book("foo").logInjectedService()
Book book3 = new Book("foo", 'bar').logInjectedService()
Book book4 = new Book("foo", 'bar', 100).logInjectedService()
Book book5 = new Book(author: "foo", title: 'bar')
println("Finished constructor Book(Map params)")
book5.logInjectedService()
}
}Output looks like this:
Started constructAndExamineBooks
Finished constructor Book()
Service libraryService is injected? -> eu.spoonman.refaktor.LibraryService@2affcce2
Finished constructor Book()
Finished constructor Book(String author)
Service libraryService is injected? -> eu.spoonman.refaktor.LibraryService@2affcce2
Finished constructor Book(String author, String title)
Service libraryService is injected? -> null
Finished constructor Book(String author, String title, int pageCount)
Service libraryService is injected? -> null
Finished constructor Book()
Finished constructor Book(Map params)
Service libraryService is injected? -> eu.spoonman.refaktor.LibraryService@2affcce2What do we see?
Always explicitily invoke empty constructor in your Grail domain classes to ensure Dependency Injection! I didn't know until today either!