TouK Hackathon – March 2019

Hackathons are prone to fail. It’s because we have very little time to create something useful, actually working and making a wow!-effect.

So this time we wanted to run “THE HACKATHON”, a well-prepared event with a very high success rate. We called it “Ship IT!” to ensure that we will focus on delivering a working product, not only on having fun while making it. We formed an organising team responsible for internal communication, collecting the needs, removing obstacles and buying some snacks and pizza for the event itself.

We thought that if we want to succeed, we’ll have to prepare well. So each team gathered before the event and talked about their plans, the technologies they intend to use and what hardware they need to order. The organizers met with all teams, exchanged details and gave some good advice.

And finally, the day has come! Seven teams started the two-day hacking session at 9AM, continued until 5PM, and showed the demos of all seven projects after 4.30PM on the second day. We gathered 25 participants total.

Here is a quick presentation of the projects we delivered.

Lidar/ROS.org based robot

robot image We always wanted to build a fully autonomous vehicle to transport sandwiches and monitor the Wi-Fi quality over the whole office space. The first step we took was creating a remote controlled platform with Lidar, which allowed to build the office map and made a foundation for future experiments. We reached this goal with ROS.org and mapped a few corridors and rooms in our office at the end of the hackathon. Now we’re almost ready to conquer Mars :D

Quak Liero-like Game

quak team Quak is a PvP platform game inspired by Liero or Soldat. We used a mix of Kotlin, libGDX and box2d (physics) to build a good-looking 2D game in less than 2 days. Destructible level as a mesh of boxes are generated from provided png layers. Additionally, we implemented some extras to diversify the player experience, such as multiple weapons, double jump, blood effects and explosions.

Quiz Game

quiz game DuckQuiz is a game we wanted to use at public event, e.g. conferences. It’s a little bit similar to the legendary Flappy Bird. Your goal is to make the title hero – a duck – go as far as possible! Correct answers give the duck a bump, so the more questions you answer correctly, the further the duck will go. Be careful, though! Three wrong answers will make a brick fall on the duck’s head! To spice things up a little, we decided to introduce a score threshold, which – once achieved – guaranteed a prize. Plus, of course, if you beat the highest score, you will make history as the reigning champion Duck Flyer. The quiz application is written in Kotlin for Android. We used RxJava and Retrofit for communication with the server. For the back-end side, we chose SpringBoot along with Kotlin. There is also a web-based “Hall of Fame” panel and a simple CMS which allows the management of the quiz questions, written in ReactJS.

Own Card Authorization system

demo4 We’re waiting for our private brand new vending machine which we’re going to use for distribution of snacks in Touk. We’ve noticed that we can communicate with it using MDB protocol. We’d like to authenticate in the machine with RFID cards which are commonly used in TouK to open doors. As a first step, we wanted to create authorization system based on RFID cards, but with additional PIN verification part added. Our system should be integrated with internal LDAP. We’ve separated our project into 4 modules: backend (spring boot), web frontend (scalajs-react), mobile app (android) and hardware device (arduino yun). Backend part connects all modules and allows matching RFID cards UID with LDAP logins. Web frontend is designed for users to set and manage their PINs. The mobile app at the moment is dedicated for our administrator to make pairing existing cards with LDAP logins easier. Finally, hardware device was created to read data from card, get PIN from user and check if PIN is correct (via backend). In the next step, we’d like to integrate our system with mentioned vending machine and make payments using it. There is also a plan to enhance mobile app and use it as mobile payment terminal and integrate web UI with Touk SelfCare system.

Notification lights

lights hackers Three suites (3 lights in red, yellow and blue) controlled by Raspberry Pi Zero W communicating with the managing service via MQTT protocol. The service handles messages from Rocket’s (our chat) outgoing web hooks and notifications from GitLab (e.g. opened merge requests).

Team gathering app (e.g. for football matches)

Team gathering app We regularly play football and volleyball after work, but we sometimes struggle with completing the teams. So far, we used a sign-up system based on Confluence pages, which is not mobile-friendly. We also wanted to allow registration for “reserve” players from outside the company. That’s why we built a simple mobile app with a sign-up form, game history and basic player ranking. The app was built with Dart using the Flutter framework and Firebase for storage and authorization.

Internal time reporting API

hr This HR project’s goal was to integrate and simplify the way we log time in our company. We’ve achieved that with a Google Calendar-like GUI and multiple microservices written in TypeScript, Python, Java and Scala!

The Grand Finale

demo1 At the end of “Ship IT!”, the teams had demonstrated the effects of their work. They shared some successes and other stories :) about the used software and hardware and the lessons learned during those two intensive days. The audience were amused when the robot started to explore the room. The Quak team had a bloody battle in their game. The lights were blinking when one of the team members wrote “kanapki” on our internal TouK chat.

demo2 demo3 demo5 demo6

It was very impressive that each team managed to deliver a viable result and show their projects in action. We can’t wait to run the next edition of The TouK Hackahton and hope there will be even more participants and surprising ideas to see.

You May Also Like

Scrum w Polsce – TouK

Nasi koledzy z Fluidcircle opublikowali właśnie studium przypadku opisujące doświadczenia TouK z wykorzystania metodyki Scrum. Wszystko w ramach programu “Scrum…

Zookeeper + Curator = Distributed sync

An application developed for one of my recent projects at TouK involved multiple servers. There was a requirement to ensure failover for the system’s components. Since I had already a few separate components I didn’t want to add more of that, and since there already was a Zookeeper ensemble running - required by one of the services, I’ve decided to go that way with my solution.

What is Zookeeper?

Just a crude distributed synchronization framework. However, it implements Paxos-style algorithms (http://en.wikipedia.org/wiki/Paxos_(computer_science)) to ensure no split-brain scenarios would occur. This is quite an important feature, since I don’t have to care about that kind of problems while using this app. You just need to create an ensemble of a couple of its instances - to ensure high availability. It is basically a virtual filesystem, with files, directories and stuff. One could ask why another filesystem? Well this one is a rather special one, especially for distributed systems. The reason why creating all the locking algorithms on top of Zookeeper is easy is its Ephemeral Nodes - which are just files that exist as long as connection for them exists. After it disconnects - such file disappears.

With such paradigms in place it’s fairly easy to create some high level algorithms for synchronization.

Having that in place, it can safely integrate multiple services ensuring loose coupling in a distributed way.

Zookeeper from developer’s POV

With all the base services for Zookeeper started, it seems there is nothing else, than just connect to it and start implementing necessary algorithms. Unfortunately, the API is quite basic and offers files and directories abstractions with the addition of different node type (file types) - ephemeral and sequence. It is also possible to watch a node for changes.

Using bare Zookeeper is hard!

Creating connections is tedious - and there is lots of things to take care of. Handling an established connection is hard - when establishing connection to ensemble, it’s necessary to negotiate a session also. During the whole process a number of exceptions can occur - these are “recoverable” exceptions, that can be gracefully handled and not break the connection.

    class="c8"><span>So, Zookeeper API is hard.</span></p><p class="c1"><span></span></p><p class="c8"><span>Even if one is proficient with that API, then there come recipes. The reason for using Zookeeper is to be able to implement some more sophisticated algorithms on top of it. Unfortunately those aren&rsquo;t trivial and it is again quite hard to implement them without bugs.</span>

And since distributed systems are hard, why would anyone want another difficult to handle tool?

Enter Curator

<p
    class="c8"><span>Happily, guys from Netflix implemented a nice abstraction for dealing with Zookeeper internals. They called it Curator and use it extensively in the company&rsquo;s environment. Curator offers consistent API for Zookeeper&rsquo;s functionality. It even implements a couple of recipes for distributed systems.</span>

File read/write

<p
    class="c8"><span>The basic use of Zookeeper is as a distributed configuration repository. For this scenario I only need read/write capabilities, to be able to write and read files from the Zookeeper filesystem. This code snippet writes a sample json to a file on ZK filesystem.</span>

<a href="#"
                                                                                                  name="0"></a>

EnsurePath ensurePath = new EnsurePath(markerPath);
ensurePath.ensure(client.getZookeeperClient());
String json = “...”;
if (client.checkExists().forPath(statusFile(core)) != null)
     client.setData().forPath(statusFile(core), json.getBytes());
else
     client.create().forPath(statusFile(core), json.getBytes());


Distributed locking

Having multiple systems there may be a need of using an exclusive lock for some resource, or perhaps some big system requires it’s components to synchronize based on locks. This “recipe” is an ideal match for those situations.

ref="#"
                                                                                    name="b0329bbbf14b79ffaba1139881914aea887ef6a3"></a>



lock = new InterProcessSemaphoreMutex(client, lockPath);
lock.acquire(5, TimeUnit.MINUTES);
… do sth …
lock.release();


 (from https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/LockingRemotely.java)

Sevice Advertisement

<p

    class="c8"><span>This is quite an interesting use case. With many small services on different servers it is not wise to exchange ip addresses and ports between them. When some of those services may go down, while other will try to replace them - the task gets even harder. </span>

That’s why, with Zookeeper in place, it can be utilised as a registry of existing services.

If a service starts, it registers into the ServiceRegistry, offering basic information, like it’s purpose, role, address, and port.

Services that want to use a specific kind of service request an access to some instance. This way of configuring easily decouples services from their configuration.

Basically this scenario needs ? steps:

<span>1. Service starts and registers its presence (</span><span class="c5"><a class="c0"
                                                                               href="https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44">https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44</a></span><span>)</span><span>:</span>



ServiceDiscovery discovery = getDiscovery();
            discovery.start();
            ServiceInstance si = getInstance();
            log.info(si);
            discovery.registerService(si);



2. Another service - on another host or in another JVM on the same machine tries to discover who is implementing the service (https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerFinder.java#L50):

<a href="#"

                                                                                                  name="3"></a>

instances = discovery.queryForInstances(serviceName);

The whole concept here is ridiculously simple - the service advertising its presence just stores a file with its whereabouts. The service that is looking for service providers just look into specific directory and read stored definitions.

In my example, the structure advertised by services looks like this (+ some getters and constructor - the rest is here: https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/model/WorkerMetadata.java):



public final class WorkerMetadata {
    private final UUID workerId;
    private final String listenAddress;
    private final int listenPort;
}


Source code

<p

    class="c8"><span>The above recipes are available in Curator library (</span><span class="c5"><a class="c0"
                                                                                                    href="http://curator.incubator.apache.org/">http://curator.incubator.apache.org/</a></span><span>). Recipes&rsquo;
usage examples are in my github repo at </span><span class="c5"><a class="c0"
                                                                   href="https://github.com/zygm0nt/curator-playground">https://github.com/zygm0nt/curator-playground</a></span>

Conclusion

<p
    class="c8"><span>If you&rsquo;re in need of a reliable platform for exchanging data and managing synchronization, and you need to do it in a distributed fashion - just choose Zookeeper. Then add Curator for the ease of using it. Enjoy!</span>


  1. image comes from: http://www.flickr.com/photos/jfgallery/2993361148
  2. all source code fragments taken from this repo: https://github.com/zygm0nt/curator-playground

An application developed for one of my recent projects at TouK involved multiple servers. There was a requirement to ensure failover for the system’s components. Since I had already a few separate components I didn’t want to add more of that, and since there already was a Zookeeper ensemble running - required by one of the services, I’ve decided to go that way with my solution.

What is Zookeeper?

Just a crude distributed synchronization framework. However, it implements Paxos-style algorithms (http://en.wikipedia.org/wiki/Paxos_(computer_science)) to ensure no split-brain scenarios would occur. This is quite an important feature, since I don’t have to care about that kind of problems while using this app. You just need to create an ensemble of a couple of its instances - to ensure high availability. It is basically a virtual filesystem, with files, directories and stuff. One could ask why another filesystem? Well this one is a rather special one, especially for distributed systems. The reason why creating all the locking algorithms on top of Zookeeper is easy is its Ephemeral Nodes - which are just files that exist as long as connection for them exists. After it disconnects - such file disappears.

With such paradigms in place it’s fairly easy to create some high level algorithms for synchronization.

Having that in place, it can safely integrate multiple services ensuring loose coupling in a distributed way.

Zookeeper from developer’s POV

With all the base services for Zookeeper started, it seems there is nothing else, than just connect to it and start implementing necessary algorithms. Unfortunately, the API is quite basic and offers files and directories abstractions with the addition of different node type (file types) - ephemeral and sequence. It is also possible to watch a node for changes.

Using bare Zookeeper is hard!

Creating connections is tedious - and there is lots of things to take care of. Handling an established connection is hard - when establishing connection to ensemble, it’s necessary to negotiate a session also. During the whole process a number of exceptions can occur - these are “recoverable” exceptions, that can be gracefully handled and not break the connection.

    class="c8"><span>So, Zookeeper API is hard.</span></p><p class="c1"><span></span></p><p class="c8"><span>Even if one is proficient with that API, then there come recipes. The reason for using Zookeeper is to be able to implement some more sophisticated algorithms on top of it. Unfortunately those aren&rsquo;t trivial and it is again quite hard to implement them without bugs.</span>

And since distributed systems are hard, why would anyone want another difficult to handle tool?

Enter Curator

<p
    class="c8"><span>Happily, guys from Netflix implemented a nice abstraction for dealing with Zookeeper internals. They called it Curator and use it extensively in the company&rsquo;s environment. Curator offers consistent API for Zookeeper&rsquo;s functionality. It even implements a couple of recipes for distributed systems.</span>

File read/write

<p
    class="c8"><span>The basic use of Zookeeper is as a distributed configuration repository. For this scenario I only need read/write capabilities, to be able to write and read files from the Zookeeper filesystem. This code snippet writes a sample json to a file on ZK filesystem.</span>

<a href="#"
                                                                                                  name="0"></a>

EnsurePath ensurePath = new EnsurePath(markerPath);
ensurePath.ensure(client.getZookeeperClient());
String json = “...”;
if (client.checkExists().forPath(statusFile(core)) != null)
     client.setData().forPath(statusFile(core), json.getBytes());
else
     client.create().forPath(statusFile(core), json.getBytes());


Distributed locking

Having multiple systems there may be a need of using an exclusive lock for some resource, or perhaps some big system requires it’s components to synchronize based on locks. This “recipe” is an ideal match for those situations.

ref="#"
                                                                                    name="b0329bbbf14b79ffaba1139881914aea887ef6a3"></a>



lock = new InterProcessSemaphoreMutex(client, lockPath);
lock.acquire(5, TimeUnit.MINUTES);
… do sth …
lock.release();


 (from https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/LockingRemotely.java)

Sevice Advertisement

<p

    class="c8"><span>This is quite an interesting use case. With many small services on different servers it is not wise to exchange ip addresses and ports between them. When some of those services may go down, while other will try to replace them - the task gets even harder. </span>

That’s why, with Zookeeper in place, it can be utilised as a registry of existing services.

If a service starts, it registers into the ServiceRegistry, offering basic information, like it’s purpose, role, address, and port.

Services that want to use a specific kind of service request an access to some instance. This way of configuring easily decouples services from their configuration.

Basically this scenario needs ? steps:

<span>1. Service starts and registers its presence (</span><span class="c5"><a class="c0"
                                                                               href="https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44">https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44</a></span><span>)</span><span>:</span>



ServiceDiscovery discovery = getDiscovery();
            discovery.start();
            ServiceInstance si = getInstance();
            log.info(si);
            discovery.registerService(si);



2. Another service - on another host or in another JVM on the same machine tries to discover who is implementing the service (https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerFinder.java#L50):

<a href="#"

                                                                                                  name="3"></a>

instances = discovery.queryForInstances(serviceName);

The whole concept here is ridiculously simple - the service advertising its presence just stores a file with its whereabouts. The service that is looking for service providers just look into specific directory and read stored definitions.

In my example, the structure advertised by services looks like this (+ some getters and constructor - the rest is here: https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/model/WorkerMetadata.java):



public final class WorkerMetadata {
    private final UUID workerId;
    private final String listenAddress;
    private final int listenPort;
}


Source code

<p

    class="c8"><span>The above recipes are available in Curator library (</span><span class="c5"><a class="c0"
                                                                                                    href="http://curator.incubator.apache.org/">http://curator.incubator.apache.org/</a></span><span>). Recipes&rsquo;
usage examples are in my github repo at </span><span class="c5"><a class="c0"
                                                                   href="https://github.com/zygm0nt/curator-playground">https://github.com/zygm0nt/curator-playground</a></span>

Conclusion

<p
    class="c8"><span>If you&rsquo;re in need of a reliable platform for exchanging data and managing synchronization, and you need to do it in a distributed fashion - just choose Zookeeper. Then add Curator for the ease of using it. Enjoy!</span>


  1. image comes from: http://www.flickr.com/photos/jfgallery/2993361148
  2. all source code fragments taken from this repo: https://github.com/zygm0nt/curator-playground

Spring Security by example: securing methods

This is a part of a simple Spring Security tutorial:

1. Set up and form authentication
2. User in the backend (getting logged user, authentication, testing)
3. Securing web resources
4. Securing methods
5. OpenID (login via gmail)
6. OAuth2 (login via Facebook)
7. Writing on Facebook wall with Spring Social

Securing web resources is all nice and cool, but in a well designed application it's more natural to secure methods (for example on backend facade or even domain objects). While we may get away with role-based authorization in many intranet business applications, nobody will ever handle assigning roles to users in a public, free to use Internet service. We need authorization based on rules described in our domain.

For example: there is a service AlterStory, that allows cooperative writing of stories, where one user is a director (like a movie director), deciding which chapter proposed by other authors should make it to the final story.

The method for accepting chapters, looks like this:

Read more »