TouK Hackathon – April 2021

byRafał Solarski
June 10, 2021
5 minute read

The last time we wanted to organize a Hackathon our plans were thwarted by “you know what”. This state of affairs has lasted so long that we just couldn’t stand it anymore and launched the next edition of The Hackathon – remotely. This time we had to cope with new conditions – not all together in an open space, but everyone at home. We launched the communicator with a separate room per each project and went into action.
Here we present a brief summary from each team.

RCB Alert

tcb sms screenshot Every now and then we have a problem in our company – we need to inform everyone that the next day some loud redecorating is to take place or our air conditioning is to be cleaned – and we had better work remotely that day.
Unfortunately, sometimes our office managers gain this knowledge after our working hours, so there is no channel to notify everyone (hopefully – no one reads emails at home).

We found a solution to this problem during our last Hackathon.
We wrote a piece of software that orders an SMS to be sent to each person subscribed to TCB Alerts (it’s a pun on our government alerts – RCB).
This was the easy part – as we already had an SMS sending service.
However, we wanted this solution to be as easy as possible, so our office managers don’t have to open their laptops, connect to a VPN, search for special forms etc.
We decided to implement it as a hook to our communicator – RocketChat.
Now, when our office managers need to inform everyone in the evening, they need only to open the RocketChat app on their phone and type a message on a special channel – and that’s it!

We hope that this solution will help us to stay at home and work remotely during the days when it is inconvenient to work from the office.

TouK AboutMe

about me screenshot Currently at TouK we have several selfcare services providing information about TouKs (people working or cooperating with TouK).
The aim of our project was to join these services into one. Of course we know the rule that if you have three separate services then the worst approach is to add a new one. For this reason we enriched the most modern of them with new features such as TouK’s search, adding new information about people, teams information management and much more.
During the hackathon our team of four (+ the business owner) learnt a lot about frontend technologies such as CORS, MongoDB and LDAP.
We all hope that our work will prove to be useful for both current and future TouKs. Our team strove to provide a user experience so seamless that users would wonder at how easy selfcare management could be. We trust that with this hackathon we are a step closer to achieving that goal.

Business Config Manager

During our deployments of Nussknacker we often have the situation that the flow of development of scenarios was done by two teams. One of them is a team called “Configurators” – people who are close to business requirements but also with quite a high level of technical skills. Those people are responsible for the development of scenarios on the Nussknacker side. On the other hand, the second team is made up of Business members, those with lower technical skills but with a good knowledge of customer needs.
Configurators want to outsource some steps of development to business – so some changes can be made faster, without involving Configurators in the process. We found out that we should make a tool that can give Configurators the ability to create definitions of some configurations and after that, Business can fill in the values of those configurations. In the end, this configuration will be used in some steps of the scenario in Nussknacker.

After the deployment of the proof of concept, we realized that it is necessary to handle some important things:

Both definitions of configurations and values should be versioned and have some audit information, such as the author and time of the change.
Migrations of changes in definitions should be painless.
We should support many types of properties: from raw strings to some date time pickers and so on.
Configurations should have a lifecycle, so new ideas can be deployed on a lower environment and after some tests can be promoted to higher env.

After brainstorming before the Hackathon, we decided to use modern stack, but with some solid, battle-tested components:

We also designed the domain level of application.

On the first day of the Hackathon, we started with pair programming. We tried to go through all layers of the application to make sure that everyone in the team has a common vision of what the architecture will look like. After that we split the work into four separate parts:

Management of definitions
Management of values
Nussknacker integration with Business Config Service
Web application Finally, we integrated all the features together and completed the MVP step.

The project ended successfully. We proved that our design was correct, prepared a solid foundation for future development, and had a lot of fun designing the architecture and testing new tools.

Nussknacker Serverless

Most of you probably know Nussknacker – a powerful platform which allows non-technical users to author and deploy streaming scenarios on Apache Flink.
But Nussknacker Designer can also describe more business rules-oriented scenarios – used e.g. in recommendations or NBA domains. In this case, the scenario is deployed as a REST microservice. During the Hackathon we decided to make this setup more Kubernetes/Serverless-oriented. We decided that a Nussknacker scenario is a good candidate for K8 CRD and that KNative will provide us with a serverless deployment platform.
In two days we reached most of our goals:

Nussknacker creating scenario ConfigMap during deployment (in the future it will be CRD)
Custom Kubernetes operator/controller which transforms scenario ConfigMap into KNative service (scaling down to zero if needed :))
Simple REST microservice image, which serves the scenario
Everything deployed via Helm/GitlabCI to our DigitalOcean K8 cluster
We are also pretty excited about the next steps – scenario observability (metrics, statuses), CRDs and making our serving image serverless ready – by using GraalVM native images.
Hopefully, in the short to medium term, all of this will be accessible with our Nussknacker offering.

MusicBox

music box screenshot We have created a minimalistic web app for generating music in a loop based on text input.
The idea is to make writing music as easy as possible. E.g. |k h s h | is the most basic percussive beat, while |Am|C| gives a basic chord progression. Furthermore, there’s a collaborative mode (think jam sessions ;).

Frontend:

TypeScript
SnowPack
WebSocket
Tone.js (music generation)

Backend:

Summary

In our opinion, The Hackathon was successful and fulfilled its task – that is, it allowed us to take a short break from our more important work and experiment with various fun technologies in good company. The fact that during the Hackathon team members were in a voice chat with each other certainly played a big role, which to some extent allowed us to build an atmosphere of cooperation in the fight against challenges.
We are already looking forward to the next hackathon – hopefully, this time on site.

If you want, here you can see what we did during the previous edition.

Rafał Solarski

— Previous article

How to find all objects that use dblinks

Next article —

Real-life Ansible - chapter one - how to run playbooks on production (more) safely

Zookeeper + Curator = Distributed sync

byMarcin Cylke
June 23, 2013

An application developed for one of my recent projects at TouK involved multiple servers. There was a requirement to ensure failover for the system’s components. Since I had already a few separate components I didn’t want to add more of that, and since there already was a Zookeeper ensemble running - required by one of the services, I’ve decided to go that way with my solution.

What is Zookeeper?

Just a crude distributed synchronization framework. However, it implements Paxos-style algorithms (http://en.wikipedia.org/wiki/Paxos_(computer_science)) to ensure no split-brain scenarios would occur. This is quite an important feature, since I don’t have to care about that kind of problems while using this app. You just need to create an ensemble of a couple of its instances - to ensure high availability. It is basically a virtual filesystem, with files, directories and stuff. One could ask why another filesystem? Well this one is a rather special one, especially for distributed systems. The reason why creating all the locking algorithms on top of Zookeeper is easy is its Ephemeral Nodes - which are just files that exist as long as connection for them exists. After it disconnects - such file disappears.

With such paradigms in place it’s fairly easy to create some high level algorithms for synchronization.

Having that in place, it can safely integrate multiple services ensuring loose coupling in a distributed way.

Zookeeper from developer’s POV

With all the base services for Zookeeper started, it seems there is nothing else, than just connect to it and start implementing necessary algorithms. Unfortunately, the API is quite basic and offers files and directories abstractions with the addition of different node type (file types) - ephemeral and sequence. It is also possible to watch a node for changes.

Using bare Zookeeper is hard!

Creating connections is tedious - and there is lots of things to take care of. Handling an established connection is hard - when establishing connection to ensemble, it’s necessary to negotiate a session also. During the whole process a number of exceptions can occur - these are “recoverable” exceptions, that can be gracefully handled and not break the connection.

    class="c8"><span>So, Zookeeper API is hard.</span></p><p class="c1"><span></span></p><p class="c8"><span>Even if one is proficient with that API, then there come recipes. The reason for using Zookeeper is to be able to implement some more sophisticated algorithms on top of it. Unfortunately those aren&rsquo;t trivial and it is again quite hard to implement them without bugs.</span>

And since distributed systems are hard, why would anyone want another difficult to handle tool?

Enter Curator

    class="c8"><span>Happily, guys from Netflix implemented a nice abstraction for dealing with Zookeeper internals. They called it Curator and use it extensively in the company&rsquo;s environment. Curator offers consistent API for Zookeeper&rsquo;s functionality. It even implements a couple of recipes for distributed systems.</span>

File read/write

    class="c8"><span>The basic use of Zookeeper is as a distributed configuration repository. For this scenario I only need read/write capabilities, to be able to write and read files from the Zookeeper filesystem. This code snippet writes a sample json to a file on ZK filesystem.</span>

<a href="#"

                                                                                                  name="0"></a>


EnsurePath ensurePath = new EnsurePath(markerPath);
ensurePath.ensure(client.getZookeeperClient());
String json = “...”;
if (client.checkExists().forPath(statusFile(core)) != null)
     client.setData().forPath(statusFile(core), json.getBytes());
else
     client.create().forPath(statusFile(core), json.getBytes());

Distributed locking

Having multiple systems there may be a need of using an exclusive lock for some resource, or perhaps some big system requires it’s components to synchronize based on locks. This “recipe” is an ideal match for those situations.

ref="#"
                                                                                    name="b0329bbbf14b79ffaba1139881914aea887ef6a3"></a>



lock = new InterProcessSemaphoreMutex(client, lockPath);
lock.acquire(5, TimeUnit.MINUTES);
… do sth …
lock.release();

(from https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/LockingRemotely.java)

Sevice Advertisement

    class="c8"><span>This is quite an interesting use case. With many small services on different servers it is not wise to exchange ip addresses and ports between them. When some of those services may go down, while other will try to replace them - the task gets even harder. </span>

That’s why, with Zookeeper in place, it can be utilised as a registry of existing services.

If a service starts, it registers into the ServiceRegistry, offering basic information, like it’s purpose, role, address, and port.

Services that want to use a specific kind of service request an access to some instance. This way of configuring easily decouples services from their configuration.

Basically this scenario needs ? steps:

<span>1. Service starts and registers its presence (</span><span class="c5"><a class="c0"
                                                                               href="https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44">https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44</a></span><span>)</span><span>:</span>



ServiceDiscovery discovery = getDiscovery();
            discovery.start();
            ServiceInstance si = getInstance();
            log.info(si);
            discovery.registerService(si);

2. Another service - on another host or in another JVM on the same machine tries to discover who is implementing the service (https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerFinder.java#L50):

<a href="#"

                                                                                                  name="3"></a>


instances = discovery.queryForInstances(serviceName);

The whole concept here is ridiculously simple - the service advertising its presence just stores a file with its whereabouts. The service that is looking for service providers just look into specific directory and read stored definitions.

In my example, the structure advertised by services looks like this (+ some getters and constructor - the rest is here: https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/model/WorkerMetadata.java):



public final class WorkerMetadata {
    private final UUID workerId;
    private final String listenAddress;
    private final int listenPort;
}

Source code

    class="c8"><span>The above recipes are available in Curator library (</span><span class="c5"><a class="c0"
                                                                                                    href="http://curator.incubator.apache.org/">http://curator.incubator.apache.org/</a></span><span>). Recipes&rsquo;
usage examples are in my github repo at </span><span class="c5"><a class="c0"
                                                                   href="https://github.com/zygm0nt/curator-playground">https://github.com/zygm0nt/curator-playground</a></span>

Conclusion

    class="c8"><span>If you&rsquo;re in need of a reliable platform for exchanging data and managing synchronization, and you need to do it in a distributed fashion - just choose Zookeeper. Then add Curator for the ease of using it. Enjoy!</span>

image comes from: http://www.flickr.com/photos/jfgallery/2993361148
all source code fragments taken from this repo: https://github.com/zygm0nt/curator-playground

What is Zookeeper?

With such paradigms in place it’s fairly easy to create some high level algorithms for synchronization.

Having that in place, it can safely integrate multiple services ensuring loose coupling in a distributed way.

Zookeeper from developer’s POV

Using bare Zookeeper is hard!

    class="c8"><span>So, Zookeeper API is hard.</span></p><p class="c1"><span></span></p><p class="c8"><span>Even if one is proficient with that API, then there come recipes. The reason for using Zookeeper is to be able to implement some more sophisticated algorithms on top of it. Unfortunately those aren&rsquo;t trivial and it is again quite hard to implement them without bugs.</span>

And since distributed systems are hard, why would anyone want another difficult to handle tool?

Enter Curator

    class="c8"><span>Happily, guys from Netflix implemented a nice abstraction for dealing with Zookeeper internals. They called it Curator and use it extensively in the company&rsquo;s environment. Curator offers consistent API for Zookeeper&rsquo;s functionality. It even implements a couple of recipes for distributed systems.</span>

File read/write

    class="c8"><span>The basic use of Zookeeper is as a distributed configuration repository. For this scenario I only need read/write capabilities, to be able to write and read files from the Zookeeper filesystem. This code snippet writes a sample json to a file on ZK filesystem.</span>

<a href="#"

                                                                                                  name="0"></a>


EnsurePath ensurePath = new EnsurePath(markerPath);
ensurePath.ensure(client.getZookeeperClient());
String json = “...”;
if (client.checkExists().forPath(statusFile(core)) != null)
     client.setData().forPath(statusFile(core), json.getBytes());
else
     client.create().forPath(statusFile(core), json.getBytes());

Distributed locking

ref="#"
                                                                                    name="b0329bbbf14b79ffaba1139881914aea887ef6a3"></a>



lock = new InterProcessSemaphoreMutex(client, lockPath);
lock.acquire(5, TimeUnit.MINUTES);
… do sth …
lock.release();

(from https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/LockingRemotely.java)

Sevice Advertisement

    class="c8"><span>This is quite an interesting use case. With many small services on different servers it is not wise to exchange ip addresses and ports between them. When some of those services may go down, while other will try to replace them - the task gets even harder. </span>

That’s why, with Zookeeper in place, it can be utilised as a registry of existing services.

If a service starts, it registers into the ServiceRegistry, offering basic information, like it’s purpose, role, address, and port.

Services that want to use a specific kind of service request an access to some instance. This way of configuring easily decouples services from their configuration.

Basically this scenario needs ? steps:

<span>1. Service starts and registers its presence (</span><span class="c5"><a class="c0"
                                                                               href="https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44">https://github.com/zygm0nt/curator-playground/blob/master/src/main/java/pl/touk/curator/WorkerAdvertiser.java#L44</a></span><span>)</span><span>:</span>



ServiceDiscovery discovery = getDiscovery();
            discovery.start();
            ServiceInstance si = getInstance();
            log.info(si);
            discovery.registerService(si);

<a href="#"

                                                                                                  name="3"></a>


instances = discovery.queryForInstances(serviceName);



public final class WorkerMetadata {
    private final UUID workerId;
    private final String listenAddress;
    private final int listenPort;
}

Source code

    class="c8"><span>The above recipes are available in Curator library (</span><span class="c5"><a class="c0"
                                                                                                    href="http://curator.incubator.apache.org/">http://curator.incubator.apache.org/</a></span><span>). Recipes&rsquo;
usage examples are in my github repo at </span><span class="c5"><a class="c0"
                                                                   href="https://github.com/zygm0nt/curator-playground">https://github.com/zygm0nt/curator-playground</a></span>

Conclusion

    class="c8"><span>If you&rsquo;re in need of a reliable platform for exchanging data and managing synchronization, and you need to do it in a distributed fashion - just choose Zookeeper. Then add Curator for the ease of using it. Enjoy!</span>

image comes from: http://www.flickr.com/photos/jfgallery/2993361148
all source code fragments taken from this repo: https://github.com/zygm0nt/curator-playground

DDevelopment & Design

Using WsLite in practice

byPiotr Jagielski
January 3, 2013

TL;DR

There is a example working GitHub project which covers unit testing and request/response logging when using WsLite.

Why Groovy WsLite ?

I’m a huge fan of Groovy WsLite project for calling SOAP web services. Yes, in a real world you have to deal with those - big companies have huge amount of “legacy” code and are crazy about homogeneous architecture - only SOAP, Java, Oracle, AIX…

But I also never been comfortable with XFire/CXF approach of web service client code generation. I wrote a bit about other posibilites in this post. With JAXB you can also experience some freaky classloading errors - as Tomek described on his blog. In a large commercial project the “the less code the better” principle is significant. And the code generated from XSD could look kinda ugly - especially more complicated structures like sequences, choices, anys etc.

Using WsLite with native Groovy concepts like XmlSlurper could be a great choice. But since it’s a dynamic approach you have to be really careful - write good unit tests and log requests. Below are my few hints for using WsLite in practice.

Unit testing

Suppose you have some invocation of WsLite SOAPClient (original WsLite example):

def getMothersDay(long _year) {
    def response = client.send(SOAPAction: action) {
       body {
           GetMothersDay('xmlns':'http://www.27seconds.com/Holidays/US/Dates/') {
              year(_year)
           }
       }
    }
    response.GetMothersDayResponse.GetMothersDayResult.text()
}

How can the unit test like? My suggestion is to mock SOAPClient and write a simple helper to test that builded XML is correct. Example using great SpockFramework:

void setup() {
   client = Mock(SOAPClient)
   service.client = client
}

def "should pass year to GetMothersDay and return date"() {
  given:
      def year = 2013
  when:
      def date = service.getMothersDay(year)
  then:
      1 * client.send(_, _) >> { Map params, Closure requestBuilder ->
            Document doc = buildAndParseXml(requestBuilder)
            assertXpathEvaluatesTo("$year", '//ns:GetMothersDay/ns:year', doc)
            return mockResponse(Responses.mothersDay)
      }
      date == "2013-05-12T00:00:00"
}

This uses a real cool feature of Spock - even when you mock the invocation with “any mark” (_), you are able to get actual arguments. So we can build XML that would be passed to SOAPClient's send method and check that specific XPaths are correct:

void setup() {
    engine = XMLUnit.newXpathEngine()
    engine.setNamespaceContext(new SimpleNamespaceContext(namespaces()))
}

protected Document buildAndParseXml(Closure xmlBuilder) {
    def writer = new StringWriter()
    def builder = new MarkupBuilder(writer)
    builder.xml(xmlBuilder)
    return XMLUnit.buildControlDocument(writer.toString())
}

protected void assertXpathEvaluatesTo(String expectedValue,
                                      String xpathExpression, Document doc) throws XpathException {
    Assert.assertEquals(expectedValue,
            engine.evaluate(xpathExpression, doc))
}

protected Map namespaces() {
    return [ns: 'http://www.27seconds.com/Holidays/US/Dates/']
}

The XMLUnit library is used just for XpathEngine, but it is much more powerful for comparing XML documents. The NamespaceContext is needed to use correct prefixes (e.g. ns:GetMothersDay) in your Xpath expressions.

Finally - the mock returns SOAPResponse instance filled with envelope parsed from some constant XML:

protected SOAPResponse mockResponse(String resp) {
    def envelope = new XmlSlurper().parseText(resp)
    new SOAPResponse(envelope: envelope)
}

Request and response logging

The WsLite itself doesn’t use any logging framework. We usually handle it by adding own sendWithLogging method:

private SOAPResponse sendWithLogging(String action, Closure cl) {
    SOAPResponse response = client.send(SOAPAction: action, cl)
    log(response?.httpRequest, response?.httpResponse)
    return response
}

private void log(HTTPRequest request, HTTPResponse response) {
    log.debug("HTTPRequest $request with content:\n${request?.contentAsString}")
    log.debug("HTTPResponse $response with content:\n${response?.contentAsString}")
}

This logs the actual request and response send through SOAPClient. But it logs only when invocation is successful and errors are much more interesting… So here goes withExceptionHandler method:

private SOAPResponse withExceptionHandler(Closure cl) {
    try {
        cl.call()
    } catch (SOAPFaultException soapEx) {
        log(soapEx.httpRequest, soapEx.httpResponse)
        def message = soapEx.hasFault() ? soapEx.fault.text() : soapEx.message
        throw new InfrastructureException(message)
    } catch (HTTPClientException httpEx) {
        log(httpEx.request, httpEx.response)
        throw new InfrastructureException(httpEx.message)
    }
}
def send(String action, Closure cl) {
    withExceptionHandler {
        sendWithLogging(action, cl)
    }
}

XmlSlurper gotchas

Working with XML document with XmlSlurper is generally great fun, but is some cases could introduce some problems. A trivial example is parsing an id with a number to Long value:

def id = Long.valueOf(edit.'@id' as String)

The Attribute class (which edit.'@id' evaluates to) can be converted to String using as operator, but converting to Long requires using valueOf.

The second example is a bit more complicated. Consider following XML fragment:

<edit id="3">
   <params>
      <param value="label1" name="label"/>
      <param value="2" name="param2"/>
   </params>
   <value>123</value>
</edit>
<edit id="6">
   <params>
      <param value="label2" name="label"/>
      <param value="2" name="param2"/>
   </params>
   <value>456</value>
</edit>

We want to find id of edit whose label is label1. The simplest solution seems to be:

def param = doc.edit.params.param.find { it['@value'] == 'label1' }
def edit = params.parent().parent()

But it doesn’t work! The parent method returns multiple edits, not only the one that is parent of given param…

Here’s the correct solution:

doc.edit.find { edit ->
    edit.params.param.find { it['@value'] == 'label1' }
}

Example

The example working project covering those hints could be found on GitHub.

DDevelopment & Design

Rozwój developera – TouK dla KarieraPlus

byAgata Madaj
July 9, 2013

Po swoim wystąpieniu na Confiturze, które dotyczyło testowania i, jak to zwykle prezentacje Jakuba, przyciągnęło masę słuchaczy, Jakub…

TouK Hackathon – April 2021

RCB Alert

TouK AboutMe

Business Config Manager

Nussknacker Serverless

MusicBox

Summary

Tags:

Rafał Solarski

How to find all objects that use dblinks

Real-life Ansible - chapter one - how to run playbooks on production (more) safely

You May Also Like

What is Zookeeper?

Zookeeper from developer’s POV

Using bare Zookeeper is hard!

Enter Curator

File read/write

Distributed locking

Sevice Advertisement

Source code

Conclusion

What is Zookeeper?

Zookeeper from developer’s POV

Using bare Zookeeper is hard!

Enter Curator

File read/write

Distributed locking

Sevice Advertisement

Source code

Conclusion

TL;DR

Why Groovy WsLite ?

Unit testing

Request and response logging

XmlSlurper gotchas

Example