DDevelopment & Design

Control your bandwidth using ntop

I was looking for tool which could help me check who is using my bandwidth. Here are requirements which I want from this kind of tool:local hosts bandwidth distribution – it is helpful when you are loosing your bandwidth and don’t know who abuse it in …

I was looking for tool which could help me check who is using my bandwidth. Here are requirements which I want from this kind of tool:

  1. local hosts bandwidth distribution – it is helpful when you are loosing your bandwidth and don’t know who abuse it in your local network
  2. remote hosts bandwidth distribution – it is useful in situation when you want to have control over DoS attacks for your public homepage or when your QoS are not set well

 

Gargoyle

My first shoot is to check what features can give me my TP-Link TL-WR941ND router. I’ve installed on it Gargoyle (modification of OpenWRT with some additional features) some time ago. It has some useful monitoring features:

  • bandwidth distribution pie charts which answer for my first requirement but I can’t check the time when bandwidth was used there
  • connections track – from this I can check two sides of connection (also remote host) and how much of data was send/received but it also doesn’t show this information in time domain and it is served in less friendly, text form

 

It was no exactly what I’m looking for. Therefor I checked what what can we find in OPKG (OpenWRT Package Management).

SNMP + NagiosGraph

I tried to find how I can link Nagios (with NagiosGraph) with my router because I already have some experience with this tools. I found out that there is check_snmp Nagios plugin which can realize this. In OPKG there is mini-snmpd package. It is light SNMP server implementation. You can run it after login by SSH to you router and execute this command:

After this you can check available from server data:
In returned MIB tree there are some useful data like server’s uptime, disk space and also interface’s bandwidth. The last one, stored in Round Robin Database and printed by NagiosGraph will give graphs of bandwidth usage in time domain. But will not show who exactly use bandwidth!

 

Other software

I continue searches in OpenWRT packages. I came across good OpenWRT wiki page: http://wiki.openwrt.org/doc/howto/bwmon describing some available stuff.

 

ntop

Among other there is mentioned ntop – extensive application written in C with many views showing statistics of network protocols usage. Installation of this application on my router with 400MHz CPU will be not the best idea. So I tried to install it on my home server and only send data to it from router by fprobe. At first I installed ntop available from ubuntu 12.04 server’s APT repository. There is available 3:4.1.0+dfsg1-1 version. After some simple configuration steps ntop start drawing graphs.

 

I simulate situation when from remote server I was downloading a big file from my home server. I was disappointed when noticed that I can’t read that this situation taking place from ntop graphs.

 

listening on interface in promiscuous mode

Some time ago I’ve done tcpdump logs analyzer on my studies. I remind that interface working in promiscuous mode can collect all data about local network traffic just like the router. To enable this mode you should exec this command:

Or if you want to set this state persistent you should edit your /etc/network/interfaces to look like this:
If the server where you want to listen for all packages is a VirutalBox vhost you should also verify that it is set promiscuous mode to ”Allow all” in their network configuration like on screenshot below.

ntop v.5.0.2

After this settings we can run ntop on any server in our local network. I give a try for a development version which you can download from ntop homepage: http://www.ntop.org/get-started/download/. Configure script led me through necessary packages that you must install before compilation. After this I run make and sudo make install. To manage ntop using init scripts I used existing /etc/init.d/ntop script and just edited a line with location of DEAMON value – setting them to /usr/local/bin/ntop value. I also removed -n 0 switch from /etc/default/ntop because I hope that bug with DNS resolution is already fixed (it is a little note in config about it).

 

I started deamon by service ntop start. In syslog there was nothing alarming – ntop started collecting traffic statistics. After login I checked available features.

  • Network load – this page shows all load in our network in four time intervals: 10mins, last hour, last day, last month
  • Top talkers – similar to network load intervals, shows how hosts were using bandwidth in past
  • Traffic maps: Region map & hosts map – ntop is connected to Google Maps and shows where are located hosts that we are talking to
  • Activity: how changes activity of hosts in every hour
  • And other – there are other useful things like Protocol statistics, Map of connections between hosts generated in dot and many more
After some tests I noticed that now I have full control about how my network is used (also find out that I have some scheduled script that every minute send unnecessary MBs of data ;-)).

 

little fix

This tests help me find out that there is a little bug in page showing top talkers of an hour. I submitted patch fixing it to ntop’s request tracker if you are interested in: http://sourceforge.net/tracker/?func=detail&aid=3559097&group_id=17233&atid=367233. This is a patch to r5644.

On the end

My adventure with traffic monitoring tools ended on ntop. It is a great tool which fits my needs. Now I know who consumes my resources and can set QoS rules which make my internet connection more responsive.

— Previous article

Rapid js + css development

Next article —

How to create a native Java App

You May Also Like
DDevelopment & Design

Writing JAXB in Groovy

Suppose you want write a jaxb class in groovy. Why? Because you do not have to write these all getters, setters and other methods. You only have to write your fields down.@XmlRootElement@HashCodeAndEquals@ToStringclass Person { String firstName String ...
DDevelopment & Design

Using WsLite in practice

TL;DR

There is a example working GitHub project which covers unit testing and request/response logging when using WsLite.

Why Groovy WsLite ?

I’m a huge fan of Groovy WsLite project for calling SOAP web services. Yes, in a real world you have to deal with those - big companies have huge amount of “legacy” code and are crazy about homogeneous architecture - only SOAP, Java, Oracle, AIX…

But I also never been comfortable with XFire/CXF approach of web service client code generation. I wrote a bit about other posibilites in this post. With JAXB you can also experience some freaky classloading errors - as Tomek described on his blog. In a large commercial project the “the less code the better” principle is significant. And the code generated from XSD could look kinda ugly - especially more complicated structures like sequences, choices, anys etc.

Using WsLite with native Groovy concepts like XmlSlurper could be a great choice. But since it’s a dynamic approach you have to be really careful - write good unit tests and log requests. Below are my few hints for using WsLite in practice.

Unit testing

Suppose you have some invocation of WsLite SOAPClient (original WsLite example):

def getMothersDay(long _year) {
    def response = client.send(SOAPAction: action) {
       body {
           GetMothersDay('xmlns':'http://www.27seconds.com/Holidays/US/Dates/') {
              year(_year)
           }
       }
    }
    response.GetMothersDayResponse.GetMothersDayResult.text()
}

How can the unit test like? My suggestion is to mock SOAPClient and write a simple helper to test that builded XML is correct. Example using great SpockFramework: 

void setup() {
   client = Mock(SOAPClient)
   service.client = client
}

def "should pass year to GetMothersDay and return date"() {
  given:
      def year = 2013
  when:
      def date = service.getMothersDay(year)
  then:
      1 * client.send(_, _) >> { Map params, Closure requestBuilder ->
            Document doc = buildAndParseXml(requestBuilder)
            assertXpathEvaluatesTo("$year", '//ns:GetMothersDay/ns:year', doc)
            return mockResponse(Responses.mothersDay)
      }
      date == "2013-05-12T00:00:00"
}

This uses a real cool feature of Spock - even when you mock the invocation with “any mark” (_), you are able to get actual arguments. So we can build XML that would be passed to SOAPClient's send method and check that specific XPaths are correct:

void setup() {
    engine = XMLUnit.newXpathEngine()
    engine.setNamespaceContext(new SimpleNamespaceContext(namespaces()))
}

protected Document buildAndParseXml(Closure xmlBuilder) {
    def writer = new StringWriter()
    def builder = new MarkupBuilder(writer)
    builder.xml(xmlBuilder)
    return XMLUnit.buildControlDocument(writer.toString())
}

protected void assertXpathEvaluatesTo(String expectedValue,
                                      String xpathExpression, Document doc) throws XpathException {
    Assert.assertEquals(expectedValue,
            engine.evaluate(xpathExpression, doc))
}

protected Map namespaces() {
    return [ns: 'http://www.27seconds.com/Holidays/US/Dates/']
}

The XMLUnit library is used just for XpathEngine, but it is much more powerful for comparing XML documents. The NamespaceContext is needed to use correct prefixes (e.g. ns:GetMothersDay) in your Xpath expressions.

Finally - the mock returns SOAPResponse instance filled with envelope parsed from some constant XML:

protected SOAPResponse mockResponse(String resp) {
    def envelope = new XmlSlurper().parseText(resp)
    new SOAPResponse(envelope: envelope)
}

Request and response logging

The WsLite itself doesn’t use any logging framework. We usually handle it by adding own sendWithLogging method:

private SOAPResponse sendWithLogging(String action, Closure cl) {
    SOAPResponse response = client.send(SOAPAction: action, cl)
    log(response?.httpRequest, response?.httpResponse)
    return response
}

private void log(HTTPRequest request, HTTPResponse response) {
    log.debug("HTTPRequest $request with content:\n${request?.contentAsString}")
    log.debug("HTTPResponse $response with content:\n${response?.contentAsString}")
}

This logs the actual request and response send through SOAPClient. But it logs only when invocation is successful and errors are much more interesting… So here goes withExceptionHandler method:

private SOAPResponse withExceptionHandler(Closure cl) {
    try {
        cl.call()
    } catch (SOAPFaultException soapEx) {
        log(soapEx.httpRequest, soapEx.httpResponse)
        def message = soapEx.hasFault() ? soapEx.fault.text() : soapEx.message
        throw new InfrastructureException(message)
    } catch (HTTPClientException httpEx) {
        log(httpEx.request, httpEx.response)
        throw new InfrastructureException(httpEx.message)
    }
}
def send(String action, Closure cl) {
    withExceptionHandler {
        sendWithLogging(action, cl)
    }
}

XmlSlurper gotchas

Working with XML document with XmlSlurper is generally great fun, but is some cases could introduce some problems. A trivial example is parsing an id with a number to Long value:

def id = Long.valueOf(edit.'@id' as String)

The Attribute class (which edit.'@id' evaluates to) can be converted to String using as operator, but converting to Long requires using valueOf.

The second example is a bit more complicated. Consider following XML fragment:

<edit id="3">
   <params>
      <param value="label1" name="label"/>
      <param value="2" name="param2"/>
   </params>
   <value>123</value>
</edit>
<edit id="6">
   <params>
      <param value="label2" name="label"/>
      <param value="2" name="param2"/>
   </params>
   <value>456</value>
</edit>

We want to find id of edit whose label is label1. The simplest solution seems to be:

def param = doc.edit.params.param.find { it['@value'] == 'label1' }
def edit = params.parent().parent()

But it doesn’t work! The parent method returns multiple edits, not only the one that is parent of given param…

Here’s the correct solution:

doc.edit.find { edit ->
    edit.params.param.find { it['@value'] == 'label1' }
}

Example

The example working project covering those hints could be found on GitHub.

DDevelopment & Design

Super Confitura Man

How Super Confitura Man came to be :)

Recently at TouK we had a one-day hackathon. There was no main theme for it, you just could post a project idea, gather people around it and hack on that idea for a whole day - drinks and pizza included.

My main idea was to create something that could be fun to build and be useful somehow to others. I’d figured out that since Confitura was just around a corner I could make a game, that would be playable at TouK’s booth at the conference venue. This idea seemed good enough to attract Rafał Nowak @RNowak3 and Marcin Jasion @marcinjasion - two TouK employees, that with me formed a team for the hackathon.

Confitura 01

The initial plan was to develop a simple mario-style game, with preceduraly generated levels, random collectible items and enemies. One of the ideas was to introduce Confitura Man as the main character, but due to time constraints, this fall through. We’ve decided to just choose a random available sprite for a character - hence the onion man :)

Confitura 02

How the game is played?

Since we wanted to have a scoreboard and have unique users, we’ve printed out QR codes. A person that would like to play the game could pick up a QR code, show it against a camera attached to the play booth. The start page scanned the QR code and launched the game with username read from paper code.

The rest of the game was playable with gamepad or keyboard.

Confitura game screen

Technicalities

Writing a game takes a lot of time and effort. We wanted to deliver, so we’ve decided to spend some time in the days before the hackathon just to bootstrap the technology stack of our enterprise.

We’ve decided that the game would be written in some Javascript based engine, with Google Chrome as a web platform. There are a lot of HTML5 game engines - list of html5 game engines and you could easily create a game with each and every of them. We’ve decided to use Phaser IO which handles a lot of difficult, game-related stuff on its own. So, we didn’t have to worry about physics, loading and storing assets, animations, object collisions, controls input/output. Go see for yourself, it is really nice and easy to use.

Scoreboard would be a rip-off from JIRA Survivor with stats being served from some web server app. To make things harder, the backend server was written in Clojure. With no experience in that language in the team, it was a bit risky, but the tasks of the server were trivial, so if all that clojure effort failed, it could be rewritten in something we know.

Statistics

During the whole Confitura day there were 69 unique players (69 QR codes were used), and 1237 games were played. The final score looked like this:

  1. Barister Lingerie 158 - 1450 points
  2. Boilerdang Custardbath 386 - 1060 points
  3. Benadryl Clarytin 306 - 870 points

And the obligatory scoreboard screenshot:

Confitura 03

Obstacles

The game, being created in just one day, had to have problems :) It wasn’t play tested enough, there were some rough edges. During the day we had to make a few fixes:

  • the server did not respect the highest score by specific user, it was just overwritting a user’s score with it’s latest one,
  • there was one feature not supported on keyboard, that was available on gamepad - turbo button
  • server was opening a database connection each time it got a request, so after around 5 minutes it would exhaust open file limit for MongoDB (backend database), this was easily fixed - thou the fix is a bit hackish :)

These were easily identified and fixed. Unfortunately there were issues that we were unable to fix while the event was on:

  • google chrome kept asking for the permission to use webcam - this was very annoying, and all the info found on the web did not work - StackOverflow thread
  • it was hard to start the game with QR code - either the codes were too small, or the lighting around that area was inappropriate - I think this issue could be fixed by printing larger codes,

Technology evaluation

All in all we were pretty happy with the chosen stack. Phaser was easy to use and left us with just the fun parts of the game creation process. Finding the right graphics with appropriate licensing was rather hard. We didn’t have enough time to polish all the visual aspects of the game before Confitura.

Writing a server in clojure was the most challenging part, with all the new syntax and new libraries. There were tasks, trivial in java/scala, but hard in Clojure - at least for a whimpy beginners :) Nevertheless Clojure seems like a really handy tool and I’d like to dive deeper into its ecosystem.

Source code

All of the sources for the game can be found here TouK/confitura-man.

The repository is split into two parts:

  • game - HTML5 game
  • server - clojure based backend server

To run the server you need to have a local MongoDB installation. Than in server’s directory run: $ lein ring server-headless This will start a server on http://localhost:3000

To run the game you need to install dependencies with bower and than run $ grunt from game’s directory.

To launch the QR reading part of the game, you enter http://localhost:9000/start.html. After scanning the code you’ll be redirected to http://localhost:9000/index.html - and the game starts.

Conclusion

Summing up, it was a great experience creating the game. It was fun to watch people playing the game. And even with all those glitches and stupid graphics, there were people vigorously playing it, which was awesome.

Thanks to Rafał and Michał for great coding experience, and thanks to all the players of our stupid little game. If you’d like to ask me about anything - feel free to contact me by mail or twitter @zygm0nt

Recently at TouK we had a one-day hackathon. There was no main theme for it, you just could post a project idea, gather people around it and hack on that idea for a whole day - drinks and pizza included.

My main idea was to create something that could be fun to build and be useful somehow to others. I’d figured out that since Confitura was just around a corner I could make a game, that would be playable at TouK’s booth at the conference venue. This idea seemed good enough to attract >Conclusion