DDevelopment & Design

Control your bandwidth using ntop

I was looking for tool which could help me check who is using my bandwidth. Here are requirements which I want from this kind of tool:local hosts bandwidth distribution – it is helpful when you are loosing your bandwidth and don’t know who abuse it in …

I was looking for tool which could help me check who is using my bandwidth. Here are requirements which I want from this kind of tool:

  1. local hosts bandwidth distribution – it is helpful when you are loosing your bandwidth and don’t know who abuse it in your local network
  2. remote hosts bandwidth distribution – it is useful in situation when you want to have control over DoS attacks for your public homepage or when your QoS are not set well

 

Gargoyle

My first shoot is to check what features can give me my TP-Link TL-WR941ND router. I’ve installed on it Gargoyle (modification of OpenWRT with some additional features) some time ago. It has some useful monitoring features:

  • bandwidth distribution pie charts which answer for my first requirement but I can’t check the time when bandwidth was used there
  • connections track – from this I can check two sides of connection (also remote host) and how much of data was send/received but it also doesn’t show this information in time domain and it is served in less friendly, text form

 

It was no exactly what I’m looking for. Therefor I checked what what can we find in OPKG (OpenWRT Package Management).

SNMP + NagiosGraph

I tried to find how I can link Nagios (with NagiosGraph) with my router because I already have some experience with this tools. I found out that there is check_snmp Nagios plugin which can realize this. In OPKG there is mini-snmpd package. It is light SNMP server implementation. You can run it after login by SSH to you router and execute this command:

After this you can check available from server data:
In returned MIB tree there are some useful data like server’s uptime, disk space and also interface’s bandwidth. The last one, stored in Round Robin Database and printed by NagiosGraph will give graphs of bandwidth usage in time domain. But will not show who exactly use bandwidth!

 

Other software

I continue searches in OpenWRT packages. I came across good OpenWRT wiki page: http://wiki.openwrt.org/doc/howto/bwmon describing some available stuff.

 

ntop

Among other there is mentioned ntop – extensive application written in C with many views showing statistics of network protocols usage. Installation of this application on my router with 400MHz CPU will be not the best idea. So I tried to install it on my home server and only send data to it from router by fprobe. At first I installed ntop available from ubuntu 12.04 server’s APT repository. There is available 3:4.1.0+dfsg1-1 version. After some simple configuration steps ntop start drawing graphs.

 

I simulate situation when from remote server I was downloading a big file from my home server. I was disappointed when noticed that I can’t read that this situation taking place from ntop graphs.

 

listening on interface in promiscuous mode

Some time ago I’ve done tcpdump logs analyzer on my studies. I remind that interface working in promiscuous mode can collect all data about local network traffic just like the router. To enable this mode you should exec this command:

Or if you want to set this state persistent you should edit your /etc/network/interfaces to look like this:
If the server where you want to listen for all packages is a VirutalBox vhost you should also verify that it is set promiscuous mode to ”Allow all” in their network configuration like on screenshot below.

ntop v.5.0.2

After this settings we can run ntop on any server in our local network. I give a try for a development version which you can download from ntop homepage: http://www.ntop.org/get-started/download/. Configure script led me through necessary packages that you must install before compilation. After this I run make and sudo make install. To manage ntop using init scripts I used existing /etc/init.d/ntop script and just edited a line with location of DEAMON value – setting them to /usr/local/bin/ntop value. I also removed -n 0 switch from /etc/default/ntop because I hope that bug with DNS resolution is already fixed (it is a little note in config about it).

 

I started deamon by service ntop start. In syslog there was nothing alarming – ntop started collecting traffic statistics. After login I checked available features.

  • Network load – this page shows all load in our network in four time intervals: 10mins, last hour, last day, last month
  • Top talkers – similar to network load intervals, shows how hosts were using bandwidth in past
  • Traffic maps: Region map & hosts map – ntop is connected to Google Maps and shows where are located hosts that we are talking to
  • Activity: how changes activity of hosts in every hour
  • And other – there are other useful things like Protocol statistics, Map of connections between hosts generated in dot and many more
After some tests I noticed that now I have full control about how my network is used (also find out that I have some scheduled script that every minute send unnecessary MBs of data ;-)).

 

little fix

This tests help me find out that there is a little bug in page showing top talkers of an hour. I submitted patch fixing it to ntop’s request tracker if you are interested in: http://sourceforge.net/tracker/?func=detail&aid=3559097&group_id=17233&atid=367233. This is a patch to r5644.

On the end

My adventure with traffic monitoring tools ended on ntop. It is a great tool which fits my needs. Now I know who consumes my resources and can set QoS rules which make my internet connection more responsive.

— Previous article

Rapid js + css development

Next article —

How to create a native Java App

You May Also Like
DDevelopment & Design

Distributed scans with HBase

HBase is by design a columnar store, that is optimized for random reads. You just ask for a row using rowId as an identifier and you get your data instantaneously. Performing a scan on part or whole table is a completely different thing. First of all, it is sequential. Meaning it is rather slow, because it doesn't use all the RegionServers at the same time. It is implemented that way to realize the contract of Scan command - which has to return results sorted by key. So, how to do this efficiently?HBase is by design a columnar store, that is optimized for random reads. You just ask for a row using rowId as an identifier and you get your data instantaneously. Performing a scan on part or whole table is a completely different thing. First of all, it is sequential. Meaning it is rather slow, because it doesn't use all the RegionServers at the same time. It is implemented that way to realize the contract of Scan command - which has to return results sorted by key. So, how to do this efficiently?
DDevelopment & Design

Spock, Java and Maven

Few months ago I've came across Groovy - powerful language for JVM platform which combines the power of Java with abilities typical for scripting languages (dynamic typing, metaprogramming).

Together with Groovy I've discovered spock framework (https://code.google.com/p/spock/) - specification framework for Groovy (of course you can test Java classes too!). But spock is not only test/specification framework - it also contains powerful mocking tools.

Even though spock is dedicated for Groovy there is no problem with using it for Java classes tests. In this post I'm going to describe how to configure Maven project to build and run spock specifications together with traditional JUnit tests.


Firstly, we need to prepare pom.xml and add necessary dependencies and plugins.

Two obligatory libraries are:
<dependency>
    <groupid>org.spockframework</groupId>
    <artifactid>spock-core</artifactId>
    <version>0.7-groovy-2.0</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupid>org.codehaus.groovy</groupId>
    <artifactid>groovy-all</artifactId>
    <version>${groovy.version}</version>
    <scope>test</scope>
</dependency>
Where groovy.version is property defined in pom.xml for more convenient use and easy version change, just like this: 
<properties>
    <gmaven-plugin.version>1.4</gmaven-plugin.version>
    <groovy.version>2.1.5</groovy.version>
</properties>

I've added property for gmaven-plugin version for the same reason ;)

Besides these two dependencies, we can use few additional ones providing extra functionality:
  • cglib - for class mocking
  • objenesis - enables mocking classes without default constructor
To add them to the project put these lines in <dependencies> section of pom.xml:
<dependency>
    <groupid>cglib</groupId>
    <artifactid>cglib-nodep</artifactId>
    <version>3.0</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupid>org.objenesis</groupId>
    <artifactid>objenesis</artifactId>
    <version>1.3</version>
    <scope>test</scope>
</dependency>

And that's all for dependencies section. Now we will focus on plugins necessary to compile Groovy classes. We need to add gmaven-plugin with gmaven-runtime-2.0 dependency in plugins section:
<plugin>
    <groupid>org.codehaus.gmaven</groupId>
    <artifactid>gmaven-plugin</artifactId>
    <version>${gmaven-plugin.version}</version>
    <configuration>
        <providerselection>2.0</providerSelection>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
                <goal>testCompile</goal>
            </goals>
        </execution>
    </executions>
    <dependencies>
        <dependency>
            <groupid>org.codehaus.gmaven.runtime</groupId>
            <artifactid>gmaven-runtime-2.0</artifactId>
            <version>${gmaven-plugin.version}</version>
            <exclusions>
                <exclusion>
                    <groupid>org.codehaus.groovy</groupId>
                    <artifactid>groovy-all</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupid>org.codehaus.groovy</groupId>
            <artifactid>groovy-all</artifactId>
            <version>${groovy.version}</version>
        </dependency>
    </dependencies>
</plugin>

With these configuration we can use spock and write our first specifications. But there is one issue: default settings for maven-surefire plugin demand that test classes must end with "..Test" postfix, which is ok when we want to use such naming scheme for our spock tests. But if we want to name them like CommentSpec.groovy or whatever with "..Spec" ending (what in my opinion is much more readable) we need to make little change in surefire plugin configuration:
<plugin>
    <groupid>org.apache.maven.plugins</groupId>
    <artifactid>maven-surefire-plugin</artifactId>
    <version>2.15</version>
    <configuration>
        <includes>
            <include>**/*Test.java</include>
            <include>**/*Spec.java</include>
        </includes>
    </configuration>
</plugin>

As you can see there is a little trick ;) We add include directive for standard Java JUnit test ending with "..Test" postfix, but there is also an entry for spock test ending with "..Spec". And there is a trick: we must write "**/*Spec.java", not "**/*Spec.groovy", otherwise Maven will not run spock tests (which is strange and I've spent some time to figure out why Maven can't run my specs).

Little update: instead of "*.java" postfix for both types of tests we can write "*.class" what is in my opinion more readable and clean:
<include>**/*Test.class</include>
<include>**/*Spec.class</include>
(thanks to Tomek Pęksa for pointing this out!)

With such configuration, we can write either traditional JUnit test and put them in src/test/java directory or groovy spock specifications and place them in src/test/groovy. And both will work together just fine :) In one of my next posts I'll write something about using spock and its mocking abilities in practice, so stay in tune.