Control your bandwidth using ntop

I was looking for tool which could help me check who is using my bandwidth. Here are requirements which I want from this kind of tool:
  1. local hosts bandwidth distribution - it is helpful when you are loosing your bandwidth and don't know who abuse it in your local network
  2. remote hosts bandwidth distribution - it is useful in situation when you want to have control over DoS attacks for your public homepage or when your QoS are not set well

Gargoyle

My first shoot is to check what features can give me my TP-Link TL-WR941ND router. I've installed on it Gargoyle (modification of OpenWRT with some additional features) some time ago. It has some useful monitoring features:
  • bandwidth distribution pie charts which answer for my first requirement but I can't check the time when bandwidth was used there

  • connections track - from this I can check two sides of connection (also remote host) and how much of data was send/received but it also doesn't show this information in time domain and it is served in less friendly, text form

It was no exactly what I'm looking for. Therefor I checked what what can we find in OPKG (OpenWRT Package Management).


SNMP + NagiosGraph

I tried to find how I can link Nagios (with NagiosGraph) with my router because I already have some experience with this tools. I found out that there is check_snmp Nagios plugin which can realize this. In OPKG there is mini-snmpd package. It is light SNMP server implementation. You can run it after login by SSH to you router and execute this command:

After this you can check available from server data:

In returned MIB tree there are some useful data like server's uptime, disk space and also interface's bandwidth. The last one, stored in Round Robin Database and printed by NagiosGraph will give graphs of bandwidth usage in time domain. But will not show who exactly use bandwidth!

Other software

I continue searches in OpenWRT packages. I came across good OpenWRT wiki page: http://wiki.openwrt.org/doc/howto/bwmon describing some available stuff.

ntop

Among other there is mentioned ntop - extensive application written in C with many views showing statistics of network protocols usage. Installation of this application on my router with 400MHz CPU will be not the best idea. So I tried to install it on my home server and only send data to it from router by fprobe. At first I installed ntop available from ubuntu 12.04 server's APT repository. There is available 3:4.1.0+dfsg1-1 version. After some simple configuration steps ntop start drawing graphs.

I simulate situation when from remote server I was downloading a big file from my home server. I was disappointed when noticed that I can't read that this situation taking place from ntop graphs.

listening on interface in promiscuous mode

Some time ago I've done tcpdump logs analyzer on my studies. I remind that interface working in promiscuous mode can collect all data about local network traffic just like the router. To enable this mode you should exec this command:

Or if you want to set this state persistent you should edit your /etc/network/interfaces to look like this:

If the server where you want to listen for all packages is a VirutalBox vhost you should also verify that it is set promiscuous mode to ''Allow all'' in their network configuration like on screenshot below.

ntop v.5.0.2

After this settings we can run ntop on any server in our local network. I give a try for a development version which you can download from ntop homepage: http://www.ntop.org/get-started/download/. Configure script led me through necessary packages that you must install before compilation. After this I run make and sudo make install. To manage ntop using init scripts I used existing /etc/init.d/ntop script and just edited a line with location of DEAMON value - setting them to /usr/local/bin/ntop value. I also removed -n 0 switch from /etc/default/ntop because I hope that bug with DNS resolution is already fixed (it is a little note in config about it).

I started deamon by service ntop start. In syslog there was nothing alarming - ntop started collecting traffic statistics. After login I checked available features.
  • Network load - this page shows all load in our network in four time intervals: 10mins, last hour, last day, last month
  • Top talkers - similar to network load intervals, shows how hosts were using bandwidth in past
  • Traffic maps: Region map & hosts map - ntop is connected to Google Maps and shows where are located hosts that we are talking to
  • Activity: how changes activity of hosts in every hour
  • And other - there are other useful things like Protocol statistics, Map of connections between hosts generated in dot and many more
After some tests I noticed that now I have full control about how my network is used (also find out that I have some scheduled script that every minute send unnecessary MBs of data ;-)).

little fix

This tests help me find out that there is a little bug in page showing top talkers of an hour. I submitted patch fixing it to ntop's request tracker if you are interested in: http://sourceforge.net/tracker/?func=detail&aid=3559097&group_id=17233&atid=367233. This is a patch to r5644.

On the end

My adventure with traffic monitoring tools ended on ntop. It is a great tool which fits my needs. Now I know who consumes my resources and can set QoS rules which make my internet connection more responsive.

Rapid js + css development

Background

Last time I had some work to do in OSGi web module written in Spring MVC. If we have application splitted to well-designed modules, back-end development in this framework run in OSGi environment is quite fast because after some modification we must update only one bundle (without dependencies). But programming in front-end is much less dynamic than in in modern frameworks like Ruby or Groovy. There is no build-in support to update resources "on the fly" after their modification (or I can't find it).

There is many plugins to web browser which help you build front-end from scratch in wysiwyg mode. But I can't find any which could modify resources of already ran application. Also it will be complicated to keep synchronized these modifications with our sources. Therefore I tried to use local links to my project in my application. I put code similar to this below in my page.



After redeploy I found in my Chromium console: Error::Not allowed to load local resource: file:///path/to/my/local/resource.js. After some googling I found solution: adding --allow-file-access-from-files switch to application. Unfortunately it doesn't work on my Chromium v.18. I also checked other switches: --disable-web-security and --allow-file-access but with no effect. I also tried  LocalLinks plugin but with the same result.



Solution

I found out that the simplest walkaround to this problem is to link my local resources directory in Apache2 web root. So i did this:



After this inclusion of script looks like:

As you can see, it is only difference in port in new location of script. So maybe there is a tool which helps in automatic replace this string?

Tampermonkey script

In Firefox I've been using Greasemonkey plugin which could do automatic code replacement like this on the fly. On Chrome there is Tampermonkey which is a port of Greasemonkey. I wrote script which do this thing for me:



What the script do?

It simply add on end of <head> script includes from location with replaced string from -> to. It also modify CSS link hrefs using the same approach. Both scripts and links are filtered using blacklist - it is helpful if our application using external resources.

Result

Now I can spend all of my time intended for development only writing a code. After any modification I only refresh a page in browser (I'm using IntelliJ Idea so instant autosaving is working for me). And what solutions of this problem you are using?

GWT has bug in Dev mode for Chrome

It happens when trying to check visibility of component. So be careful with
component.isVisible()
as it fails on Chrome.

Same thing is for GXT (extgwt from Sencha) as it calls underlaying GWT's isVisible()


(UncaughtExceptionHandler.java:23) 2012-07-30 12:55:02,147 [ERROR] java.lang.ClassCastException: java.lang.String cannot be cast to com.google.gwt.dom.client.Element
java.lang.ClassCastException:
java.lang.String cannot be cast to com.google.gwt.dom.client.Element
    at com.google.gwt.dom.client.DOMImpl.getParentElement(DOMImpl.java)
    at com.google.gwt.dom.client.Node$.getParentElement$(Node.java:184)
    at com.google.gwt.user.client.DOM.getParent(DOM.java:892)
    at com.extjs.gxt.ui.client.core.El.getParent(El.java:1274)
    at com.extjs.gxt.ui.client.core.El.isVisible(El.java:1889)
    at com.extjs.gxt.ui.client.core.El.isVisible(El.java:1874)
    at com.extjs.gxt.ui.client.core.El.isVisible(El.java:1898)
    at com.extjs.gxt.ui.client.widget.Component.isVisible(Component.java:767)
    at com.extjs.gxt.ui.client.widget.Component.isVisible(Component.java:754)
    at pl.touk.mnp.frontend.webapp.client.widgets.RichDateComboFilter.handleBeforeLoadEvent(RichDateComboFilter.java:258)

When using development mode, better use Firefox. In production Chrome works well without bugs.

EDIT:
When using shallow visibility checking
component.isVisible(false);
the error can be avoided. GWT isn't traversing through DOM tree and it doesn't blow with an exception but you have to check whether shallow visibility checking is sufficient (in most cases it is).

New Firefox 14 ruined all my GWT work (Ubuntu 12.04)

That's because there's no GWT Development Mode plugin for Firefox 14. And now I can't work!

Wait! There's a solution. Instead of fighting with ubuntu with reversing installed Firefox to 13. I can have parallel versions of FF.

So how to install previous version of Firefox on ubuntu?
Go there


http://mirror.dacentec.com/mozilla/firefox/releases/

and download your binary. Be cautious with 32 and 64 version. If you don't know what version do you have, call


uname -m

and it'll tell you.

Then unpack your version to desired directory (other than current Firefox's dir).
Close any running Firefox instances and run your downloaded version. Beware of upgrading both firefox and any plugins. Don't do this. This copy should remain as is.


If you're looking for unofficial GWT Development Plugin versions (for FF12, FF13 and soon for FF14)  you might looking at my gwt entries.

How to set gnome-terminal tab title. Permanently…

I've been looking for this a few times without success. There are some nasty proposals with setting PS1 variable or other tricks with .bashrc mods. Weak and doesn't actually work.
I needed a way to make a bash script that opens several tabs with fixed titles and with fixed start directories and with already started programs (for example tail/less of log files). Or solutions are very complicated.

There's a simple way.

There's a nice option for gnome-terminal -t "Tab title" which allows you to set a title for a tab. But in default profile (a switchable terminal behavior configuration) any program or bash prompt can change tab's title. So actually setting tab title doesn't work permanently.

A good solution is to create a new profile (let's call it "NoTitleChange") with options set to not to allow tab change. To do this, in gnome-terminal go to Edit->profiles... click New, apply new name. Then in tab Title and Command from drop box (When commands set own title...) choose Keep initial title.


Now you can invoke

gnome-terminal --tab-with-profile=NoTitileChange -t "My Permanent title"


and nothing will change the tab's title.

You can play with these and other options to create many tabs in one window.

gnome-terminal --tab-with-profile=NoTitleChange -t "PERMANENT" --tab-with-profile=NoTitleChange -t "TABS ARE"  --tab-with-profile=NoTitleChange -t "AWESOME"


I'm using these other options
-e, --command=STRING
                 Execute the argument to this option inside the terminal.

--working-directory=DIRNAME
                 Set the terminal's working directory to DIRNAME.

--geometry=GEOMETRY
                 X geometry specification (see "X" man page), can be specified once per window to be opened.

For example

gnome-terminal --tab-with-profile=NoTitleChange -t "src" --working-directory=/opt/workspace/play/mnp/repo/console_gxt2/mnpc  --tab-with-profile=NoTitleChange -t "DevMode" --working-directory=/opt/workspace/play/mnp/repo/console_gxt2/mnpc --command="mvn initialize exec:exec -Pdevel"


Now you can create start script that opens proper working dirs (with source code for example) and with server logs open. And you're ready to go!

GWT Hosted mode on 64bit linux

GWT for linux is build against 32bit architecture. It contains some SWT/GTK 32bit modules. So if you try to run it with 64bit java it fails

Exception in thread "main" java.lang.UnsatisfiedLinkError: /opt/tools/sdk/gwt/gwt-linux-1.5.3/libswt-pi-gtk-3235.so: /opt/tools/sdk/gwt/gwt-linux-1.5.3/libswt-pi-gtk-3235.so: wrong ELF class: ELFCLASS32 (Possible cause: architecture word width mismatch)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1807)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1703)
    at java.lang.Runtime.load0(Runtime.java:770)
    at java.lang.System.load(System.java:1003)
    at org.eclipse.swt.internal.Library.loadLibrary(Library.java:132)
    at org.eclipse.swt.internal.gtk.OS.<clinit>(OS.java:22)
    at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:63)
    at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:54)
    at org.eclipse.swt.widgets.Display.<clinit>(Display.java:126)
    at com.google.gwt.dev.GWTShell.<clinit>(GWTShell.java:301)
Could not find the main class: com.google.gwt.dev.GWTShell.  Program will exit.
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR


You have two choices. Find 64bit SWT/GTK modules with same build version (3235 in this case) - good luck! or download 32bit JRE.

I chose second option and it took me 3 mins to resolve the problem.
Find proper JRE version on Oracle site I suggest bin file instead rpm. It unpacks jre to own dir. Move that directory to some convenient location (it doesn't matter where). Edit gwt.properties and set java.executable to java exec located in 32bit JRE.
Now run your GWT hosted mode and be unstoppable developer!




You may have some warnings from GTK, such as

/usr/lib/gtk-2.0/2.10.0/menuproxies/libappmenu.so: wrong ELF class: ELFCLASS64
(GWT:351): Gtk-WARNING **: Failed to load type module: /usr/lib/gtk-2.0/2.10.0
/menuproxies/libappmenu.so


But it has no consequences for me, so far...



Some sources say that you should set an environment var:

export LIBXCB_ALLOW_SLOPPY_LOCK=1

to block bugs in X display layer but I don't know what does it mean :)


In this case I had to hadle with GWT version 1.5.3 (old corporate project)