Home
NetMate PDF Print E-mail
Written by Administrator   

NETMATE

+

NetMate + NmRsh

Content



Introduction

NETMATE (Network Measurement and Accounting System) is a flexible and extensible network measurement tool (meter). It can be used for accounting, delay/loss measurement, packet capturing and much more. The main advantage over other existing tools is that it can be easily extended due to its modular (class-based) structure and dynamic loadable packet processing and information export modules. A GUI for controlling multiple meters and displaying measurement results is currently under development.

NMRSH is the NetMate Remote Shell which allows to remote control NetMate meters.

Project-Site:

NetMate Features

  • Flexibility and Extensibility
    • Runtime loadable metric and export modules
    • Modular architecture (C++ classes)
    • Extensible Ruleset Format (XML-based)
  • Portable Implementation
    • GNU autotools
    • OS tested: Linux (SuSE, Debian, Redhat), FreeBSD, Solaris
  • Open Source (GPL)
  • Configurable Multithreading
  • IPv4 and IPv6 Support
  • Multiple Classification Algorithms
  • Automatic flow generation based on arbitrary packet attribute combinations
  • Packet Sampling Support
  • Secure Control Interface
    • SSL Encryption
    • Host-based Authentication (DNS, IP address)
    • User-based Authentication (HTTP)
  • Packet capturing using libpcap
    • Support simultaneous measurement on multiple interfaces
    • Currently only Ethernet, IPv4/IPv6, ICMP, TCP, UDP, data layer support
    • Extensible to everything libpcap can capture
  • Metric Modules
    • Counter, bandwidth, jitter, port usage, packet length, RTP packet loss, packet ID generation (crc32 and md5), capture (tcpdump file), RTT (ICMP echo), text output (similar to tcpdump output), DNS latency, HTTP performance, TCP connection setup latency
  • Export Modules
    • Text file, binary file, SQL (under development), IPFIX (under development)
  • Remote Control via Shell Tool or Standard Web Browser
  • Interactive or batch processing of meter commands


Automatic Flow Generation

Automatic flow generation is a new feature introduced in NetMate version 0.8. This page provides a quick introduction. Automatic flow generation allows to automatically separate all traffic matching a classification rule into flows as defined by the filter attributes. Packet processing modules are then applied on each single flow.

Flow are separated by the filter attributes. The sum of all filter attributes is called the flow key. To create different flows at least on of the filter attributes must match on different values. This can be achieved by using wilcard, ranges or set matches or by specifying masks. The following shows an example on how to use automatic flow detection.

Example: Suppose we want to account for a certain host all flows between that host and the different destinations. All we need to do is to specify a rule matching all packets from the host going to any host and tell NetMate that this is a rule which should use automatic flow generation. This is done by setting the preference auto:

 

<RULE ID="2">

<!-- match all packets -->
<FILTER NAME="IPVer">0x40</FILTER>
<FILTER NAME="SrcIP">192.168.41.2</FILTER>
<FILTER NAME="DstIP">*</FILTER>

<ACTION NAME="count">
</ACTION>

<EXPORT NAME="text_file">
<PREF NAME="Filename">/tmp/accouting_test</PREF>
</EXPORT>

<PREF NAME="auto">yes</PREF>
<PREF NAME="FlowTimeout">60</PREF>
</RULE>

This rule will create the following output in /tmp/accounting_test:

 

task: 2, module: count, tstamp: May 03 11:22:58.225847, rows: 7, columns: 9, colnames: 
dstip ipver srcip packets volume first_time first_time_us last_time last_time_us
192.168.41.201, 4, 192.168.41.2, 16, 1386, 1083576161, 308775, 1083576176, 673356
192.186.41.5, 4, 192.168.41.2, 1, 106, 1083576163, 922731, 1083576163, 922731
192.168.41.116, 4, 192.168.41.2, 64, 5616, 1083576146, 370879, 1083576156, 615516
192.168.41.149, 4, 192.168.41.2, 3, 294, 1083576161, 314353, 1083576163, 332723
192.168.41.250, 4, 192.168.41.2, 4, 392, 1083576167, 992018, 1083576171, 22726
192.168.41.9, 4, 192.168.41.2, 3, 578, 1083576161, 232661, 1083576161, 240737

As expected all packet from 192.168.41.2 have been separated into flows by the destination addresses and the count module has counted the number of packets and bytes for each flow.


Getting Started with NetMate

You will need to download and unpack the NetMate archive from the download page first.

NetMate has been successfully compiled on ix86/Linux 2.4, ix86/FreeBSD 4 and Sparc/Solaris 8.

Requirements

The following libraries are required:

The following may be required depending on the build configuration:

libxslt and libcurl are only required for the interactive remote shell meter control tool (called nmrsh) and not for the meter itself.

Installation

A normal unix installation is made in three steps (after you've unpacked the source archive):
	./configure
make
make install
You probably need to be root when doing the last command.

If you have checked out the sources from the CVS repository, read the CVS-INFO on how to proceed.

Get a full listing of all available configure options by invoking it like:

 

	./configure --help
If you want to install NetMate in a different file hierarchy than /usr/local, you need to specify that already when running configure:

 

	./configure --prefix=/path/to/NetMate/tree

If you happen to have write permission in that directory, you can do 'make install' without being root. An example of this would be to make a local install in your own home directory:

 

	./configure --prefix=$HOME
make
make install
By default NetMate will be compiled without SSL and thread support. SSL can be enabled:

 

	./configure --enable-ssl
If you have OpenSSL installed in the default search path for your compiler/linker, you don't need to do anything special. If you have OpenSSL installed somewhere else (for example, /opt/OpenSSL,) you can run configure like this:

 

	./configure --with-ssl=/opt/OpenSSL
Thread support requires a functional pthread library and can be enabled:

 

	./configure --enable-threads
If you have OpenSSL installed, but with the libraries in one place and the header files somewhere else, you have to set the LDFLAGS and CPPFLAGS environment variables prior to running configure. Something like this should work:

 

  • (with the Bourne shell and its clones):
    CPPFLAGS="-I/path/to/ssl/include" LDFLAGS="-L/path/to/ssl/lib" ./configure
  • (with csh, tcsh and their clones):
    env CPPFLAGS="-I/path/to/ssl/include" LDFLAGS="-L/path/to/ssl/lib" ./configure
If your SSL library was compiled with rsaref (usually for use in the United States), you may also need to set:

 

	LIBS=-lRSAglue -lrsaref
If you want want to compile NetMate for debugging use:

 

	./configure --enable-debug

Configuration

Before starting NetMate the default configuration file should be adapted. The deafult configuration file is <install_path>/etc/NetMate/NetMate.conf.xml. When building your own configuration file it is encouraged to copy and modify the default file.

Running

After the successful installation of the toolkit you may run NetMate (or nmrsh, nmctl) as any other command line tool from the shell prompt via:
	cd <install_path>/bin  ;  ./NetMate
or
	<install_path>/bin/NetMate
For convenience you may add the NetMate bin directory to your PATH environment variable or alternatively generate a soft in the form of link from:
	ln -s <install_path>/bin/NetMate /usr/local/bin/
You will probably require root permissions to generate such a soft link. Given that /usr/local/bin is part of your PATH environment variable you now can start NetMate via:
	NetMate [<options>]
The same techniques apply to starting the nmrsh tool and the nmctl script. When started without any parameters the NetMate tool will start up but run without any installed metering rules. This means that no traffic analysis is performed and no output is generated. There are three ways to add analysis rules to NetMate:
  1. start NetMate with a ruleset parameter (-r option). This will read rules from a rule file on startup and install them. See section 3.2.3.1 for details
  2. configure (add/remove) new tasks on-line after startup using the nmrsh tool. This allows adding or removing analysis rules to/from NetMate at runtime from a remote host.
  3. configure (add/remove) new tasks on-line after startup using a web browser. With a standard web browser it is possible to connect to the built-in web server of NetMate and add or delete rules on-line from a remote host.
All of these three methods can be used in combination. The typical command line for starting NetMate is:
	NetMate -c <configfile> -r <rulefile>
If started successfully this prints out something like:
	NetMate version 0.9.5, (c) 2003-2009 Fraunhofer Institute FOKUS, Germany
NetMate logfile is: /var/log/NetMate.log
Up and running.
NetMate reads its settings from the supplied configuration file. If no file is supplied then the default file <install_path>/etc/NetMate/NetMate.conf.xml is used. This file contains attribute-value pairs that alter the behaviour of the software.
The rulefile parameter tells NetMate what analysis rule(s) it shall install and run right from the start. The rules describe packet filters and metrics to apply to traffic flows as well as how to export the metric data. Some example rule files to start with are installed in <install_path>/etc/NetMate. Their names have the format "example_rulesXYZ.xml". Copy and modify these files when starting to experiment with ruleset files. An example rulefile which demonstrates how rules are specified can be viewed here.

NetMate has very sparse output to the terminal. Its progress is written to the log file /var/run/NetMate.log. This file is best viewed with tools such as less or with tail -f <logfile> which constantly updates the output when new logging messages appear.


NetMate Documentation

Manuals

The complete NetMate user and developer manual is available in HTML, PDF and postscript (PS) format.

Man Pages

Man pages are available for NetMate, nmctrl and nmrsh.

Technical Reports

List of reports and papers covering NetMate.


Release Notes

Date                 
Version 
Notes/Changes
Sep 10, 2009 0.9.5 - latest NetMate ChangeLog
Jul 5, 2006 0.9.4 - improve command line parameter handling and usage output
(including fixing one incorrect long option name)
- fixed the doxygen test in configure.in
- fixed a bug in nmrsh that caused the sending of wrong meter commands to netmate
- some internal changes to make gcc 4 happy (hash_map moved etc.)
Dec 7, 2005 0.9.3 - can read mutiple trace files if specified on the command line
as comma separated list
- flow record status (0=interim, 1=final) is now optionally exported
put an yes in the export configuration
- now exits and prints pcap error if pcap_dispatch fails
- fixed ctext_file module name in debug output of module
- enabled large file support for export modules
- fixed bug in handling flow timeout which occured if a trace files
contains data from a single source destination pair
- fixed BINARY output (e.g. MAC addresses) in all export modules
- fixed variance computation in jitter module
- fixed module loading (recent libtool generates .so files again)
- fixed a bug in packet processor that caused a packet to be assigned
to the forward direction also it is in fact backward; this problem
only occured with the hack that swaps flow directions if the first
packet comes from a port<1024 and goes to a port>1024
May 6, 2005 0.9.2 - fixed missing # in nmctl copyright
- fixes to make netmate compile with gcc 3.4
- flow ids are now "unique" and optionally exported to files
(flow ids are 2^64 bit so eventually they wrap around)
put an yes in the export configuration
- fixed a bug hich caused all export data to be ouput twice, which
occured if reading tracefiles and specifying interval or stop
time in the rule set
- rule data will now always exported when rule expires or is terminated through ctrlcomm
- changed realm name shown in the username/password browser dialog to "NetMate"
- fixed c_str() coversion bugs in ctrlcomm messages (ends inserts a \0
for ostringstream :( )
- the password callback option has been removed from libcurl 7.11.
therefore we can't use it any longer. the getpass function from
libcurl has been included as lib to provide the same functionality.
Dec 3, 2004 0.9 - fixed export of string/binary flow keys
- fixed bug where the filter definition mask wasn't applied to the
flowkeys
- added automatic termination of the meter when reading tcpdump files after reaching the end of the file
- added data export when the meter terminates (EOF or CTRL-C)
- fixed bug in exporter for binary data types which resulted in div/0
- byte-sized flowkeys (such as IP version) are exported shifted
downwards if their mask only uses higher order bits of the byte
- added support for VLAN (802.1Q) ether type
- fixed host byte order conversion for 16 and 32 bit flow key values
- replaced deprecated strstream by sstream
- fixed a bug in flow key to host byte order conversion which produced
error when multiple export modules where used in the same rule
- added support for ATM RFC1483 link layer
- added better trace file support; now this tracefiles for all event
handling instead of the wallclock time the time from the tracfile
is used (as stored in the pcap packet headers). this allows the
proper use of rule start, stop, duration and flow timeout.
- optimized some internal data structures
- fixed a bug where auto flows starting with a reverse matched packet
(according to the classifier) where incorrectly handled
- when auto flows are used now the rule filters appear in the output
file in exactly the same order as specified in the rule (previously they were alphabetically sorted)
April 3, 2004 0.8.1 - added support for using raw IP (no link level headers) as input
- fixed a bug when reading from a packet savefile, in this case filter rules must be installed before the net tap is active
May 3, 2004 0.8 - added multiple interface support (NetMate -i eth0,eth1 works)
- added automatic flow creation: this feature allows an automatic flow detection within a rule (all packet processing modules are applied on each flow)
- changed the interface through which export modules access the metric data
- the text_file module does no longer delete files anymore when new rules are started
- text_file module now creates new directories if necessary
- changed rule parsing functions so that there are not DNS queries anymore for numeric IP adresses used in rulesets
- fixed a bug in configure.in which caused wrong default paths to be set in config.h when configure was started with --prefix
- change build logic so that if libxslt or libcurl are not found nmrsh won't be build
- added check for termcap in configure.in otherwise the readline check fails (on Redhat)
- added check for curl_free in configure.in (old curl version do not have that function)
- fixed warning which occur when creating documentation with new
doxygen 1.3.2
- fixed set socket buffer size in NetTapPcap (now also works if socket is blocking)
- fixed a bug in debugging code of ClassifierRFC
- fixed a memory leak which occured when parsing IP adresses after a DNS timeout
- fixed a bug in the internal rule id usage
- fixed a bug which prevented make dist with VPATH
Feb 23, 2004 0.7 First beta release


Related Projects


Contact

If you have further questions, wishes or suggestions please contact the NetMate core team.

NetMate Core Team

Carsten Schmoll - Modules architecture, packet processing and export, logging, manual, logo
Sebastian Zander - Chief architect, make system, task scheduling, classification, control protocol

Contributors

Elisa Boschi - Nice idea for the logo
Tanja Zseby - Packet ID modules
Sebastian Ludwig    
- Web page
Last Updated on Tuesday, 26 January 2010 11:20