NetMate + NmRsh
NETMATE (Network Measurement and Accounting System) is a flexible and extensible network measurement tool (meter). It can be used for accounting, delay/loss measurement, packet capturing and much more. The main advantage over other existing tools is that it can be easily extended due to its modular (class-based) structure and dynamic loadable packet processing and information export modules. A GUI for controlling multiple meters and displaying measurement results is currently under development.
NMRSH is the NetMate Remote Shell which allows to remote control NetMate meters.
- Flexibility and Extensibility
- Runtime loadable metric and export modules
- Modular architecture (C++ classes)
- Extensible Ruleset Format (XML-based)
- Portable Implementation
- GNU autotools
- OS tested: Linux (SuSE, Debian, Redhat), FreeBSD, Solaris
- Open Source (GPL)
- Configurable Multithreading
- IPv4 and IPv6 Support
- Multiple Classification Algorithms
- Automatic flow generation based on arbitrary packet attribute combinations
- Packet Sampling Support
- Secure Control Interface
- SSL Encryption
- Host-based Authentication (DNS, IP address)
- User-based Authentication (HTTP)
- Packet capturing using libpcap
- Support simultaneous measurement on multiple interfaces
- Currently only Ethernet, IPv4/IPv6, ICMP, TCP, UDP, data layer support
- Extensible to everything libpcap can capture
- Metric Modules
- Counter, bandwidth, jitter, port usage, packet length, RTP packet loss, packet ID
generation (crc32 and md5), capture (tcpdump file), RTT (ICMP echo), text
output (similar to tcpdump output), DNS latency, HTTP performance, TCP
connection setup latency
- Export Modules
- Text file, binary file, SQL (under development), IPFIX (under development)
- Remote Control via Shell Tool or Standard Web Browser
- Interactive or batch processing of meter commands
Automatic Flow Generation
Automatic flow generation is a new feature introduced in NetMate version 0.8. This page provides
a quick introduction. Automatic flow generation allows to automatically separate all traffic
matching a classification rule into flows as defined by the filter attributes. Packet processing
modules are then applied on each single flow.
Flow are separated by the filter attributes. The sum of all filter attributes is called the flow
key. To create different flows at least on of the filter attributes must match on different values.
This can be achieved by using wilcard, ranges or set matches or by specifying masks. The following
shows an example on how to use automatic flow detection.
Example: Suppose we want to account for a certain host all flows between that host and the different
destinations. All we need to do is to specify a rule matching all packets from the host going to
any host and tell NetMate that this is a rule which should use automatic flow generation. This
is done by setting the preference auto:
<!-- match all packets -->
This rule will create the following output in /tmp/accounting_test:
task: 2, module: count, tstamp: May 03 11:22:58.225847, rows: 7, columns: 9, colnames:
dstip ipver srcip packets volume first_time first_time_us last_time last_time_us
192.168.41.201, 4, 192.168.41.2, 16, 1386, 1083576161, 308775, 1083576176, 673356
220.127.116.11, 4, 192.168.41.2, 1, 106, 1083576163, 922731, 1083576163, 922731
192.168.41.116, 4, 192.168.41.2, 64, 5616, 1083576146, 370879, 1083576156, 615516
192.168.41.149, 4, 192.168.41.2, 3, 294, 1083576161, 314353, 1083576163, 332723
192.168.41.250, 4, 192.168.41.2, 4, 392, 1083576167, 992018, 1083576171, 22726
192.168.41.9, 4, 192.168.41.2, 3, 578, 1083576161, 232661, 1083576161, 240737
As expected all packet from 192.168.41.2 have been separated into flows by the destination addresses
and the count module has counted the number of packets and bytes for each flow.
Getting Started with NetMate
You will need to download and unpack the NetMate archive from the
download page first.
NetMate has been successfully compiled on ix86/Linux 2.4, ix86/FreeBSD 4
and Sparc/Solaris 8.
The following libraries are required:
The following may be required depending on the build configuration:
libxslt and libcurl are only required for the interactive remote shell
meter control tool (called nmrsh) and not for the meter itself.
A normal unix installation is made in three steps (after you've
unpacked the source archive):
You probably need to be root when doing the last command.
If you have checked out the sources from the CVS repository, read the
CVS-INFO on how to proceed.
Get a full listing of all available configure options by invoking it like:
If you want to install NetMate in a different file hierarchy than /usr/local,
you need to specify that already when running configure:
If you happen to have write permission in that directory, you can do 'make
install' without being root. An example of this would be to make a local
install in your own home directory:
By default NetMate will be compiled without SSL and thread support. SSL
can be enabled:
If you have OpenSSL installed in the default search
path for your compiler/linker, you don't need to do anything special.
If you have OpenSSL installed somewhere else (for example, /opt/OpenSSL,)
you can run configure like this:
Thread support requires a functional pthread library and can be enabled:
If you have OpenSSL installed, but with the libraries in one place and the
header files somewhere else, you have to set the LDFLAGS and CPPFLAGS
environment variables prior to running configure. Something like this
If your SSL library was compiled with rsaref (usually for use in the United
States), you may also need to set:
If you want want to compile NetMate for debugging use:
Before starting NetMate the default configuration file should be adapted. The deafult configuration
file is <install_path>/etc/NetMate/NetMate.conf.xml. When building your own configuration file it is
encouraged to copy and modify the default file.
After the successful installation of the toolkit you may run NetMate (or nmrsh, nmctl) as any
other command line tool from the shell prompt via:
cd <install_path>/bin ; ./NetMate
For convenience you may add the NetMate bin directory to your PATH environment variable or
alternatively generate a soft in the form of link from:
ln -s <install_path>/bin/NetMate /usr/local/bin/
You will probably require root permissions to generate such a soft link. Given that /usr/local/bin
is part of your PATH environment variable you now can start NetMate via:
The same techniques apply to starting the nmrsh tool and the nmctl script.
When started without any parameters the NetMate tool will start up but run without any installed
metering rules. This means that no traffic analysis is performed and no output is generated. There
are three ways to add analysis rules to NetMate:
All of these three methods can be used in combination.
The typical command line for starting NetMate is:
- start NetMate with a ruleset parameter (-r option). This will read rules from a rule file on
startup and install them. See section 18.104.22.168 for details
- configure (add/remove) new tasks on-line after startup using the nmrsh tool. This allows adding
or removing analysis rules to/from NetMate at runtime from a remote host.
- configure (add/remove) new tasks on-line after startup using a web browser. With a standard web
browser it is possible to connect to the built-in web server of NetMate and add or delete rules
on-line from a remote host.
NetMate -c <configfile> -r <rulefile>
If started successfully this prints out something like:
NetMate version 0.9.5, (c) 2003-2009 Fraunhofer Institute FOKUS, Germany
NetMate reads its settings from the supplied configuration file. If no file is supplied then the default
file <install_path>/etc/NetMate/NetMate.conf.xml is used. This file contains attribute-value pairs that
alter the behaviour of the software.
NetMate logfile is: /var/log/NetMate.log
Up and running.
The rulefile parameter tells NetMate what analysis rule(s) it shall install and run right from the start.
The rules describe packet filters and metrics to apply to traffic flows as well as how to export
the metric data. Some example rule files to start with are installed in <install_path>/etc/NetMate.
Their names have the format "example_rulesXYZ.xml". Copy and modify these files when starting to experiment
with ruleset files. An example rulefile which demonstrates how rules are specified can be viewed
NetMate has very sparse output to the terminal. Its progress is written to the log file /var/run/NetMate.log.
This file is best viewed with tools such as less or with tail -f <logfile> which constantly updates the
output when new logging messages appear.
The complete NetMate user and developer manual is available in HTML, PDF and postscript
Man pages are available for NetMate, nmctrl and nmrsh.
List of reports and papers covering NetMate.
|Sep 10, 2009
||- latest NetMate ChangeLog
|Jul 5, 2006
||- improve command line parameter handling and usage output
(including fixing one incorrect long option name)
- fixed the doxygen test in configure.in
- fixed a bug in nmrsh that caused the sending of wrong meter commands to netmate
- some internal changes to make gcc 4 happy (hash_map moved etc.)
|Dec 7, 2005
||- can read mutiple trace files if specified on the command line
as comma separated list
- flow record status (0=interim, 1=final) is now optionally exported
put an yes in the export configuration
- now exits and prints pcap error if pcap_dispatch fails
- fixed ctext_file module name in debug output of module
- enabled large file support for export modules
- fixed bug in handling flow timeout which occured if a trace files
contains data from a single source destination pair
- fixed BINARY output (e.g. MAC addresses) in all export modules
- fixed variance computation in jitter module
- fixed module loading (recent libtool generates .so files again)
- fixed a bug in packet processor that caused a packet to be assigned
to the forward direction also it is in fact backward; this problem
only occured with the hack that swaps flow directions if the first
packet comes from a port<1024 and goes to a port>1024
|May 6, 2005
||- fixed missing # in nmctl copyright
- fixes to make netmate compile with gcc 3.4
- flow ids are now "unique" and optionally exported to files
(flow ids are 2^64 bit so eventually they wrap around)
put an yes in the export configuration
- fixed a bug hich caused all export data to be ouput twice, which
occured if reading tracefiles and specifying interval or stop
time in the rule set
- rule data will now always exported when rule expires or is terminated through ctrlcomm
- changed realm name shown in the username/password browser dialog to "NetMate"
- fixed c_str() coversion bugs in ctrlcomm messages (ends inserts a \0
for ostringstream :( )
- the password callback option has been removed from libcurl 7.11.
therefore we can't use it any longer. the getpass function from
libcurl has been included as lib to provide the same functionality.
|Dec 3, 2004
||- fixed export of string/binary flow keys
- fixed bug where the filter definition mask wasn't applied to the
- added automatic termination of the meter when reading tcpdump files after reaching the end of the file
- added data export when the meter terminates (EOF or CTRL-C)
- fixed bug in exporter for binary data types which resulted in div/0
- byte-sized flowkeys (such as IP version) are exported shifted
downwards if their mask only uses higher order bits of the byte
- added support for VLAN (802.1Q) ether type
- fixed host byte order conversion for 16 and 32 bit flow key values
- replaced deprecated strstream by sstream
- fixed a bug in flow key to host byte order conversion which produced
error when multiple export modules where used in the same rule
- added support for ATM RFC1483 link layer
- added better trace file support; now this tracefiles for all event
handling instead of the wallclock time the time from the tracfile
is used (as stored in the pcap packet headers). this allows the
proper use of rule start, stop, duration and flow timeout.
- optimized some internal data structures
- fixed a bug where auto flows starting with a reverse matched packet
(according to the classifier) where incorrectly handled
- when auto flows are used now the rule filters appear in the output
file in exactly the same order as specified in the rule (previously they were alphabetically sorted)
|April 3, 2004
||- added support for using raw IP (no link level headers) as input
- fixed a bug when reading from a packet savefile, in this case filter rules must be installed before the net tap is active
|May 3, 2004
||- added multiple interface support (NetMate -i eth0,eth1 works)
- added automatic flow creation: this feature allows an automatic flow detection within a rule (all packet processing modules are applied on each flow)
- changed the interface through which export modules access the metric data
- the text_file module does no longer delete files anymore when new rules are started
- text_file module now creates new directories if necessary
- changed rule parsing functions so that there are not DNS queries anymore for numeric IP adresses used in rulesets
- fixed a bug in configure.in which caused wrong default paths to be set in config.h when configure was started with --prefix
- change build logic so that if libxslt or libcurl are not found nmrsh won't be build
- added check for termcap in configure.in otherwise the readline check fails (on Redhat)
- added check for curl_free in configure.in (old curl version do not have that function)
- fixed warning which occur when creating documentation with new
- fixed set socket buffer size in NetTapPcap (now also works if socket is blocking)
- fixed a bug in debugging code of ClassifierRFC
- fixed a memory leak which occured when parsing IP adresses after a DNS timeout
- fixed a bug in the internal rule id usage
- fixed a bug which prevented make dist with VPATH
|Feb 23, 2004
||First beta release
If you have further questions, wishes or suggestions please contact the
NetMate core team.
NetMate Core Team
||- Modules architecture, packet processing and export, logging, manual, logo
||- Chief architect, make system, task scheduling,
classification, control protocol
||- Nice idea for the logo
||- Packet ID modules
||- Web page