Recent Changes - Search:

SPAN

News and Events

Teaching

Projects

People

Publications

Data & Tools

Schedule

Links

CFP

Map-tools

Abstract

Map-tools allows quick and easy visualization of very high-dimensional data. In particular, it has been applied to detect spatial anomalies in network backbone traffic.

Introduction

This page shows example visualizations of network traffic data, and descriptions of the tools used to generate them. Most visualization of networks is based on connectivity, rather than similarity of the traffic that travels between them. Further, traffic tools for detecting (and displaying) anomalies typically watch for temporal anomalies. This tool can be used also to detect and display spatial anomalies.

Reference

N. Patwari, A. O. Hero, and A. Pacholski, Manifold Learning Visualization of Network Traffic Data, in Proceedings of the 2005 Workshop on Mining Network Data (MineNet'05), Philadelphia, PA, August 26, 2005, pp. 191-196.

Presentation Slides

Slides from the lecture, "Network Visualization," made by Neal Patwari to CS 5480, Computer Networks, also provide some motivation and background into the network data visualization problem, along with an introduction to manifold learning.

Map-tools code

Map-tools is a set of C-code and bash-script utilities for command-line processing of NetFlow data. There are several tools used to process NetFlow data into sensor map visualizations. The flow of the several tools is shown below.

Code Download

The code is freely available for download. Download and install proceeds as follows.

  1. Follow the instructions to download and install wnlib.
  2. Set a environment variable for WNLIB to the directory in which you just installed the wnlib package.
  3. Create a directory for nplib.
     $ cd < directory where nplib is to reside >
     $ mkdir nplib
     $ cd nplib
  1. Download nplib.tar.gz. This gzipfile contains about 94 kbytes of data compressed. You will need about 2M of free disk space to build everything.
  2. Decode the gz file. Type
     $ gunzip nplib.tar.gz

you probably have 'gunzip', if you don't, it can be obtained from http://www.gzip.org or http://www.gnu.org.

  1. Un-tar the code. Type
     $ tar xvf nplib.tar
  1. Compile the code. Type
     $ make all

Program Flow Chart:

Executable Description

Executable NameDescription
sensorRouter, sensorPort, and sensorTimeThese bash shell scripts run flow-tools and extract the desired data when sensors are either routers, ports, or time. The measurements can be either flows, octets, or packets, separated in any way that flow-stat is able. For example, traffic can be divided by source or destination port, IP address, or autonomous system (AS). An arbitrary filter using flow-filter or flow-nfilter can also be applied to limit, for example, the ports or IP addresses of the input traffic. (Flow-tools was created by Mark Fullmer and information is available online.) The output is the sparse data vectors in a two-column text format.
spl2distThis C-code executable inputs the two-column sparse data vectors and outputs the distance between each pair of vectors. When N sparse data vectors are input, an N by N matrix is output. The data vectors can be optionally normalized, to use percent of total rather than absolute traffic numbers. Distance is calculated as L2 (Euclidean) distance.
wmdsThis C-code executable inputs the N by N distance matrix and outputs low-dimensional coordinates. The number of dimensions defaults to 2, but can be set to any positive integer. The dimension reduction is done using the weighted multi-dimensional scaling (wMDS) method, as described in the paper. Arbitrary prior coordinates can be set, along with the weights and weighting scheme. Neighbors can be selected via K-nearest-neighbors, with an arbitrary integer for K.
coords2epsThis C-code executable inputs N 2-dimensional coordinates and residuals ei, and produces an EPS file which plots the sensor map. The axis limits can be chosen automatically or set on the command line.

The code was developed in part using Will Naylor and Bill Chapman's WNLIB subroutine library, which is a free, unrestricted ANSI C subroutine library.

Example Usage of Map-tools

  • This line will output a distance matrix.
    $ cat Thursday, January 6, 2005.1755.f9.S1.sdat | spl2dist > temp.dst
  • Take a distance matrix and output a set of 2-D coordinates.
    $ cat temp.dst | wmds -n 11 -K 5 -p fourWeekJanAvg.f8.S1.K5.r10-3.crds 
      -r 0.001 -w loess -ND  > temp.crds
  • Take a coordinate list and output an eps graphic.
    $ cat temp.crds | coords2eps -n 11 -m fourWeekJanAvg.f8.S1.K5.r10-3.crds 
      -z -c abilenePrior.conn > temp.eps
  • You can now view temp.eps using gview.
  • Taking all commands together,
    $ cat Thursday, January 6, 2005.1755.f9.S1.sdat | spl2dist  |    wmds -n 11 -K 5 -p 
      fourWeekJanAvg.f8.S1.K5.r10-3.crds -r 0.001 -w loess -ND | coords2eps 
      -n 11 -m fourWeekJanAvg.f8.S1.K5.r10-3.crds -z -c abilenePrior.conn 
      > temp.eps

See the .man files associated with each command, included in the map-tools code download, for a detailed description of all command-line options.

Image Database

NetFlow data was collected from January 2 to January 29, 2005 from the 11 routers in the Abilene backbone network. Sample visualizations are given at http://www.ece.utah.edu/~npatwari/mnd05/.

ATLAAtlanta
CHINChicago North
DNVRDenver
HSTNHouston
IPLSIndianapolis
KSCYKansas City
LOSALos Angeles
NYCMNew York City
SNVASunnyvale
STTLSeattle
WASHWashington
Edit - History - Print - Recent Changes - Search
Page last modified on October 17, 2007, at 12:11 PM MST