LOBSTER

This article is about a network technology. For other uses, see Lobster (disambiguation).

LOBSTER was a European network monitoring system, based on passive monitoring of traffic on the internet. Its functions were to gather traffic information as a basis for improving internet performance, and to detect security incidents.

Objectives

• To build an advanced pilot European Internet traffic monitoring infrastructure based on passive network monitoring sensors.
• To develop novel performance and security monitoring applications, enabled by the availability of the passive network monitoring infrastructure, and to develop the appropriate data anonymisation tools for prohibiting unauthorised access or tampering of the original traffic data.

History

The project originated from SCAMPI, a European project active in 2004–5, aiming to develop a scalable monitoring platform for the Internet. LOBSTER was funded by the European Commission and ceased in 2007. It fed into "IST 2.3.5 Research Networking testbeds", which aimed to contribute to improving internet infrastructure in Europe.

36 LOBSTER sensors were deployed in nine countries across Europe by several organisations. At any one time the system could monitor traffic across 2.3 million IP addresses. It was claimed that more than 400,000 Internet attacks were detected by LOBSTER.

Passive monitoring

LOBSTER was based on passive network traffic monitoring. Instead of collecting flow-level traffic summaries or actively probing the network, passive network monitoring records all IP packets (both headers and payloads) that flow through the monitored link. This enables passive monitoring methods to record complete information about the actual traffic of the network, which allows for tackling monitoring problems more accurately compared to methods based on flow-level statistics or active monitoring.

The passive monitoring applications running on the sensors were developed on top of MAPI (Monitoring Application Programming Interface), an expressive programming interface for building network monitoring applications, developed in the context of the SCAMPI and LOBSTER projects. MAPI enables application programmers to express complex monitoring needs, choose only the amount of information they are interested in, and therefore balance the monitoring overhead with the amount of the received information. Furthermore, MAPI gives the ability for building remote and distributed passive network monitoring applications that can receive monitoring data from multiple remote monitoring sensors.

Developed applications

The LOBSTER sensors operated by the various organisations monitored the network traffic using different measurement applications. All applications were developed within the LOBSTER project using MAPI, according to the needs of each organisation.

• Appmon, an application for Accurate Per-Application Network Traffic Classification.
• Stager, a system for aggregating and presenting network statistics.
• ABW, an application written on top of LOBSTER DiMAPI (Distributed Monitoring Application Interface) and tracklib library.