AbuseHelper
 | This article has an unclear citation style. (December 2015) |
AbuseHelper is an open-source project initiated by the computer emergency response teams (CERTs) of Finland and Estonia with ClarifiedNetworks to automatically process incidents notifications.
This tool is being developed for CERTs and Internet Service Providers (ISPs) to help them in their daily job of following and treating a wide range of high-volume information sources. The framework can also be used for automatically processing (standardized) information from a wide range of sources.
Context
[edit]CERTs and ISPs have to handle a very high-volume of notifications (E-mail spam, Botnets, ...). These notifications are often normalized per feed (each feed typically uses different formats to report). There is also a lot of information about Internet abuse, available by different feed providers (Zone-H Zone-H[1], DShield Dshield[2], Zeus Tracker Zeus (Trojan horse) [3]...). This information is not well utilized, as the amount of information is too big for manual processing. AbuseHelper follows a number of sources and produces actionable reports and dashboard for the people that need to treat all these notifications. AbuseHelper also automates the enriching of information, such as finding the owners of reported IP addresses from public databases (such as Whois).
History
[edit]Technical developments that led to collaborative effort on solving the automated collection of Abuse Information
- 2005 CERT-FI Autoreporter gen1, implemented with Perl
- 2006-2007 CERT-FI Autoreporter generations 2&3 (incremental updates to gen1). Plans to rewrite
- 2008-2009 CERT-FI Autoreporter gen4, proof-of-concept implementation using sh. Paper describing the prototype won the joint FIRST.org & CERT/CC contest for the best practices and advances in safeguarding the security of computer systems and networks in 2009
- 2009 CERT-FI gen5, implemented with Python. Full rewrite
- 2009-10 Clarified Networks & CERT-EE Abusehelper collaboration starts
- 2009-11 CERT-FI joins.
- 2010-01 AbuseHelper first public release
- 2010-01 First training @ TF-CSIRT event in Germany
- 2010-03 CERT.BE (Belgium) / BELNET CERT joined.
- 2011-07 CERT.IS (Iceland) joined
Architecture
[edit]AbuseHelper is written in Python and developed relying on XMPP protocol (not mandatory) and agents. The base principle is to control agents via a central chat room where all bots are listening. Agents are exchanging information in subrooms. AbuseHelper relies on a modular approach to workflows and attempts to be scalable by keeping every agent simple.
Sources
[edit]The goal of AbuseHelper is to handle a large panel of sources and try to extract useful information to follow-up on events. Currently, AbuseHelper is able to parse the following types of sources:
- email containing an ARF attachment (Abuse Report Format) (like CleanMX or abusix or cyscon's C-SIRT);
- email with a CSV (could be zipped) attachment or an URL to a CSV file (like ShadowServer);
- IRC events (live feed);
- XMPP events (live feed);
- DShield events.
The community is working on being able to handle more type of input formats. Each type of input is handled by a dedicated bot.
Internal information processing
[edit]AbuseHelper is more than a pipe. In the workflow, it could be decided to add extra informations coming from other sources like:
- Whois to retrieve abuse contact (typically people you have to contact when something security related happened);
- CRM (Customer Relation Management) to retrieve the same kind of info than for Whois.
Output
[edit]By default, AbuseHelper produces the following kind of reports:
- Mail reports with digests of events and a CSV attachments with all observed events for a time frame following some conditions;
- Wiki report - AbuseHelper wrote the incidents to a wiki;
- SQL report - AbuseHelper writes all events to a SQL database.
Generic agents
[edit]At all steps, there are standard agents:
- Roomgraph to transport events for one chat room to another based on some rules;
- Historian to log all events observed in each chat room.
Community
[edit]AbuseHelper is developed by an open-source community composed by:
- Clarified Networks [4]
- CERT-FI (Finland) [5]
- CERT.EE (Estonia) [6]
- CERT.BE (Belgium) / BELNET CERT [7] / [8]
- CSC/FUNET (Finland)
- University of Oulu (OUSPG)
References
[edit]As of this edit, this article uses content from "Workshops BruCON 2010", which is licensed in a way that permits reuse under the Creative Commons Attribution-ShareAlike 3.0 Unported License, but not under the GFDL. All relevant terms must be followed.
General references
[edit]- https://github.com/abusesa/abusehelper
- https://web.archive.org/web/20100922054126/http://2010.hack.lu/index.php/Workshops#AbuseHelper_Workshop
- http://2010.brucon.org/index.php/Workshops
- https://www.clarifiednetworks.com/collab/abusehelper[permanent dead link] CollabWiki of AbuseHelper (Invitation-based access)
- Bitbucket source code repository: [9]
- CERT.BE [10]