Syslog Collector

Syslog Collector listen for syslog messages and parse them following some regular expressions. It then increments some counters which are exposed with specific OIDs in a simple SNMP Agent interface

You should use Syslog Collector if you want to match some regular expressions in your syslog server messages as they are arriving, and you want to expose the counter result of this matching throughout the SNMP protocol.

More details

With Syslog Collector you have the opportunity to:

• configure some regular expressions to match in your syslog messages
• get them matched as messages are arriving
• get counter and last_matched values exposed through SNMP protocol

Configuration file is settings.py. Please refer to Configuration section of the manual for details.

Regex are defined by user in pythonic regex syntax. If regex is matched, specific counters are incremented. You can set the exit_if_matched flag in order to make the parsing suddenly stop if the corresponding regex has been matched. Normal behaviour is to match all regex with each line.

Counters live in memory. If service is restarted, counters restarts too... as almost network management support services works !

How it works

Here you find an overview of how Syslog Collector works. For some hints on using it see Usage.

Syslog Collector overview

There is only one main process which initializes the regular expressions list (i.e: regex_list) basing on settings.MSG_REGEX_LIST.

Then pass reactor (Twisted event manager) to modules:

• syslog_handler
• subagent

They configure events they listen to, and corresponding callbacks.

• Syslog handler receives syslog messages and match regular expressions. If regex matches, corresponding counter and last matched timestamp are updated.

• Subagent is a real SNMP Agent based on Twisted SNMP. It configures its own OIDs and listen for SNMP requests on configured port.

  Most often this is a subagent contacted through a main SNMP agent configured with the proxy directive

In the end run the reactor.

Registered OIDs

The OIDs registered in SNMP Agent resides in memory. Which OIDs are being exported by SNMP Agent ?

In SNMP Agent you will find OIDs starting with SUBAGENT_BASE_OID + “.” + one of the following indexes + <index of regex in MSG_REGEX_LIST +1 >:

• 1 –> name as configured in MSG_REGEX_LIST
• 2 –> regex as configured in MSG_REGEX_LIST
• 3 –> exit_if_matched as configured in MSG_REGEX_LIST
• 4 –> last_matched
• 5 –> value (counter32: how many times regex matched)

The SNMP agent exposes also some MIB2.system useful information.

Notes

Syslog Collector relies on Twisted for Syslog listener and SNMP Agent.

It uses TwistedSNMP in order to register OIDs and expose them in a (simple) SNMP Agent or, most often, in a subagent identified by the proxy directive in your main SNMP agent configuration file.

Twisted is a network framework and its server-side implementation (reactor) provides asynchronous I/O out of the box.

I have chosen Twisted because I’ve heard about it for a long time, and I’d like to learn developing with it. It has been very useful for the simple SNMP Agent implementation, whereas almost useless for Syslog listener.

In the whole application Twisted has been very useful after I’ve learned that I can use the same reactor to listen to more than one socket (thanks godog for your hints :) ) as it is a generic and flexible event manager.