TwistedSNMP

TwistedSNMP is a set of SNMP protocol implementations for Python's Twisted Matrix networking framework using the PySNMP project.  It provides the following:

• get, set, getnext and getbulk Manager-side queries
• get, set, getnext and getbulk Agent-side services

Eventual goals of the system:

• provide access to all v1 and v2 SNMP functionality for writing Agent and Manager services
• provide convenient testing mechanisms for SNMP Agent/Manager development (e.g. mirroring an SNMP Agent's OID tree for local query testing)

You can download TwistedSNMP from the project page.  If you have feedback, feel free to contact the author.

Installation

TwistedSNMP is distributed as a Python distutils package.  You can install it by unpacking the source distribution and running:

In the top level of the install directory.  You will need both Twisted and PySNMP 3.x or PySNMP-SE installed.  The PySNMP-SE package (available from the TwistedSNMP download page) allows for faster scanning, but returns tuple-based OIDs.  PySNMP-SE is recommended for use for all new TwistedSNMP development.

Documentation

At the moment, most of the documentation for the module is in the form of the auto-generated PyDoc reference API, which includes auto-generated documentation for PySNMP.  You can find usage examples in the test sub-directory of the distribution.

Usage

TwistedSNMP is designed to provide natural APIs for dealing with remote SNMP Agents under the assumption that the code will be used to implement Manager-side applications.  At the moment, it does not support Trap handling.

For manager-side operations, we can use the port function in the snmpprotocol module to create a new Twisted port object with an attached SNMPProtocol object.  With this, we can create any number of AgentProxy objects which provide the API we use for querying the SNMP Agent.

port = snmpprotocol.port()
proxy = agentproxy.AgentProxy(
	ipAddress, 161,
	community = sys.argv[2],
	snmpVersion = 'v2',
	protocol = port.protocol,
	allowCache = True,
)

Once we have the AgentProxy, we have three major entry-points.  The first is "get", which retrieves single oid values.  You can see a full usage example in simpleget.py:

df = proxy.get(
	oids, timeout=.25, retryCount=5
)

the deferred object returned from the get method will fire with a dictionary mapping { OID: value } for each oid in oids.  OIDs which were not present on the agent will return None for their value.  If the Agent does not return a value, or returns an error, then the deferred will fire it's errback chain with the returned error.

The second entry-point is "getTable", which retrieves all sub-oids for each oid in the list of oids passed in.  You can see a full usage example in simplegettable.py:

df = proxy.getTable(
	oids, timeout=.25, retryCount=5
)

the deferred object returned from the getTable method will fire with a dictionary mapping { rootOID: { OID : value} } for each rootOID in oids.  rootOIDs which were not present on the agent will return None instead of the dictionary of { OID : value} pairs.  If the Agent does not return a value, or returns an error, then the deferred will fire it's errback chain with the returned error.

The third entry-point is "set", which takes a dictionary of { OID : value} pairs and uploads the values to the Agent.  Usage example in simpleset.py:

df = proxy.set(
	oidSet, timeout=.25, retryCount=5
)

At the moment the return value for the deferred object is just the raw SNMP message returned from the agent.  This may be changed to be something useful at some point.

For Agent-side operations, hopefully the simpleserver.py sample will get you started on how to set up testing agents (keep in mind that the goal for these agents in the current codebase is solely to test the Manager-side implementations, so they are rather primitive).  There are bisect (in-memory) and BSDDB-based OIDStores available.

Changes

Version 0.3.12

• Latency fixes to make loading of very large tables less likely to produce long hangs in applications
• Beginning of the work to provide a PySNMPv4 version of the library (unfinished, may never actually be finished)

Version 0.3.11

• Optimise the integrateNewRecord method used with pysnmp-se to reduce the number of total iterations required to process the incoming records

Version 0.3.10

• Fix bug in the Agent code, was trimming the set of OIDs returned when a particular OID reached end-of-MIB-view, now fills the rest of the value-sets with end-of-MIB-view objects, so OID sets are always indexed by the requested OIDs.
  

Version 0.3.9

• Allow retrieval of the "tail" of a table by specifying marker OIDs, that is, retrieve from the item beyond the marker OID to the end of the table, to allow for querying "new items" from a table given known previous items (assuming new items are appended to the end of the table)
  

Version 0.3.8

• Include missing module from the 0.3.8 release (and actually test with 3.4.x to be sure it works with that version)
• Release the 3.5 extensions (PySNMP-SE) so that others can play with them
• Repackage with somewhat cleaner distutils script
  

Version 0.3.7

• Provide method to cache OID encodings (when this feature is available in PySNMP (currently only in 3.5 extensions))

Version 0.3.6

• Fix bug/undesirable behaviour relating to the caching mechanism, was using the RequestId object, instead of the value of that object as the request ID key
  

Version 0.3.5

• Provides simple mechanism for caching PySNMP get and getbulk request objects to optimise queries
• Tightens up the allowed values for "oids" and "roots" parameters throughout, now must be either a sequence of OIDs (or OID-compatible values) for get/getTable or a sequence of (OID,value) pairs for set
  

Version 0.3.0

• Allows for patches to PySNMP 3.4 to use tuple-based OID objects for representing OIDs instead of strings, if the patch is installed then TwistedSNMP will produce pysnmp.ans1.oid.OID instances for all OID return values, otherwise it will continue to produce formatted-string OIDs.
• The effect of the patch on performance is dependent primary on how much tabular retrieval you do.
• Note that if you install the patch, you will almost certainly need to alter your client code to deal with OID objects rather than strings.
  

Version 0.2.15

• Uses psyco, when available, to bind various classes in PySNMP which have been identified as causing significant slowdowns.
• Splits out the OID-setting code from the message interpretation code in Agent so that it's easier to programmatically set OIDs on an Agent.
  

Version 0.2.14

• Eliminates potential memory leaks by reworking the __del__ method on BSDOIDStore objects
• Note: There is another memory leak for which I have a work-around available, but this leak is in Twisted itself (particularly the deferred object), so it's not part of the TwistedSNMP release
• Reduce log level for "Unexpected request key" message, as it is extremely common and so floods application logs
  

Version 0.2.13

• Adds ability to insert sub-OIDStores in BisectOIDStore objects, this allows you to compose OIDStores such that a given (shared) sub-storage is responsible for common application-level values
  
• Adds ability to insert callable objects, this allows you to insert functions that return calculated values for a given OID.  Functions have the signature: def callback( oid, storage ): returning a value appropriate to the given OID
  

Version 0.2.12

• Work around what appears to be a memory leak in deferred objects holding references to objects which also hold references to the deferred
• Fix for memory leak where tabular retrieval times out, was doing an incorrect test to determine whether to delete the key, so never deleted it
• Switch to use of Python's standard logging module for all debug output
• Note: This will require that you set up a logging handler, e.g. by calling logging.basicConfig() during your application's initialisation procedure
• Switch to using an internal registration point for PySNMP modules (to avoid problems when PySNMP 4 comes out with an entirely new organisation), shouldn't affect client modules save if they want to create v2c objects, if so, importing v2c from twistedsnmp.pysnmpproto is the way to go to keep in synch with the version being used by TwistedSNMP.
• Note: PySNMP 4.0 alpha is not supported yet, I'm waiting for the API to stabilise before trying to support it (it looks like a lot of non-trivial changes that will require rewriting large tracts of TwistedSNMP).
• A bit more documentation including usage examples.
  

Version 0.2.11

• Allow for passing in maxRepetitions for getBulk requests, this simply controls the size of the window requested from the Agent, which should allow you to reduce overhead in cases where you know you only want a small table
  

Version 0.2.10

• Rewrite table-retrieval results-handling to be more robust and simple
  

Version 0.2.9

• Catch socket errors during message sending and report as Twisted failures (otherwise the errors simply print exceptions and are ignored, which can hang client code that expects (rightly) an eventual return value from something which calls a TwistedSNMP method).
  

Version 0.2.8

• BSDOIDStore.close method added to actually save data to disk (useful, that :) )
• Modifications to MirrorAgent to call close on OID store
• Bug-fixes in both OIDStore classes (old BSDDB data-files will be incompatible as a result)
• Essentially, the OIDs were being stored in too naive an order, so the oid 0.1.2 would sort after 0.1.12, i.e. using simple string sorting
• BisectOIDStore now uses a tuple of integers while BSDOIDStore uses a packed binary string representation
  

Version 0.2.7

• Refactoring of Agent class to allow for easier construction of mock objects that can simulate agent dialogues without needing to go through the SNMP encoding/decoding stages
• Introduction of stand-alone errors module
• Use of errors module in OIDStore classes to indicate failed OID lookup
  

Version 0.2.6

• Mostly just documentation changes to reflect new project location URLs and the like.
  

Version 0.2.5

• Altered timeout methods to catch exceptions within themselves and relay to the caller via the registered deferred object
• Documentation enhancements
• Patch to work around Python 2.3.x bug in the BSDDB wrapper class
• Eliminated old algorithms for MassRetriever
• Reduced default MassRetriever iteration delay to 0.005 instead of 0.01 seconds
  

Version 0.2.4

• Fixed bug with table request processing.
  

Version 0.2.3

• Rework of mass retriever algorithm (yet again) to be more robust when dealing with larger AgentProxy sets
• Much simpler mechanism
• Throttles to initiate less than 200 queries/second by default, but ignores the number in play, which, as it turns out, seems to be somewhat useless for managing load.  This is not ideal, as, at least on our testing systems, the hardware can readily handle a few thousand queries/second (it's Twisted's structure that requires the throttling).
  
• Internal rework so that timeouts are cancelled ASAP for messages received
• To prevent timeouts issuing before the system gets around to calling the callback
• Discovered (as-yet-unfixed) bug in Agent's (as distinct from AgentProxy) processing of table requests
• Basically it doesn't properly handle tables of different lengths.
  

Version 0.2.2

• Bug-fix in utilities/mirroragent.py
  
• Test module cleanup for cross-platform operation
  
• Allow for operation when no bsddb module is available, as there are platforms on which it's not available by default
• Use non-privileged port if the privileged 161 is not bindable
  
• Added check to twineTables for NULL OID set (passing in such an OID set makes no sense, and would cause silent, difficult-to-track-down errors)
  

Version 0.2.1

• Cleaned up and reworked mass-query mechanism;
• patterned loosely after the table-query mechanism
  
• fixed at least one logic bug in the batching machinery
  
• added adaptive heuristics for determining number of queries to keep in play
• changes the api for retrieval in a backward-incompatible way
  
• Fixed shallow bug in TableRetriever's handling of responses with unequal rootOID and (OID,value) set lengths
  

Version 0.2.0

• Complete refactoring of the client/manager-side API.  Instead of requiring one port per managed agent, is able to manage any number of agents from a single port. Most of the machinery of the old protocol object is now part of an AgentProxy class.  You *will* need to modify all client code to work with the new API.  Mostly just a matter of a different initialisation pattern and then using the AgentProxy object in the same manner as you used the old SNMPProtocol object.
• Crude mass-query mechanism created.  As of yet only tested against a local server, and doesn't yet have balancing heuristics to throttle load in response to network response, but does appear to work.
  
• Dropped the snmpports module.  It's no longer necessary with the new structure.
• Bug was discovered in Python 2.3's BSDDB module which affects the BSDDB storages, bug report submitted to the Python project, but for now, will not pass tests under Python 2.3.
  

Version 0.1.5

• Refactored the test-suite into its own package w/ sub-modules for the different test-types
• Fixed bug in usage of oidStore API for Agent set-requests
  

Version 0.1.4

• Utilities:
• mirroragent -- mirrors an agent to local bsddb shelve for testing (still very rough, just a sketch, really)
  
• BSDOIDStore -- a new OIDStore implementation for use in utilities
  
• Major refactoring of the Agent-side mechanisms:
• AgentProtocol -- low-level send and dispatch of messages to the Agent
• Agent -- SNMP logic for get/getNext/getTable/set implementation
• OIDStore -- interface for storage/retrieval of ordered OID tree
• BisectOIDStore -- the demo OIDStore implementation for testing purposes using the bisect module
  

Version 0.1.3

• Agent and Manager-side set protocol implementation
• Minimal sample code for Agent-side set code, doesn't yet do the error-negotiation stuff
  

Version 0.1.2

• Test code
  
• Agent-side protocol sample code extended to include getnext and getbulk
  

Version 0.1.1

• Beginning of work on an Agent-side protocol and some sample Agent code

Version 0.1

• Initial release
  

TwistedSNMP  is a Open-Source Project