Multi-line snmp events not supported by SNMP Modular Input App?

[krusty]

Hi there,
we use the SNMP Modular Input App to get some application events to splunk.
For single line events the App works great.
If the splunk indexer received multi-line events we only get an event with hex data in there.

Is it possible that the App only works with single-line events?
If this is true, are there any plans to increase this functionality to multi-line events?
Or does anybody know what we have to change that we can receive readable multiline events?

Kind regards,
Tom

[Damien_Dallimor · Accepted solution]

If you require custom response handling (such as specialized character handling), then you should add a custom response handler to snmp_ta/bin/responsehandlers.py and declare it to be applied in your setup stanza. responsehandlers.py has some example custom handlers in it that you could just copy paste to get started.Should be quite easy.

[krusty]

I finaly found out how to fix the problem with the special characters. Now I tried to implement the solution into a custom reponse handler.
I just copy the original responsehandlers.py script and add my changes and save it as responsehandlers_test.py.

Then I go to the frontend and include responsehandlers_test into the field "Response Handler". Should be right?!

Do I have to do something else?

If I have a look into the splunkd.log I see this ERROR message.

03-16-2015 13:17:02.035 +0100 ERROR ExecProcessor - message from "python /splunk/opt/splunk/etc/apps/snmp_ta/bin/snmp.py" AttributeError: 'module' object has no attribute 'responsehandlers_test'

Something went wrong but I've no idea what and where.
Which attribute does snmp.py expect?

Unfortunately there is no further documentation where I can read something about the process in the Background. 😞

[Damien_Dallimor]

That is not correct.

As I wrote "...then you should add a custom response handler to snmp_ta/bin/responsehandlers.py...."

So you add a custom response handler class to the existing responsehandler.py module.

And the name of this class is what you configure.

As per the screenshot above the class name is MyCustomResponseHandler.

[krusty]

Hi Damien,

thanks for you support.
I implement a new responsehandler and with this I can modify the incoming traps.

Regards,
Tom

[Damien_Dallimor]

Assuming the mib to py translation succeeded, maybe the source mib is inaccurate hence it cant figure out how to decode the octet string,or maybe the trap sender is failing to adhere to the value constraints defined in the mib. Perhaps try hacking the constraints in the mib file ?

[krusty]

Me again,
I found out that the problem is how the snmp.py script handle the data.
The module pyasn1 from python is responsible to decode the values of the snmp data.
This module could only handle "normal" characters in this values. If there is a line-feed or some other special character the module didn't translate the data. It will only shown as a hex value.

I tried to do some testing with a shorter script on my testing server but since I'm no python expert I'm not able to fix the problem.

@Damien, could you please check if it is possible to implement a check for special characters?
I tried something like this:

if isinstance(val, OctetString):
    val = val.asOctets()

With asOctets() you are able to read also the special charaters.

[krusty]

If I read the mib file correct then the part of interest is this one.
saaEventDescription OBJECT-TYPE
SYNTAX OCTET STRING
ACCESS read-only
STATUS mandatory
DESCRIPTION "Event description"
::= { saatrap 9 }

That describes the discription where multi-line data is in.

Is it right, that the SYNTAX "OCTET STRING" can handle such multi-line events?
I could not find any explanations which SYNTAX should be used to handle multi-line events.

Have you any idea or tip?

[Damien_Dallimor]

I'm guessing. I can't possibly know every MIB. But there is definitely some sort of validation error going on with the trap data received vs the validation rule in the MIB you compiled. Perhaps the "saatrap 9" refers to the "ValueSizeConstraint(9, 9)" in the error. Dunno , purely guessing.But it is definitely a validation problem. At least the OID is resolving now though.So that is progress.

[krusty]

Yes indeed, it's a progress. 😉

I just thinking about a Workaround but not sure if it could work or not.

When the snmp event receives the splunk indexer, does the transform.conf and props.conf working before the snmp.py script runs? If yes, then I could try to search and replace for the line feeds and wordwraps in the events.

Do you think that makes sense?

I know that you couldn't know any MIB file. And I'm really happy that you nevertheless try to find a solution for my problem.
Thank you very much for this help!

[Damien_Dallimor]

does the transform.conf and props.conf
working before the snmp.py script runs

No. SNMP Mod Input trap listener receives the trap , MIB logic is applied , output is passed to the response handler in responsehandlers.py , output is written to STDOUT , Splunk reads from STDIN , data goes through the Splunk indexing pipeline.

[seanel]

Damien, does that mean that transform and props confs are ignored? i.e. only response handler is used?

[krusty]

Today I did a short test on another linux server.
I configured this server that the snmp Events are stored in a file.
When I start grabbing the events from the application I can read all of them. But I can't see any line feeds.
For me the problem is located at the SNMP Mod Input trap listener.

Is it possible to modify this listener to get some debug output?

I see that you do some replacements in the responsehandlers.py script. Is it right that you search into to raw data for the special characters ("&", "\n", ... ) ?

[Damien_Dallimor]

Assuming you are sending text (not binary)....

1) any error messages , search : index=_internal ExecProcessor error snmp.py

2) post examples of currrent and expected output

3) have you tried applying event line breaking rules in props.conf for your sourcetype ?

[krusty]

Hi Damien,

thanks for you quick reply. Yes we are sending text. All other events are looking good. Only the multi-line events are not fine.
1) We saw a lot of error messages in splunkd.log. But this happens because of the missing custom MIB file.
As you can see in the link we are not able to generate a custom mib.py file for the swift mibs. I tried everything but nothing works so far.

Error Messages:
- ERROR ExecProcessor - message from "python /splunk/opt/splunk/etc/apps/snmp_ta/bin/snmp.py" Exception resolving MIB value in the caught trap: local variable 'modName' referenced before assignment snmp_stanza:snmp://SWIFT_EVENT
- ERROR ExecProcessor - message from "python /splunk/opt/splunk/etc/apps/snmp_ta/bin/snmp.py" Exception resolving MIB name in the caught trap: NoSuchObjectError({'str': 'No MIB registered that defines 1.3.6.1.4.1.18494.2.1.9 object, closest known parent is 1.3.6.1.4.1 (SNMPv2-SMI::enterprises)'}) snmp_stanza:snmp://SWIFT_EVENT

2)
Hex value, indexed by splunk:
0x4175746f6d61746963204261636b7570206f66206d65737361676520617263686976652873290a443a5c4241434b55505c4d46415c4d4541525f32303135303232320a7375636365737366756c6c7920636f6d706c657465642e

Readable message (with newlines):
Automatic Backup of message archive(s)
D:\BACKUP\MFA\MEAR_201502222
successfully completed.

[krusty]

3) Until now I didn't use any line breaking rules.
I read the article behind your link, but I'm not sure how to configure it right. I think I have to play a littel bit with the configuration.

I think that snmp.py expect "0d0a" (DOS newline character) but the application sends "0a" as the newline character which the snmp.py didn't understand. Could this be the Problem?

[Damien_Dallimor]

Errors with parsing custom mibs to py modules is usually because you are not satisfying MIB dependencies when running smidump, in my experience anyway.

So look inside your custom MIB for any imports and get any others MIBS that are needed.

Follow some guidelines in this answer for using smidump and resolving MIB dependencies :

http://answers.splunk.com/answers/216296/cannot-resolve-node-name-for-custom-mibs.html

[krusty]

I follow the instruction of the provided link.
But now I got a couple of other error Messages in splunkd.log

ERROR ExecProcessor - message from "python /splunk/opt/splunk/etc/apps/snmp_ta/bin/snmp.py" Exception resolving MIB value in the caught trap: ConstraintsIntersection(ConstraintsIntersection(ConstraintsIntersection(), ValueSizeConstraint(0, 65535)), ValueSizeConstraint(9, 9)) failed at: "ValueSizeConstraint(9, 9) failed at: "ObjectSyntax().setComponentByPosition(0, SimpleSyntax().setComponentByPosition(1, OctetString('10:48:02')))"" at OctetString snmp_stanza:snmp://SWIFT_EVENT

And unfortunately the main problem still exists. The muli-line events are still shown in hex.