Why props.conf configuration with unarchive_cmd on udp input doesn't work?

[HansWurscht]

Hi,

i'm trying to implement a custom Charset-Decoder for an udp input. I'm using the following settings in props.conf:

[source::udp:515]
sourcetype=my_sourcetype
unarchive_cmd=/usr/bin/perl $SPLUNK_HOME/etc/apps/foo/bin/decode.pl

[my_sourcetype]
invalid_cause = archive

However this doesn't work. The incoming network data won't be processed by my decode-script. Is my splunk configuration correct?
I also tried other variations of the configuration, like putting unarchive_cmd in the sourcetype-stanza. But they all won't work.

I can't see any words in the documentation of unarchive_cmd and invalid_cause why this isn't supposed to work.

Any Ideas?

Thanks!

[guilmxm]

I think too.

You solution is modular input:

http://answers.splunk.com/answers/108190/how-to-index-a-binary-data-from-udp

[martin_mueller]

Looking at http://wiki.splunk.com/Community:HowIndexingWorks it seems you're right - the archiveProcessor is not part of the processing pipeline for network inputs but rather for files.

[HansWurscht]

I've created a support case to clarify this in the documentation.

[HansWurscht]

I restarted splunk after changing the configuration files.

I think the unarchive_cmd simply won't work on network inputs.

The sourcetype is set correctly, so i don't think there is a problem with that.

[guilmxm]

What about your inputs.conf config ? Do you specify the sourcetype in it ?

[guilmxm]

I see, i massively use unarchive_cmd in my App (http://apps.splunk.com/app/1753/) to decode nmon data through party converter (initially Perl then Python) you may sometimes fight with it 🙂

I mean i think your stanza does not match (like some kind of regex that will not match a pattern), but as it is, it's seems correct.

I guess you've reloaded Splunk after each setting change ?

[martin_mueller]

On the invalid_cause - that can only be set for sourcetype stanzas, not for source or host stanzas as per http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/propsconf

[HansWurscht]

@guilmxm:
Sorry, i don't understand that.

My data appears in splunk with source=udp:515 and sourcetype=my_sourcetype. The data in splunk is undecoded (shows up in a hex format \xF1\xF2\xF3...). The unarchive_cmd isn't working.

unarchice_cmd and invalid_cause are listed under the source stanza in props.conf:

[source::udp:515]
NO_BINARY_CHECK=true
sourcetype=my_sourcetype
invalid_cause = archive
unarchive_cmd=/usr/bin/perl $SPLUNK_HOME/etc/apps/foo/bin/decode.pl

Thanks!

[guilmxm]

@HansWurscht: If you have the data undecoded, then for sure your stanza is not matched, whatever you will change in configuration this won't change.
When you achieve a search against your data like "| stats by source" what is the source reported by splunk ?

[HansWurscht]

Thanks for the answers.
The data appears in splunk, just the decoding part won't work.

@guilmxm:
1. the perl-Script will read from STDIN and print to STDOUT
2. no coressponding entries in splunkd.log. I could increase the debugging level
3. I added the NO_BINARY_CHECK=true to the source stanza in props.conf. But this doesn't change anything 😕
4. I moved invalied_cause from [my_sourcetype] to [source::udp:515] but this also didn't change the behaviour 😕

@martin_mueller:
My data is coming to UDP 515. I'm testing with netcat:
cat /tmp/binary_to_splunk | netcat -u splunk.home.lan 515

[martin_mueller]

Additionally, make sure your data actually is coming in on UDP port 515 instead of the much more common syslog port 514.

[guilmxm]

And you should try moving the invalid_clause = archive from your sourcetype to your source stanza, this is where it needs to be located

[guilmxm]

Hi, I have no experience with using unarchive_cmd within UDP flow, but i do with unarchive_cmd itself.

1. Is your perl script built to read data from stdin ? (this is the way Splunk will send data to your script)
2. If the data is not proceeded by your script, this is probably that your source stanza in props.conf does not match your input, have you any trace in splunkd.log ?
3. Not sure this is relevant with UDP, but with files you have to add NO_BINARY_CHECK = true to your source stanza in props.conf