topic Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work? in Getting Data In

Why props.conf configuration with unarchive_cmd on udp input doesn't work?

HansWurscht — Mon, 28 Sep 2020 17:20:10 GMT

Hi,

i'm trying to implement a custom Charset-Decoder for an udp input. I'm using the following settings in props.conf:

[source::udp:515]
sourcetype=my_sourcetype
unarchive_cmd=/usr/bin/perl $SPLUNK_HOME/etc/apps/foo/bin/decode.pl

[my_sourcetype]
invalid_cause = archive

However this doesn't work. The incoming network data won't be processed by my decode-script. Is my splunk configuration correct?
I also tried other variations of the configuration, like putting unarchive_cmd in the sourcetype-stanza. But they all won't work.

I can't see any words in the documentation of unarchive_cmd and invalid_cause why this isn't supposed to work.

Any Ideas?

Thanks!

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

guilmxm — Mon, 28 Sep 2020 17:20:12 GMT

Hi, I have no experience with using unarchive_cmd within UDP flow, but i do with unarchive_cmd itself.

Is your perl script built to read data from stdin ? (this is the way Splunk will send data to your script)
If the data is not proceeded by your script, this is probably that your source stanza in props.conf does not match your input, have you any trace in splunkd.log ?
Not sure this is relevant with UDP, but with files you have to add NO_BINARY_CHECK = true to your source stanza in props.conf

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

guilmxm — Thu, 14 Aug 2014 12:10:26 GMT

And you should try moving the invalid_clause = archive from your sourcetype to your source stanza, this is where it needs to be located

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

martin_mueller — Thu, 14 Aug 2014 12:13:10 GMT

Additionally, make sure your data actually is coming in on UDP port 515 instead of the much more common syslog port 514.

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

HansWurscht — Mon, 28 Sep 2020 17:20:15 GMT

Thanks for the answers.
The data appears in splunk, just the decoding part won't work.

@guilmxm:
1. the perl-Script will read from STDIN and print to STDOUT
2. no coressponding entries in splunkd.log. I could increase the debugging level
3. I added the NO_BINARY_CHECK=true to the source stanza in props.conf. But this doesn't change anything 😕
4. I moved invalied_cause from [my_sourcetype] to [source::udp:515] but this also didn't change the behaviour 😕

@martin_mueller:
My data is coming to UDP 515. I'm testing with netcat:
cat /tmp/binary_to_splunk | netcat -u splunk.home.lan 515

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

guilmxm — Thu, 14 Aug 2014 12:38:31 GMT

@HansWurscht: If you have the data undecoded, then for sure your stanza is not matched, whatever you will change in configuration this won't change.
When you achieve a search against your data like "| stats by source" what is the source reported by splunk ?

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

HansWurscht — Mon, 28 Sep 2020 17:20:17 GMT

@guilmxm:
Sorry, i don't understand that.

My data appears in splunk with source=udp:515 and sourcetype=my_sourcetype. The data in splunk is undecoded (shows up in a hex format \xF1\xF2\xF3...). The unarchive_cmd isn't working.

unarchice_cmd and invalid_cause are listed under the source stanza in props.conf:

[source::udp:515]
NO_BINARY_CHECK=true
sourcetype=my_sourcetype
invalid_cause = archive
unarchive_cmd=/usr/bin/perl $SPLUNK_HOME/etc/apps/foo/bin/decode.pl

Thanks!

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

martin_mueller — Thu, 14 Aug 2014 12:57:55 GMT

On the invalid_cause - that can only be set for sourcetype stanzas, not for source or host stanzas as per http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/propsconf

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

guilmxm — Thu, 14 Aug 2014 13:09:01 GMT

I see, i massively use unarchive_cmd in my App (http://apps.splunk.com/app/1753/) to decode nmon data through party converter (initially Perl then Python) you may sometimes fight with it 🙂

I mean i think your stanza does not match (like some kind of regex that will not match a pattern), but as it is, it's seems correct.

I guess you've reloaded Splunk after each setting change ?

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

guilmxm — Thu, 14 Aug 2014 13:14:19 GMT

What about your inputs.conf config ? Do you specify the sourcetype in it ?

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

HansWurscht — Thu, 14 Aug 2014 13:17:01 GMT

I restarted splunk after changing the configuration files.

I think the unarchive_cmd simply won't work on network inputs.

The sourcetype is set correctly, so i don't think there is a problem with that.

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

martin_mueller — Thu, 14 Aug 2014 13:25:06 GMT

Looking at http://wiki.splunk.com/Community:HowIndexingWorks it seems you're right - the archiveProcessor is not part of the processing pipeline for network inputs but rather for files.

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

guilmxm — Thu, 14 Aug 2014 13:30:31 GMT

I think too.

You solution is modular input:

http://answers.splunk.com/answers/108190/how-to-index-a-binary-data-from-udp

Re: Why props.conf configuration with unarchive_cmd on udp input doesn't work?

HansWurscht — Thu, 14 Aug 2014 13:45:47 GMT

I've created a support case to clarify this in the documentation.