Getting Data In

Why props.conf configuration with unarchive_cmd on udp input doesn't work?

HansWurscht
Path Finder

Hi,

i'm trying to implement a custom Charset-Decoder for an udp input. I'm using the following settings in props.conf:

[source::udp:515]
sourcetype=my_sourcetype
unarchive_cmd=/usr/bin/perl $SPLUNK_HOME/etc/apps/foo/bin/decode.pl

[my_sourcetype]
invalid_cause = archive

However this doesn't work. The incoming network data won't be processed by my decode-script. Is my splunk configuration correct?
I also tried other variations of the configuration, like putting unarchive_cmd in the sourcetype-stanza. But they all won't work.

I can't see any words in the documentation of unarchive_cmd and invalid_cause why this isn't supposed to work.

Any Ideas?

Thanks!

0 Karma

guilmxm
Influencer
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Looking at http://wiki.splunk.com/Community:HowIndexingWorks it seems you're right - the archiveProcessor is not part of the processing pipeline for network inputs but rather for files.

0 Karma

HansWurscht
Path Finder

I've created a support case to clarify this in the documentation.

0 Karma

HansWurscht
Path Finder

I restarted splunk after changing the configuration files.

I think the unarchive_cmd simply won't work on network inputs.

The sourcetype is set correctly, so i don't think there is a problem with that.

0 Karma

guilmxm
Influencer

What about your inputs.conf config ? Do you specify the sourcetype in it ?

0 Karma

guilmxm
Influencer

I see, i massively use unarchive_cmd in my App (http://apps.splunk.com/app/1753/) to decode nmon data through party converter (initially Perl then Python) you may sometimes fight with it 🙂

I mean i think your stanza does not match (like some kind of regex that will not match a pattern), but as it is, it's seems correct.

I guess you've reloaded Splunk after each setting change ?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

On the invalid_cause - that can only be set for sourcetype stanzas, not for source or host stanzas as per http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/propsconf

0 Karma

HansWurscht
Path Finder

@guilmxm:
Sorry, i don't understand that.

My data appears in splunk with source=udp:515 and sourcetype=my_sourcetype. The data in splunk is undecoded (shows up in a hex format \xF1\xF2\xF3...). The unarchive_cmd isn't working.

unarchice_cmd and invalid_cause are listed under the source stanza in props.conf:

[source::udp:515]
NO_BINARY_CHECK=true
sourcetype=my_sourcetype
invalid_cause = archive
unarchive_cmd=/usr/bin/perl $SPLUNK_HOME/etc/apps/foo/bin/decode.pl

Thanks!

0 Karma

guilmxm
Influencer

@HansWurscht: If you have the data undecoded, then for sure your stanza is not matched, whatever you will change in configuration this won't change.
When you achieve a search against your data like "| stats by source" what is the source reported by splunk ?

0 Karma

HansWurscht
Path Finder

Thanks for the answers.
The data appears in splunk, just the decoding part won't work.

@guilmxm:
1. the perl-Script will read from STDIN and print to STDOUT
2. no coressponding entries in splunkd.log. I could increase the debugging level
3. I added the NO_BINARY_CHECK=true to the source stanza in props.conf. But this doesn't change anything 😕
4. I moved invalied_cause from [my_sourcetype] to [source::udp:515] but this also didn't change the behaviour 😕

@martin_mueller:
My data is coming to UDP 515. I'm testing with netcat:
cat /tmp/binary_to_splunk | netcat -u splunk.home.lan 515

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Additionally, make sure your data actually is coming in on UDP port 515 instead of the much more common syslog port 514.

0 Karma

guilmxm
Influencer

And you should try moving the invalid_clause = archive from your sourcetype to your source stanza, this is where it needs to be located

0 Karma

guilmxm
Influencer

Hi, I have no experience with using unarchive_cmd within UDP flow, but i do with unarchive_cmd itself.

  1. Is your perl script built to read data from stdin ? (this is the way Splunk will send data to your script)
  2. If the data is not proceeded by your script, this is probably that your source stanza in props.conf does not match your input, have you any trace in splunkd.log ?
  3. Not sure this is relevant with UDP, but with files you have to add NO_BINARY_CHECK = true to your source stanza in props.conf
0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...