Splunk Search

Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Log_wrangler
Builder

My override index confs are breaking and I cannot find the cause...

Currently I have logs from two sources (A and B) coming in on (port TCP 666) going to one index_A.

Event logs containing: pipe two separate words pipe, like this ---> | Foo Bar | need to go into index_B.

Inputs.conf

[TCP://666] 
disabled = 0
connection_host = dns
index = index_A
sourcetype = st_A

To override I created:
in Props.conf

[source::TCP://666]

TRANSFORMS-Indx_B = SEND_TO_Index_B

in Transforms.conf

[SEND_TO_Index_B]
REGEX = |Foo Bar|
DEST_KEY = _MetaData:Index
FORMAT = Index_B

When I edit both confs and restart, I don't receive any conf errors on restart, but any events containing |foo bar| are lost or dropped from both indexes.

I grep for either index in splunkd.log in /opt/splunk/var/log/splunk, but I am not finding any clues.

Am I missing an error in my confs?
Is there a specific log that might identify were the events are going?

Please advise
Thank you

Tags (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi Log_wrangler,
there are some issue to check:

where you have props.conf and transforms.conf?
they must be on Heavy Forwarders (if present) or on Indexers.

It's better to use sourcetype in props.conf

[st_A]
TRANSFORMS-Indx_B = SEND_TO_Index_B

Check the regex in transforms.conf: pipe is a special char for regex:

[SEND_TO_Index_B]
 REGEX = \|Foo Bar\|
 DEST_KEY = _MetaData:Index
 FORMAT = Index_B

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Log_wrangler,
there are some issue to check:

where you have props.conf and transforms.conf?
they must be on Heavy Forwarders (if present) or on Indexers.

It's better to use sourcetype in props.conf

[st_A]
TRANSFORMS-Indx_B = SEND_TO_Index_B

Check the regex in transforms.conf: pipe is a special char for regex:

[SEND_TO_Index_B]
 REGEX = \|Foo Bar\|
 DEST_KEY = _MetaData:Index
 FORMAT = Index_B

Bye.
Giuseppe

View solution in original post

0 Karma

Log_wrangler
Builder

Thank you, I will try your suggestions and let you know.

0 Karma

Log_wrangler
Builder

still not working, so I removed the pipes, now its just

REGEX = Foo Bar

does that require quotes or anything special because its two words?

Thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

No, you con use space or \s and you don't need quotes.

Only one additional question:
I remember from your previous question that you overrided also sourcetype, so what's the event's sourcetype now, the old or the new one?
so in props.conf put the one you have or try both.
[st_A]
TRANSFORMS-Indx_B = SEND_TO_Index_B
or

[st_B]
 TRANSFORMS-Indx_B = SEND_TO_Index_B

Bye.
Giuseppe

0 Karma

Log_wrangler
Builder

Apparently there was an issue with the logs not flowing from the source device, which I interpreted as I made a fatal config. However the escape | foo bar | works fine.

Still testing a double override, Index and sourcetype. Override Index works fine, wondering if there will be a performance hit if I do two overrides. But I will make that a separate question.

Thank you

0 Karma

Log_wrangler
Builder

fyi, it did retain the old/wrong sourcetype but I will fix that later.

0 Karma

p_gurav
Champion

In props.conf, can you try to give sourcetype name and in transforms.conf edit REGEX to \|Foo Bar\|

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!