Splunk Search
Highlighted

Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Builder

My override index confs are breaking and I cannot find the cause...

Currently I have logs from two sources (A and B) coming in on (port TCP 666) going to one index_A.

Event logs containing: pipe two separate words pipe, like this ---> | Foo Bar | need to go into index_B.

Inputs.conf

[TCP://666] 
disabled = 0
connection_host = dns
index = index_A
sourcetype = st_A

To override I created:
in Props.conf

[source::TCP://666]

TRANSFORMS-Indx_B = SEND_TO_Index_B

in Transforms.conf

[SEND_TO_Index_B]
REGEX = |Foo Bar|
DEST_KEY = _MetaData:Index
FORMAT = Index_B

When I edit both confs and restart, I don't receive any conf errors on restart, but any events containing |foo bar| are lost or dropped from both indexes.

I grep for either index in splunkd.log in /opt/splunk/var/log/splunk, but I am not finding any clues.

Am I missing an error in my confs?
Is there a specific log that might identify were the events are going?

Please advise
Thank you

Tags (3)
0 Karma
Highlighted

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Champion

In props.conf, can you try to give sourcetype name and in transforms.conf edit REGEX to \|Foo Bar\|

0 Karma
Highlighted

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Legend

Hi Log_wrangler,
there are some issue to check:

where you have props.conf and transforms.conf?
they must be on Heavy Forwarders (if present) or on Indexers.

It's better to use sourcetype in props.conf

[st_A]
TRANSFORMS-Indx_B = SEND_TO_Index_B

Check the regex in transforms.conf: pipe is a special char for regex:

[SEND_TO_Index_B]
 REGEX = \|Foo Bar\|
 DEST_KEY = _MetaData:Index
 FORMAT = Index_B

Bye.
Giuseppe

View solution in original post

0 Karma
Highlighted

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Builder

Thank you, I will try your suggestions and let you know.

0 Karma
Highlighted

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Builder

still not working, so I removed the pipes, now its just

REGEX = Foo Bar

does that require quotes or anything special because its two words?

Thank you

0 Karma
Highlighted

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Legend

No, you con use space or \s and you don't need quotes.

Only one additional question:
I remember from your previous question that you overrided also sourcetype, so what's the event's sourcetype now, the old or the new one?
so in props.conf put the one you have or try both.
[stA]
TRANSFORMS-Indx
B = SENDTOIndex_B
or

[st_B]
 TRANSFORMS-Indx_B = SEND_TO_Index_B

Bye.
Giuseppe

0 Karma
Highlighted

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Builder

Apparently there was an issue with the logs not flowing from the source device, which I interpreted as I made a fatal config. However the escape | foo bar | works fine.

Still testing a double override, Index and sourcetype. Override Index works fine, wondering if there will be a performance hit if I do two overrides. But I will make that a separate question.

Thank you

0 Karma
Highlighted

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) Part 2

Builder

fyi, it did retain the old/wrong sourcetype but I will fix that later.

0 Karma