Getting Data In

Transforms.conf to reroute logs to different index.

Habanero
Explorer

 

Hello community,

 

I am trying to "reroute" specific logs (based on Regex match) to a different index. This is done on the heavy-forwarder. It is ingested via syslog.

Both props and transform are in the correct folder where syslogs events are ingested.

I have created a ruleset in props.conf:

 

[vmware]
TRANSFORMS-include = reroute_to_indexA

 

And here is the config from transform.conf:

 

[reroute_to_indexA]
SOURCE_KEY = _raw
REGEX = ^.*2300-.*$
DEST_KEY = _MetaData:Index
FORMAT = index-a

 

Last but not least here is a sample of the logs I am working with:

 

Jul  5 09:02:11 10.32.37.214 1 2022-07-05T09:02:11.339-04:00 2300-RDSH-1-2 View - 1009 [View@6876 Severity="INFO" Module="Agent" EventType="AGENT_DISCONNECTED" UserSID="omitted" UserDisplayName="omitted" PoolId="2300-rdsh-farm1" MachineId="omitted" MachineName="2300-RDSH-1-2" MachineDnsName="2300-rdsh-1-2" CurrentSessionLength="180" TotalLoginLength="180" SessionType="APPLICATION"] User omitted has disconnected from machine 2300-RDSH-1-2

 

 

At this point I would have expected to see the logs being written to index-a.

What have I done so far as troubleshooting:

  • Remove SOURCE_KEY
  • Replace SOURCE_KEY = _raw with field:MachineDnsName
  • Replace SOURCE_KEY = _raw with fields:MachineDnsName
  • Substituted the REGEX for .*2300.* and .*2300-.*

Nothing have helped so far; any help or pointers would be greatly appreciated.

 

Thank you,

 

 

 

 

0 Karma
1 Solution

gcusello
Legend

Hi @Habanero,

some questions:

  • where are you making syslog ingestion, on the same HF where you located the transformation on in another HF?
  • what is the sourcetype you're using to ingest vmware logs, are you sure that there isn't any sourcetype overriding and the sourcetype is still "vmware"?

If you have syslog ingestion in another HF, you have to put the transformation in that HF.

About SOURCE_KEY, it isn't relevant because it's _raw, so you can also omit it.

About the regex, I'd try with a simpler one:

REGEX = 2300-

Ciao.

Giuseppe

View solution in original post

gcusello
Legend

Hi @Habanero,

some questions:

  • where are you making syslog ingestion, on the same HF where you located the transformation on in another HF?
  • what is the sourcetype you're using to ingest vmware logs, are you sure that there isn't any sourcetype overriding and the sourcetype is still "vmware"?

If you have syslog ingestion in another HF, you have to put the transformation in that HF.

About SOURCE_KEY, it isn't relevant because it's _raw, so you can also omit it.

About the regex, I'd try with a simpler one:

REGEX = 2300-

Ciao.

Giuseppe

Habanero
Explorer

Hello @gcusello

Thank you the the quick reply 🙂

  • Yes, ingestion is done on the same HF. We only have one in our environment.
  • As per our Search Heads the sourcetype is classified as "vmw-syslog". The index is "vmware"

To expand on your second point, I though what was put between the square brackets (stanza?) could define either a index or a sourcetype [vmware] or [vmware:vmw-syslog]

In any case, I have modified the value inside the square brackets for the props, and followed your suggestion for the transforms.conf.

Unfortunately, It is not a log source that is super noisy. I will report back once data comes in.

 

Thank you,

 

 

 

0 Karma

gcusello
Legend

Hi @Habanero,

when you don't have any explicit field name, you have always sourcetype, so use "vmware:vmw-syslog".

Anyway, in the transfrom, you can use source or host or sourcetype, never index.

Ciao.

Giuseppe

 

Habanero
Explorer

I see thank you for clarifying that.

I am please to report that it is finally working. I suspect the biggest problem was the value inside the [] in my props.conf that was the problem.

For anyone that stumble upon this post in the feature

Here's the working config:

props.conf:

[vmw-syslog]
TRANSFORMS-include = reroute_to_indexA

transforms.conf:

[reroute_to_indexA]
REGEX = 2300-
DEST_KEY = _MetaData:Index
FORMAT = index-a

 

Thank you for your help @gcusello!

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...