Getting Data In

Transforms.conf to reroute logs to different index.

Habanero
Explorer

 

Hello community,

 

I am trying to "reroute" specific logs (based on Regex match) to a different index. This is done on the heavy-forwarder. It is ingested via syslog.

Both props and transform are in the correct folder where syslogs events are ingested.

I have created a ruleset in props.conf:

 

[vmware]
TRANSFORMS-include = reroute_to_indexA

 

And here is the config from transform.conf:

 

[reroute_to_indexA]
SOURCE_KEY = _raw
REGEX = ^.*2300-.*$
DEST_KEY = _MetaData:Index
FORMAT = index-a

 

Last but not least here is a sample of the logs I am working with:

 

Jul  5 09:02:11 10.32.37.214 1 2022-07-05T09:02:11.339-04:00 2300-RDSH-1-2 View - 1009 [View@6876 Severity="INFO" Module="Agent" EventType="AGENT_DISCONNECTED" UserSID="omitted" UserDisplayName="omitted" PoolId="2300-rdsh-farm1" MachineId="omitted" MachineName="2300-RDSH-1-2" MachineDnsName="2300-rdsh-1-2" CurrentSessionLength="180" TotalLoginLength="180" SessionType="APPLICATION"] User omitted has disconnected from machine 2300-RDSH-1-2

 

 

At this point I would have expected to see the logs being written to index-a.

What have I done so far as troubleshooting:

  • Remove SOURCE_KEY
  • Replace SOURCE_KEY = _raw with field:MachineDnsName
  • Replace SOURCE_KEY = _raw with fields:MachineDnsName
  • Substituted the REGEX for .*2300.* and .*2300-.*

Nothing have helped so far; any help or pointers would be greatly appreciated.

 

Thank you,

 

 

 

 

0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @Habanero,

some questions:

  • where are you making syslog ingestion, on the same HF where you located the transformation on in another HF?
  • what is the sourcetype you're using to ingest vmware logs, are you sure that there isn't any sourcetype overriding and the sourcetype is still "vmware"?

If you have syslog ingestion in another HF, you have to put the transformation in that HF.

About SOURCE_KEY, it isn't relevant because it's _raw, so you can also omit it.

About the regex, I'd try with a simpler one:

REGEX = 2300-

Ciao.

Giuseppe

View solution in original post

gcusello
Esteemed Legend

Hi @Habanero,

some questions:

  • where are you making syslog ingestion, on the same HF where you located the transformation on in another HF?
  • what is the sourcetype you're using to ingest vmware logs, are you sure that there isn't any sourcetype overriding and the sourcetype is still "vmware"?

If you have syslog ingestion in another HF, you have to put the transformation in that HF.

About SOURCE_KEY, it isn't relevant because it's _raw, so you can also omit it.

About the regex, I'd try with a simpler one:

REGEX = 2300-

Ciao.

Giuseppe

Habanero
Explorer

Hello @gcusello

Thank you the the quick reply 🙂

  • Yes, ingestion is done on the same HF. We only have one in our environment.
  • As per our Search Heads the sourcetype is classified as "vmw-syslog". The index is "vmware"

To expand on your second point, I though what was put between the square brackets (stanza?) could define either a index or a sourcetype [vmware] or [vmware:vmw-syslog]

In any case, I have modified the value inside the square brackets for the props, and followed your suggestion for the transforms.conf.

Unfortunately, It is not a log source that is super noisy. I will report back once data comes in.

 

Thank you,

 

 

 

0 Karma

gcusello
Esteemed Legend

Hi @Habanero,

when you don't have any explicit field name, you have always sourcetype, so use "vmware:vmw-syslog".

Anyway, in the transfrom, you can use source or host or sourcetype, never index.

Ciao.

Giuseppe

 

Habanero
Explorer

I see thank you for clarifying that.

I am please to report that it is finally working. I suspect the biggest problem was the value inside the [] in my props.conf that was the problem.

For anyone that stumble upon this post in the feature

Here's the working config:

props.conf:

[vmw-syslog]
TRANSFORMS-include = reroute_to_indexA

transforms.conf:

[reroute_to_indexA]
REGEX = 2300-
DEST_KEY = _MetaData:Index
FORMAT = index-a

 

Thank you for your help @gcusello!

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...