topic Re: Transforms conf to reroute logs to different index. in Getting Data In

Transforms.conf to reroute logs to different index.

Habanero — Tue, 05 Jul 2022 13:25:28 GMT

Hello community,

I am trying to "reroute" specific logs (based on Regex match) to a different index. This is done on the heavy-forwarder. It is ingested via syslog.

Both props and transform are in the correct folder where syslogs events are ingested.

I have created a ruleset in props.conf:

[vmware] TRANSFORMS-include = reroute_to_indexA

And here is the config from transform.conf:

[reroute_to_indexA] SOURCE_KEY = _raw REGEX = ^.*2300-.*$ DEST_KEY = _MetaData:Index FORMAT = index-a

Last but not least here is a sample of the logs I am working with:

Jul 5 09:02:11 10.32.37.214 1 2022-07-05T09:02:11.339-04:00 2300-RDSH-1-2 View - 1009 [View@6876 Severity="INFO" Module="Agent" EventType="AGENT_DISCONNECTED" UserSID="omitted" UserDisplayName="omitted" PoolId="2300-rdsh-farm1" MachineId="omitted" MachineName="2300-RDSH-1-2" MachineDnsName="2300-rdsh-1-2" CurrentSessionLength="180" TotalLoginLength="180" SessionType="APPLICATION"] User omitted has disconnected from machine 2300-RDSH-1-2

At this point I would have expected to see the logs being written to index-a.

What have I done so far as troubleshooting:

Remove SOURCE_KEY
Replace SOURCE_KEY = _raw with field:MachineDnsName
Replace SOURCE_KEY = _raw with fields:MachineDnsName
Substituted the REGEX for .*2300.* and .*2300-.*

Nothing have helped so far; any help or pointers would be greatly appreciated.

Thank you,

Re: Transforms conf to reroute logs to different index.

gcusello — Tue, 05 Jul 2022 13:26:49 GMT

Hi @Habanero,

some questions:

where are you making syslog ingestion, on the same HF where you located the transformation on in another HF?
what is the sourcetype you're using to ingest vmware logs, are you sure that there isn't any sourcetype overriding and the sourcetype is still "vmware"?

If you have syslog ingestion in another HF, you have to put the transformation in that HF.

About SOURCE_KEY, it isn't relevant because it's _raw, so you can also omit it.

About the regex, I'd try with a simpler one:

REGEX = 2300-

Ciao.

Giuseppe

Re: Transforms conf to reroute logs to different index.

Habanero — Tue, 05 Jul 2022 13:46:59 GMT

Hello @gcusello

Thank you the the quick reply 🙂

Yes, ingestion is done on the same HF. We only have one in our environment.
As per our Search Heads the sourcetype is classified as "vmw-syslog". The index is "vmware"

To expand on your second point, I though what was put between the square brackets (stanza?) could define either a index or a sourcetype [vmware] or [vmware:vmw-syslog]

In any case, I have modified the value inside the square brackets for the props, and followed your suggestion for the transforms.conf.

Unfortunately, It is not a log source that is super noisy. I will report back once data comes in.

Thank you,

Re: Transforms conf to reroute logs to different index.

gcusello — Tue, 05 Jul 2022 14:06:33 GMT

Hi @Habanero,

when you don't have any explicit field name, you have always sourcetype, so use "vmware:vmw-syslog".

Anyway, in the transfrom, you can use source or host or sourcetype, never index.

Ciao.

Giuseppe

Re: Transforms conf to reroute logs to different index.

Habanero — Tue, 05 Jul 2022 14:12:23 GMT

I see thank you for clarifying that.

I am please to report that it is finally working. I suspect the biggest problem was the value inside the [] in my props.conf that was the problem.

For anyone that stumble upon this post in the feature

Here's the working config:

props.conf:

[vmw-syslog] TRANSFORMS-include = reroute_to_indexA

transforms.conf:

[reroute_to_indexA] REGEX = 2300- DEST_KEY = _MetaData:Index FORMAT = index-a

Thank you for your help @gcusello!