topic Re: How to direct incoming data from heavy forwarder to index by host name? in Getting Data In

How to direct incoming data from heavy forwarder to index by host name?

jgorman_THG — Tue, 29 Sep 2020 10:53:43 GMT

Hi,

I have data coming in from multiple hosts using either syslog, or a universal forwarder, going into 3 heavy forwarders, and then forwarding to SplunkCloud.

I've created 3 indexes - Financial, Infrastructure, and Security - and I would like to separate the data by host name

So I want data from "financial_server1" to go to the "financial" index, and data from "Firewall_1" to go to the "Security" index.

Can someone give me an example of how this would be done?

Thanks,

Re: How to direct incoming data from heavy forwarder to index by host name?

dshpritz — Wed, 07 Sep 2016 16:59:43 GMT

Are the items that you want to separate the data by available in the data? That is, are the hostnames in the eventdata itself?

Re: How to direct incoming data from heavy forwarder to index by host name?

jgorman_THG — Wed, 07 Sep 2016 17:01:58 GMT

Yes! Splunk recognizes and sets the host for each device.

Thanks

Re: How to direct incoming data from heavy forwarder to index by host name?

somesoni2 — Wed, 07 Sep 2016 17:09:16 GMT

This is how you can override the index name based on a sourcetype (sourcetype used here is "mysourcetype"). You can configure this based on host name as well by replacing "mysourcetype" in props.conf with "host::YourHostName".
On your indexer or heavy forwarder:
etc/system/local/transforms.conf

 [overrideindex]
 DEST_KEY =_MetaData:Index
 REGEX = .
 FORMAT = my_new_index

etc/system/local/props.conf

 [mysourcetype]
 TRANSFORMS-index = overrideindex

Re: How to direct incoming data from heavy forwarder to index by host name?

jgorman_THG — Thu, 08 Sep 2016 14:38:18 GMT

Ok Cool.

So if I wanted "host1" to go to index "Infrastructure" I would do

Transforms.conf:

[overrideindex]
DEST_KEY=_MetaData:index
REGEX = .
FORMAT = Infrastructure

Props.conf:

[host::host1]
TRANSFORMS-index = overrideindex

Correct?

Do I need to restart splunk after making this change? Will I need to do seperate entries for each host, or is there a way where I can enter all the applicable hosts in the same [host::***] line?

Since I have multiple indexes, in transforms would I put for example [overrideindex] [overrideindex1] [overrideindex2]?

Also does this also work for wineventlog data?

Thanks a lot! I really appreciate the help.

Re: How to direct incoming data from heavy forwarder to index by host name?

somesoni2 — Thu, 08 Sep 2016 15:23:52 GMT

The configuration looks correct (for moving data from host1 to Infrastructure index).
Yes, you would need to restart your HF after making this change.
You can re-use the entry in transforms.conf for each host that you need to more to same index.
You need to define separate transforms.conf stanza for each index.
It'll work for any type of data coming to HF (from UF OR syslog).

Re: How to direct incoming data from heavy forwarder to index by host name?

jgorman_THG — Mon, 12 Sep 2016 16:05:24 GMT

So I made the following changes in the etc/system/local on all 3 heavy forwarders, I then ran on all 3 heavy forwarders:

| extract reload=T

This had no effect so I then restarted each one and still there was no change.

The heavy forwarders are feeding into SplunkCloud. The security index exists on splunk cloud. does it need to be on the heavy forwarders as well?

What am I doing wrong? Should this not be sending events from xx.xx.x.xx to index "security" in SplunkCloud?

Transforms.conf

[redirect_to_security]
DEST_KEY = _MetaData:Index
Regex = .
FORMAT = security

Props:

[host::xx\.xx\.x\.xx]
TRANSFORMS-index = redirect_to_security

Re: How to direct incoming data from heavy forwarder to index by host name?

somesoni2 — Mon, 12 Sep 2016 16:57:30 GMT

It should. You would need to restart Splunk services on Heavy forwarder for these changes to take effect (bin/splunk restart).

Re: How to direct incoming data from heavy forwarder to index by host name?

jgorman_THG — Mon, 12 Sep 2016 17:33:38 GMT

So strange. I restarted all the heavy forwarders and still have the same result. There's nothing special that needs to be done since it is forwarding to SplunkCloud?

Thanks