topic Re: Change Index for some HEC data in Installation

How to Change Index for some HEC data?

karthikm — Mon, 28 Aug 2023 18:54:21 GMT

I have a HEC and I am receiving logs from CloudWatch and the default index is set to "aws". From the same HEC token I am also receiving Firewall logs from CloudWatch and these logs are also going to the index "aws".

How can I transform the Firewall logs coming from the same HEC token from a different source to be assigned to index "paloalto"?

I tried using the below config but it doesn't work

props.conf

[source::syslogng:dev/syslogng/*]
TRANSFORMS-hecpaloalto = hecpaloalto
disabled = false

transforms.conf

[hecpaloalto]
DEST_KEY = _MetaData:Index
REGEX = (.*)
FORMAT = palo_alto

I created the index palo_alto in the cluster master indexes.conf, applied cluster bundles to the indexers.

And also applied the above config using deployment server to the Indexers.

For some reason the logs are still going to the aws index.

Re: Change Index for some HEC data

gcusello — Mon, 28 Aug 2023 10:25:08 GMT

Hi @karthikm,

I suppose that you're speaking of an on-premise installation.

Which Add-On are you using for the data ingestion?

if I correctly remember, it's possible to define the index for each data source by GUI, anyway, you could see the inputs.conf in tha used Add-On and see if the inputs (as tey should be!) are in two different stanzas.

If not, you can override the index value finding a regex that identifies the Firewall Logs and follow the configurations described in my previous answer https://community.splunk.com/t5/Splunk-Search/How-to-change-index-based-on-MetaData-Source/m-p/619936 or other answers in Community.

Ciao.

Giuseppe

Re: Change Index for some HEC data

karthikm — Mon, 28 Aug 2023 13:00:46 GMT

Hi @gcusello, yes it's distributed on-prem installation. I am not using any add-on for ingesting data.

I am using HTTP Event Collector Token to send AWS Cloudwatch logs to Splunk indexers (using load balancing).

From the GUI it's possible to select multiple indexes but use only the default index as the log index. So far all the logs are going to the default index and I don't see an option in the HEC settings or GUI where I can change the index name for partial logs coming through the HEC.

I tried overriding the index value as you mentioned, but it doesn't work. Any idea what's wrong in the below config?

props.conf

[source::syslogng:dev/syslogng/*]
TRANSFORMS-hecpaloalto = hecpaloalto
disabled = false

transforms.conf

[hecpaloalto]
DEST_KEY = _MetaData:Index
REGEX = (.*)
FORMAT = palo_alto

Re: Change Index for some HEC data

gcusello — Mon, 28 Aug 2023 15:50:57 GMT

Hi @karthikm,

ok for the input phase, but you need an Ad-On for the parsing phase, so you must have an Add-On otherwise, you have to manually create all the parsing rules.

Anyway, the approach is the one I described: you have to override the index value.

In addition I hint to analyze the Splunk Add-On for Amazon Web Services (AWS) at https://splunkbase.splunk.com/app/1876 because maybe it could help you.

Ciao.

Giuseppe