I am using HEC to receive various logs from Firehose, HEC is allowed to use index names AWS & palo_alto. The default index is set to AWS. All the logs coming from the HEC are assigned to default index AWS and default sourcetype aws:firehose I am using the below config to change the sourcetype and index name of the logs.      props.conf [source::syslog:dev/syslogng/*] TRANSFORMS-hecpaloalto = hecpaloalto, hecpaloalto_in disabled = false transforms.conf [hecpaloalto] REGEX = (.*) DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::pan:log [hecpaloalto_in] REGEX = (.*) DEST_KEY = _MetaData:Index FORMAT = palo_alto   The sourcetype has changed to pan:log and its as intended but the Index name is still displaying as AWS, instead of changing to palo_alto. HEC config have default index as aws and selected indexes are aws, palo_alto Is there anything wrong in my config? ... View more

Here is my sample log    2024-07-08T04:43:32.468537+00:00 dxx1-dbxxxs.xxx.net MSSQLSERVER[0] {"EventTime":"2024-07-08 04:43:32","Hostname":"dx1-dbxxxs.xxx.net","Keywords":45035996273704960,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":44444,"SourceName":"MSSQLSERVER","Task":5,"RecordNumber":1234343410,"ProcessID":0,"ThreadID":0,"Channel":"Application","Message":"Audit event:lkjfd:sdfkjhf:Askjhdfsdf","Category":"None","EventReceivedTime":"2024-07-08 04:43:32","SourceModuleName":"default-inputs","SourceModuleType":"im_msvistalog"}#015   Here is my config props.conf [dbtest:test] #mysourcetype TRANSFORMS-extract_kv_pairs = extract_json_data transforms.conf   [extract_json_data] REGEX = "(\w+)":"?([^",}]+)"? FORMAT = $1::$2 WRITE_META = true The same Regex is working in Regex101 here is the test link https://regex101.com/r/rt3bly/1 I am not sure why its not working in my log extraction.  Any help is highly appreciated. Thanks   ... View more

Is it possible to have WarmData stored partially on local indexers' storage and partially on remote storage? My total retention period is 90 days but I would like to keep data for the first 45 days on local indexers as warm buckets and then send the warm buckets to AWS S3 bucket to store for another 45 days. I am using the below settings and the data sent to the S3 afrer 90 days and is stored in frozen instead of warm or cold. How can I set warm or cold data in S3? [default] remotePath = volume:remote_store/$_index_name repFactor = auto frozenTimePeriodInSecs = 7776000 If I change the frozenTimePeriodInSecs = 1555200 (45 days) the data will be sent to S3 after 45 days but it will be sent in frozen buckets. I need to send data either as warm or cold buckets. If I setup  [volume:local_store] maxVolumeDataSizeMB = 1000000 That will send data to S3 only after filling 1 TB in local storage.  How can I maintain a time based storage with 45 days on local storage and 45 days on remote storage? ... View more

I have a HEC and I am receiving logs from CloudWatch and the default index is set to "aws". From the same HEC token I am also receiving Firewall logs from CloudWatch and these logs are also going to the index "aws". How can I transform the Firewall logs coming from the same HEC token from a different source to be assigned to index "paloalto"? I tried using the below config but it doesn't work props.conf [source::syslogng:dev/syslogng/*] TRANSFORMS-hecpaloalto = hecpaloalto disabled = false transforms.conf [hecpaloalto] DEST_KEY = _MetaData:Index REGEX = (.*) FORMAT = palo_alto I created the index palo_alto in the cluster master indexes.conf, applied cluster bundles to the indexers. And also applied the above config using deployment server to the Indexers. For some reason the logs are still going to the aws index. ... View more

I have a HEC token sending various logs from AWS Cloudwatch. HEC token is set to have two indexes paloalto and aws. And all the data is going to the default index aws. I want all the data from the source=syslogng to be sent to index=paloalto instead of index=aws. Should I change the inputs.conf or props.conf for the splunk_httpinput in the deployment apps? ... View more