We have a architecture of 3 site multi cluster which contains 6 indexers (2 in each site), 3 search heads (one in each), 1 Deployment server, 2 CMs(active and standby), Deployer, and these instances residing in AWS cloud and we are in Splunk enterprise 9.1.4 Our requirement is AWS logs will be pushed to our Splunk environment. So they are pushing Cloudwatch logs from Kinesis data streams and Lambda configured (no idea on all these). But they asked our team to create HEC token and give token details and endpoint details to them in order to push logs to Splunk. They don't want Amazon firehouse add-on here. They prefer only through HEC. Now my doubt is where to configure this HEC token? I want all my 6 indexers to be load balances data receiving from AWS (considering it will be huge data). I have gone through HEC docs but it is unclear. Can someone help me with the step by step procedure how can I acheive it? Like where to create HEC token and all these? Thank you. ... View more

I was newly aligned into a project and didn't have proper KT from the left ones. I have queries regarding my current architecture and configurations and I am not well versed with advanced admin concepts. Please help me in these queries: We have 6 indexers (hosted on AWS cloud as EC2 but not Splunk cloud) with 6.9TB disk storage and 1.5GB/day license. Is this ok? I am checking for retention period but nowhere set with frozentimeperiodinsecs or maxTotalDataSizeMB in local. But in default it is there... I am also looking whether archival location is set or not. indexes.conf in Cluster Manager: [new_index] homePath   = volume:primary/$_index_name/db coldPath   = volume:primary/$_index_name/colddb thawedPath = $SPLUNK_DB/$_index_name/thaweddb   volumes indexes.conf:   [volume:primary] path = $SPLUNK_DB #maxVolumeDataSizeMB = 6000000   there is one more app which is pushing to indexers with indexes.conf: (not at all aware of this)   [default] remotePath = volume:aws_s3_vol/$_index_name maxDataSize = 750 [volume:aws_s3_vol] storageType = remote path = s3://conn-splunk-prod-smartstore/ remote.s3.auth_region = eu-west-1 remote.s3.bucket_name = conn-splunk-prod-smartstore remote.s3.encryption = sse-kms remote.s3.kms.key_id = XXXX remote.s3.supports_versioning = false   and I don't see coldToFrozenDir and coldToFrozenScript is also not mentioned anywhere.   Now are we storing archival data in S3 bucket now? but there mentioned maxDataSize which is related to hot to warm. So apart from hot bucket data, rest all data is storing in s3 bucket now?    So how will Splunk take the data from S3 bucket to search and make queries? ... View more

we got a requirement to on-board new platform logs to Splunk. They will have 1.8 TB/day data to be ingested. As of now our license is 2 TB/day and we already have other platform data on-boarded. Now these new ones accepted to uplift our license with 2TB/day more so now our total becomes 4TB/day. But here they said that their normal ingestion is 1.8 TB/day, but during DDOS attack it can go in double digits. We got surprised by this. Total itself is 4TB/day, how come we can handle double digits TB of data, which in return this project might impact the on-boarding of other projects. My manager asked me to investigate on this whether we can accommodate this requirement? If yes, he want the action plan. If not, he want the justification to share it with them. I am not much aware of these licensing things in Splunk, but as per my knowledge this is very dangerous because 4TB and 10/20TB per day is huge difference. My understanding is, if we breach 4TB/day (may be 200gb of data more), new indexing stops but still old searches can be accessed.  Our infrastructure: multi site cluster with 3 sites ... 2 indexers in each (total 6), 3 SHs one in each, 1 deployment server, 2 CMs (active and standby), 1 deployer (which is license master.) Can anyone please help me on this topic how to proceed on it? ... View more