Hello, I'm attempting to change the sourcetype and host on a single event. The tricky part is I want the second transform based on the change from the first transform For Example, My data comes in as   index=main host=heavy_forwarder sourcetype=aws:logbucket   I want the data to change to   index=main host=amazonfsx.host sourcetype=XmlWinEventLog   The catch is that I have other sourcetypes coming in as aws:logbucket and getting transformed to various other sourcetypes (cloudtrail, config, etc). On these events I do not want to run the regex to change the host value   If I have a props.conf file that states TRANSFORMS-modify_data = aws_fsx_sourcetype, aws_fsx_host And a transforms.conf of [aws_fsx_sourcetype] SOURCE_KEY = MetaData:Source REGEX = ^source::s3:\/\/fsxbucket\/.* FORMAT = sourcetype::XmlWinEventLog DEST_KEY = MetaData:Sourcetype [aws_fsx_host] REGEX = <Computer>([^.<]+).*?<\/Computer> FORMAT = host::$1 DEST_KEY = MetaData:Host   I'm worried this will have unexpected results on the other sourcetypes that aws:logucket has, like cloudtrail and config. If I break it out with two separate transforms, like this   TRANSFORMS-modify_data = aws_fsx_sourcetype TRANSFORMS-modify_data2 = aws_fsx_host   I'm worried the typing pipeline won't see the second transform. What is the most effective way to accomplish this?   Thanks, Nate ... View more

Hello, I'm looking to change our indexing architecture We have dozens of AWS accounts. We use the Splunk AWS app to ingest the data from a SQS queue. Currently, we have a single SQS-based input type for each individual AWS account that grabs all the data and applies the index and a catch-all sourcetype named aws:logbucket. From there, we route the data to a more specific sourcetype based on the type of data. aws:logbucket will be changed to aws:cloudwatch:vpcflowlogs, aws:cloudtrail, aws:config, etc. This has worked well enough for us, but I now have a new requirement. For each of these AWS accounts, I want a separate index for the specific AWS service by AWS account. ie) awsaccount1-vpcflow, awsaccount1-cloudtrail, awsaccount2-vpcflow, etc. We use S2, so storing aws:cloudtrail with aws:cloudwatch:vpcflow hurts the performance of aws:cloudtrail data. Searching for aws:cloudtrail data requires us to write back all aws:cloudwatch:vpcflow data back to disk. This has accounted for 120x more buckets required written to disk for aws:cloudtrail since it's stored with VPCFlow. Expanding these indexes to be more specific will have huge performance improvements for my Splunk environment I would like to use a lookup table to match the source of the SQS-based S3 to specify the index and sourcetype. I am unable to do this using regex and FORMAT, since the bucket names and index names are not a 1-1 match. ie) for s3://acc1/cloudtrail/..., I would like to have a lookup table that tells it to route to index account1 and sourcetype aws:cloudtrail, for s3://acc2/config/... I would like to have it route to index account2 and sourcetype aws:config. After that long summary... how do I technically implement this and how will a lookup with ~300-400 different rows affect performance? Thank you, Nate       ... View more

Hello, I'm am wondering how other security service providers have handled this issue or what is best practice To plan for least privilege, indexes would be separated out by group. We could store all data related to the group in a respective index. Network traffic, network security, antivirus, Windows Event Data, etc all in a single index for the group and give that group permissions to the index. An issue with this scenario is search performance. Searches may be performed on network traffic. Or on host data. Or on antivirus data. But Splunk will have to search the bucket containing all of the other unrelated data. If antivirus data is only producing 5GB a day, but network traffic is producing 10TB a day, this will have a huge negative affect on searches for antivirus data. This will be compounded with SmartStore (S2) where IOPS will be used to write the bucket back to disk. If least privilege isn't an issue, it would be optimal to create a bucket for the specific data. Network traffic would have it's own index. Windows hosts would have their own index. But the crux of architecting in this fashion is how to implement least privilege. One group cannot be able to see the host data of the other group. One idea to get around this is to limit the search capability by host, but that would require much work from the Splunk team and is not 100% certain if wildcards are used. Another idea is to simply create a separate index for each data type for each group. My concern with this is scaling. If we have 10 separate groups that require 10 indexes, that's 100 indexes. If we have 50, that's 500 indexes. 100 is 1000. This does not scale well. Thank you in advance for your help     ... View more