So we have are pulling host logs on an EC2 instance and dropping them in an S3 Bucket. Our Splunk Heavy Forwarder is grabbing the logs and pushing them to Splunk Cloud. As we can easily pull the JSON log files into Splunk, there is one specific field inside the JSON document that we want to extract and push to another index.
So want complete json to be in one index and just the message into another index? I don't believe there is an option at index time to send same event or portion of a single event to two indexes. My suggestion would to have the data fully ingested in one index and then setup a summary indexing search to filter only the relevant portion and index it to other index. Generally summary indexing is used for optimal searching/reporting but it's design helps in this kind of use-cases as well (save processed data to a different index). See below link for more details on summary indexing. http://docs.splunk.com/Documentation/Splunk/6.5.2/Knowledge/Usesummaryindexing