No, if you need to elaborate your logs before indexing, you have to run a pre-parsing script outside Splunk and index the output file.
We had to encrypt a field in a log file using a key, because our customer wanted to archive this data in encrypted format, but they also wanted the possibility to recreate the original value using the encryption key.
To do this, we used a script that parsed the log file and after we indexed it.
To do this is easy if you have a syslog data flow or a file on the Splunk server, but less easy if you receive logs via Forwarder, because, you have to distribute the external script in every Forwarder.
In addition, you lose the real time monitoring because there is always a delay between the log arrive and the indexing time.
We asked to Splunk to insert the possibility to run a script before indexing, but not yet.
I am using splunk api (push mech) to retrieve all index notables, so i can see them in the ticketing system!!
First i send a query (using splunk api) to the index notables to get all the notables within the timeframe of 5 mins and assume that i have 20 notables and all the 20 notables have the "drill_down" search command. This is just one query.
Now i have to hit 20 times in the search head to retrieve all the related events for each notables incidents. (aka) you know the drill-down provide the search for Contributing Events (Drill-down). right ?
now i dont want to hit 20 times to get all the drill-down search. A slight change in the approach,
can i hold all the events which matched the correlation search of splunk es app, before it get indexed in the notable index. ?
so thats like --> Cor.search runs --> (store all the contributing events in a file) --> then allow splunk to index in the index=notable disk.
this is where a custom pipeline inside the indexer pipeline should help me to achieve.