Getting Data In

Filtering data before indexing with script

alisaf
New Member

Hi all,
I have a very longs logs that I would like to filter before indexing/ I have some patterns that interesting me, however, I don't think that using props.conf and transforms.conf will help me because I want only this event, and they are very different in the strings. I thought to use a bash script that excludes only these events, and I create one that works well.
Currently, I uploading the logs and in the future, I will send them form Amazon S3 cloud.

I have 2 questions:
1. How I define this script in Splunk? how does it know form which folder takes the log and which script to run?
2. If 1 is not possible, maybe using transforms.conf regex is possible?
for example, these are the only logs pattern that I want to be uploaded:

ShutdownThread: shutdown reason is: userrequested
SystemInfo: boot
XCMP : Tx: PUI broadcast source:0x1 type:0x0 id:0x1 state:0x1
XCMP : Tx: PUI broadcast source:0x1 type:0x0 id:0x1 state:0x0
CAP:MediaRecorder: Start video recording
CAP:MediaRecorder: Stop video recording
(I have like 10 different events with ON/OFF string)
Thank you!

0 Karma

richgalloway
SplunkTrust
SplunkTrust
  1. Define the script as a scripted input (Settings->Data inputs->Scripts). Anything the script writes to stdout will be indexed by Splunk. There are no arguments passed to the script so it must have its own logic to know which log to read.
  2. Yes, you can use transforms.conf. Use a REGEX statement to identify the events you want to index and send them to FORMAT = indexQueue; send the rest to FORMAT = nullQueue.
---
If this reply helps you, Karma would be appreciated.
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...

Network to App: Observability Unlocked [May & June Series]

In today’s digital landscape, your environment is no longer confined to the data center. It spans complex ...

SPL2 Deep Dives, AppDynamics Integrations, SAML Made Simple and Much More on Splunk ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...