Getting Data In

Filtering data before indexing with script

alisaf
New Member

Hi all,
I have a very longs logs that I would like to filter before indexing/ I have some patterns that interesting me, however, I don't think that using props.conf and transforms.conf will help me because I want only this event, and they are very different in the strings. I thought to use a bash script that excludes only these events, and I create one that works well.
Currently, I uploading the logs and in the future, I will send them form Amazon S3 cloud.

I have 2 questions:
1. How I define this script in Splunk? how does it know form which folder takes the log and which script to run?
2. If 1 is not possible, maybe using transforms.conf regex is possible?
for example, these are the only logs pattern that I want to be uploaded:

ShutdownThread: shutdown reason is: userrequested
SystemInfo: boot
XCMP : Tx: PUI broadcast source:0x1 type:0x0 id:0x1 state:0x1
XCMP : Tx: PUI broadcast source:0x1 type:0x0 id:0x1 state:0x0
CAP:MediaRecorder: Start video recording
CAP:MediaRecorder: Stop video recording
(I have like 10 different events with ON/OFF string)
Thank you!

0 Karma

richgalloway
SplunkTrust
SplunkTrust
  1. Define the script as a scripted input (Settings->Data inputs->Scripts). Anything the script writes to stdout will be indexed by Splunk. There are no arguments passed to the script so it must have its own logic to know which log to read.
  2. Yes, you can use transforms.conf. Use a REGEX statement to identify the events you want to index and send them to FORMAT = indexQueue; send the rest to FORMAT = nullQueue.
---
If this reply helps you, Karma would be appreciated.
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Agent Mode Engaged! Enchaining Agentic Operations with Splunk AI Assistant 2.0

    Are you ready to transform how your team handles complex data requests? We invite you to our upcoming ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...