Getting Data In

event filtering using transforms and props

ajaysamantbms
Explorer

My event data contains the foll

POST:....
...
ffffff
ABCD
EFG

WERT
SDF
... and so on

As you see some lines are non-xml and some are xml..

my props.conf file has right set of rules

[sourcetype]
BREAK_ONLY_BEFORE = ||
KV_MODE = xml
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
pulldown_type = 1
TRANSFORMS-set = setnull, setparsing

and in transforms.xml

[setnull]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ||
DEST_KEY = queue
FORMAT = indexQueue

Will this work? ...the lines outside XML tags as shown in example should not be indexed...

Tags (1)
0 Karma

kristian_kolb
Ultra Champion

Er, well, no. Your events will be broken where you've indicated, i.e. like this;

Event 1:

<transaction>ffffff</transaction> 
ABCD EFG 

Event 2:

<access></access> 
WERT SDF

Thus you should consider whether these are the types of event that you want. Btw, do they even contain timestamps?

As for the second part of your setup, the nullQueueing will work, but maybe not as you've expected. The ordering of the transforms is correct - all events (the regex dot) get queue=nullQueue in the first transform, but since all events (at least according to your sample events) will also match on the second transform, they get the queue set back to indexQueue. In short, the order of the transforms matters since each event for your sourcetype will pass through all transforms - in order - before being processed further.

If you want to filter out parts of an event, you should probably look at some other options, like SEDCMD. Perhaps this can be of use;

http://docs.splunk.com/Documentation/Splunk/6.0.1/Data/Anonymizedatausingconfigurationfiles

Hope this helps,

K

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...