Splunk Dev

Extract fields fomr Python Dictionary text file

Claw
Splunk Employee
Splunk Employee

How do I create an extract that handles variable numbers of fields.

I am generating events that are time stamped Python dictionaries. The list of items are emitted to the event using Python logging where everything in curly brackets was from a single Python dictionary. The dictionary is rendered using built-in rend(d) where d is a dict. The list of items in the {} brackets are variable, and the order of items is arbitrary.

2013/07/18 19:51:00.090 UTC [DssSplunkHostAgent] INFO: {'rtime': 31556.303133832, 'ABI': 32, 'egid': 20000, 'uid': 101, 'mrcv': 0, 'pctcpu': 0.0, 'pctmem': 0.5523681640625, 'args': 'python -W ignore::DeprecationWarning -m DssCore', 'pid': 9452, 'Taskname': 'DssCore', 'start': datetime.datetime(2013, 7, 18, 11, 5, 5, 785854), 'gid': 20000, 'euid': 101, 'fname': 'python', 'time': 1.244510313, 'oublk': 0, 'dmodel': 1, 'inblk': 0, 'ppid': 9289, 'msnd': 0, 'ctime': 256.16}

0 Karma

Claw
Splunk Employee
Splunk Employee

Here is a very clean answer. It assumes that the sourcetype is called python dictionary. Change this to meet your needs.

The oddball issue here is the "start" field so we set up automated extractions for every thing else using DELIMS and handle the exceptions with regular extracts.

Props.conf

[pythondictionary]
EXTRACT-PD_MessageLevel = (?i)[.?] (?P\w+)(?=:)
EXTRACT-PD_Agent = (?i)^[^[]
[(?P[^]]+)
REPORT-pythondictionary = PD-Extract
EXTRACT-PD_Message = {(?P.+)(?=})
EXTRACT-start = \'start\':\s(?.*?),\s\'

Transforms.conf

[PD-Extract]
SOURCE_KEY = PD_Message
DELIMS = ",", ":"

Kudos for this solution goes to Dritan Bitincka in Splunk ps.

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...