Splunk Dev

Extract fields fomr Python Dictionary text file

Claw
Splunk Employee
Splunk Employee

How do I create an extract that handles variable numbers of fields.

I am generating events that are time stamped Python dictionaries. The list of items are emitted to the event using Python logging where everything in curly brackets was from a single Python dictionary. The dictionary is rendered using built-in rend(d) where d is a dict. The list of items in the {} brackets are variable, and the order of items is arbitrary.

2013/07/18 19:51:00.090 UTC [DssSplunkHostAgent] INFO: {'rtime': 31556.303133832, 'ABI': 32, 'egid': 20000, 'uid': 101, 'mrcv': 0, 'pctcpu': 0.0, 'pctmem': 0.5523681640625, 'args': 'python -W ignore::DeprecationWarning -m DssCore', 'pid': 9452, 'Taskname': 'DssCore', 'start': datetime.datetime(2013, 7, 18, 11, 5, 5, 785854), 'gid': 20000, 'euid': 101, 'fname': 'python', 'time': 1.244510313, 'oublk': 0, 'dmodel': 1, 'inblk': 0, 'ppid': 9289, 'msnd': 0, 'ctime': 256.16}

0 Karma

Claw
Splunk Employee
Splunk Employee

Here is a very clean answer. It assumes that the sourcetype is called python dictionary. Change this to meet your needs.

The oddball issue here is the "start" field so we set up automated extractions for every thing else using DELIMS and handle the exceptions with regular extracts.

Props.conf

[pythondictionary]
EXTRACT-PD_MessageLevel = (?i)[.?] (?P\w+)(?=:)
EXTRACT-PD_Agent = (?i)^[^[]
[(?P[^]]+)
REPORT-pythondictionary = PD-Extract
EXTRACT-PD_Message = {(?P.+)(?=})
EXTRACT-start = \'start\':\s(?.*?),\s\'

Transforms.conf

[PD-Extract]
SOURCE_KEY = PD_Message
DELIMS = ",", ":"

Kudos for this solution goes to Dritan Bitincka in Splunk ps.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Kubernetes Alerts That Work Out of the Box with AutoDetect detectors in Splunk ...

A deployment rolls out slowly, one pod sits not ready for a few minutes, and whoever is on call gets an alert ...

Observability at Your Fingertips Directly From Splunk Platform—for Free

At .conf, we announced our roadmap to bring observability foundations to Splunk Platform. Bringing metrics and ...

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...