Splunk Dev

Extract fields fomr Python Dictionary text file

Claw
Splunk Employee
Splunk Employee

How do I create an extract that handles variable numbers of fields.

I am generating events that are time stamped Python dictionaries. The list of items are emitted to the event using Python logging where everything in curly brackets was from a single Python dictionary. The dictionary is rendered using built-in rend(d) where d is a dict. The list of items in the {} brackets are variable, and the order of items is arbitrary.

2013/07/18 19:51:00.090 UTC [DssSplunkHostAgent] INFO: {'rtime': 31556.303133832, 'ABI': 32, 'egid': 20000, 'uid': 101, 'mrcv': 0, 'pctcpu': 0.0, 'pctmem': 0.5523681640625, 'args': 'python -W ignore::DeprecationWarning -m DssCore', 'pid': 9452, 'Taskname': 'DssCore', 'start': datetime.datetime(2013, 7, 18, 11, 5, 5, 785854), 'gid': 20000, 'euid': 101, 'fname': 'python', 'time': 1.244510313, 'oublk': 0, 'dmodel': 1, 'inblk': 0, 'ppid': 9289, 'msnd': 0, 'ctime': 256.16}

0 Karma

Claw
Splunk Employee
Splunk Employee

Here is a very clean answer. It assumes that the sourcetype is called python dictionary. Change this to meet your needs.

The oddball issue here is the "start" field so we set up automated extractions for every thing else using DELIMS and handle the exceptions with regular extracts.

Props.conf

[pythondictionary]
EXTRACT-PD_MessageLevel = (?i)[.?] (?P\w+)(?=:)
EXTRACT-PD_Agent = (?i)^[^[]
[(?P[^]]+)
REPORT-pythondictionary = PD-Extract
EXTRACT-PD_Message = {(?P.+)(?=})
EXTRACT-start = \'start\':\s(?.*?),\s\'

Transforms.conf

[PD-Extract]
SOURCE_KEY = PD_Message
DELIMS = ",", ":"

Kudos for this solution goes to Dritan Bitincka in Splunk ps.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...