Extract fields fomr Python Dictionary text file

Claw — Fri, 26 Jul 2013 22:52:13 GMT

How do I create an extract that handles variable numbers of fields.

I am generating events that are time stamped Python dictionaries. The list of items are emitted to the event using Python logging where everything in curly brackets was from a single Python dictionary. The dictionary is rendered using built-in rend(d) where d is a dict. The list of items in the {} brackets are variable, and the order of items is arbitrary.

2013/07/18 19:51:00.090 UTC [DssSplunkHostAgent] INFO: {'rtime': 31556.303133832, 'ABI': 32, 'egid': 20000, 'uid': 101, 'mrcv': 0, 'pctcpu': 0.0, 'pctmem': 0.5523681640625, 'args': 'python -W ignore::DeprecationWarning -m DssCore', 'pid': 9452, 'Taskname': 'DssCore', 'start': datetime.datetime(2013, 7, 18, 11, 5, 5, 785854), 'gid': 20000, 'euid': 101, 'fname': 'python', 'time': 1.244510313, 'oublk': 0, 'dmodel': 1, 'inblk': 0, 'ppid': 9289, 'msnd': 0, 'ctime': 256.16}

Re: Extract fields fomr Python Dictionary text file

Claw — Mon, 28 Sep 2020 14:26:41 GMT

Here is a very clean answer. It assumes that the sourcetype is called python dictionary. Change this to meet your needs.

The oddball issue here is the "start" field so we set up automated extractions for every thing else using DELIMS and handle the exceptions with regular extracts.

Props.conf

[pythondictionary]
EXTRACT-PD_MessageLevel = (?i)[.?] (?P\w+)(?=:)
EXTRACT-PD_Agent = (?i)^[^[][(?P[^]]+)
REPORT-pythondictionary = PD-Extract
EXTRACT-PD_Message = {(?P.+)(?=})
EXTRACT-start = \'start\':\s(?.*?),\s\'

Transforms.conf

[PD-Extract]
SOURCE_KEY = PD_Message
DELIMS = ",", ":"

Kudos for this solution goes to Dritan Bitincka in Splunk ps.

topic Extract fields fomr Python Dictionary text file in Splunk Dev

Extract fields fomr Python Dictionary text file

Re: Extract fields fomr Python Dictionary text file