topic How to extract fields from data that has NO key-value pair (data provides just the value) in Splunk Search

How to extract fields from data that has NO key-value pair (data provides just the value)

seanlon11 — Fri, 14 May 2010 02:22:21 GMT

The data below has NO key-value pairs, but the bold data is very important. If I could somehow extract the bold data below into the following fields, it would be very helpful:

(in order of boldness, and separated by a space. the bold data can obviously be different with each log entry, but could also be repeated)

PID
Session ID
Request ID

Any suggestions?

Thanks, Sean

BTW - this is a Cognos log

192.168.27.13:9081 9892 2010-05-13 04:59:28.439 -6 ECD46F7C56E2D4F6E57931FA643C78980F4264F2 Ch9v29jvh228Glw8w8jd9M2jwC8M9wMvj448j8s2 Ch9v29jvh228Glw8w8jd9M2jwC8M9wMvj448j8s2 3104 RSVP 1198 3 Audit.RTUsage.RSVP Execute Report /content/folder[@name='Vendor Reports']/folder[@name='Mars']/report[@name='DSVND05 - DISTRIBUTION REPORT XLS'] Success

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

Lowell — Fri, 14 May 2010 02:48:44 GMT

You can extract this at search time (interactively) using rex. Once you have a good regular expression defined, then you can stick it into your props.conf file to match against either a source or sourcetype that this data is found in. Or, you can use the interactive field extractor.

Option 1: Interactive extraction

your search ... | rex "^\S+ (?P<pid>\d+) \S+ \S+ \S+ (?P<session_id>\S+) (?P<request_id>\S+)"

Options 2: Setup and entry in props.conf:

[my_source_type]
EXTRACT-key_fields = ^\S+ (?P<pid>\d+) \S+ \S+ \S+ (?P<session_id>\S+) (?P<request_id>\S+)

Option 3: Use the interactive field extractor. See the Fields extraction tutorial.

Hope that helps you get started.

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

dwaddle — Fri, 14 May 2010 02:49:01 GMT

You can assign an arbitrary field name as part of a search time extraction. Try something similar to this...

In ${SPLUNK_HOME}/etc/system/local/transforms.conf:

[cognosfields]
REGEX=^[^\s]+ ([0-9]+) [^\s]+ [^\s]+ [^\s]+ ([^\s]+) ([^\s]+)
FORMAT= pid::$1 sessionid::$2 requestid::$3

In ${SPLUNK_HOME}/etc/system/local/props.conf:

[cognos_type]
REPORT-cognos=cognosfields

Reload the configs by doing a search in the splunkweb UI of "| kv reload=true" - and see if your fields show up.

http://www.splunk.com/base/Documentation/4.0.10/Knowledge/Addfieldsatsearchtime#Add_fields_at_search_time_through_configuration_file_edits

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

gkanapathy — Fri, 14 May 2010 02:49:27 GMT

Presumably this data can be identified by a regular expression or delimiters between fields. In that case, you have a variety of ways to perform extractions. Please see: http://www.splunk.com/base/Documentation/latest/Knowledge/Addfieldsatsearchtime and http://www.splunk.com/base/Documentation/latest/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

Lowell — Fri, 14 May 2010 02:53:51 GMT

Dwaddle, I think you mean REPORT-cognos=cognosfields, not EXTRACT I think they used to be equivalent in Splunk 3.x, but they aren't in 4.0

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

seanlon11 — Fri, 14 May 2010 03:17:13 GMT

Splunkerific! Thanks for the help - IFX worked like a charm.

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

dwaddle — Fri, 14 May 2010 03:21:46 GMT

Right you are... I use REPORT- in all of my configs, but goofed here... /me fixes

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

Jason — Mon, 18 Oct 2010 22:49:24 GMT

As of 4.1.x, the reload command is not necessary - just run another search.

Re: How to extract fields from data that has NO key-value pair (data provides just the value)

Pierceyuk — Wed, 12 Feb 2014 11:22:00 GMT

Just thought I would add the way I did this which can be found in my question here: http://answers.splunk.com/answers/112243/delimited-field-extractions-for-cognos-data
sometime you can just see the data is separated by tabs and can name the columns rather than using regex!

Build a transforms as follows:

[cognos-fields]
DELIMS="t"
FIELDS="Host ID","Process ID","Time","Time Zone","Session ID","Request ID","SubRequest ID","Step ID","Thread","Component ID","Build Number","Level","Logger","Operation","Object Type","Object Path","Status","Message","Log Data"

much easier I find 🙂