topic Re: How do I parse a value from a log message? in Splunk Search

How do I parse a value from a log message?

oriches — Thu, 04 Jul 2013 13:00:12 GMT

I have the log messages in the following structure, the one shown represents a heardbeat from the application.

How can I parse the ProcessMemory(KB) value?

Is it best to change the log message structure so all values use an equals '=' as a seperator?

TimeStamp=2013-07-04 13:48:45.733 +01:00 | UserInterface | Level=Info | Spec=UserInterface | Level=Debug | Spec=Heartbeat | Attributes=Status:Running, Utilisation(%):2, AvailableMemory(KB):10076160, ProcessUtilisation(%):0, ProcessMemory(KB):194668, ProcessPeakMemory(KB):194732, ProcessorCount:12 | Type=Heartbeat | SessionId=8d26ee12-f40f-471d-bea4-5836fce72362 | ThreadName=61 | Status=Instant

Re: How do I parse a value from a log message?

Linegod — Thu, 04 Jul 2013 13:33:00 GMT

Extract Fields with search commands

Manage Search Time Field Extractions

Re: How do I parse a value from a log message?

grijhwani — Thu, 04 Jul 2013 13:35:58 GMT

You don't necessarily need to. Use the field definition tool on sample data. Select the drop-down arrow against any sample log entry, take the "extract field" option, and follow through the dialogue. This will (attempt to) automatically created a regex for reliably locating your required field. You may need to refine the regex manually if the generated form pulls unexpected values from unintended records, but you have ample opportunity to refine and test.

Re: How do I parse a value from a log message?

MHibbin — Thu, 04 Jul 2013 13:55:10 GMT

Why would you want to change the separator to "=" that's already in use within the logging. In terms of the log itself, they have different structure relevance.

Re: How do I parse a value from a log message?

oriches — Thu, 04 Jul 2013 13:59:16 GMT

I don't particularly, I'm new to splunk and trying to workout what's the best way to extract the data i want

Re: How do I parse a value from a log message?

Gilberto_Castil — Thu, 04 Jul 2013 14:58:24 GMT

The best way to extract these values is to group the desired field name and corresponding value, detached by the delimiter. To preserve the actual field name as closely as possible, a regular expression will help.

In your props.conf define a report

#props.conf
[answers-1372947346]
REPORT-get_perf_fields = get_perf_fields

In transforms.conf define the extraction method:

#transforms.conf
[get_perf_fields]
REGEX = ([a-zA-Z\(\)\%]+)\:([a-zA-Z0-9]+)
FORMAT = $1::$2
MV_ADD = true

This should automatically load the fields at search time. Note that the non-alphabetic characters will be replaced with an underscore character. That means ProcessUtilisation(%) will become ProcessUtilisation___.

PS: If you are unsure as to where to place the props.conf or transforms.conf files, open or create them under $SPLUNK_HOME/etc/apps/search/local/ in Linux/UNIX or %SPLUNK_HOME%\etc\apps\local in Windows.

Re: How do I parse a value from a log message?

Gilberto_Castil — Thu, 04 Jul 2013 15:06:14 GMT

You do not need to change the separator. This is known as Semantic Logging and it is a recommended approach not just for Splunk but as a general best practice. Splunk will automatically recognize key value pairs with an equals delimiter. Splunk is very flexible and will work with what you have, regardless of the format.

Re: How do I parse a value from a log message?

venki08 — Tue, 26 Jul 2016 05:34:31 GMT

hi in my splunk sonic firewall log the source address with source port in single header so how to do parse . (Example. src:192.168.1.2:5545:XA1) I WANT( SOURCE ADDRESS=192.168.1.2 SOURCE PORT 5545 OTHER XA1) kindly suggest how to do ..?