How do I parse a value from a log message?

[oriches]

I have the log messages in the following structure, the one shown represents a heardbeat from the application.

How can I parse the ProcessMemory(KB) value?

or

Is it best to change the log message structure so all values use an equals '=' as a seperator?

TimeStamp=2013-07-04 13:48:45.733 +01:00 | UserInterface | Level=Info | Spec=UserInterface | Level=Debug | Spec=Heartbeat | Attributes=Status:Running, Utilisation(%):2, AvailableMemory(KB):10076160, ProcessUtilisation(%):0, ProcessMemory(KB):194668, ProcessPeakMemory(KB):194732, ProcessorCount:12 | Type=Heartbeat | SessionId=8d26ee12-f40f-471d-bea4-5836fce72362 | ThreadName=61 | Status=Instant

[Gilberto_Castil]

You do not need to change the separator. This is known as Semantic Logging and it is a recommended approach not just for Splunk but as a general best practice. Splunk will automatically recognize key value pairs with an equals delimiter. Splunk is very flexible and will work with what you have, regardless of the format.

[venki08]

hi in my splunk sonic firewall log the source address with source port in single header so how to do parse . (Example. src:192.168.1.2:5545:XA1) I WANT( SOURCE ADDRESS=192.168.1.2 SOURCE PORT 5545 OTHER XA1) kindly suggest how to do ..?

[Gilberto_Castil]

The best way to extract these values is to group the desired field name and corresponding value, detached by the delimiter. To preserve the actual field name as closely as possible, a regular expression will help.

In your props.conf define a report

#props.conf
[answers-1372947346]
REPORT-get_perf_fields = get_perf_fields

In transforms.conf define the extraction method:

#transforms.conf
[get_perf_fields]
REGEX = ([a-zA-Z\(\)\%]+)\:([a-zA-Z0-9]+)
FORMAT = $1::$2
MV_ADD = true

This should automatically load the fields at search time. Note that the non-alphabetic characters will be replaced with an underscore character. That means ProcessUtilisation(%) will become ProcessUtilisation___.

---

PS: If you are unsure as to where to place the props.conf or transforms.conf files, open or create them under $SPLUNK_HOME/etc/apps/search/local/ in Linux/UNIX or %SPLUNK_HOME%\etc\apps\local in Windows.

[oriches]

I don't particularly, I'm new to splunk and trying to workout what's the best way to extract the data i want

[MHibbin]

Why would you want to change the separator to "=" that's already in use within the logging. In terms of the log itself, they have different structure relevance.

[grijhwani]

You don't necessarily need to. Use the field definition tool on sample data. Select the drop-down arrow against any sample log entry, take the "extract field" option, and follow through the dialogue. This will (attempt to) automatically created a regex for reliably locating your required field. You may need to refine the regex manually if the generated form pulls unexpected values from unintended records, but you have ample opportunity to refine and test.

[Linegod · Accepted solution]

Extract Fields with search commands

Manage Search Time Field Extractions