How to extract keywords and associated values with rex from raw logs and display data in a table?

[satya2p]

I have a request input output logged by various sourcetypes in XML and other similar below format. I tried multiple options to extract exact keywords and associated values to display in a table, but I have been unable to do so. Please help.

Sample Data:

Somecontent.ssn=XXXXXXXXX.Somecontent.SomeOthercontent
in XML: <SSN> XXXXXXXXX </SSN>

Options Tried:

| rex field=_raw "SSN=(?P\d\d\d-?\d\d-?\d\d\d\d)" | stats count by SSN
| rex field=_raw=\W+SSN "(?\w+)" | stats count by SSN
| rex field=_raw "SSN\=(?\d{9})+"| stats count by SSN

[richgalloway · Accepted solution]

Your rex commands are not extracting fields. Therefore, there is nothing for downstream commands to work with. Try this:

... | rex "SSN\>\s*(?P<SSN>\d{9})" | stats count by SSN

The first "SSN" is an eyecatcher to help rex find the right data in your XML. The second "SSN" is a field name which can be used by the stats command.

[satya2p]

Thank you.. I tried, its just reading all SSN keyword(considering false positive) not extracting the values and listing out.

[richgalloway]

Let's try something simpler. This should display all the SSN values found.

... | rex "SSN\>\s*(?P<SSN>\d{9})" | table SSN

[satya2p]

I tried this option earlier, table getting generated for each event logged by keyword but data is not populating. seems its unable to extract values. I am using splunk 5.0.9, is xml filed extraction is available in this release.

[richgalloway]

Yes, field extraction using the rex command is the same in version 5. Is the sample data in your OP accurate? Regex strings can be very sensitive to differences in white space, case, etc.