topic Re: How to parse and extract field from my raw data in Splunk Search

How to parse and extract field from my raw data?

imam28 — Mon, 21 Nov 2022 03:44:20 GMT

I Have a log like this, how do I Parse it into fields??

Is there a way to use Splunk to parse this and extract one value? If so, how?

Thank you in advance.

Regards,

Imam

Re: How to parse and extract field from my raw data

bowesmana — Mon, 21 Nov 2022 02:25:20 GMT

Looking at that data, it appears to be field name/field value pairs separated by line feeds, so a simple mechanism in SPL is to do

| extract kvdelim="," pairdelim="\n"

which will extract the field names/values from the _raw field

However, you should really extract these at ingest time, but that will depend on how your data is being ingested. Is the data coming in as a multi-row event. If you take a file containing a number of events, you can use the 'Add data' option to play with ingesting that data, so the fields are extracted automatically

Re: How to parse and extract field from my raw data?

yuanliu — Mon, 21 Nov 2022 09:50:00 GMT

The screenshot doesn't show any comma, or anything that clearly signifies a separation of field name from field value for @bowesmana's method to succeed. If that is the raw log, your best bet is to go back to developers of the code and ask for better logging. There is no excuse to write such poorly formatted logs. If it is not raw log, you should illustrate raw data (in text, anonymize as needed).

There are some workarounds if certain assumptions holds, but they will not work 100%.

First, I assume that each line that contains a string like abc.efg represents a field name, and lines that do not contain this pattern are values.

This will (kinda) work till it gets to the line db.0. The next line is overhead.hashtable.main. (With two dots.) Is this a field name or a field value? Using the above method, you'll get something like db.0=overhead.hashtable.main=14912, which the kv (extract) command will extract into field name db_0, value overhead_hashtable_main=14912.

An less versatile method could be to assume that each "odd numbered" line is a field name, other lines will be field value.

Again, this kinda works till it gets to the line db.0. It will transform into db.0=overhead.hashtable.main, and 14912. Splunk will then extract field name db_0 with value overhead.hashtable.main, and field name 14912 with null value.

Either way, these are very tortured methods. Application developers ought to do better logging than this.

Re: How to parse and extract field from my raw data

bowesmana — Mon, 21 Nov 2022 22:22:06 GMT

My bad reading your screenshot, I saw commas - however, although you don't have commas in your data, the extract command can still separate out the line feed separators, as in this example

| makeresults | eval _raw="peak.allocated 31660 total.allocated 423084723094 c 2" | extract kvdelim="\n" pairdelim="\n"

but it would rely on well formatted data. However, this solution is really only a hack solution to read already ingested data like that.

Re: How to parse and extract field from my raw data?

imam28 — Fri, 25 Nov 2022 06:41:42 GMT

where i write this command , in search or transform.conf?

Re: How to parse and extract field from my raw data?

bowesmana — Fri, 25 Nov 2022 06:45:19 GMT

These are all search time examples

Re: How to parse and extract field from my raw data

imam28 — Fri, 25 Nov 2022 07:44:20 GMT

I have followed the command, but there are some that still haven't extracted what is the solution?

Re: How to parse and extract field from my raw data

yuanliu — Fri, 25 Nov 2022 23:06:22 GMT

That is the exact problem @bowesmana and I pointed out: Poorly written logs. You really need to ask coders to write logs that leaves no ambiguity, that is conducive to extraction by mechanical tools like Splunk.

Re: How to parse and extract field from my raw data

bowesmana — Sun, 27 Nov 2022 21:40:28 GMT

@imam28

The problem you have is that db.0 has no 'value' following it on the next row, so no simple regular extraction will work.

Can you post the _raw view of that event.

Just wondering what it's showing, whether it's a multivalue field, where perhaps the "value" for db.0 is a null value, in which case it may be possible to extract it.

Re: How to parse and extract field from my raw data

imam28 — Mon, 28 Nov 2022 04:08:37 GMT

Correct, db.0 in the next raw has no value in the next raw

Re: How to parse and extract field from my raw data

imam28 — Mon, 28 Nov 2022 04:13:41 GMT

it is log event , only db.0 haven't value