topic Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf? in Splunk Search

Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

ishangajera — Mon, 10 Aug 2015 12:42:54 GMT

I have created a shell script (script input) which is attached. It gives me information about status of threads in Linux.

Sample output of the script is as below:

totThreads      runThreads      slpThreads
       433               2             431

Now I want extract fields from these events.

In transforms.conf:

[fields_for_threads_sh]
REGEX = \s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)
FORMAT = totThreads::"$1" runThreads::"$2" slpThreads::"$3"

In props.conf:

[threads]
LINE_BREAKER=(^$|[\r\n]+[\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT

REPORT-0kv_for_threads = fields_for_threads_sh
FIELDALIAS-system_threads_count_for_threads = totThreads as system_threads_count
FIELDALIAS-sleeping_threads_count_for_threads = slpThreads as wait_threads_count
FIELDALIAS-running_threads_count_for_threads = runThreads as running_threads_count

Now when all this is configured, my events are generated with all three values as 0.

totThreads  runThreads  slpThreads
         0           0           0

Please help where I am going wrong?

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

woodcock — Mon, 10 Aug 2015 17:30:48 GMT

Try this:

REGEX = \s*(\d+)\s+(\d+)\s+(\d+)
FORMAT = totThreads::$1 runThreads::$2 slpThreads::$3

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

ishangajera — Tue, 11 Aug 2015 04:30:10 GMT

Hi,

Tried changing REGEX and FORMAT as you specified. But still no change. The values are still 0.

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

woodcock — Tue, 11 Aug 2015 14:10:14 GMT

I don't understand. The configurations we are talking about have NOTHING to do with the data inside the raw events (they can never "case" anything to become "0"). What does this show?

... | table _raw totThreads runThreads slpThreads

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

ishangajera — Wed, 12 Aug 2015 04:38:11 GMT

Hi
Please check the output as shown in image:

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

woodcock — Wed, 12 Aug 2015 13:27:31 GMT

Since it is multiline, try this:

REGEX = (?m)\s*(\d+)\s+(\d+)\s+(\d+)
FORMAT = totThreads::$1 runThreads::$2 slpThreads::$3

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

ishangajera — Thu, 13 Aug 2015 04:50:13 GMT

Hi,

It is still not working. I have another configuration for another field in the app.
That is also multiline event. But for that we are getting the data in events.

Details of that field are as below:

Ouput of shell script:

 [root@splunkitsi bin]$ ./vmstat.sh
    memTotalMB   memFreeMB   memUsedMB  memFreePct  memUsedPct   pgPageOut  swapUsedPct   pgSwapOut   cSwitches  interrupts       forks   processes     threads  loadAvg1mi
         12863       12122         741        94.2         5.8   338955619          0.9       12696  2302182698  3016592159    15420967         130         407        1.08

Content of Transforms.conf

#memTotalMB   memFreeMB   memUsedMB  memFreePct  memUsedPct   pgPageOut  swapUsedPct   pgSwapOut   cSwitches  interrupts       forks   processes     threads  loadAvg1mi
    #      8192        4153        4039        50.7        49.3     1585619          5.0           ?           ?           ?           ?          82         566        0.72
    [fields_for_vmstat_sh]
    REGEX = \s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)\s+([0-9.?]+)
    FORMAT = memTotalMB::"$1" memFreeMB::"$2" memUsedMB::"$3" memFreePct::"$4" memUsedPct::"$5" pgPageOut::"$6" swapUsedPct::"$7" pgSwapOut::"$8" cSwitches::"$9" interrupts::"$10" forks::"$11" processes::"$12" threads::"$13" loadAvg1mi::"$14"

Contents of Props.conf

[vmstat]
    LINE_BREAKER=(^$|[\r\n]+[\r\n]+)
    TRUNCATE=1000000
    DATETIME_CONFIG = CURRENT
    REPORT-0kv_for_vmstat = fields_for_vmstat_sh,vmstat_linux,vmstat_osx
    FIELDALIAS-dest_for_vmstat = host as dest
    EVAL-mem = if(isnotnull(memFreeMB) AND isnotnull(memUsedMB),(memFreeMB*1048576)+(memUsedMB*1048576),null())
    EVAL-mem_free = if(isnotnull(memFreeMB),memFreeMB*1048576,null())
    EVAL-mem_used = if(isnotnull(memUsedMB),memUsedMB*1048576,null())
    FIELDALIAS-src_for_vmstat = host as src

Event output in Search query:

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

woodcock — Thu, 13 Aug 2015 18:17:29 GMT

I am stumped.

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

ishangajera — Tue, 18 Aug 2015 13:44:02 GMT

Hi,

I am still not able to find the solution. Can anyone please help?

Re: Why am I unable to extract event values using REGEX and FORMAT in transforms.conf and REPORT in props.conf?

landen99 — Fri, 11 Sep 2015 15:21:07 GMT

Obviously he is extracting 0's because the raw data only has 0's. Everything is working as expected. The issue is in the raw data. Look into why your box is sending all 0's

Also, I am not seeing multi-line events. That should not be a consideration in this case.