topic Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it? in Getting Data In

How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Wed, 20 Jun 2018 16:55:41 GMT

Hi Splunk experts,

Just want to know how can I remove events which does not contain any information in it?
Example

sample events which have information on them.

I know that, we have to configure Route and Filter data in Props/transforms to achieve this but not sure about the Regex.

Props.conf details:
[who]
TRANSFORMS-null= setnull

Transforms.conf details:
[setnull]
 REGEX = 
 DEST_KEY = queue
 FORMAT = nullQueue

Kindly guide me on this

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

somesoni2 — Wed, 20 Jun 2018 18:17:19 GMT

The REGEX will depend upon how your event looks like when it has information. Could you provide some sample events which have information on them?

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

cpetterborg — Wed, 20 Jun 2018 19:08:26 GMT

If the Question contains the example data, then perhaps something like:

REGEX = ^USERNAME\s+LINE\s+HOSTNAME\s+TIME$

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Wed, 20 Jun 2018 19:31:23 GMT

Hi Somesoni2, thanks for your effort on this, we are monitoring wtmpx file from the unix machines using the Splunk Add-on for Unix. Below are the sample events contains the information.

Index=unix sourcetype=who host=*

6/20/18
3:06:05.000 PM

USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
host = ttbmt02 source = who sourcetype = who
6/20/18
3:03:35.000 PM

USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Wed, 20 Jun 2018 19:33:26 GMT

Hi cpetterborg, thanks for your effort on this, I had added the sample events containing the data, so can I use the above regex to remove events which does not contain any information in it.

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

cpetterborg — Wed, 20 Jun 2018 19:38:09 GMT

If you are making the events be multi-line events (one for each time the command is run), then you can use the SEDCMD in props.conf to remove only the header lines.

If you are making each line be a separate event (as it appeared in your question), you can use something like my suggestion.

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Wed, 20 Jun 2018 19:49:18 GMT

Hey we are splitting multiple events in to single individual events by using the line_breaker stanza in props.conf. And also we wanted to remove the events which does not contain any information in it.

1) To split multiple events in to single individual events.

[who]
 SHOULD_LINEMERGE=false
 LINE_BREAKER=([\r\n]+)
 TRUNCATE=1000000
 DATETIME_CONFIG = CURRENT

2) For removing the events which does not contain any information.

Props.conf
[who]
 TRANSFORMS-null= setnull

 Transforms.conf details:
 [setnull]
  REGEX = ^USERNAME\s+LINE\s+HOSTNAME\s+TIME$
  DEST_KEY = queue
  FORMAT = nullQueue

Kindly guide me on this.

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

somesoni2 — Wed, 20 Jun 2018 21:38:05 GMT

Give this a try (changed strategy from 'Drop specific events' to 'Keep specific event and drop remaining').
Reference: http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Edit props.conf and add the following:

[who]
TRANSFORMS-set= setnull,setparsing

Edit transforms.conf and add the following:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

#basically keeping anything that has header and at least one data row
[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Thu, 21 Jun 2018 09:40:22 GMT

Hi SomeSoni2,

Yes I had given a try with the above set of configuration in my test environment by uploading the data in raw text format via Add data -->upload --> select sourcetype but it did not fetch the required output.

Instead getting the below message.
No results found. Please change source type, adjust source type settings, or check your source file

Props.conf details:
[who]
  SHOULD_LINEMERGE=false
  LINE_BREAKER=([\r\n]+)
  TRUNCATE=1000000
  DATETIME_CONFIG = CURRENT
TRANSFORMS-set= setnull,setparsing

Transforms.conf details:
[setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

 #basically keeping anything that has header and at least one data row
 [setparsing]
 REGEX = ^(USERNAME).+[\r\n]+\w+
 DEST_KEY = queue
 FORMAT = indexQueue

Kindly guide me on this.

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Thu, 21 Jun 2018 14:50:08 GMT

Hi Somesoni2, by applying the above stanza in props.conf and transforms.conf, the entire event information are removed from splunk and it throws the below message.

No results found. Please change source type, adjust source type settings, or check your source file

Could you please guide me on this to events which does not contain any information in it.

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

somesoni2 — Thu, 21 Jun 2018 14:58:11 GMT

YOu said you're uploading it from an instance. Did you setup these (props and transforms) on the same server/instance? If this a test environment, try setting up a monitoring (using inputs.conf) and test the ingestion, instead of using Add Data wizard.

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Thu, 21 Jun 2018 15:38:13 GMT

thanks Somesoni2, let me try that.

Re: How to configure in props/transforms.conf to remove the event data which does not contain any information in it?

Hemnaath — Thu, 21 Jun 2018 17:26:12 GMT

Hey it did not work, could you please guide me on this

USERNAME LINE HOSTNAME TIME

host = tt3crp00 source = who sourcetype = who

6/21/18
1:11:25.000 PM

USERNAME LINE HOSTNAME TIME

solarwinds pts/21 vmswpep02.XXXX.com Jun 21 13:11

host = tt3dev00 source = who sourcetype = who

6/21/18
1:11:25.000 PM

USERNAME LINE HOSTNAME TIME
solarwinds pts/0 vmswpep05.xxxx.com 2018-06-21 13:11
solarwinds pts/1 vmswpep05.xxxx.com 2018-06-21 13:11

the above props/transforms.conf are placed in the Heavy forwarder instances.

Kindly guide me to fix this issue.