Hi Splunk experts,
Just want to know how can I remove events which does not contain any information in it?
Example
sample events which have information on them.
I know that, we have to configure Route and Filter data in Props/transforms to achieve this but not sure about the Regex.
Props.conf details:
[who]
TRANSFORMS-null= setnull
Transforms.conf details:
[setnull]
REGEX =
DEST_KEY = queue
FORMAT = nullQueue
Kindly guide me on this
Give this a try (changed strategy from 'Drop specific events' to 'Keep specific event and drop remaining').
Reference: http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad#Keep_specific_event...
Edit props.conf and add the following:
[who]
TRANSFORMS-set= setnull,setparsing
Edit transforms.conf and add the following:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
#basically keeping anything that has header and at least one data row
[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue
Hi SomeSoni2,
Yes I had given a try with the above set of configuration in my test environment by uploading the data in raw text format via Add data -->upload --> select sourcetype but it did not fetch the required output.
Instead getting the below message.
No results found. Please change source type, adjust source type settings, or check your source file
Props.conf details:
[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
TRANSFORMS-set= setnull,setparsing
Transforms.conf details:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
#basically keeping anything that has header and at least one data row
[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue
Kindly guide me on this.
YOu said you're uploading it from an instance. Did you setup these (props and transforms) on the same server/instance? If this a test environment, try setting up a monitoring (using inputs.conf) and test the ingestion, instead of using Add Data wizard.
thanks Somesoni2, let me try that.
Hey it did not work, could you please guide me on this
USERNAME LINE HOSTNAME TIME
host = tt3crp00 source = who sourcetype = who
6/21/18
1:11:25.000 PM
USERNAME LINE HOSTNAME TIME
solarwinds pts/21 vmswpep02.XXXX.com Jun 21 13:11
host = tt3dev00 source = who sourcetype = who
6/21/18
1:11:25.000 PM
USERNAME LINE HOSTNAME TIME
solarwinds pts/0 vmswpep05.xxxx.com 2018-06-21 13:11
solarwinds pts/1 vmswpep05.xxxx.com 2018-06-21 13:11
the above props/transforms.conf are placed in the Heavy forwarder instances.
Kindly guide me to fix this issue.
Hi Somesoni2, by applying the above stanza in props.conf and transforms.conf, the entire event information are removed from splunk and it throws the below message.
No results found. Please change source type, adjust source type settings, or check your source file
Could you please guide me on this to events which does not contain any information in it.
If the Question contains the example data, then perhaps something like:
REGEX = ^USERNAME\s+LINE\s+HOSTNAME\s+TIME$
Hi cpetterborg, thanks for your effort on this, I had added the sample events containing the data, so can I use the above regex to remove events which does not contain any information in it.
If you are making the events be multi-line events (one for each time the command is run), then you can use the SEDCMD
in props.conf
to remove only the header lines.
If you are making each line be a separate event (as it appeared in your question), you can use something like my suggestion.
Hey we are splitting multiple events in to single individual events by using the line_breaker stanza in props.conf. And also we wanted to remove the events which does not contain any information in it.
1) To split multiple events in to single individual events.
[who]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT
2) For removing the events which does not contain any information.
Props.conf
[who]
TRANSFORMS-null= setnull
Transforms.conf details:
[setnull]
REGEX = ^USERNAME\s+LINE\s+HOSTNAME\s+TIME$
DEST_KEY = queue
FORMAT = nullQueue
Kindly guide me on this.
The REGEX will depend upon how your event looks like when it has information. Could you provide some sample events which have information on them?
Hi Somesoni2, thanks for your effort on this, we are monitoring wtmpx file from the unix machines using the Splunk Add-on for Unix. Below are the sample events contains the information.
Index=unix sourcetype=who host=*
6/20/18
3:06:05.000 PM
USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53
host = ttbmt02 source = who sourcetype = who
6/20/18
3:03:35.000 PM
USERNAME LINE HOSTNAME TIME
DBB019 pts/1 w442xty1.XXXX.com Jun 19 18:40
AMM007 pts/5 yb33gnn1.XXXX.com Jun 19 08:53