topic Re: Query for exclude words in a raw data in Splunk Search

Query for exclude words in a raw data

phanichintha — Fri, 02 Apr 2021 08:21:39 GMT

Hello!

As shown in the below picture, those are the events with a timestamp. I want when a "Kafka" service or "Jps" services are down, I will get an alert. How to write a search query for this when any of the below services are down, I will get an alert.

Re: Query for exclude words in a raw data

gcusello — Fri, 02 Apr 2021 08:48:47 GMT

Hi @phanichintha,

if you have few words to search, you can insert them in your main search:

<your_search> (Kafka OR Jps OR <other_words>)

if these words are in a field, you can use the field to have more performat searches (e.g. they are in a field called "service"):

<your_search> (service=Kafka OR service=Jps OR servide=<other_words>)

If instead you have many words to check, you can put them in a lookup (called e.g. "patterns.csv" with a single column called "pattern"), if you haven't them in a field:

<your_search> [ | inputlookup patterns | rename pattern AS query | fields query]

if you have them in a field:

<your_search> [ | inputlookup patterns | fields pattern ]

At the end I hint to follow the basic training in Splunk : Fundamentals i course (https://www.splunk.com/en_us/training/free-courses/splunk-fundamentals-1.html) that's a free course and the Search Tutorial (https://docs.splunk.com/Documentation/Splunk/8.1.0/SearchTutorial/WelcometotheSearchTutorial) that help you to understand how Splunk works.

Ciao.

Giuseppe

Re: Query for exclude words in a raw data

phanichintha — Mon, 05 Apr 2021 06:26:45 GMT

@gcusello there is no fields to segregate.

Actually, the question is In a Linux machined using JPS command some services is are running, ex: Kafka, JPS etc with PID, if any services are stopped we need to get an alert.

Here some tricky idea I have, so if the keyword "Kafka" is not seen in events for more than 1 minute I want to get that alert, so based on this the application team to know oh! the Kafka services are not running in that particulate host.

Here is the Query:
index="main" host="linux machine" source="logs" "Kafka"

Please suggest the query to get the alert when "Kafka" word is seen more than 1 minitue.

Re: Query for exclude words in a raw data

gcusello — Mon, 05 Apr 2021 07:00:23 GMT

Hi @phanichintha,

as I said, if you have to search the presence of few word (e.g. only "Kafka"), you can use the search you shared:

index="main" host="linux machine" source="logs" "Kafka" earliest-1m@m latest now

and save it as an alert scheduled to run every minute (cron * * * * *)

then configure it to send an email or make another action.

Only one meditation: meybe a time period of one minute is too frequent and not efficient, because you and your team probably haven't a reaction time of one minute, so you could also use a little larger time frame (e.g. 5 minutes).

Ciao.

Giuseppe

Re: Query for exclude words in a raw data

phanichintha — Mon, 05 Apr 2021 07:26:42 GMT

@gcusello Thanks for helping!

If you see the below query and latest log in that able to see the "Kafka" is running, So in the same case I need if the service "Kafka" is not present in that list I want to know with alert.

Re: Query for exclude words in a raw data

gcusello — Mon, 05 Apr 2021 07:31:47 GMT

Hi @phanichintha,

you have to use as trigger condition: Number of results = 0.

Ciao.

Giuseppe

Re: Query for exclude words in a raw data

phanichintha — Mon, 05 Apr 2021 09:20:14 GMT

@gcusello

what are all you suggesting it's not working for me, you can see the sample alert I got.
Please help, if the keyword in the below list is not in any events will get mail. Can you provide a solution for this?

Re: Query for exclude words in a raw data

gcusello — Mon, 05 Apr 2021 09:50:21 GMT

Hi @phanichintha,

sorry but I don't understand your need, so I try to summarize it:

you have one or more words (as kafka or Jps) that are usually present in your logs,
you need to have an alert when one of these words isn't present in your logs in one minute;

is it correct?

If this is your need the below search is correct:

index="main" host="linux machine" source="logs" (Kafka OR Jps) earliest=-1m@m latest=now

and you have to configure your alert to trigger when there's no result.

What's your problem:

has the above search always results?
have you problems to configure the trigger condition in the alert?

Ciao.

Giuseppe

Re: Query for exclude words in a raw data

phanichintha — Mon, 05 Apr 2021 10:57:48 GMT

Sorry @gcusello this is not working, based on this the alert triggers still "Kafka" included logs.

Re: Query for exclude words in a raw data

gcusello — Mon, 05 Apr 2021 16:25:54 GMT

Hi @phanichintha,

which trigger condition did you setted in your alert?

the correct one is: Number of results is equal to 0

Ciao.

Giuseppe

Re: Query for exclude words in a raw data

phanichintha — Tue, 06 Apr 2021 04:20:15 GMT

Yes, I did the same what you suggested, but no luck. If you can share the zoom meeting, please let me know will discuss it live.

Re: Query for exclude words in a raw data

phanichintha — Tue, 06 Apr 2021 05:07:46 GMT

T@gcusello: The application team creates manually.

#: var/log/services.txt
services.txt file contains, below logs, it looks like this. These logs are ingested with Splunk. So based on this I can't able to correlate the file because of logs like this.

Re: Query for exclude words in a raw data

bowesmana — Tue, 06 Apr 2021 05:17:59 GMT

I suspect the problem is that, based on your data, that your event contains 1...n of the processes that are present, so event 1 may contain

123 Kafka
345 Bootstrap
567 Jps

and you are trying to see if Jps OR Kafka OR Bootstrap OR other OR other are not present for a minute

I suggest you do the following

Create a CSV lookup file with the list of the process names you are interested in, in a single column called process_name
Create a search that does something like

This is extracting your PID/Process name from the _raw event (you will have to confirm that this creates a Splunk multivalue field with all the process names in it, per event).

Then it expands all the process names to their separate events.

It then counts the occurrences of each process.

To then work out which ones are missing you just append your lookup file to the end of the results with a count of 0 and then look for the largest count value per process and if it's 0, you have your list.

Re: Query for exclude words in a raw data

phanichintha — Tue, 06 Apr 2021 06:02:18 GMT

I created CSV like this:

Next created query like this:

The process_names shows like this:

Re: Query for exclude words in a raw data

bowesmana — Tue, 06 Apr 2021 10:43:31 GMT

OK, so the key is the rex statement and the regex that extracts the PID and process names from the _raw message.

Assuming there will be no space in the process name, then this should work

| rex field=_raw max_match=0 "(?<pid>\d+)\s(?<process_name>[\w]*)\s?"

which is saying extract

pid = sequence of digits

followed by a single whitespace

followed by

process_name = sequence of word characters followed by

optional whitespace

If you do a

| table _raw pid process_name

after the rex statement without the rest of the query, you can see what the rex is extracting. If that shows the pid and process names as multi values in the field then it's good and the rest of the query will work.