Alerting

Output search results from alert to syslog. Retrieving search output vs. alert statistics from variables. (Linux)

kairobin
Path Finder

Hi all

I am using RedHat Linux on Our Splunk installation.
On our search head, we are using alerts a lot and I am wondering if anyone out here has
a example of how to get the search result into the syslog of the RedHat server?

The reason for this is because we are using a ticket system which can monitor the syslog.

I now use this search:

searchstring |  outputcsv result.csv

And I am trying to make a script that will send the data inside the file into syslog, like this:

RedHat Script

logger -p -t Name_of_alert -f /opt/splunk/var/run/splunk/resultdata.csv

This will pass the result into syslog, but the alert does not use this when it is scripted.
If I do it manually, then it will work,

Using a script for this does not give me the result in cleartext, only metadata about it.
0 SPLUNK_ARG_0 Script name

1 SPLUNK_ARG_1 Number of events returned

2 SPLUNK_ARG_2 Search terms

3 SPLUNK_ARG_3 Fully qualified query string

4 SPLUNK_ARG_4 Name of report

5 SPLUNK_ARG_5 Trigger reason
For example, "The number of events was greater than 1."

6 SPLUNK_ARG_6 Browser URL to view the report.

7 SPLUNK_ARG_7 Not used for historical reasons.

8 SPLUNK_ARG_8 File in which the results for the search are stored.

Thanks in advance to splunk and all answers.

1 Solution

rsennett_splunk
Splunk Employee
Splunk Employee

The result of an alert is not the result of the search that triggered it. So resultdata.csv just contains the environmental variables (you've listed). In other words...

This:

logger -p user.info -t Name_of_alert -f $8
would be more what you want... except the path in SPLUNK_ARG_8 leads you to a gzipped file... logger can only do text

So you can use the contents of the SPLUNK_ARG_8 and whatever other environmental variables you want to include in your syslog lines, to create your log line... meaning... you have to grab the name/location of the search results, open the file (csv which is gzipped) and iterate through it... sending the lines to syslog

In other words... your alert triggers a script that
- Grabs the values you want environmental variable(s) ( in python it would be: rawEventsFile = os.environ['SPLUNK_ARG_8'] )
- takes the variable rawEventsFile which is now set to something like: /opt/splunk/var/run/splunk/dispatch/scheduler__myusername__myappname__RMD5020f581c1fe74122_at_1428534600_143/results.csv.gz' and unpack the gzip
- takes the resulting .csv file and iterates through, line by line... sending the resulting lines to syslog

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

View solution in original post

CreieR
New Member

what would be the alternative for Windows ?

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

The result of an alert is not the result of the search that triggered it. So resultdata.csv just contains the environmental variables (you've listed). In other words...

This:

logger -p user.info -t Name_of_alert -f $8
would be more what you want... except the path in SPLUNK_ARG_8 leads you to a gzipped file... logger can only do text

So you can use the contents of the SPLUNK_ARG_8 and whatever other environmental variables you want to include in your syslog lines, to create your log line... meaning... you have to grab the name/location of the search results, open the file (csv which is gzipped) and iterate through it... sending the lines to syslog

In other words... your alert triggers a script that
- Grabs the values you want environmental variable(s) ( in python it would be: rawEventsFile = os.environ['SPLUNK_ARG_8'] )
- takes the variable rawEventsFile which is now set to something like: /opt/splunk/var/run/splunk/dispatch/scheduler__myusername__myappname__RMD5020f581c1fe74122_at_1428534600_143/results.csv.gz' and unpack the gzip
- takes the resulting .csv file and iterates through, line by line... sending the resulting lines to syslog

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

kairobin
Path Finder

It Works now as It was planned:
First I made an saved search.

1:
Searchstring:
index=etc something something | outputcsv result.csv

2:
This makes a New file called result.csv

3:
A script is triggered looking like this:

logger -f /opt/splunk/var/run/splunk/result.csv

What this script does it puts all the info inside the file and post it into syslog.
That way Our ticket program can look for certan types of text and react to it.

Thank you very much for the help.

rsennett_splunk
Splunk Employee
Splunk Employee

Very good option! Interesting approach. Changing title to reflect the contents a bit more clearly for searchers.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

kairobin
Path Finder

This is the walktrough I needed, Thank you.

0 Karma

kairobin
Path Finder

I wil add the Complete code here when I have made it to work.

0 Karma

lguinn2
Legend

So did you write a script for the alert to execute? Did the script contain the logger line that you show above?

0 Karma

kairobin
Path Finder

In the search string I use to pipe the result in OUTPUTCSV.
The I can get the result in Clear text.
And inside the script that is running when the alert i set off. It wil use logger and send the text in the csv file into syslog.
This only works manually

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...