Python api for splunk to gather data

[swagatam1308]

Hi All,

We need to write a python script  to pull data for below query ,using script below but no output is showing.

Please advice how we can do it with python script as AND operation seems not working 

Index="ti-p_plasma" sourcetype="plasma: ops-gateway" earliest=-1h source ="/home/zsvg9ky/deployments/ops-gateway/ops-gateway/logs/access*" | search ogw_uri!=.js AND ogw_uri!=.css AND ogw_uri!=.gif AND ogw_uri!=.jpeg AND ogw_uri!=.png AND ogw_uri!=.jpg AND ogw_uri!=.fonts AND ogw_uri!=.assets/ | Rex field=ogw_uri " ^/(?<end_point_services>[A-Za-z0-9_-]+)[/|?].*$" | chart count by end_point_services, ogw_status_code | field -"201","405","206"

I am using below python script but output is not producing nothing

from __future__ import print_function
from future import standard_library
standard_library.install_aliases()
import urllib.request, urllib.parse, urllib.error
import httplib2
from xml.dom import minidom
baseurl = 'https://3.131.162.26:8089'
userName = 'admin'
password = 'India@nic'
searchQuery = 'index=main host="splunk1" source="/var/log/secure"|stats'
# Authenticate with server.
# Disable SSL cert validation. Splunk certs are self-signed.
serverContent = httplib2.Http(disable_ssl_certificate_validation=True).request(baseurl + '/services/auth/login',
'POST', headers={}, body=urllib.parse.urlencode({'username':userName, 'password':password}))[1]
sessionKey = minidom.parseString(serverContent).getElementsByTagName('sessionKey')[0].childNodes[0].nodeValue
# Remove leading and trailing whitespace from the search
searchQuery = searchQuery.strip()
# If the query doesn't already start with the 'search' operator or another
# generating command (e.g. "| inputcsv"), then prepend "search " to it.
if not (searchQuery.startswith('search') or searchQuery.startswith("|")):
searchQuery = 'search ' + searchQuery
print(searchQuery)
# Run the search.
# Again, disable SSL cert validation.
print(httplib2.Http(disable_ssl_certificate_validation=True).request(baseurl + '/services/search/jobs','POST',
headers={'Authorization': 'Splunk %s' % sessionKey},body=urllib.parse.urlencode({'search': searchQuery}))[1])

Which genereting job sid then using below script to show the output 

curl -k -H "Authorization:Splunk $token" https://3.131.35.127:8089/services/search/jobs/$jobid/results_preview --get -d output_mode=csv

[tscroggins]

@swagatam1308 

These may just be typos in your message, but just in case:

1. Field names are case-sensitive. "index" is all lowercase.
2. Your regular expression includes a space before the caret.
3. The "fields" command is plural.
4. Your fields command refers to fields that don't exist in your chart output. Was this meant to be a filter?

In your code sample, you also have this:

searchQuery = 'index=main host="splunk1" source="/var/log/secure"|stats'

That works and produces a set of preselected statistics for all fields, but it's probably not what you intended.

"AND" is implied in Splunk searches, and you can include search terms directly in the base search. There's no need to use a separate "search" command unless you've used one intentionally when constructing your search in code.

I'm assuming your source type really does include a space after the colon: plasma: ops-gateway. You can correct it if it does not. Try this:

index="ti-p_plasma" sourcetype="plasma: ops-gateway" earliest=-1h source="/home/zsvg9ky/deployments/ops-gateway/ops-gateway/logs/access*" ogw_uri!=.js ogw_uri!=.css ogw_uri!=.gif ogw_uri!=.jpeg ogw_uri!=.png ogw_uri!=.jpg ogw_uri!=.fonts ogw_uri!=.assets/ ogw_status_code!=201 ogw_status_code!=405 ogw_status_code!=206
| rex field=ogw_uri "^/(?<end_point_services>[A-Za-z0-9_-]+)[/|?].*$"
| chart count by end_point_services, ogw_status_code

You can also make the search a little easier to read:

index="ti-p_plasma" sourcetype="plasma: ops-gateway" earliest=-1h source="/home/zsvg9ky/deployments/ops-gateway/ops-gateway/logs/access*" NOT ogw_uri IN (.js .css .gif .jpeg .png .jpg .fonts .assets/) NOT ogw_status_code IN (201 405 206)
| rex field=ogw_uri "^/(?<end_point_services>[A-Za-z0-9_-]+)[/|?].*$"
| chart count by end_point_services, ogw_status_code

If ogw_uri is a complete URI and not just an extension, you probably meant to include wildcards:

index="ti-p_plasma" sourcetype="plasma: ops-gateway" earliest=-1h source="/home/zsvg9ky/deployments/ops-gateway/ops-gateway/logs/access*" NOT ogw_uri IN (*.js *.css *.gif *.jpeg *.png *.jpg *.fonts *.assets/) NOT ogw_status_code IN (201 405 206)
| rex field=ogw_uri "^/(?<end_point_services>[A-Za-z0-9_-]+)[/|?].*$"
| chart count by end_point_services, ogw_status_code

If your search works in the Splunk web user interface, it should work over REST as well. Try breaking your problem down into three distinct steps:

1. Test search using Splunk web UI.
2. Test search using cURL.
3. Test search using Python.

[swagatam1308]

@tscroggins  thanks for your reply ,

The query is running from Gui but from python script below attached it giving  0 output saying chart option is saying no result.

PFA the output of the SID generated from splunk python api script from GUI

chart count option is not working from REST API script but it is working from GUI search panel.

Please advice.

[tscroggins]

@swagatam1308 

To expedite testing, I would use a much smaller time window, e.g. Last 15 minutes.

What values of the split-by field exist in your results without the chart command? The chart result implies the field is NULL in all events despite the base search returning results.

[swagatam1308]

@tscroggins 

Thanks for your repl.

The values are same like pasted in conversation and we are getting result from GUI but inspecting job id with sid generated from API is showing syntax error as pasted in last chat.

upto below it is executing fine without chart from API but with chart option from API splunk call of python it is returning no result 

Please advice

index="ti-p_plasma" sourcetype="plasma: ops-gateway" earliest=-1h source="/home/zsvg9ky/deployments/ops-gateway/ops-gateway/logs/access*" NOT ogw_uri IN (.js .css .gif .jpeg .png .jpg .fonts .assets/) NOT ogw_status_code IN (201 405 206)
| rex field=ogw_uri "^/(?<end_point_services>[A-Za-z0-9_-]+)[/|?].*$" 

[swagatam1308]

I observed from splunk api script of python only general fields are printing output but interesting fields are not as attached in screenshot.

Any idea how to get interesting fields output results also via splunk python script to gather data ?

[swagatam1308]

any update ?

[tscroggins]

@swagatam1308 

The interesting fields are displayed in the UI when you execute a search in Smart or Verbose mode. If you execute a search in Fast mode, the results will be similar to what you're seeing by default via the API.

You can add a fields or table command to the end of your search that includes the fields you want to return.

[swagatam1308]

Can you please give the command.

From GUI it is fine but same splunk query not working from cli command thet you pasted unfortunately.

Please help!

[tscroggins]

@swagatam1308 

Here's a cURL example that returns the _time (always returned by default), log_level, component, and event_message fields:

curl -k -u admin 'https://localhost:8089/services/search/jobs' -d search="search index=_internal sourcetype=splunkd source=*/splunkd.log* component=* | head 1 | fields _time log_level component event_message | fields - _bkt _cd _indextime _kv _raw _si _sourcetype _subsecond" -d earliest_time=-24h -d latest_time=now -d exec_mode=oneshot

With exec_mode=oneshot, the API blocks until the search completes and returns the result instead of the sid, but you can use the same search in an asynchronous call.

Here's a similar example using the splunk search command:

/opt/splunk/bin/splunk search "search index=_internal sourcetype=splunkd source=*/splunkd.log* component=* earliest=-24h latest=now | head 1 | fields _time log_level component event_message | fields - _bkt _cd _indextime _kv _raw _si _sourcetype _subsecond"

[swagatam1308]

Thanks for your reply.

Would the same work for any other app configured on splunk server?

I meant if index=app_name[configured on splunk] would that work?

In my case if I gave index=app_name then special fields not coming on api result from cli.

[tscroggins]

@swagatam1308 

Your search must include fields present in your events. For example, if you want to retrieve fields called foo, bar, and baz you would include the following command:

| fields foo bar baz