Developing for Splunk Enterprise

How to install a python script to call the fireeye HX API and GET all HX json data into Splunk?

Contributor

I need help installing a py script to call the fireeye HX API and GET all HX json data (more data than collected from the FireEye App and Add-on for Splunk Enterprise) into Splunk.

I got an awesome python script written by Ruairi O'Mahony for HX. When I use it in pycharm terminal (free edition), it returns the data I want. So now I want to test it in my standalone Dev instance of Splunk. However ultimately the script will reside on a HF (Heavy Forwarder) in a distributed production environment.

I am new to scripting in Splunk so I am not sure what the best way to go about this.

The script has options as to what data you wish to collect.

I run the command in the pycharm terminal with an option like this...

C:\Users\myname\PycharmProject\HX_API> C:\Python27\python.exe C:/Users/myname/PycharmProject/HX_API/RO_hx_api_script.py 1

as you can see there is a number one at the end which correlates to one of the following options

To use this script append one of the following options to the end of the command line: 

0 -  All options -  Run all options
1 -  HX version info -  v2/version
2 -  List all alerts -  v2/alerts?limit=200000
3 -  List source alerts from another appliance for all hosts -  v2/source_alerts?limit=200000
4 -  List of hosts -  v2/hosts?limit=200000
5 -  List of acqusitions -  v2/acqs/files?limit=200000
6 -  List of bulk acqusitions -  v2/acqs/bulk?limit=200000
7 -  List of Triages -  v2/acqs/triages?limit=200000
8 -  List all searches -  v2/searches?limit=200000
9 -  List all scripts -  v2/scripts?limit=200000
10 -  Download all scripts -  v2/scripts.zip
11 -  Indicator caregories -  v2/indicator_categories?limit=200000
12 -  Custom indicators -  v2/indicator_categories/custom?limit=200000
13 -  FireEye indicators -  v2/indicator_categories/fireeye?limit=200000
14 -  FireEye-CMS indicators -  v2/indicator_categories/fireeye_cms?limit=200000
15 -  Imported indicators -  v2/indicator_categories/imported?limit=200000
16 -  DTI intel from Mandiant -  v2/indicator_categories/mandiant?limit=200000
17 -  Containment states -  v2/containment_states?limit=200000

Here is a copy of the full script (permitted to share by the author)...

#!/usr/bin/python
# Rev. 20170313.1
# Copyright 2016-2017, Ruairi O'Mahony 
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#### import libraries
import urllib2
import ssl
import json
import csv
import base64
import getpass
import sys
import time
import os

#### define variables for loggin
appliance = "your ip adress goes here"
username = "your HX api analyst username goes here"
password = "your HX api analyst pswd goes here"



#### set-up options list for use in menu
options = ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17"]
name = ["All options", "HX version info", "List all alerts", "List source alerts from another appliance for all hosts",
        "List of hosts", "List of acqusitions", "List of bulk acqusitions", "List of Triages", "List all searches",
        "List all scripts", "Download all scripts", "Indicator caregories", "Custom indicators", "FireEye indicators",
        "FireEye-CMS indicators", "Imported indicators", "DTI intel from Mandiant", "Containment states"]
uri = ["Run all options", "v2/version", "v2/alerts?limit=200000", "v2/source_alerts?limit=200000",
       "v2/hosts?limit=200000", "v2/acqs/files?limit=200000", "v2/acqs/bulk?limit=200000",
       "v2/acqs/triages?limit=200000", "v2/searches?limit=200000", "v2/scripts?limit=200000", "v2/scripts.zip",
       "v2/indicator_categories?limit=200000", "v2/indicator_categories/custom?limit=200000",
       "v2/indicator_categories/fireeye?limit=200000", "v2/indicator_categories/fireeye_cms?limit=200000",
       "v2/indicator_categories/imported?limit=200000", "v2/indicator_categories/mandiant?limit=200000",
       "v2/containment_states?limit=200000"]


#### Functions used

# http handler
def build_request(url, tests):
    handler = urllib2.HTTPHandler()
    opener = urllib2.build_opener(handler)
    urllib2.install_opener(opener)
    data = tests
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    request = urllib2.Request(url, data=data)
    return request, ctx


# get token from HX
def authorization(appliance, encoded_authorization_string):
    request = build_request('https://' + appliance + ':3000/hx/api/v1/token', None)
    get_request = request[0]
    ctx = request[1]
    get_request.add_header('Accept', 'application/json')
    get_request.add_header('Authorization', 'Basic ' + encoded_authorization_string)
    get_request.get_method = lambda: 'GET'

    try:
        response = urllib2.urlopen(get_request, context=ctx)
    except urllib2.HTTPError as e:
        print e.read()
    except urllib2.URLError as e:
        print 'Failed to connect to HX API server.'
        print 'Reason: ', e.reason
    else:
        return response.headers['X-FeApi-Token']


# get json
def get_json(token, appliance, uri, string):
    request = build_request('https://' + appliance + ':3000/hx/api/' + uri + '', None)
    put_request = request[0]
    ctx = request[1]
    put_request.add_header('X-FeApi-Token', token)
    put_request.add_header('Accept', 'application/json,application/octet-stream')
    put_request.add_header('Content-Type', 'application/json')
    put_request.get_method = lambda: 'GET'

    try:
        response = urllib2.urlopen(put_request, context=ctx)
    except urllib2.HTTPError as e:
        print e.read()
    except urllib2.URLError as e:
        print 'Failed to connect to HX API server.'
        print 'Reason: ', e.reason
    else:
        print '############### Result for: ' + string + ' (https://' + appliance + ':3000/hx/api/' + uri + ')' + ' ###############\n'
        result = response.read()
        parsed = json.loads(result)
        result = 0
        result = json.dumps(parsed, indent=2, sort_keys=True)
        print result
        print '\n\t\t\t############### Result end ###############\n'
        return result


# function to download all the scripts
def script_get(token, appliance, uri, string):
    request = urllib2.Request('https://' + appliance + ':3000/hx/api/v2/scripts.zip', None)
    context = ssl._create_unverified_context()
    context.check_hostname = False
    context.verify_mode = ssl.CERT_NONE
    request.add_header('X-FeApi-Token', token)
    request.add_header('Accept', 'application/octet-stream')
    request.get_method = lambda: 'GET'

    try:
        response = urllib2.urlopen(request, context=context)
    except urllib2.HTTPError as e:
        print e.read()
    except urllib2.URLError as e:
        print 'Failed to connect to HX API server.'
        print 'Reason: ', e.reason
    else:
        scripts = response.read()
        op = open(filepath, 'w')
        op.write(scripts)
        op.close()
        op1 = open(filepath1, 'w')
        info = "Script was run on: " + now + "\nScript was ran by: " + user
        op1.write(info)
        op1.close()
        print '\n############### Result for: ' + string + ' (https://' + appliance + ':3000/hx/api/' + uri + ')' + ' ###############\n'
        print 'The scripts have been downloaded to \n' + filepath + '\nand stored in xml format \n'  # note that file opens easier with an app like unrar
        print '\t\t\t############### Result end ###############\n'


# print the options for the script
def print_useage():
    print "To use this script append one of the following options to the end of the command line: \n"
    for x, y, z in map(None, options, name, uri):
        print str(x) + " - ", str(y) + " - ", str(z)
    print "\n"
    print "Option 10 will be stored in the following location: " + filepath + "\n"


#### main execution

# read login details, will check that default details have been removed, if not it will use them
if appliance == "":
    appliance = raw_input("HX IP address: ")
if username == "":
    print "**User needs to be api_analyst role**"
    username = raw_input("Username: ")
if password == "":
    password = getpass.getpass()  # the get pass function keeps user input off the screen

# create the auth string using base 64 encoding
decoded_authorization_string = username + ":" + password
encoded_authorization_string = base64.b64encode(decoded_authorization_string)

put_request = 0
filepath = os.path.join(os.path.expanduser('~'), 'HX_Scripts.zip')  # create new folder to store results
filepath1 = os.path.join(os.path.expanduser('~'), 'HX_Timestamp.txt')
token = authorization(appliance, encoded_authorization_string)
now = time.strftime("%c")  # get current time
user = getpass.getuser()  # currently logged in user account
print "\n"
if len(sys.argv) > 1:  # check if arguments passed into script
    n = sys.argv[1]
    m = len(options)
    if n in options:  # check the argument passed is in the options range
        if n == "0":  # if argument == 0 run all the options
            for o in range(0, m):
                if o != 10:
                    get_json(token, appliance, uri[o], name[o])
                elif o == 10:
                    script_get(token, appliance, uri[o], name[o])
                o += 1
        elif n == "10":
            for o in range(0, m):
                if int(n) == o:
                    script_get(token, appliance, uri[o], name[o])
                    o += 1
        elif n != "0":  # if argument != 0 run the required option
            for o in range(0, m):
                if int(n) == o:
                    get_json(token, appliance, uri[o], name[o])
                    o += 1
    else:
        print("No option selected")
else:
    print_useage()

I am not sure of best way to run this or where to place the script. Any advice or guidance or reference is much appreciated.
I have been using this reference http://docs.splunk.com/Documentation/Splunk/6.6.2/Data/Getdatafromscriptedinputs
Any other suggestions as to how to run this script with the different option choices?
I am getting the script to run but still need guidance on where to append the option number (that I choose) in the conf files. Unless my only option is to hard code the script....

So there is a way to hard code the script. For example if you want option 2 - list all alerts....

add option = "2" to #### define variables for login

and

after #### main execution > # check if arguments passed into script,

change
if option > 1:

n = option

Thank you

0 Karma
1 Solution

Communicator

Ok, how about we try something with the commands.conf https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Commandsconf
create a commands.conf inside of your /local for your app and create a Stanza named hxget

[defaults]
 supports_getinfo = true
 supports_rawargs = true
 outputheader = true

 [hxget]
 #filename = This is where the python script is located that was written to accept arguments
 filename = $SPLUNK_HOME/etc/apps/HX_API_Script/bin/HX_API.py

Now inside of Splunk you can create a dashboard with a dropdown menu for your argument options and add that variable after "*|hxget 2(arg from dropdown list)" removing the quotes and the *| is searching everything before pipe so drop time down to 60 seconds or so.

Let me know if this dosn't make sense and I can try to go into my thought process further.

View solution in original post

0 Karma

Communicator

Ok, how about we try something with the commands.conf https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Commandsconf
create a commands.conf inside of your /local for your app and create a Stanza named hxget

[defaults]
 supports_getinfo = true
 supports_rawargs = true
 outputheader = true

 [hxget]
 #filename = This is where the python script is located that was written to accept arguments
 filename = $SPLUNK_HOME/etc/apps/HX_API_Script/bin/HX_API.py

Now inside of Splunk you can create a dashboard with a dropdown menu for your argument options and add that variable after "*|hxget 2(arg from dropdown list)" removing the quotes and the *| is searching everything before pipe so drop time down to 60 seconds or so.

Let me know if this dosn't make sense and I can try to go into my thought process further.

View solution in original post

0 Karma

Communicator

@packet_hunter are you looking for a saved search or an alert to initiate some portion of the script by supplying the argument?

0 Karma

Contributor

Thanks for the reply.

So what the script does is supplies considerably more event data (depending on the option chosen) than what you would get if you just used the FireEye _TA. HX is new-ish, and there is a ton of usefully data within the management console that aggregates all the HX agent info. The script calls the HX management console API and gets all the json data, for splunk to ingest.
The script is currently running in a custom app (I built) on a schedule.
My question is how to pass an option number to the above py script in Splunk.
I could hard code the option I want in the py script but would rather have an option dropdown or field.
I hope that makes sense.

0 Karma

Contributor

ultimately, I will create an alert based on the data the py script gets from the API.

0 Karma

Communicator

Ok, how about we try something with the commands.conf https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Commandsconf

create a commands.conf inside of your /local for your app and create a Stanza named hxget

[defaults]
supports_getinfo = true
supports_rawargs = true
outputheader = true

[hxget]
#filename = C:/Users/myname/PycharmProject/HX_API/RO_hx_api_script.py
filename = C:/Users/myname/PycharmProject/HX_API/RO_hx_api_script.py

Now inside of Splunk you can create a dashboard with a dropdown menu for your argument options and add that variable after "*|hxget 2(arg from dropdown list)"

Let me know if this dosn't make sense and I can try to go into my thought process further.

0 Karma

Contributor

So I think your idea is feasible, but I am having no luck running the script with the command hxget.

Currently the script is running on a schedule of every 60 seconds, but it is only giving me the default list of options, when an option number is not appended to the script as stated above. When I try following your commands.conf example (which I may be not writing it correctly as the py script is in the folder ...apps/HX_API_Script/bin/HX_API.py) I don't understand how to invoke the script with the command, I have been looking for an example, but no luck. Can it be run from search and reporting?

0 Karma

Communicator

I understand,

Lets do some verification your commands.conf is located at ...apps/HX_API_Script/local/commands.conf and it looks like this

 [defaults]
 supports_getinfo = true
 supports_rawargs = true
 outputheader = true

 [hxget]
 #filename = ...apps/HX_API_Script/bin/HX_API.py
 filename = ...apps/HX_API_Script/bin/HX_API.py

I would then suggest restarting your dev search at this point.

You should then be able to run the hxget command using
*| hxget 2
the *| will return everything since it just a pipe and if the script is running successfully a search for * should also take place returning back everything in your Index so you may want to limit to say last 60 seconds or something.

This should run successfully with no errors. Of course this should be running ...apps/HX_API_Script/bin/HX_API.py 2

I did a test of my own where I created a file and did some input to that file via python script using the above method and it did work.

0 Karma

Communicator

The hxget command is initiating the python script already and adding the 1 is the argument as it calls.

So in searching and reporting running the search command

" *| hxget 1 " removing the " " is telling Splunk to run the below command

$SPLUNK_HOME/etc/apps/HX_API_Script/bin/HX_API.py 1 (OR the exact thing you put after filename = in ...apps/HX_API_Script/local/commands.conf )

0 Karma

Contributor

I should mention that the HX console updates so I need to GET data from the API about every minute...

0 Karma

Communicator

Once the search command [hxget] is working then a saved search can be setup to run every 60 seconds on a cron schedule. This will run the script with option 1 or whatever argument you need every 60 seconds by Splunk.

0 Karma

Contributor

ah, sorry now I follow what you are saying... I will keep poking around at it.
Thank you for the support!

0 Karma

Contributor

btw you need to convert your comments to answer so I can send you some karma points...

0 Karma

Communicator

Went ahead and submitted my first post with a few mods as the Answer post.

0 Karma

Contributor

thank you for the reply, checking it out now

0 Karma

Contributor

good news that "*|hxget 1" does not return errors but no events or data

if I drop the quotes I get:

Error in 'hxget' command: Cannot find program 'hxget' or script 'hxget'.

Getting close... any other ideas...

Actually I know why I am not getting any data... the app I created is stopping at the if then statement at line 176.

So I need to retool the app, and just have it run every 60 seconds with an option.

Unless you have a way to kick off the script from the search&reporting app with the option number directly... because that is what I am not familiar with.

0 Karma

Communicator

OK lets take a look at the error message

Error in 'hxget' command: Cannot find program 'hxget' or script 'hxget'.

can you verify under setttings > advanced search > search commands > HX_API_Script - hxget is enabled and permissions set for Global and all view?

If so, then it is mad at the location
filename = python location

the filename location should be exactly what the command would be to run the script outside of Splunk and minus the C:\Users\myname\PycharmProject\HX_API> C:\Python27\python.exe

0 Karma

Contributor

Awesome, thank you I will give it a try and let you know... thank you!

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!