Hi all. I have a report set up in Splunk producing a visualisation we're embedding on our website. A member of the public has asked if they can instead get the raw data in JSON format. I don't want to create them a user in the system, and I'd really rather link them through our Azure API portal, where we have our other API end points for retrieving customer data. So, what I'm really wanting to do is work out how I can get the scheduled report data out of Splunk and into the Azure API. I'm aware this is not a pure Splunk data, but TBH, I thought people here would be most likely to have the most experience, especially as I seem to need to 2-step the REST queries to find the search IDs and then get the results and....  It all ended up being quite a lot more complicated than I expected it to be. I couldn't find any relevant how-to guides online either, so thought this  ... View more

Hi all  I'm creating a custom visualisation for my org, so I'm developing it into an app, following this tutorial: https://docs.splunk.com/Documentation/Splunk/8.0.5/AdvancedDev/CustomVizTutorial Everything's great so far and working perfectly on my local machine when I'm referencing a background image in my CSS using: background-image: url( '/static/app/********/images/hot-water-icon.gif' ); (obviously ******** is the name of my app, on my local machine). This is great, until I sent an upgrade to test on another machine, where it got installed into an app directory called "******** - Copy" (I assume because that was the name of the file of the folder I copied to zip & send thru, because I didn't want to send some other things in the /static/app/********/bin/ folder I was still working on at the time....) So... How do I access the URI for the named app install folder? Can it be done? or do I just need to presume it'll always be as I set it? Thanks in advance, Phil ... View more

TL;DR - the python scripts runing from the $SPLUNK_HOME$/bin/splunk cmd python throw errors that aren't thrown using /usr/bin/python. So, I've got a load of data which has X/Y coordinates stored in NZTM2000 format, and I want to enrich the data by converting to a Lat/Lng so that I can plot onto a map visualisation, or output to other systems that use this format. I've created an app (replaced with in the examples below), and dropped a Python script file in there as a proof of concept. The Python script I'm using to convert requires a couple of additional libraries (installed using "pip install pyproj") which I installed as the Splunk user (who's home directory is also set to $SPLUNK_HOME$). The problem is that the script works fine from the command line: splunk@standalone-vm:~/etc/apps/<appname>/bin$ /usr/bin/python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py (-40.981429307189785, 174.95992442484228) However, if I try running it using the Splunk binary route, I get an error.... splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py Traceback (most recent call last): File "/opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py", line 14, in <module> from pyproj import Proj, transform File "/opt/splunk/.local/lib/python2.7/site-packages/pyproj/__init__.py", line 69, in <module> from pyproj._list import ( # noqa: F401 ImportError: /opt/splunk/.local/lib/python2.7/site-packages/pyproj/_list.so: undefined symbol: PyUnicodeUCS4_FromStringAndSize So I thought it might be something to do with the 2.7 version, but it seems not: splunk@standalone-vm:~/etc/apps/<appname>/bin$ /usr/bin/python --version Python 2.7.17 splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd python --version Python 2.7.17 Which seems to rule that out. So, I tried a different route: splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd /usr/bin/python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py (-40.981429307189785, 174.95992442484228) So if I get Splunk to call the /usr/bin/python route, everything works as expected. So this points to a problem with the shipped version of python shipped with Splunk, but that seems.... Unlikely. So, help, please? Appendix: Here's a copy of the python file I'm trying to run (Note: you'll need the pyproj libary too...) #!/usr/bin/env python """nztm_to_lat_lng.py: Takes an NZTM2000 X/Y coordinate system and outputs a lat/lng""" __author__ = "Phil Tanner" __version__ = "0.0.1" __status__ = "Prototype" # If we're running from Splunk command, patch our local path so we can load modules; hack! import os splunk_home = os.getenv('SPLUNK_HOME') if splunk_home is not None: import sys # Local app path sys.path.append(os.path.join(os.environ['SPLUNK_HOME'],'etc','apps','<appname>','bin')) # Splunk user path - note this is fixed to the 2.7 to pull in the pip installed pyproj files, without which it fails horribly sys.path.append(os.path.join(os.environ['SPLUNK_HOME'],'.local','lib','python2.7','site-packages')) from pyproj import Proj, transform inProj = Proj("+proj=tmerc +lat_0=0 +lon_0=173 +k=0.9996 +x_0=1600000 +y_0=10000000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs") outProj = Proj('epsg:4326') # Note: hardcoded sample, literally chosen at random! x1,y1 = 1764883.4618,5461454.6101 # Convert between NZTM & Lat/Lng x2,y2 = transform(inProj,outProj,x1,y1) print (x2,y2) And for reference, here's the Linux version I'm running the server on (which is the standard MS Azure Splunk instance): splunk@standalone-vm:~/etc/apps/<appname>/bin$ uname -a Linux standalone-vm 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ... View more

Hi all - bit of a weird one! I've run out of ideas. Help please! I'm trying to index some CSV files. However, the first line does not contain headers, it contains CSV info about the file itself (such as the start/end dates for the data covered in the file, the original sending organisation, and the type of data it relates to), and a different number of "columns" than the following rows. I would like to parse this header row to extract these values, and append them as fields to each event contained within that file. The Header & Detail rows are easily located (they are the first & every other row respectively, and start with HDR or DET respectively). Sample data here would look like this: HDR,ICPHHAB,10,XXX,XXX,YYYY,7/03/2019,15:04:34,20190307,1344,201902,E,I DET,############,ZZZZZZ,F,1/02/2019,1,44.597,0,,X,,, DET,############,ZZZZZZ,F,1/02/2019,2,44.972,0,,X,,, DET,############,ZZZZZZ,F,1/02/2019,3,44.972,0,,X,,, DET,############,ZZZZZZ,F,1/02/2019,4,44.947,0,,X,,, And what I'm wanting to achieve would be to not include the HDR rows in the search results, but rather to pick up those bits of data & append them to each of the DET rows that follow. Because these are files we are receiving, and they need to follow a certain industry standard, I would like to avoid doing any python style pre-processing (which I agree would be easiest route) in order to leave an audit trail of the original files. I've tried various things from the props.conf like PREAMBLE_REGEX = ^HDR\,(.*) and transforms.conf REGEX = (?=[^H]*(?:HDR|H.*HDR))^\w+,(?P<field1>\w+),(?P<`field2>\d+\.\d+),(?P<field3 >\w+), etc etc etc But the HDR files are returning in my search, the fields from the HDR row are only appearing for those rows (not appending to DET ones), and the field extractions I'm doing in the props.conf for the DET rows are being polluted with HDR row (like col3 contains all values from col3, regardless of if it was a DET or HDR row). ... View more