scripted input (python) - splunk not ingesting program stdout

[keiran_harris]

Hi guys,
[FYI, im running splunk 6.3.2 on OSX, dev box so SH, UF, IND are all on the same tin]

im trying to setup my first scripted input. So apologies if this is a n00b question, but all my reading/searching and troubleshooting has left me at a dead end, and im at pull out hair stages! I must be doing something stupid....

basically my python runs fine, but NOTHING appears in my splunk index. When i execute the script manually, i get my nicely formatted CSV output, via a simple python "print" to stdout (ready for splunk ingestion... when that finally works!):

keiran_harris@pro15:~[502]$ python /Applications/Splunk/etc/apps/kscriptedinputapp/bin/ksplunkscriptedinput.py

['DNSqryTime=0.00141596794128', 'BOMjsonDlTime=0.203664064407', 'BOMdateTime=20170416183000', 'BOMcloud=-', 'BOMwindDir=SE', 'BOMwindSpd=13', 'BOMwindGst=19', 'BOMrain=0.0', 'BOMhumid=52', 'BOMtempAir=20.3', 'BOMtempFeel=17.9']

Heres my inputs.conf that invokes the above script every 60 secs...

keiran_harris@pro15:/Applications/Splunk/etc/apps/kscriptedinputapp/local[532]$ sudo cat inputs.conf

[script://$SPLUNK_HOME/etc/apps/kscriptedinputapp/bin/ksplunkscriptedinput.py]
interval = 60
disabled = 0
source = ksplunkscriptedinput.py
sourcetype = csv
index = main
host = pro15

Ive added some simple debugging (as i wasnt sure if the program was getting invoked at all!)- this debugging appends a timestamp to a particular file each time the program is invoked, so i can see splunk is calling the code to execute as per the 60 sec interval set in inputs.conf:

hi there myProgStartTime: Sun Apr 16 18:32:24 2017
hi there myProgStartTime: Sun Apr 16 18:33:24 2017
hi there myProgStartTime: Sun Apr 16 18:34:24 2017
hi there myProgStartTime: Sun Apr 16 18:35:24 2017
hi there myProgStartTime: Sun Apr 16 18:36:24 2017
hi there myProgStartTime: Sun Apr 16 18:37:24 2017 

but NOTHING ingested in splunk!! see screenshot:

Before anyone asks, the _internal ExecProc error log (search: 'index=_internal ExecProcessor sourcetype=splunkd ERROR') is squeaky clean. No hits.

any help MOST appreciated!
thanks in advance guys..
Keiran

[dhirendra761]

Hi,

Sorry for my late answer.

You can try with updating  with "path"

[script://./bin/ksplunkscriptedinput.py]

 It will work.

[jonmargulies]

Have you tried searching "All Time"? It would be easy for Splunk to mistakenly index this data with the wrong time, as you have the correct time field buried in the middle and aren't telling Splunk what time to assign.

[keiran_harris]

Hi, thanks for the suggestion, but sadly no. Nothing shows when i set to 'all time'

[richgalloway]

Is your 'python' command running the Splunk instance of python or a system instance. It should be the latter.

Your output is not CSV. CSV would look more like this:

DNSqryTime, BOMjsonDlTime, BOMdateTime, BOMcloud, BOMwindDir, BOMwindSpd, BOMwindGst, BOMrain, BOMhumid, BOMtempAir, BOMtempFeel
0.00141596794128, 0.203664064407, 20170416183000, -, SE, 13, 19, 0.0, 52, 20.3, 17.9

If you can't change your script, define a new sourcetype that matches your data format.

[keiran_harris]

Thanks! Ill read up on creating my own datatype. I (naively?) thought a comma seperated string containing elements of variable=value pairs would be CSV. But i guess not. Is that the reason splunk doesnt ingest though?

Re which python, i didnt realise splunk shipped with a bundled python! That was one of my earlier queuries actually if in inputs.conf, i had to speficy the sys path to python (for osx this is /usr/bin/python) but when i tried that it error'd. How do I explicitly mandate it uses the system python?

[richgalloway]

Don't. You should use the Splunk-provided python.
To run your script manually run /Applications/Splunk/bin/splunk cmd python ....