Getting Data In

scripted input (python) - splunk not ingesting program stdout

keiran_harris
Path Finder

Hi guys,
[FYI, im running splunk 6.3.2 on OSX, dev box so SH, UF, IND are all on the same tin]

im trying to setup my first scripted input. So apologies if this is a n00b question, but all my reading/searching and troubleshooting has left me at a dead end, and im at pull out hair stages! I must be doing something stupid....

basically my python runs fine, but NOTHING appears in my splunk index. When i execute the script manually, i get my nicely formatted CSV output, via a simple python "print" to stdout (ready for splunk ingestion... when that finally works!):

keiran_harris@pro15:~[502]$ python /Applications/Splunk/etc/apps/kscriptedinputapp/bin/ksplunkscriptedinput.py

['DNSqryTime=0.00141596794128', 'BOMjsonDlTime=0.203664064407', 'BOMdateTime=20170416183000', 'BOMcloud=-', 'BOMwindDir=SE', 'BOMwindSpd=13', 'BOMwindGst=19', 'BOMrain=0.0', 'BOMhumid=52', 'BOMtempAir=20.3', 'BOMtempFeel=17.9']

Heres my inputs.conf that invokes the above script every 60 secs...

keiran_harris@pro15:/Applications/Splunk/etc/apps/kscriptedinputapp/local[532]$ sudo cat inputs.conf

[script://$SPLUNK_HOME/etc/apps/kscriptedinputapp/bin/ksplunkscriptedinput.py]
interval = 60
disabled = 0
source = ksplunkscriptedinput.py
sourcetype = csv
index = main
host = pro15

Ive added some simple debugging (as i wasnt sure if the program was getting invoked at all!)- this debugging appends a timestamp to a particular file each time the program is invoked, so i can see splunk is calling the code to execute as per the 60 sec interval set in inputs.conf:

hi there myProgStartTime: Sun Apr 16 18:32:24 2017
hi there myProgStartTime: Sun Apr 16 18:33:24 2017
hi there myProgStartTime: Sun Apr 16 18:34:24 2017
hi there myProgStartTime: Sun Apr 16 18:35:24 2017
hi there myProgStartTime: Sun Apr 16 18:36:24 2017
hi there myProgStartTime: Sun Apr 16 18:37:24 2017 

but NOTHING ingested in splunk!! see screenshot:
alt text

Before anyone asks, the _internal ExecProc error log (search: 'index=_internal ExecProcessor sourcetype=splunkd ERROR') is squeaky clean. No hits.

any help MOST appreciated!
thanks in advance guys..
Keiran

0 Karma

dhirendra761
Contributor

Hi,

Sorry for my late answer.

You can try with updating  with "path"

[script://./bin/ksplunkscriptedinput.py]

 It will work.

Tags (1)
0 Karma

jonmargulies
Path Finder

Have you tried searching "All Time"? It would be easy for Splunk to mistakenly index this data with the wrong time, as you have the correct time field buried in the middle and aren't telling Splunk what time to assign.

0 Karma

keiran_harris
Path Finder

Hi, thanks for the suggestion, but sadly no. Nothing shows when i set to 'all time'

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is your 'python' command running the Splunk instance of python or a system instance. It should be the latter.

Your output is not CSV. CSV would look more like this:

DNSqryTime, BOMjsonDlTime, BOMdateTime, BOMcloud, BOMwindDir, BOMwindSpd, BOMwindGst, BOMrain, BOMhumid, BOMtempAir, BOMtempFeel
0.00141596794128, 0.203664064407, 20170416183000, -, SE, 13, 19, 0.0, 52, 20.3, 17.9

If you can't change your script, define a new sourcetype that matches your data format.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

keiran_harris
Path Finder

Thanks! Ill read up on creating my own datatype. I (naively?) thought a comma seperated string containing elements of variable=value pairs would be CSV. But i guess not. Is that the reason splunk doesnt ingest though?

Re which python, i didnt realise splunk shipped with a bundled python! That was one of my earlier queuries actually if in inputs.conf, i had to speficy the sys path to python (for osx this is /usr/bin/python) but when i tried that it error'd. How do I explicitly mandate it uses the system python?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Don't. You should use the Splunk-provided python.
To run your script manually run /Applications/Splunk/bin/splunk cmd python ....

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...