topic Re: scripted input not working in Getting Data In

scripted input not working

blbr123 — Fri, 17 Nov 2023 11:22:06 GMT

Hi All,

I have a requirement to Onboard Data from a website like http://1.1.1.1:1234/status/v2 and its a vendor managed API url so Application team cannot use the HEC Token option.

so I have prepared the script to get the Data and tested it Locally and the script works as expected.

I have created a forwarder app with bin folder and kept the script in that and pushed the App to one of our Integration Forwarder but unable to get any data in Splunk.

I have tested the connectivity between our IF and the URL and its successful( Did a Curl to that URL and able to see the URL content)

I have checked firewall and permissions , all seems to be ok but still I am unable to get data in splunk.

Also I checked internal index but don't find anything there.

Can someone guide me what else I need to check in order to get this fixed.

Below is my inputs:

[monitor://./bin/abc.sh]

index=xyz

disabled=false

interval = 500

sourcetype=script:abc

source=abc.sh

I have also created props as below:

[script:abc

DATETIME_CONFIG = CURRENT

SHOULD_LINEMERGE = true

Re: scripted input not working

richgalloway — Fri, 17 Nov 2023 15:33:49 GMT

The monitor stanza in inputs.conf is looking for updates the abc.sh file - something unlikely to happen often. To run a scripted input, use a script stanza

[script://./bin/abc.sh] interval = 500 index = xyz sourcetype = script:abc

Re: scripted input not working

blbr123 — Fri, 17 Nov 2023 14:50:46 GMT

I did not understand the difference between the two stanzas can you please explain

Re: scripted input not working

richgalloway — Fri, 17 Nov 2023 15:35:54 GMT

A [monitor] stanza reads a file and indexes new data written to that file.

A [script] stanza runs a script and indexes the output of it.

Re: scripted input not working

blbr123 — Mon, 20 Nov 2023 04:04:38 GMT

Hi @richgalloway

I have changed the stanza to script from monitor but still unable to see any data in splunk?

Is there anything else I have to check?

Re: scripted input not working

richgalloway — Mon, 20 Nov 2023 14:02:12 GMT

Verify the script runs correctly when run manually

splunk cmd python <<your script>>

Are you trying to run the script on a heavy forwarder or universal forwarder? UFs cannot run python scripts because they don't have in interpreter.

Confirm the forwarder successfully connects to the indexer(s), by verifying the forwarder's logs are in the _internal index.

Tell us how you are trying to find the data in Splunk.

Re: scripted input not working

PickleRick — Mon, 20 Nov 2023 14:21:44 GMT

Check your _internal index for any events from that forwarder regarding that script (or look for those events in splunkd.log directly on that forwarder). That might tell you more.

Re: scripted input not working

blbr123 — Tue, 28 Nov 2023 13:03:31 GMT

My script name was access-abc.sh , I just removed hyphen and renamed it to accessabc.sh and that fixed the issue and able to see the Data in Splunk.

But now I have issue with event Formatting, Actual website data I am ingesting is shown below:

##### BEGIN STATUS #####

#LAST UPDATE : Tue, 28 Nov 2023 11:00:16 +0000

Abcstatus.status=ok

Abcstatus.lastupdate=17xxxxxxxx555

### ServiceStatus ###

xxxxx

xxxxxx

xxxx

### SystemStatus ###

XXXX'

XXXX

### xyxStatus ###

XXX

So on....

But in splunk below lines are coming as a seperate events instead of being part of one complete event:

##### FIRST STATUS ##### - is coming as seperate event

Abcstatus.status=ok - this is also coming as a separate event

Below all events coming as one event which is correct and the above two lines should also be part of this one event:

Abcstatus.lastupdate=17xxxxxxxx555

### ServiceStatus ###

xxxxx

xxxxxx

xxxx

### SystemStatus ###

So on....

##### END STATUS #####

Below is my props:

DATETIME_CONFIG = CURRENT SHOULD_LINEMERGE=TRUE BREAK_ONLY_AFTER = ^#{5}\s{6}END\sSTATUS\s{6}\#{5} MUST_NOT_BREAK_AFTER=\#{5}\s{5}BEGIN\sSTATUS\s{5}\#{5} TIME_PREFIX=^#\w+\s\w+\w+\s:\s MAX_TIMESTAMP_LOOKAHEAD=200

Can you please help me with the issue?

Re: scripted input not working

richgalloway — Tue, 28 Nov 2023 13:58:39 GMT

This new question should be a new posting.