topic Re: How do you run a Python script on (or before) an index? in Splunk Dev

How do you run a Python script on (or before) an index?

agro1986001 — Sun, 07 Jun 2020 18:38:38 GMT

Hi,

Is it possible to create a custom app on Splunk so that will run a Python script on a custom source (or sourcetype) before a new item is indexed? Specifically, I would also like to access the data that is incoming.

Suppose I have this event coming into splunk:

eventName=newUser firstName=henry lastName=adams

I would like to intercept it and then perhaps add fullName="henry adams"

PS: on my use case, I have to do the processing on/before index, so I cannot use real time alerts.

Best regards

Re: How do you run a Python script on (or before) an index?

vishaltaneja070 — Wed, 23 Jan 2019 13:26:42 GMT

Hello @agro1986001
I think the below example can be achieved using props and transform using regex

In Splunk using regex, you can replace the data inside the event.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Configureindex-timefieldextraction

Re: How do you run a Python script on (or before) an index?

agro1986001 — Wed, 23 Jan 2019 13:34:08 GMT

Hi @vishaltaneja07011993

I gave a simple example of reading data, but unfortunately what I'm doing is not just that. Let's say for example that my python script wants to write to a database (mysql, redis, etc.), which cannot be done using just splunk (only an example. the point is I really want a python script to be called). I want to know whether it's technically possible or not.

Thanks a lot!

Re: How do you run a Python script on (or before) an index?

vishaltaneja070 — Wed, 23 Jan 2019 13:37:59 GMT

@agro1986001

Okay. Yes you can call python script through splunk using inputs.conf.

https://docs.splunk.com/Documentation/Splunk/7.2.3/AdvancedDev/ScriptedInputsIntro

And secondly, if we forward data to database from Splunk, you can relay on db connect as well.

Re: How do you run a Python script on (or before) an index?

agro1986001 — Wed, 23 Jan 2019 13:46:54 GMT

Thanks, but that's different than what I want to accomplish.

I'm not trying to make a script that inputs data to splunk.

I already have data flowing into splunk. I just want a script to be called for every event before that event gets indexed.

Re: How do you run a Python script on (or before) an index?

vishaltaneja070 — Wed, 23 Jan 2019 13:57:15 GMT

@agro1986001

Sorry that doesn't seem to feasible using Splunk.

After indexing, i think still it is possible if you save it as alert but not before indexing.

Re: How do you run a Python script on (or before) an index?

rameshprasad — Tue, 10 Dec 2019 10:12:11 GMT

Hi, I have a similar requirement where I want to intercept the event and want to modify the value of a field which will again come from a REST call. Basically I want to execute a script before sending the fields to index. I am getting data through HTTP Event Collector. Is this possible to do in Splunk?