topic Re: discard few fields and ingest required data using scripted input in Getting Data In

discard few fields and ingest required data using scripted input

ips_mandar — Fri, 14 Jun 2019 10:41:33 GMT

I want to discard few fields from monitoring input so not increase license usage
What will be best way to do it
It can be possible with SEDCMD but I am trying to know using scripted input
I am newbie in script writing ..can anyone guide me to write python script to take only required data in splunk.
What are the stepsto follow?
Thanks in advance. I am using Splunk 7.3 on Windows server.

Re: discard few fields and ingest required data using scripted input

richgalloway — Fri, 14 Jun 2019 12:20:21 GMT

Splunk will index whatever a scripted input writes to stdout. Your script can read any data at all, but the key is write only the fields you want in Splunk.

Re: discard few fields and ingest required data using scripted input

ips_mandar — Fri, 14 Jun 2019 13:47:13 GMT

Thanks @richgalloway Can you please help me with sample script like python . for example I have csv file in which I want only field 2 ,field 3 ,field 5 to be extracted... Since I never written any script can you please help to provide sample script which will work like mentioned above.

Re: discard few fields and ingest required data using scripted input

richgalloway — Fri, 14 Jun 2019 14:28:56 GMT

Google can provide lots of examples. Here's one I crafted from the first result.

import csv

with open('my_csv.txt', mode='r') as csv_file:
    csv_reader = csv.DictReader(csv_file)
    for row in csv_reader:
        print(f'{row[2]},{row[3]},{row[5]}')

Re: discard few fields and ingest required data using scripted input

woodcock — Fri, 14 Jun 2019 16:51:09 GMT

If you are using a scripted input, then you can either edit the script to modify what it outputs, or, if you already have a SEDCMD that works, you can just add | sed "Your SEDCMD here" to the end of the command line.

Re: discard few fields and ingest required data using scripted input

ips_mandar — Mon, 17 Jun 2019 09:13:42 GMT

Thanks @woodcock
For example I have below props.conf

SEDCMD-aremoveheader = s/^\<\?xml[^\>]*\>\n*//g

Then What I need to write in script to run above in script(will it by .py?).
Note: the above props.conf is in Indexer and if I run only |sed ""using script it will not fetch the data from remote server. Since I want to fetch data from remote server.

Re: discard few fields and ingest required data using scripted input

ips_mandar — Mon, 17 Jun 2019 09:14:24 GMT

Thanks @richgalloway I will give this try and will keep posted.

Re: discard few fields and ingest required data using scripted input

woodcock — Mon, 17 Jun 2019 13:40:48 GMT

You have something like this in your inputs.conf:

 [script:///path/to/your_script.sh]

Change it to this:

[script:///path/to/your_script.sh | sed "s/^\<\?xml[^\>]*\>\n*//g"]

Re: discard few fields and ingest required data using scripted input

woodcock — Mon, 17 Jun 2019 16:48:18 GMT

You might need to specify the full path to the sed binary.