topic Re: Question on shell script for linux in Splunk Dev

Question on shell script for linux

Splunkometry88 — Wed, 16 Sep 2020 22:54:04 GMT

Hi Splunkers

I have a CSV download with URL threat intel which is a flat file with URLs listed, I will import these into the Splunk Enteprise Security App however I need to add more columns to the file to allow the import

Currently the file is flat showing just URLs as below

I need to add the columns as below and have the URLs land in column 4 (url) as shown below

description,http_referrer,http_user_agent,url,weight

More of a *nix question but if anyone can assist with how I might edit this CSV to add those columns would be great.

I download the file via a CRON job so I'll create another job to run the import script after the download

Any help appreciated

Thanks

Re: Question on shell script for linux

thambisetty — Thu, 17 Sep 2020 09:24:51 GMT

I advice you to index data and create lookup using the data and you can have field headers the way you want.

Re: Question on shell script for linux

Splunkometry88 — Thu, 17 Sep 2020 23:01:24 GMT

Thanks for the suggestion,

I have progressed a little bit, as the file needs to be imported by Splunk ES i've done the below to create a new CSV file with the correct delimiting fields and then import the records from a downloaded file

I’ve managed to get the python working to create the new CSV file and I’ve got the command right to read the file but I need to get this imported into the url field in the newly created file

The script below creates the file but I’m just printing to shell at the end where I should be importing the df = pd.read_csv command into the newly created CSV file in the field 'url'

I'm looking at using pandas

import csv import pandas as pd

with open('destinationfile.csv', 'wb') as csvfile: filewriter = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL) filewriter.writerow(['description', 'http_referrer', 'http_user_agent', 'url', 'wei$

df = pd.read_csv (r'/filepath/sourcefile.csv)' print df

Have a think about this, maybe there’s an easier way to do it

Re: Question on shell script for linux

bdiego_splunk — Thu, 17 Sep 2020 23:25:43 GMT

Hi Splunkometry88,

Try the following.

Create a file with the following content:

#!/bin/bash echo "description,http_referrer,http_user_agent,url,weight" > dest.csv sed -E 's/(.*)/,,,\1,/' source.txt >> dest.csv

Save it as create_csv.sh.

Then make it executable:

chmod 744 create_csv.sh

Assuming 'source.txt' has your list of URLs, issuing the command:

./create_csv.sh

will produce a file named "dest.csv" that looks like:

description,http_referrer,http_user_agent,url,weight ,,,https://testurl.com, ,,,https://testurl1.com, ,,,https://testurl2.com, ,,,https://testurl3.com,

Hope that is what you were seeking.

Regards,

-bd-

Re: Question on shell script for linux

thambisetty — Fri, 18 Sep 2020 08:06:39 GMT

if your API doesn't have auth then you can directly call API request from Splunk Enterprise security. you can refer fields with field names.

Look at existing feeds to get an idea.