Splunk Dev

Question on shell script for linux

Splunkometry88
Explorer

Hi Splunkers

I have a CSV download with URL threat intel which is a flat file with URLs listed, I will import these into the Splunk Enteprise Security App however I need to add more columns to the file to allow the import

Currently the file is flat showing just URLs as below

https://testurl.com

https://testurl1.com

https://testurl2.com

https://testurl3.com

I need to add the columns as below and have the URLs land in column 4 (url) as shown below

description,http_referrer,http_user_agent,url,weight

More of a *nix question but if anyone can assist with how I might edit this CSV to add those columns would be great.

I download the file via a CRON job so I'll create another job to run the import script after the download

Any help appreciated

Thanks

Labels (2)
0 Karma

bdiego_splunk
Splunk Employee
Splunk Employee

Hi Splunkometry88,

Try the following.

Create a file with the following content:

#!/bin/bash
echo "description,http_referrer,http_user_agent,url,weight" > dest.csv
sed -E 's/(.*)/,,,\1,/' source.txt >> dest.csv

Save it as create_csv.sh.

Then make it executable:

chmod 744 create_csv.sh

 Assuming  'source.txt' has your list of URLs, issuing the command:

./create_csv.sh

will produce a file named "dest.csv" that looks like:

description,http_referrer,http_user_agent,url,weight
,,,https://testurl.com,
,,,https://testurl1.com,
,,,https://testurl2.com,
,,,https://testurl3.com,

Hope that is what you were seeking.

Regards,

-bd-

0 Karma

thambisetty
SplunkTrust
SplunkTrust

I advice you to index data and create lookup using the data and you can have field headers the way you want.

————————————
If this helps, give a like below.

Splunkometry88
Explorer

Thanks for the suggestion, 

I have progressed a little bit, as the file needs to be imported by Splunk ES i've done the below to create a new CSV file with the correct delimiting fields and then import the records from a downloaded file

I’ve managed to get the python working to create the new CSV file and I’ve got the command right to read the file but I need to get this imported into the url field in the newly created file

The script below creates the file but I’m just printing to shell at the end where I should be importing the df = pd.read_csv command into the newly created CSV file in the field 'url'

I'm looking at using pandas

import csv import pandas as pd

with open('destinationfile.csv', 'wb') as csvfile: filewriter = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL) filewriter.writerow(['description', 'http_referrer', 'http_user_agent', 'url', 'wei$

df = pd.read_csv (r'/filepath/sourcefile.csv)' print df

Have a think about this, maybe there’s an easier way to do it

0 Karma

thambisetty
SplunkTrust
SplunkTrust

if your API doesn't have auth then you can directly call API request from Splunk Enterprise security. you can refer fields with field names.

Look at existing feeds to get an idea.

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...