Building for the Splunk Platform

Question on shell script for linux

Splunkometry88
Explorer

Hi Splunkers

I have a CSV download with URL threat intel which is a flat file with URLs listed, I will import these into the Splunk Enteprise Security App however I need to add more columns to the file to allow the import

Currently the file is flat showing just URLs as below

https://testurl.com

https://testurl1.com

https://testurl2.com

https://testurl3.com

I need to add the columns as below and have the URLs land in column 4 (url) as shown below

description,http_referrer,http_user_agent,url,weight

More of a *nix question but if anyone can assist with how I might edit this CSV to add those columns would be great.

I download the file via a CRON job so I'll create another job to run the import script after the download

Any help appreciated

Thanks

Labels (2)
0 Karma

bdiego_splunk
Splunk Employee
Splunk Employee

Hi Splunkometry88,

Try the following.

Create a file with the following content:

#!/bin/bash
echo "description,http_referrer,http_user_agent,url,weight" > dest.csv
sed -E 's/(.*)/,,,\1,/' source.txt >> dest.csv

Save it as create_csv.sh.

Then make it executable:

chmod 744 create_csv.sh

 Assuming  'source.txt' has your list of URLs, issuing the command:

./create_csv.sh

will produce a file named "dest.csv" that looks like:

description,http_referrer,http_user_agent,url,weight
,,,https://testurl.com,
,,,https://testurl1.com,
,,,https://testurl2.com,
,,,https://testurl3.com,

Hope that is what you were seeking.

Regards,

-bd-

0 Karma

thambisetty
Super Champion

I advice you to index data and create lookup using the data and you can have field headers the way you want.

————————————
If this helps, give a like below.

Splunkometry88
Explorer

Thanks for the suggestion, 

I have progressed a little bit, as the file needs to be imported by Splunk ES i've done the below to create a new CSV file with the correct delimiting fields and then import the records from a downloaded file

I’ve managed to get the python working to create the new CSV file and I’ve got the command right to read the file but I need to get this imported into the url field in the newly created file

The script below creates the file but I’m just printing to shell at the end where I should be importing the df = pd.read_csv command into the newly created CSV file in the field 'url'

I'm looking at using pandas

import csv import pandas as pd

with open('destinationfile.csv', 'wb') as csvfile: filewriter = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL) filewriter.writerow(['description', 'http_referrer', 'http_user_agent', 'url', 'wei$

df = pd.read_csv (r'/filepath/sourcefile.csv)' print df

Have a think about this, maybe there’s an easier way to do it

0 Karma

thambisetty
Super Champion

if your API doesn't have auth then you can directly call API request from Splunk Enterprise security. you can refer fields with field names.

Look at existing feeds to get an idea.

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...