Hi Splunkers
I have a CSV download with URL threat intel which is a flat file with URLs listed, I will import these into the Splunk Enteprise Security App however I need to add more columns to the file to allow the import
Currently the file is flat showing just URLs as below
I need to add the columns as below and have the URLs land in column 4 (url) as shown below
description,http_referrer,http_user_agent,url,weight
More of a *nix question but if anyone can assist with how I might edit this CSV to add those columns would be great.
I download the file via a CRON job so I'll create another job to run the import script after the download
Any help appreciated
Thanks
Hi Splunkometry88,
Try the following.
Create a file with the following content:
#!/bin/bash
echo "description,http_referrer,http_user_agent,url,weight" > dest.csv
sed -E 's/(.*)/,,,\1,/' source.txt >> dest.csv
Save it as create_csv.sh.
Then make it executable:
chmod 744 create_csv.sh
Assuming 'source.txt' has your list of URLs, issuing the command:
./create_csv.sh
will produce a file named "dest.csv" that looks like:
description,http_referrer,http_user_agent,url,weight
,,,https://testurl.com,
,,,https://testurl1.com,
,,,https://testurl2.com,
,,,https://testurl3.com,
Hope that is what you were seeking.
Regards,
-bd-
I advice you to index data and create lookup using the data and you can have field headers the way you want.
Thanks for the suggestion,
I have progressed a little bit, as the file needs to be imported by Splunk ES i've done the below to create a new CSV file with the correct delimiting fields and then import the records from a downloaded file
I’ve managed to get the python working to create the new CSV file and I’ve got the command right to read the file but I need to get this imported into the url field in the newly created file
The script below creates the file but I’m just printing to shell at the end where I should be importing the df = pd.read_csv command into the newly created CSV file in the field 'url'
I'm looking at using pandas
import csv import pandas as pd
with open('destinationfile.csv', 'wb') as csvfile: filewriter = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL) filewriter.writerow(['description', 'http_referrer', 'http_user_agent', 'url', 'wei$
df = pd.read_csv (r'/filepath/sourcefile.csv)' print df
Have a think about this, maybe there’s an easier way to do it
if your API doesn't have auth then you can directly call API request from Splunk Enterprise security. you can refer fields with field names.
Look at existing feeds to get an idea.