All Apps and Add-ons
Highlighted

AWS Billing App - index aws-bill not getting populated

Path Finder

Hi All

I have installed and configured SplunkAppforAWSBilling, I can run:
fetchdetailedreport.py
followed by
processdetailedreport.py

This returns with no errors.

I noiced in the csv directory I have files:
**

xxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2014-05.csv
xxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2014-05.processed.csv

**

As far as I understand, this entry in "inputs.conf"

[monitor://$SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling/csv/*.processed.csv]
crcSalt = <SOURCE>
disabled=0
index=aws-bill
source=aws-csv
sourcetype=csv

Should be picking up the processed csv and importing it in to aws-bill index.

This is not happening. How can I trouble shoot this?

SPLUNK_HOME is default of "/opt/splunk"

Thanks
Mark

0 Karma
Highlighted

Re: AWS Billing App - index aws-bill not getting populated

Path Finder

Also, when I check in the GUI, under Settings > Data Inputs > Files & Directories

I can see the associated line, and when I create dummy files, they are getting procesed and the Number of Files column is incrementing.

BUT still nothing in the aws-bill index.

SO where is the data going?

0 Karma
Highlighted

Re: AWS Billing App - index aws-bill not getting populated

Engager

a couple of suggestions:

  1. try looking in the billing app's log file for any errors. it is located in

    //$SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling/log/detailed_bill_errors.txt
    
  2. this app normally pulls logs for the current month only. you can tell it to pull/process logs for previous months and then see if the index aws-bill gets populated (this is what solved the problem for me). use the helper scripts located under //$SPLUNK__HOME/etc/apps/SplunkAppforAWSBilling/bin:

    fetch_older_report.py 2014 08
    

    (this will pull the report for August 2014).

followed by:

    process_older_report.py 2014 08 

(process the report for August 2014).

running these scripts will poplate the index with older data that you fetched.

  1. another problem could be with file naming: I had to fix the script fetcholderreport.py because it looked for files ending with .csv.zip instead of .zip. I changed line 78 from:

    blah blah blah +".csv.zip" 
    

    to

    blah blah blah +".zip"
    

I hope that helps.

don't forget to restart Splunk after you make these changes!

View solution in original post

Highlighted

Re: AWS Billing App - index aws-bill not getting populated

Path Finder

I am totally new to this add-on .I configured the aws.yaml file in the local folder of the app.
Still i dont see any logs or csv files .

0 Karma
Highlighted

Re: AWS Billing App - index aws-bill not getting populated

Path Finder

Hi , I am not sure if this is still and issue, but I have released a newer version of the application that avoids these import problems. The application was very sensitive to file permissions so quite often files would not import if they did not have the correct ownership.

The incremental approach to importing files was not ideal and could cause duplicates; this has also been replaced.