All Apps and Add-ons

AWS Billing App - index aws-bill not getting populated

mjm295
Path Finder

Hi All

I have installed and configured SplunkAppforAWSBilling, I can run:
fetch_detailed_report.py
followed by
process_detailed_report.py

This returns with no errors.

I noiced in the csv directory I have files:
**

xxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2014-05.csv
xxxxxxxx-aws-billing-detailed-line-items-with-resources-and-tags-2014-05.processed.csv

**

As far as I understand, this entry in "inputs.conf"

[monitor://$SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling/csv/*.processed.csv]
crcSalt = <SOURCE>
disabled=0
index=aws-bill
source=aws-csv
sourcetype=csv

Should be picking up the processed csv and importing it in to aws-bill index.

This is not happening. How can I trouble shoot this?

SPLUNK_HOME is default of "/opt/splunk"

Thanks
Mark

0 Karma
1 Solution

liork1
Engager

a couple of suggestions:

  1. try looking in the billing app's log file for any errors. it is located in

    //$SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling/log/detailed_bill_errors.txt
    
  2. this app normally pulls logs for the current month only. you can tell it to pull/process logs for previous months and then see if the index aws-bill gets populated (this is what solved the problem for me). use the helper scripts located under //$SPLUNK__HOME/etc/apps/SplunkAppforAWSBilling/bin:

    fetch_older_report.py 2014 08
    

    (this will pull the report for August 2014).

followed by:

    process_older_report.py 2014 08 

(process the report for August 2014).

running these scripts will poplate the index with older data that you fetched.

  1. another problem could be with file naming: I had to fix the script fetch_older_report.py because it looked for files ending with .csv.zip instead of .zip. I changed line 78 from:

    blah blah blah +".csv.zip" 
    

    to

    blah blah blah +".zip"
    

I hope that helps.

don't forget to restart Splunk after you make these changes!

View solution in original post

monkee
Path Finder

Hi , I am not sure if this is still and issue, but I have released a newer version of the application that avoids these import problems. The application was very sensitive to file permissions so quite often files would not import if they did not have the correct ownership.

The incremental approach to importing files was not ideal and could cause duplicates; this has also been replaced.

liork1
Engager

a couple of suggestions:

  1. try looking in the billing app's log file for any errors. it is located in

    //$SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling/log/detailed_bill_errors.txt
    
  2. this app normally pulls logs for the current month only. you can tell it to pull/process logs for previous months and then see if the index aws-bill gets populated (this is what solved the problem for me). use the helper scripts located under //$SPLUNK__HOME/etc/apps/SplunkAppforAWSBilling/bin:

    fetch_older_report.py 2014 08
    

    (this will pull the report for August 2014).

followed by:

    process_older_report.py 2014 08 

(process the report for August 2014).

running these scripts will poplate the index with older data that you fetched.

  1. another problem could be with file naming: I had to fix the script fetch_older_report.py because it looked for files ending with .csv.zip instead of .zip. I changed line 78 from:

    blah blah blah +".csv.zip" 
    

    to

    blah blah blah +".zip"
    

I hope that helps.

don't forget to restart Splunk after you make these changes!

View solution in original post

Nadhiyaa
Path Finder

I am totally new to this add-on .I configured the aws.yaml file in the local folder of the app.
Still i dont see any logs or csv files .

0 Karma

mjm295
Path Finder

Also, when I check in the GUI, under Settings > Data Inputs > Files & Directories

I can see the associated line, and when I create dummy files, they are getting procesed and the Number of Files column is incrementing.

BUT still nothing in the aws-bill index.

SO where is the data going?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!