Splunk Dev

How do I return AWS storage use by region/month -- only returns most recent month no matter what I try?

brockwebb
Explorer

earliest=-360d aws-description-resource( (aws_account_id="*") , (region="*") , "ec2_volumes") |convert timeformat="%Y-%m-%d %H:%M:%S" ctime(_time) AS c_time |timechart span=1months sum(size) as total_size by region

I can get this info using the right filters in AWS cost explorer, but drives me batty because I can't seem to do it in splunk... above is the query I'm running, and I've tried many variants on the theme of aggregating by month with no success.

Tags (1)
0 Karma
1 Solution

brockwebb
Explorer

Problem was a pre-defined macro:
"aws-description-resource" macro was set up to only look at current data within the last 55 minutes, so trying to use that macro for historical data was never going to work. Instead I modified the search calling out the index vs the macro and now the data is coming in as it appears it should.

Query:

((index="aws" OR index="default") sourcetype="aws:description" aws_account_id="" region="" source="*:ec2_volumes") | eventstats latest(_time) as latest_time | eval latest_time=relative_time(latest_time,"-55m") | dedup id sortby -_time | bin _time span=1mon@mon0 | stats sum(size) by _time, region

This works!

View solution in original post

0 Karma

brockwebb
Explorer

Problem was a pre-defined macro:
"aws-description-resource" macro was set up to only look at current data within the last 55 minutes, so trying to use that macro for historical data was never going to work. Instead I modified the search calling out the index vs the macro and now the data is coming in as it appears it should.

Query:

((index="aws" OR index="default") sourcetype="aws:description" aws_account_id="" region="" source="*:ec2_volumes") | eventstats latest(_time) as latest_time | eval latest_time=relative_time(latest_time,"-55m") | dedup id sortby -_time | bin _time span=1mon@mon0 | stats sum(size) by _time, region

This works!

0 Karma

renjith_nair
Legend

@brockwebb, is it worth to check the timestamps in splunk ? i.e. data for previous months are ingested with previous months timestamps or as a batch with single timestamp of current month?

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma

adonio
Ultra Champion

can you check if you have older data? you are looking for data 360 days ago, not sure your index supports that retention

0 Karma

brockwebb
Explorer

We have all the data, problem was a pre-defined macro for AWS plugins ... we corrected. Posted answer and description. Thanks for all your help looking into this!!!

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...