Splunk Log Priortization

[amimulahasun]

My Core Problem is-

• Log categorization

• Log prioritization

• License optimization

Your main constraint:

 Splunk SIEM license limited to 50 GB per day

The major challenge is that informational and operational logs consume a large portion of the license, leaving limited capacity for high-value security logs.

 What I Need to Achieve

I need to:

1. Categorize logs across multiple device types
   (Windows, Linux, AD, Firewalls, Proxy, VPN, PAM, IDS/IPS, etc.)

2. Prioritize logs into:

   • MUST (critical for detection)

   • OPTIONAL (use-case driven)

   • EXCLUDED (non-security / high-volume noise)

3. Control ingestion so that:

   • Only security-relevant logs are indexed

   • Informational logs do not exhaust license

   • Detection capability is preserved

   • The 50GB/day limit is sustainable

4. Clearly document:

   • What is collected

   • What is not collected

   • Why each decision was made

   • How inclusion/exclusion is technically implemented

   • Where filtering should occur (Device vs UF vs HF vs Indexer)

---

Architectural Clarification I Needed

I wanted clarity on:

• Whether logs should be filtered at the device level or Splunk layer

• The role of Universal Forwarder vs Heavy Forwarder

---

The Real Underlying Issue

Your environment likely generates far more logs than 50GB/day can handle if:

• Firewall allow logs are fully enabled

• Proxy full browsing logs are ingested

• Full Windows auditing is enabled

• Informational logs are not filtered

So my real problem is:

How to design a detection-focused SIEM model under strict license constraints without breaking visibility.

[amimulahasun]

@kknairr Thanks for your response 

[kknairr]

@amimulahasun Hope it was helpful. Happy Splunking!

[PickleRick]

Well, this is not really a Splunk question. It's about knowing your data and knowing what to ingest and it's a question of a properly designed onboarding process.

Noone knows your data better than you.

For example, some organizations consider full firewall logs important, some don't. Some use them in their detections and/or threat hunting, some don't.

So noone can tell you what you should be ingesting. It depends on your use cases, maybe on your compliance needs. It's one of those things I always say you should go through internally and/or with your local friendly Splunk Partner.

[amimulahasun]

@PickleRick Thanks for your reply

[kknairr]

@amimulahasun I have done similar activity to optimize license usage and were successful to certain extent. Considering your scenario, the log sources (Windows, Linux, AD, Firewalls, Proxy, VPN, PAM, IDS/IPS, etc.) you mentioned are data heavy and depending on the environment, I presume 50 GB per day would not be sufficient even if you optimize or filter security-only events. The key is to categorize logs by device type, prioritize them, and enforce ingestion controls so that only security-relevant data consumes your 50GB/day license.

Regarding the architectural consideration, 

• Whether logs should be filtered at the device level or Splunk layer

Filter as close to the source as possible (device level), then use Splunk forwarders for fine-grained control. This ensures you don’t waste license on low-value logs and reduce processing burden on Splunk.

• The role of Universal Forwarder vs Heavy Forwarder

Universal Forwarder is a lightweight agent, designed only to collect and forward raw data. UF has filtering capability before forwarding to Splunk infrastructure using inputs.conf  - whitelist/blacklist specific files, directories, or Windows Event IDs. You can also use limits.conf  to control event size or throughput. 

For example, to filter Windows Security logs using specific event IDs, 
[WinEventLog:Security]
disabled = 0
whitelist = 4624,4625,4672,4688
blacklist = 4634,4662

For advanced filtering options, you can use regex patterns to omit event patterns. Like, to ignore Windows Event Code 4662 events whose Message field contains events with the value Account Name: "example account", add the following line to the inputs.conf file:
[WinEventLog:Security]
blacklist1 = EventCode = "4662" Message = "Account Name:\s+(example account)"

Heavy Forwarder is a full Splunk Enterprise instance, capable of parsing and forwarding. Filtering capability can be achieved via creating custom props.conf & transforms.conf. Supports Regex-based filtering, routing, masking, anonymization.

For strategy perspective, you can filter at the source, enforce forwarder-level controls, and only index logs that directly support detection use cases.  Data onboarding should be purely use-case driven. Your categorization - MUST/OPTIONAL/EXCLUDED is good, use this to categorize the existing data sources and excluded is a definite drop, MUST can be ingested and further filtered using the advanced regex patterns. This way, you preserve detection capability while keeping the 50GB/day license sustainable.

For log filtering reference, 

Include or exclude specific incoming data | Splunk Cloud Platform (last updated 2025-07-04T00:47:59....

> Marking the answer and giving Karma helps others find solutions faster!