topic What kind of things do you view as "bad config"? in Splunk Enterprise

What kind of things do you view as "bad config"?

muebel — Tue, 19 Apr 2022 12:48:44 GMT

This is kind of open ended, but essentially I'm looking for things that you view as bad config, or at least configuration settings that should be flagged for review.

Some ideas I've had so far:

- Indexes with a very short retention period (100 seconds or the like)
- Searches with `index=*` in them
- A deployment server targeturi that doesn't match the name of your actual DS

What other sorts of config would you flag as concerning? Do you have any automated checks for anything like this in house?

Re: What kind of things do you view as "bad config"?

TRex — Tue, 19 Apr 2022 12:54:47 GMT

Something that always catches my attention, and requires review:

[default]

the default stanza

Re: What kind of things do you view as "bad config"?

muebel — Tue, 19 Apr 2022 12:58:36 GMT

ah yeah. Seems like it might be appropriate in some settings. Are there config files where it's particularly egregious?

Re: What kind of things do you view as "bad config"?

isoutamo — Fri, 22 Apr 2022 15:14:36 GMT

some other issues which I always try to avoid/fix

indexes.conf without volumes
KO stored under search app
No naming policy for Splunk KOs
use system/local instead of separate apps for local configurations
non indexer layer nodes without internal log forwarding to indexers
no MC installed / configured
no email configured
no kvstore backups
no own Splunk Apps for different groups/business apps
no separate test/prod environments
no index / sourcetype policies
no source system integration catalog from where you could see what and why you have stuff and to whom contact based on issues
IPs have used instead of FQDN, use always at least meaningful CNAME for all nodes

Definitely there are a lot more items, here was what comes my mind without thinking.

r. Ismo

Re: What kind of things do you view as "bad config"?

PickleRick — Fri, 22 Apr 2022 17:10:11 GMT

TLS certs issued to IP addresses

"Sharing" certs among multiple clients (multiple servers too but I can find legitimate use cases for it).

All scheduled searches with the same schedule.

No sound method of source health monitoring (that's more of an organizational/policy issue than config but some internal splunk-based automation would surely be helpful here).

Searches with tons of rex commands (create the extractions already!).

Abuse of append in searches (usualy by users "extending" old searches in dashboards without really understanding SPL).

Searches with eval _time=something. Often that's a badly onboarded source (but can be legit use case so it's up for review, not necessarily automatic red flag).

Using obsolete tls version and/or weak cipher suites.

Using default certs when tls is enabled.