Deployment Architecture

Same source file across multiple hosts - split by sourcetype

Barty
Explorer

Afternoon guys & gals,

The scenario I face is as such - we have multiple hosts all feeding data into a single index. The same source file(s), by name, are present on each host but relate to different information depending on the envronment that host represents - so for example, SYS1, DEV2. etc.

To assist in identifying what data is relevant to what host (and therefore environment), we would like to associate a sourcetype to the data driven by the parent host. So it would be essentially:-

[monitor:///test/logs/anylog*.log]
index=same_index
sourcetype=dev_log

[monitor:///test/logs/anylog*.log]
index=same_index
sourcetype=sys_log

[monitor:///test/logs/anylog*.log]
index=same_index
sourcetype=uat_log

Having reviewed varying docs, it seems this is achievable utilising a combination of props & tranforms .conf, but I'm struggling to find examples where this has been used before or an article that makes sense to be able to assist me. We've taken over existing configuration so learning on the spot every day! Thanks in advance for any assistance anyone can provide.

Tags (1)
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Wow. The decision to put production and test data into the same index introduces an amazing level of risk. It also violates several security standards, since splunk security is primarily index-based.

Professional security standards and practices require limiting personnel in what they can access to those things that they have a valid business need to see. There will be many DEV or QA personnel who have a role requirement to be able to see DEV information, but who have no valid need to see PROD info. However, if the data is in the same index, there will be no simple and effective way to keep them from looking. (There are LOTS of complicated, ineffective ways...)

To reiterate - ** you cannot assign security by sourcetype.**


That being said, you can do what you are asking. (You shouldn't, but you can.)

Your simplest method would be to default the sourcetype based on the most numerous one, then have a stanza based on host name for each of the other two to reassign the sourcetype. Obviously, you can't use the monitored folder to differentiate, because it is not different so it wouldn't differentiate anything.

That's going to be something like

--props.conf--

 [monitor:///test/logs/anylog*.log]
 index=same_index
 sourcetype=sys_log
 TRANSFORMS-force_st_for_others = force_devlog, force_uatlog

--transforms.conf--

 [force_devlog]
 SOURCE_KEY = host
 REGEX = some regex that captures the dev hosts
 DEST_KEY = MetaData:Sourcetype
 FORMAT = sourcetype::dev_log

 [force_uatlog]
 SOURCE_KEY = host
 REGEX = some regex that captures the uat hosts
 DEST_KEY = MetaData:Sourcetype
 FORMAT = sourcetype::uat_log
0 Karma

adonio
Ultra Champion

why not put different data in different indexes?
that will allow you to leverage all the search time knowledge items by sourcetype

0 Karma

Barty
Explorer

Sadly the decision has been to keep the index the same, although I shall ask once more in case I can persuade them differently

0 Karma

adonio
Ultra Champion

most of the time when there are different tiers of data from environments - test, prod etc. it is better to segment.
also your props.conf and transforms.conf will be easier to align and you ll be able to easily extract fields across all data and if you need to compare, than split by index...

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...