Hello Everyone,
I have an environment consisting of three VPC's (say x, y, and z). Each VPC holds Linux, Windows and AWS logs. I have successfully set-up the AWS log ingest using separate indexes (aws_vpcx, aws_vpcy, aws_vpcz). However, I'm struggling to get the Linux/Windows data to index the same way. The unique identifier I'm using is hostnames. The following holds true for all hostnames per VPC,
For Linux logs I tried to add the following :
props.conf
[host::vpcY*]
TRANSFORMS-osVpcY = osVpcYTrans
[host::vpcZ*]
TRANSFORMS-osVpcZ = osVpcZTrans
transforms.conf
[osVpcYTrans]
REGEX = vpcX.+
DEST_KEY = _MetaData:Index
FORMAT = os_vpcy
[osVpcZTrans]
REGEX = vpcY.+
DEST_KEY = _MetaData:Index
FORMAT = os_vpcz
My second question is the same but for the Windows add-on..this seems more difficult with the single inputs.conf file having multiple indexes in it.
Is there a way for me to specify more than one 'unique' thing about the stanza? For example, this is the default windows inputs.conf containing multiple indexes...I will need the windows index to go to either windows, windows_vpcY, or windows_vpcZ depending on the host that's sending the logs..but then I will also need that same separation for the wineventlog data (wineventlog, wineventlog_vpcY, wineventlog_vpcZ).
###### WinEventLog Inputs for DNS ######
[WinEventLog://DNS Server]
disabled = 0
renderXml=true
index = wineventlog
###### DHCP ######
[monitor://$WINDIR\System32\DHCP]
disabled = 0
whitelist = DhcpSrvLog*
crcSalt = <SOURCE>
sourcetype = DhcpSrvLog
index = windows
Thanks in advance to anyone that can help!
For Linux logs, you need to add the attribute SOURCE_KEY to both stanzas in transforms.conf.
[osVpcYTrans]
SOURCE_KEY = MetaData:Host
REGEX = vpcX.+
DEST_KEY = _MetaData:Index
FORMAT = os_vpcy
For windows, I don't have much idea. You can refer to this answer https://community.splunk.com/t5/Getting-Data-In/How-do-I-configure-Splunk-to-index-Windows-Event-Log...
You can use the same transforms configurations in windows, if the host/source/sourcetype values are different VPCs logs in the wineventlog index.
If this reply helps you, an upvote/like would be appreciated.
This solution worked until I needed to add more indexes to the VPC. Since the host have multiple indexes I've changed the files to go by source and am trying to indicate the prefix of the hostnames in the transforms REGEX section. My current problem and set-up is,
VPC Name | Hostnames Prefix per VPC | Inputs | Indexes |
vpcX | ab- | Linux and Windows | os, windows, wineventlog, msad, perfmon |
vpcY | cd- | Linux and Windows | os_cd, windows_cd, wineventlog_cd, msad_cd, perfmon_cd |
vpcZ | ef- | Linux and Windowws | os_ef, windows_ef, wineventlog_ef, msad_ef, perfmon_ef |
My current props.conf for Splunk_TA_nix is,
[source::/var/*]
TRANSFORMS-routing = osCd
TRANSFORMS-routing = osEf
[source::/etc/*]
TRANSFORMS-routing = osCd
TRANSFORMS-routing = osEf
[source::Linux*]
TRANSFORMS-routing = osCd
TRANSFORMS-routing = osEf
(etc. I've added every source found in the Splunk_TA_nix add-on)
and transforms.conf is where the REGEX is referencing the hostname prefix,
[osCd]
SOURCE_KEY = MetaData:Source
REGEX = .+cd.+
DEST_KEY = _MetaData:Index
FORMAT = os_cd
[osEf]
SOURCE_KEY = MetaData:Source
REGEX = .+ef.+
DEST_KEY = _MetaData:Index
FORMAT = os_ef
Am I writing the REGEX correctly to search on the hostname in addition to the source that is referenced in props.conf? Or is there another parameter/method to specify this?
You don't need multiple stanzas in transforms.conf. In the below transform $1 is the output of REGEX match (cd or ef) and is used in the index name.
props.conf
[source::/var/*]
TRANSFORMS-routing = overrideindex
transforms.conf
[overrideindex]
SOURCE_KEY = MetaData:Host
REGEX = (cd|ef).+
DEST_KEY = _MetaData:Index
FORMAT = os_$1
I really like that idea and am looking into it moving forward....however, with multiple apps (windows/linux) it still didn't separate the logs correctly.
What I ended up doing is more tedious but worked. I created the following apps and then created 6 serverclasses to break them down by OS and host.
I'd like to change this in the future though if anyone using props/transforms for this scenario is willing to share alternative methods.