I have a typical scenario that could be resolved with a UF on syslog-ng, however that is a future resolution.
At the moment, I have 2 data sources (A and B) coming in on a common port (e.g. TCP 666).
Each data source needs its own sourcetype name and reside in a different index.
So far, I have been trying to follow the docs and I created the following in .../opt/splunk/etc/system/local
Inputs.conf
[TCP://666] <------------------- this is the original data source stanza
Disabled = 0
index = indx_A
sourcetype = st_A
Props.conf
[sourcetype_B] <--------------------- I added this to accomodate data source B
TRANSFORMS-routing = sourcetype_B_routing
Transforms.conf
[sourcetype_B_routing] <--------------------- I added this to accomodate data source B
REGEX=|String_between_pipes|
DEST_KEY=_TCP_ROUTING
FORMAT=Everything, Subsidiary
Outputs.conf
I used two existing stanzas to direct the data to two different locations.
[tcpout:Everything] <----- splunk destination
disabled = false
server = x.x.x.x, x.x.x.x <---------------------I have multiple destinations
autoLB = true
[tcpout:Subsidiary] <----------- 3rd party destination, data to be sent raw
disabled = false
sendCookedData=false
server = x.x.x.x:port
I need some guidance on connecting the inputs to the props>tranforms>outputs.
How do I edit the inputs.conf stanza (or other location) to define index_B and sourcetype_B, and tie the inputs.conf to the other .confs?
Thank you
Hi Log_wrangler,
At first, if you can use different ports to send logs to Splunk you can easily manage your two inputs setting two stanzas one for each port, having in this way for each flow own index and sourcetype.
In inputs.conf
[TCP://xx.xx.xx.xx:666]
Disabled = 0
index = indx_A
sourcetype = st_A
[TCP://yy.yy.yy.yy:667]
Disabled = 0
index = indx_B
sourcetype = st_B
Then if you need to send data via syslog to external systems you can see at http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Forwarddatatothird-partysystemsd
wher you can find a guide to configure inpus.conf, props.conf, transforms.conf and outputs.conf.
Bye.
Giuseppe
Hi Log_wrangler,
At first, if you can use different ports to send logs to Splunk you can easily manage your two inputs setting two stanzas one for each port, having in this way for each flow own index and sourcetype.
In inputs.conf
[TCP://xx.xx.xx.xx:666]
Disabled = 0
index = indx_A
sourcetype = st_A
[TCP://yy.yy.yy.yy:667]
Disabled = 0
index = indx_B
sourcetype = st_B
Then if you need to send data via syslog to external systems you can see at http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Forwarddatatothird-partysystemsd
wher you can find a guide to configure inpus.conf, props.conf, transforms.conf and outputs.conf.
Bye.
Giuseppe
Thank you so much for the response, and if I could use different port numbers, I definitely would do exactly as your inputs.conf. Unfortunately, both sources are coming in to the same port and I cannot define them in the source device.
Is there a way to define two indexes and two source types in the inputs.conf?
e.g.
[TCP://666]
disabled = 0
then look at events, if has REGEX string - > send as index_B and sourcetype_B routing and everything else goes to index_A and sourcetype_A routing?
Thank you.
If you can identify flows from sourceA and source B using a regular expression, you can override sourcetype and index:
To override sourcetype see http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/Advancedsourcetypeoverrides
to override index try something similar on your indexer or heavy forwarder:
in props.conf
[mysourcetype]
TRANSFORMS-index = overrideindex
in transforms.conf
[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index
Bye.
Giuseppe
There's another option:
if your sources come from different hosts you could identify flows from IP address or hostname:
[TCP://xx.xx.xx.xx:666]
Disabled = 0
index = indx_A
sourcetype = st_A
[TCP://yy.yy.yy.yy:666]
Disabled = 0
index = indx_B
sourcetype = st_B
The only problem is that you must set this configuration only handly modifying conf files, because web gui gives an error.
Bye.
Giuseppe
Thank you for those two options. I understand Option 1 completely, but have a question about Option 2.
Option 1
(edit confs, GUI gives errors)
inputs***************
[TCP://xx.xx.xx.xx:666] *OR [TCP://hostname_A.com:666]
Disabled = 0
index = indx_A
sourcetype = st_A
[TCP://yy.yy.yy.yy:666] *OR [TCP://hostname_B.com:666]
Disabled = 0
index = indx_B
sourcetype = st_B
Option 2
inputs***************
[TCP://666]
Disabled = 0
index = indx_A
sourcetype = st_A
props******************
[st_B]
TRANSFORMS-index = st_B_override
transforms*****************
[st_B_override]
DEST_KEY = MetaData:Sourcetype
REGEX=|String_between_pipes|
FORMAT = index_B
So I am inferring that FORMAT = index_B will define the index to route to in indexes.conf, but where would I define sourcetype = st_B? Can I add ( FORMAT = st_B ) below ( FORMAT = index_B ) ? Or do I add st_B somewhere else? Or does splunk look back at the props stanza [st_B] and know that is the sourcetype?
Hopefully that is my last question and I will accept your answer.
Thank you very much!
Option 1 is the easiest!
Anyway, there's a mistake:
with DEST_KEY = MetaData:Sourcetype you override sourcetype not index so:
inputs***************
[TCP://666]
Disabled = 0
index = indx_A
sourcetype = st_A
props******************
[st_A]
TRANSFORMS-sourcetype = sourcetype_override
TRANSFORMS-index = index_override
transforms*****************
[sourcetype_override]
DEST_KEY = MetaData:Sourcetype
REGEX=|String_between_pipes|
FORMAT = st_B
[index_override]
DEST_KEY = _MetaData:Index
REGEX=|String_between_pipes|
FORMAT = index_B
I'm not sure, usually I avoid to override at the same time index and sourcetype: you should test it.
Bye.
Giuseppe
Ok thank you very much for the extensive answers.