topic Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option) in Getting Data In

Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

Log_wrangler — Tue, 29 Sep 2020 18:53:37 GMT

I have a typical scenario that could be resolved with a UF on syslog-ng, however that is a future resolution.

At the moment, I have 2 data sources (A and B) coming in on a common port (e.g. TCP 666).

Each data source needs its own sourcetype name and reside in a different index.

So far, I have been trying to follow the docs and I created the following in .../opt/splunk/etc/system/local

Inputs.conf

[TCP://666] <------------------- this is the original data source stanza
Disabled = 0
index = indx_A
sourcetype = st_A

Props.conf

[sourcetype_B] <--------------------- I added this to accomodate data source B
TRANSFORMS-routing = sourcetype_B_routing

Transforms.conf

[sourcetype_B_routing] <--------------------- I added this to accomodate data source B
REGEX=|String_between_pipes|
DEST_KEY=_TCP_ROUTING
FORMAT=Everything, Subsidiary

Outputs.conf
I used two existing stanzas to direct the data to two different locations.

[tcpout:Everything] <----- splunk destination
disabled = false
server = x.x.x.x, x.x.x.x <---------------------I have multiple destinations
autoLB = true

[tcpout:Subsidiary] <----------- 3rd party destination, data to be sent raw
disabled = false
sendCookedData=false
server = x.x.x.x:port

I need some guidance on connecting the inputs to the props>tranforms>outputs.

How do I edit the inputs.conf stanza (or other location) to define index_B and sourcetype_B, and tie the inputs.conf to the other .confs?

Thank you

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

gcusello — Fri, 06 Apr 2018 06:49:13 GMT

Hi Log_wrangler,
At first, if you can use different ports to send logs to Splunk you can easily manage your two inputs setting two stanzas one for each port, having in this way for each flow own index and sourcetype.
In inputs.conf

[TCP://xx.xx.xx.xx:666]
Disabled = 0
index = indx_A
sourcetype = st_A
[TCP://yy.yy.yy.yy:667]
Disabled = 0
index = indx_B
sourcetype = st_B

Then if you need to send data via syslog to external systems you can see at http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Forwarddatatothird-partysystemsd
wher you can find a guide to configure inpus.conf, props.conf, transforms.conf and outputs.conf.

Bye.
Giuseppe

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

Log_wrangler — Tue, 29 Sep 2020 18:54:15 GMT

Thank you so much for the response, and if I could use different port numbers, I definitely would do exactly as your inputs.conf. Unfortunately, both sources are coming in to the same port and I cannot define them in the source device.

Is there a way to define two indexes and two source types in the inputs.conf?

e.g.

[TCP://666]
disabled = 0

then look at events, if has REGEX string - > send as index_B and sourcetype_B routing and everything else goes to index_A and sourcetype_A routing?

Thank you.

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

gcusello — Mon, 09 Apr 2018 06:45:59 GMT

If you can identify flows from sourceA and source B using a regular expression, you can override sourcetype and index:
To override sourcetype see http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/Advancedsourcetypeoverrides
to override index try something similar on your indexer or heavy forwarder:
in props.conf

[mysourcetype]
TRANSFORMS-index = overrideindex

in transforms.conf

[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index

Bye.
Giuseppe

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

gcusello — Mon, 09 Apr 2018 06:53:37 GMT

There's another option:
if your sources come from different hosts you could identify flows from IP address or hostname:

[TCP://xx.xx.xx.xx:666]
 Disabled = 0
 index = indx_A
 sourcetype = st_A

 [TCP://yy.yy.yy.yy:666]
 Disabled = 0
 index = indx_B
 sourcetype = st_B

The only problem is that you must set this configuration only handly modifying conf files, because web gui gives an error.

Bye.
Giuseppe

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

Log_wrangler — Tue, 29 Sep 2020 18:54:34 GMT

Thank you for those two options. I understand Option 1 completely, but have a question about Option 2.

Option 1

(edit confs, GUI gives errors)

inputs***************  

[TCP://xx.xx.xx.xx:666]   *OR [TCP://hostname_A.com:666]
  Disabled = 0
  index = indx_A
  sourcetype = st_A

[TCP://yy.yy.yy.yy:666]  *OR [TCP://hostname_B.com:666] 
  Disabled = 0
  index = indx_B
  sourcetype = st_B

Option 2

  inputs***************

    [TCP://666]
    Disabled = 0
    index = indx_A
    sourcetype = st_A


    props******************

   [st_B]
   TRANSFORMS-index = st_B_override


    transforms*****************

    [st_B_override]
    DEST_KEY = MetaData:Sourcetype
    REGEX=|String_between_pipes|
    FORMAT = index_B

So I am inferring that FORMAT = index_B will define the index to route to in indexes.conf, but where would I define sourcetype = st_B? Can I add ( FORMAT = st_B ) below ( FORMAT = index_B ) ? Or do I add st_B somewhere else? Or does splunk look back at the props stanza [st_B] and know that is the sourcetype?

Hopefully that is my last question and I will accept your answer.

Thank you very much!

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

gcusello — Mon, 09 Apr 2018 15:01:49 GMT

Option 1 is the easiest!
Anyway, there's a mistake:
with DEST_KEY = MetaData:Sourcetype you override sourcetype not index so:
inputs***************

[TCP://666]
Disabled = 0
index = indx_A
sourcetype = st_A

props******************

[st_A]
TRANSFORMS-sourcetype = sourcetype_override
TRANSFORMS-index = index_override

transforms*****************

[sourcetype_override]
DEST_KEY = MetaData:Sourcetype
REGEX=|String_between_pipes|
FORMAT = st_B
[index_override]
DEST_KEY = _MetaData:Index
REGEX=|String_between_pipes|
FORMAT = index_B

I'm not sure, usually I avoid to override at the same time index and sourcetype: you should test it.
Bye.
Giuseppe

Re: Route and Filter Data from syslog (and syslog-ng is NOT an immediate option)

Log_wrangler — Mon, 09 Apr 2018 20:00:40 GMT

Ok thank you very much for the extensive answers.