topic Route to index based on source IP/Dest IP in log (Not source host) in Getting Data In

Route to index based on source IP/Dest IP in log (Not source host)

seankoniarz — Thu, 16 May 2019 15:03:13 GMT

I cannot seem to get this to work so I assume I am doing something wrong. We are about to start a POC for splunk but we wanted to get a head start on some of our use cases.

We need to route specific data coming in to different indexes for our clients. Proxy and Firewall logs. The actual host sending us the logs could be the same for 100 clients so we need to do the routing based on Source or Dest with in the log.

Samples are below. But we basically want to route that data into the index called 1000. We would then want to make more that does different regex for other CIDR ranges. From what I am reading, this appears it should be at least close to what I want.

Props.conf

    [cisco:asa]
    TRANSFORMS-1000 = 1000cisco

Transforms.conf

[1000cisco]
REGEX = :10\.1\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))
DEST_KEY = _MetaData:Index
FORMAT = 1000

Sample Log

<172>May 16 10:51:17 hostip %ASA-4-106023: Deny tcp src fwinterface:10.1.1.57/64176 dst outside:172.217.7.14/443(cloud.google.com) by access-group "aclname" [0x0, 0x0]

Re: Route to index based on source IP/Dest IP in log (Not source host)

koshyk — Thu, 16 May 2019 20:56:07 GMT

Just out of curiosity, Why are you redirecting after you receive the data? You could have specified correct index in inputs.conf if you collect by source ip? i.e. if you have syslog or something, log it based on IP address/hostname, and then index using inputs.conf and send to individual databases as it is very efficient in performance. If you do at props/transforms, Splunk have to check at event level to make that decision

Said that, your props/transforms seems correct. The only thing is, the regex may be bit complicated and you try

[1000cisco]
 REGEX = ^.+:10\.1\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5])).+$
 DEST_KEY = _MetaData:Index
 FORMAT = 1000

Re: Route to index based on source IP/Dest IP in log (Not source host)

seankoniarz — Thu, 16 May 2019 22:08:37 GMT

So I might not need to redirect it after receiving the data if that is an option. But I guess let me better explain the use case.

We act as an ISP essentially for some clients. They use traverse our firewalls and out to the internet.

1 FW for Multiple clients

So the source device could be 10.1.1.6. But that device is going to have multiple client CIDR ranges.

10.2.0.0/16 = client1
10.3.0.0/16 = client2
10.4.0.0/16 = client3

sample log style
Device(sourceHost) source dest
1. 10.1.1.6 client1 to google
2. 10.1.1.6 client2 to bing
3. 10.1.1.6 client3 to amazon

Log #1 to index client1
Log #2 to index client2
Log #3 to index client3

Does that make more sense? If they all had separate firewalls this would be very simple, but that is not the case. If I can do this in a more efficient way I am all about doing that.