topic Re: How to separate and get indexed logs coming from two different hosts in Getting Data In

How to separate and get indexed logs coming from two different hosts

thambisetty — Wed, 03 Sep 2014 16:37:49 GMT

Hi Splunkers,
I getting two types of logs: 1>fireeye 2>dlp on the same port(514). two logs are being indexed to main index. now i want to separate those two to different indexes are 1>fireeye 2>dlp. at the same time i need to override source type also for both.
please help me on this.........

Re: How to separate and get indexed logs coming from two different hosts

aholzer — Wed, 03 Sep 2014 17:24:21 GMT

For the sourcetype bit look at transforms.conf. You can use regex to identify the message type and change the value of the sourcetype.

Not sure if this is doable for the index as well. But this should get you started.

Hope this helps

Re: How to separate and get indexed logs coming from two different hosts

sk314 — Wed, 03 Sep 2014 17:28:20 GMT

You can configure the inputs.conf file on the splunk forwarders for this purpose.

Add an additional entry for index, sourcetype for each log type like so:

[<stanza associated with the logs>]
index=fireeye           
sourcetype=fireeye_logs

You need to have those indexes created on the splunk indexer before you begin forwarding data.

Re: How to separate and get indexed logs coming from two different hosts

strive — Wed, 03 Sep 2014 17:54:46 GMT

You need some identifier to distinguish the two logs. Say for example, the FireEye log formats CEF, LEEF, CSV all have FireEye word. Like CEF:0|FireEye, CSV:0:FireEye. So you can use FireEye as the regex.

Note: May be you need to think for better regexes. I am just showing you how it can be done.

transforms.conf

[set_sourcetype_FireEye]  
REGEX = FireEye  
FORMAT = sourcetype::fireeye_log  
DEST_KEY = MetaData:Sourcetype 

[set_index_FireEye]  
REGEX = FireEye  
FORMAT = fireeye_index  
DEST_KEY = _MetaData:Index

Similarly create transforms for dlp logs. Then,

props.conf

[source::udp:514]  
TRANSFORMS-include = set_sourcetype_FireEye, set_index_FireEye, set_sourcetype_dlp, set_index_dlp

The following links will help you with more details:

http://answers.splunk.com/answers/70027/spliting-multiple-feed-that-use-udp514
http://answers.splunk.com/answers/7125/sending-to-different-index-based-on-host-from-syslog-stream#answer-7127

Re: How to separate and get indexed logs coming from two different hosts

strive — Mon, 28 Sep 2020 17:28:59 GMT

There is also an app for FireEye. I think you should have a look at it. They have many regexes which you can reuse.
For example, for setting sourcetype:

[fix_FireEye_CEF_st]
REGEX=\|FireEye\|
DEST_KEY=MetaData:Sourcetype
FORMAT=sourcetype::FireEye_CEF

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Thu, 04 Sep 2014 06:36:20 GMT

only once i got the data to index. not getting data continuously to index

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Mon, 28 Sep 2020 17:29:02 GMT

transforms.con
[set_sourcetype_dlp]
SOURCE_KEY = MetaData:Host
REGEX=^host::(10.215.2.\55+)$
FORMAT = sourcetype::dlp

DEST_KEY = MetaData:Sourcetype
[set_index_dlp]
SOURCE_KEY = MetaData:Host
REGEX=^host::(10.215.2.\55+)$

FORMAT = dlp

DEST_KEY = _MetaData:Index
[set_sourcetype_fireeye_malware_alerts]

SOURCE_KEY = MetaData:Host
REGEX=^host::(10.201.50.\240+)$
FORMAT = sourcetype::fireeye_malware_alerts

DEST_KEY = MetaData:Sourcetype
[set_index_fireeye_csv]

SOURCE_KEY = MetaData:Host
REGEX=^host::(10.201.50.\240+)$

FORMAT = fireeye_csv

DEST_KEY = _MetaData:Index

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Mon, 28 Sep 2020 17:29:04 GMT

props.conf

[source::udp:514]

TRANSFORMS-include = set_sourcetype_dlp, set_index_dlp, set_sourcetype_fireeye_malware_alerts, set_index_fireeye_csv

Re: How to separate and get indexed logs coming from two different hosts

strive — Thu, 04 Sep 2014 09:59:05 GMT

You mean to say you received data into two separate indexes once?

As per your regex, will you always have host::(10.xx.xx.xx+) at the start on all your log lines?

I think this might be causing problem. You need to revisit your regex.

Can you post some log samples for both fireeye and dlp

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Fri, 05 Sep 2014 03:43:12 GMT

dlp:
Sep 5 09:10:37 10.215.2.55 Sep 5 09:10:37 DLP-BLR-ENF-SEC.wipro.com ID: 9713957, Policy Violated: SMTP Last Working Day, Count: 3, Protocol: SMTP, Recipient: haripriya.elangovan@gmail.com,kannan_shakthi@ymail.com, Sender: kalaiselvi.s19@wipro.com, Severity: 1:High, Subject: FW: Mega Employee Referral Walk-In Opportunities for Associate, Target: N/A, Filename: N/A, Blocked: Passed, Endpoint: N/A

Sep 5 09:07:56 10.215.2.55 Sep 5 09:07:56 DLP-BLR-ENF-SEC.wipro.com ID: 9713945, Policy Violated: SMTP Customer Data Monitoring, Count: 12, Protocol: SMTP, Recipient: dinesh.maksat@gmail.com,niklesh.gupta2@wipro.com,Devender.Thakur@airtel.com,abinish.bhakhan1@wipro.com,amit.tanwar5@wipro.com,a_vivek.tanwar@in.airtel.com,wajahat.shreemal@gmail.com,rajeev.sharma4@wipro.com,rohit.kumar44@wipro.com,ashutosh.singh20@wipro.com, Sender: amit.bhardwaj3@wipro.com, Severity: 1:High, Subject: TO BE ADD IN NMS, Target: N/A, Filename: N/A, Blocked: Passed, Endpoint: N/A

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Fri, 05 Sep 2014 03:54:23 GMT

my regex correct only, getting dlp logs from only one host:10.215.2.55 and fireeye logs from only one host:10.201.50.240

Re: How to separate and get indexed logs coming from two different hosts

strive — Fri, 05 Sep 2014 17:48:11 GMT

Try to set Regex for some field that is present in your log events. Say for example: DLP-

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Mon, 28 Sep 2020 17:31:34 GMT

regex=DLP for dlp and regex=fenotify for fireeye_csv. dlp is working, data getting indexed to dlp and fireeye_csv is not.

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Mon, 28 Sep 2020 17:31:37 GMT

[set_sourcetype_dlp]
REGEX= DLP
FORMAT = sourcetype::dlp

DEST_KEY = MetaData:Sourcetype

[set_index_dlp]
REGEX= DLP
FORMAT = dlp

DEST_KEY = _MetaData:Index

[set_sourcetype_fireeye_malware_alerts]

REGEX= fenotify
FORMAT = sourcetype::fireeye_malware_alerts

DEST_KEY = MetaData:Sourcetype

[set_index_fireeye_csv]

REGEX= fenotify
SOURCE_KEY = MetaData:Host

FORMAT = fireeye_csv

DEST_KEY = _MetaData:Index

Re: How to separate and get indexed logs coming from two different hosts

thambisetty — Mon, 28 Sep 2020 17:31:39 GMT

inputs.conf

[udp://514]
connection_host = ip
index = main
source =
sourcetype = dlp_fireeye
disabled = 0
on 514 im getting two types of logs if i change index in inputs.conf both logs get indexed to that index.