How to separate and get indexed logs coming from two different hosts

[thambisetty]

Hi Splunkers,
I getting two types of logs: 1>fireeye 2>dlp on the same port(514). two logs are being indexed to main index. now i want to separate those two to different indexes are 1>fireeye 2>dlp. at the same time i need to override source type also for both.
please help me on this.........

[strive · Accepted solution]

You need some identifier to distinguish the two logs. Say for example, the FireEye log formats CEF, LEEF, CSV all have FireEye word. Like CEF:0|FireEye, CSV:0:FireEye. So you can use FireEye as the regex.

Note: May be you need to think for better regexes. I am just showing you how it can be done.

transforms.conf

[set_sourcetype_FireEye]  
REGEX = FireEye  
FORMAT = sourcetype::fireeye_log  
DEST_KEY = MetaData:Sourcetype 

[set_index_FireEye]  
REGEX = FireEye  
FORMAT = fireeye_index  
DEST_KEY = _MetaData:Index

Similarly create transforms for dlp logs. Then,

props.conf

[source::udp:514]  
TRANSFORMS-include = set_sourcetype_FireEye, set_index_FireEye, set_sourcetype_dlp, set_index_dlp

The following links will help you with more details:

http://answers.splunk.com/answers/70027/spliting-multiple-feed-that-use-udp514
http://answers.splunk.com/answers/7125/sending-to-different-index-based-on-host-from-syslog-stream#a...

[thambisetty]

[set_sourcetype_dlp]
REGEX= DLP
FORMAT = sourcetype::dlp

DEST_KEY = MetaData:Sourcetype

[set_index_dlp]
REGEX= DLP
FORMAT = dlp

DEST_KEY = _MetaData:Index

[set_sourcetype_fireeye_malware_alerts]

REGEX= fenotify
FORMAT = sourcetype::fireeye_malware_alerts

DEST_KEY = MetaData:Sourcetype

[set_index_fireeye_csv]

REGEX= fenotify
SOURCE_KEY = MetaData:Host

FORMAT = fireeye_csv

DEST_KEY = _MetaData:Index

[thambisetty]

regex=DLP for dlp and regex=fenotify for fireeye_csv. dlp is working, data getting indexed to dlp and fireeye_csv is not.

[strive]

Try to set Regex for some field that is present in your log events. Say for example: DLP-

[thambisetty]

my regex correct only, getting dlp logs from only one host:10.215.2.55 and fireeye logs from only one host:10.201.50.240

[thambisetty]

dlp:
Sep 5 09:10:37 10.215.2.55 Sep 5 09:10:37 DLP-BLR-ENF-SEC.wipro.com ID: 9713957, Policy Violated: SMTP Last Working Day, Count: 3, Protocol: SMTP, Recipient: haripriya.elangovan@gmail.com,kannan_shakthi@ymail.com, Sender: kalaiselvi.s19@wipro.com, Severity: 1:High, Subject: FW: Mega Employee Referral Walk-In Opportunities for Associate, Target: N/A, Filename: N/A, Blocked: Passed, Endpoint: N/A

Sep 5 09:07:56 10.215.2.55 Sep 5 09:07:56 DLP-BLR-ENF-SEC.wipro.com ID: 9713945, Policy Violated: SMTP Customer Data Monitoring, Count: 12, Protocol: SMTP, Recipient: dinesh.maksat@gmail.com,niklesh.gupta2@wipro.com,Devender.Thakur@airtel.com,abinish.bhakhan1@wipro.com,amit.tanwar5@wipro.com,a_vivek.tanwar@in.airtel.com,wajahat.shreemal@gmail.com,rajeev.sharma4@wipro.com,rohit.kumar44@wipro.com,ashutosh.singh20@wipro.com, Sender: amit.bhardwaj3@wipro.com, Severity: 1:High, Subject: TO BE ADD IN NMS, Target: N/A, Filename: N/A, Blocked: Passed, Endpoint: N/A

[strive]

You mean to say you received data into two separate indexes once?

As per your regex, will you always have host::(10.xx.xx.xx+) at the start on all your log lines?

I think this might be causing problem. You need to revisit your regex.

Can you post some log samples for both fireeye and dlp

[thambisetty]

props.conf

[source::udp:514]

TRANSFORMS-include = set_sourcetype_dlp, set_index_dlp, set_sourcetype_fireeye_malware_alerts, set_index_fireeye_csv

[thambisetty]

transforms.con
[set_sourcetype_dlp]
SOURCE_KEY = MetaData:Host
REGEX=^host::(10.215.2.\55+)$
FORMAT = sourcetype::dlp

DEST_KEY = MetaData:Sourcetype
[set_index_dlp]
SOURCE_KEY = MetaData:Host
REGEX=^host::(10.215.2.\55+)$

FORMAT = dlp

DEST_KEY = _MetaData:Index
[set_sourcetype_fireeye_malware_alerts]

SOURCE_KEY = MetaData:Host
REGEX=^host::(10.201.50.\240+)$
FORMAT = sourcetype::fireeye_malware_alerts

DEST_KEY = MetaData:Sourcetype
[set_index_fireeye_csv]

SOURCE_KEY = MetaData:Host
REGEX=^host::(10.201.50.\240+)$

FORMAT = fireeye_csv

DEST_KEY = _MetaData:Index

[thambisetty]

only once i got the data to index. not getting data continuously to index

[strive]

There is also an app for FireEye. I think you should have a look at it. They have many regexes which you can reuse.
For example, for setting sourcetype:

[fix_FireEye_CEF_st]
REGEX=\|FireEye\|
DEST_KEY=MetaData:Sourcetype
FORMAT=sourcetype::FireEye_CEF

[sk314]

You can configure the inputs.conf file on the splunk forwarders for this purpose.

Add an additional entry for index, sourcetype for each log type like so:

[<stanza associated with the logs>]
index=fireeye           
sourcetype=fireeye_logs 

You need to have those indexes created on the splunk indexer before you begin forwarding data.

[thambisetty]

inputs.conf

[udp://514]
connection_host = ip
index = main
source =
sourcetype = dlp_fireeye
disabled = 0
on 514 im getting two types of logs if i change index in inputs.conf both logs get indexed to that index.

[aholzer]

For the sourcetype bit look at transforms.conf. You can use regex to identify the message type and change the value of the sourcetype.

Not sure if this is doable for the index as well. But this should get you started.

Hope this helps