Getting Data In

How to separate and get indexed logs coming from two different hosts

thambisetty
SplunkTrust
SplunkTrust

Hi Splunkers,
I getting two types of logs: 1>fireeye 2>dlp on the same port(514). two logs are being indexed to main index. now i want to separate those two to different indexes are 1>fireeye 2>dlp. at the same time i need to override source type also for both.
please help me on this.........

————————————
If this helps, give a like below.
0 Karma
1 Solution

strive
Influencer

You need some identifier to distinguish the two logs. Say for example, the FireEye log formats CEF, LEEF, CSV all have FireEye word. Like CEF:0|FireEye, CSV:0:FireEye. So you can use FireEye as the regex.

Note: May be you need to think for better regexes. I am just showing you how it can be done.

transforms.conf

[set_sourcetype_FireEye]  
REGEX = FireEye  
FORMAT = sourcetype::fireeye_log  
DEST_KEY = MetaData:Sourcetype 

[set_index_FireEye]  
REGEX = FireEye  
FORMAT = fireeye_index  
DEST_KEY = _MetaData:Index

Similarly create transforms for dlp logs. Then,

props.conf

[source::udp:514]  
TRANSFORMS-include = set_sourcetype_FireEye, set_index_FireEye, set_sourcetype_dlp, set_index_dlp

The following links will help you with more details:

http://answers.splunk.com/answers/70027/spliting-multiple-feed-that-use-udp514
http://answers.splunk.com/answers/7125/sending-to-different-index-based-on-host-from-syslog-stream#a...

View solution in original post

0 Karma

strive
Influencer

You need some identifier to distinguish the two logs. Say for example, the FireEye log formats CEF, LEEF, CSV all have FireEye word. Like CEF:0|FireEye, CSV:0:FireEye. So you can use FireEye as the regex.

Note: May be you need to think for better regexes. I am just showing you how it can be done.

transforms.conf

[set_sourcetype_FireEye]  
REGEX = FireEye  
FORMAT = sourcetype::fireeye_log  
DEST_KEY = MetaData:Sourcetype 

[set_index_FireEye]  
REGEX = FireEye  
FORMAT = fireeye_index  
DEST_KEY = _MetaData:Index

Similarly create transforms for dlp logs. Then,

props.conf

[source::udp:514]  
TRANSFORMS-include = set_sourcetype_FireEye, set_index_FireEye, set_sourcetype_dlp, set_index_dlp

The following links will help you with more details:

http://answers.splunk.com/answers/70027/spliting-multiple-feed-that-use-udp514
http://answers.splunk.com/answers/7125/sending-to-different-index-based-on-host-from-syslog-stream#a...

0 Karma

thambisetty
SplunkTrust
SplunkTrust

[set_sourcetype_dlp]
REGEX= DLP
FORMAT = sourcetype::dlp

DEST_KEY = MetaData:Sourcetype

[set_index_dlp]
REGEX= DLP
FORMAT = dlp

DEST_KEY = _MetaData:Index

[set_sourcetype_fireeye_malware_alerts]

REGEX= fenotify
FORMAT = sourcetype::fireeye_malware_alerts

DEST_KEY = MetaData:Sourcetype

[set_index_fireeye_csv]

REGEX= fenotify
SOURCE_KEY = MetaData:Host

FORMAT = fireeye_csv

DEST_KEY = _MetaData:Index

————————————
If this helps, give a like below.
0 Karma

thambisetty
SplunkTrust
SplunkTrust

regex=DLP for dlp and regex=fenotify for fireeye_csv. dlp is working, data getting indexed to dlp and fireeye_csv is not.

————————————
If this helps, give a like below.
0 Karma

strive
Influencer

Try to set Regex for some field that is present in your log events. Say for example: DLP-

0 Karma

thambisetty
SplunkTrust
SplunkTrust

my regex correct only, getting dlp logs from only one host:10.215.2.55 and fireeye logs from only one host:10.201.50.240

————————————
If this helps, give a like below.
0 Karma

thambisetty
SplunkTrust
SplunkTrust

dlp:
Sep 5 09:10:37 10.215.2.55 Sep 5 09:10:37 DLP-BLR-ENF-SEC.wipro.com ID: 9713957, Policy Violated: SMTP Last Working Day, Count: 3, Protocol: SMTP, Recipient: haripriya.elangovan@gmail.com,kannan_shakthi@ymail.com, Sender: kalaiselvi.s19@wipro.com, Severity: 1:High, Subject: FW: Mega Employee Referral Walk-In Opportunities for Associate, Target: N/A, Filename: N/A, Blocked: Passed, Endpoint: N/A

Sep 5 09:07:56 10.215.2.55 Sep 5 09:07:56 DLP-BLR-ENF-SEC.wipro.com ID: 9713945, Policy Violated: SMTP Customer Data Monitoring, Count: 12, Protocol: SMTP, Recipient: dinesh.maksat@gmail.com,niklesh.gupta2@wipro.com,Devender.Thakur@airtel.com,abinish.bhakhan1@wipro.com,amit.tanwar5@wipro.com,a_vivek.tanwar@in.airtel.com,wajahat.shreemal@gmail.com,rajeev.sharma4@wipro.com,rohit.kumar44@wipro.com,ashutosh.singh20@wipro.com, Sender: amit.bhardwaj3@wipro.com, Severity: 1:High, Subject: TO BE ADD IN NMS, Target: N/A, Filename: N/A, Blocked: Passed, Endpoint: N/A

————————————
If this helps, give a like below.
0 Karma

strive
Influencer

You mean to say you received data into two separate indexes once?

As per your regex, will you always have host::(10.xx.xx.xx+) at the start on all your log lines?

I think this might be causing problem. You need to revisit your regex.

Can you post some log samples for both fireeye and dlp

0 Karma

thambisetty
SplunkTrust
SplunkTrust

props.conf

[source::udp:514]

TRANSFORMS-include = set_sourcetype_dlp, set_index_dlp, set_sourcetype_fireeye_malware_alerts, set_index_fireeye_csv

————————————
If this helps, give a like below.
0 Karma

thambisetty
SplunkTrust
SplunkTrust

transforms.con
[set_sourcetype_dlp]
SOURCE_KEY = MetaData:Host
REGEX=^host::(10.215.2.\55+)$
FORMAT = sourcetype::dlp

DEST_KEY = MetaData:Sourcetype
[set_index_dlp]
SOURCE_KEY = MetaData:Host
REGEX=^host::(10.215.2.\55+)$

FORMAT = dlp

DEST_KEY = _MetaData:Index
[set_sourcetype_fireeye_malware_alerts]

SOURCE_KEY = MetaData:Host
REGEX=^host::(10.201.50.\240+)$
FORMAT = sourcetype::fireeye_malware_alerts

DEST_KEY = MetaData:Sourcetype
[set_index_fireeye_csv]

SOURCE_KEY = MetaData:Host
REGEX=^host::(10.201.50.\240+)$

FORMAT = fireeye_csv

DEST_KEY = _MetaData:Index

————————————
If this helps, give a like below.
0 Karma

thambisetty
SplunkTrust
SplunkTrust

only once i got the data to index. not getting data continuously to index

————————————
If this helps, give a like below.
0 Karma

strive
Influencer

There is also an app for FireEye. I think you should have a look at it. They have many regexes which you can reuse.
For example, for setting sourcetype:

[fix_FireEye_CEF_st]
REGEX=\|FireEye\|
DEST_KEY=MetaData:Sourcetype
FORMAT=sourcetype::FireEye_CEF

0 Karma

sk314
Builder

You can configure the inputs.conf file on the splunk forwarders for this purpose.

Add an additional entry for index, sourcetype for each log type like so:

[<stanza associated with the logs>]
index=fireeye           
sourcetype=fireeye_logs 

You need to have those indexes created on the splunk indexer before you begin forwarding data.

0 Karma

thambisetty
SplunkTrust
SplunkTrust

inputs.conf

[udp://514]
connection_host = ip
index = main
source =
sourcetype = dlp_fireeye
disabled = 0
on 514 im getting two types of logs if i change index in inputs.conf both logs get indexed to that index.

————————————
If this helps, give a like below.
0 Karma

aholzer
Motivator

For the sourcetype bit look at transforms.conf. You can use regex to identify the message type and change the value of the sourcetype.

Not sure if this is doable for the index as well. But this should get you started.

Hope this helps

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...