How to get data data from ArcSight Connectors

[Paolo_Prigione]

The integrating Splunk with Arcsight document, states it is possible to feed Splunk with data coming straight from a Connector. Do you have any idea how this is possible?

The ArcSight website is not as full of infos as Splunk's...
And, yes, I know this might not be the right community, but it's the one I happen to trust.

Paolo

[dflodstrom]

If you use a connector appliance to manage your ArcSight connectors you can just add a new destination and point it at your Splunk server.

Add Destination > Create a new destination > Raw Syslog. Enter IP/Host, Port, Protocol (UDP), and select 'false' for metadata.

Enable a UDP syslog listener on the port you specified for your destination and have Splunk read the file.

[marko_CD]

Hi, dflodstrom,

would you know what the ArcSight Connector Raw Syslog output looks like for non-syslog event data?

I've got an externally-hosted Windows Domain Controller cluster sending WinEventLog data to my Splunk deployment, and because the service provider uses ArcSight for log collection, the only way that data's coming to me is from an ArcSight Connector - deploying Splunk UF's to harvest the DC logs is not an option.

Currently, the output is in CEF, and I'd like to eliminate all the complications that brings, and ingest the data in raw format... except I can't find a straight answer as to what "raw format" means when it comes to non-syslog event data.

[dflodstrom]

@marko_CD,

I believe the 'raw syslog' format is a way to send the events without applying any formatting to them. Our events look very similar to how they come from the UF ... but they're definitely not the same. I'll do my best to paste a copy of an event received from ArcSight here:

EventlogType=Security
EventIndex=17712236
WindowsVersion=Windows Server 2008 R2
WindowsKeyMapFamily=Windows 2008 R2
WindowsParserFamily=Windows 2008 R2|2008|7|Vista
DetectTime=2016-4-13 8:38:17
EventSource=Microsoft-Windows-Security-Auditing
EventID=4647
EventType=Audit_success
EventCategory=12545
User=
ComputerName=<fully.qualified.hostname>
Description=User initiated logoff
Message=
Subject:Security ID=<some.long.string>
Subject:Account Name=<account_name>
Subject:Account Domain=<domain>
Subject:Logon ID=<some_hex_number>

We collect these events with a syslog daemon and use custom props/transforms. Here is our props:

[source::<path_to_source_file>]

NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=EventlogType=
TIME_PREFIX=DetectTime=
TIME_FORMAT=%Y-%m-%d %H:%M:%S
SEDCMD-remove-syslog-header=s/(?mis)(\w+\s+\d+\s+\d+:\d+:\d+\s+\d+\.\d+\.\d+\.\d+\s)//g
SEDCMD-aaaaaaalinebreaker = s/&&/\n/g
SEDCMD-removemessageEventID-1202 = s/(?mis)(Advanced help for this problem is available on.*)\n/\n/g
SEDCMD-removemessageEventID-4624 = s/(This event is generated.*?)\n/\n/g
SEDCMD-removemessageEventID-4688 = s/(?mis)(Token Elevation Type indicates.*?)\n/\n/g

In addition to the parameters identified above we also define some transforms in props.conf to set the host, change index/sourcetype/source for each of the log types, and to null-queue certain event IDs.

YMMV but I hope this helps!

[marko_CD]

@dflodstrom,

that's more insight into the matter than even multiple people in polo shirts embroidered with the HP logo have been able to provide... I might just owe you lunch...

I figured there was no way WinEventLog data was making it through ArcSight unaltered, but at least it's, as you say, close enough to use the stock parsers as a starting point, and it forgoes all the CEF padding.

Thanks a bunch.

[dflodstrom]

Glad I could help! I'll let you buy me lunch at .conf this year 😉 I've converted my comment to an answer as I feel it truly is the answer to this question.

[marko_CD]

They'll have to let me out of the trenches long enough to send me to .conf first, but if they do, you're on...

Meanwhile, the latest from our hosting provider is that ArcSight Connector can output raw direct to a file - no need for a syslog listener intermediary. If that turns out to be true, I'll report back.

[dflodstrom]

You're saying the ArcSight connector will write the file locally and a Splunk Universal Forwarder will read the file and send it off? I like that idea as well!

[dflodstrom]

One option is to create a RAW syslog output destination on your ArcSight connector. On your Splunk server use rsyslog or similar to listen for the incoming syslog feed from the ArcSight connector. Use Splunk to monitor the file it writes. The file is still written to if you stop or restart Splunk.

Do you need more specific instruction?

[rakeshmukherjee]

Hi, I would appreciate if you can send me the ArcSight connector to SPLUNK configuration instruction. Also what is a good architecture for an MSSP environment, Sending data from connector to SPLUNK or sending data from Logger to SPLUNK?

[Claw]

Please ignore the REGEXES above, the editor screws them up.

We will get the proper examples posted in CEF (Common Event Format) Extraction Utilities App

[Claw]

Some updates to this thread
The CEF app needs to be updated with some small corrections. It will work with any Splunk version 4.1 or later.

Those corrections are listed below. props.conf and transforms.com will work nicely exactly as they are below. All of these are minor improvements and corrections on the advice above.

The default way to send data from an Arcsight Connector with be to a port. The default Arcsight Connector port is 8443

This is what is should look like.

props.conf

[cefevents]
MAX_TIMESTAMP_LOOKAHEAD = 350
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TIME_PREFIX = \s(end|rt)=
pulldown_type = 1
REPORT-cefevents = cefHeaders cefKeys

transforms.conf

[cefHeaders]
REGEX = CEF:(?\d+)|(?[^|])|(?[^|])|(?[^|])|(?[^|])|(?[^|])|(?[^|])

[cefKeys]
REGEX = (?:_+)?(?<_KEY_1>[\w.:,[]]+)=(?<_VAL_1>.*?(?=(?:\s[\w.:,[]]+=|$||)))
REPEAT_MATCH = True
CLEAN_KEYS = 1

[dotan_patrich77]

I'm a total noob, and trying to figured out how to configure the cef extraction utils app (http://splunk-base.splunk.com/apps/22280/cef-common-event-format-extraction-utilities) but cannot understand how to work it out

Can you help out in understanding what does it mean to do the following:
"Add REPORT-cefvenets = cefHeaders,cefKeys to relevant stanzas in order for this add-on to parse the events"
what file should i edit?

and secondly, can i apply the app on content that is loaded to splunk via the oneshot rest api?

[gooza]

add to your props.conf file under the relevent stanza the row:

REPORT-cefevents = cefHeaders,cefKeys

you can read more on props.conf at

splunk Documentation

[gooza · Accepted solution]

If you meant how to configure the arcsight agent to send the data out to splunk , let me know and I'll send you instructions on how to ...

run the command ..installdir\current\bin\arcsight agentsetup

choose yes to start the wizardmode
choose I want to add/remove/modify arcsight Manager destinations
choose add new destination
choose raw syslog
add the information of the splunk input you prepared choose the protocol.

hope this helps.

[betelgeuze]

Hi Gooza,

Do you still have the documents on Integrating Splunk with ArcSight. I have my raw data in the ArcSight logger? Is there a way to push from the logger to Splunk?

And is there any documents i can read. i tried looking on the link given in the post above but to no avail.

[phil_wang]

Hi Gooza,

Can you forward the document on how to configure the Arcsight agent to send data out to Splunk to me as well please?

Many thanks,

[woojacky]

Hey Phil, I didn't get any information from Gooza as well. Nevertheless I worked it out internally with my network folks to eventually send data to Splunk from Arcsight. If you can wait till next Friday when I am back in office I will gladly share the information to you.

[tnoelOTS]

Woojacky,
Would you be willing to share how you set up Arcsight to send information to Splunk with me please?
Thank you

[betelgeuze]

Can you send to me too please

[omerr]

Hi woojacky,

I will very happy if you can share with my the configuration we need to do for connecting our Arcsight to Splunk:

how to define to with index arcsight events will written?

how to define the right sourcetype so splunk can parse the data in the right way?

Thanks.