All Apps and Add-ons

Why are fields not parsed in the Bit9 Security Platform app?

todd_miller
Communicator

Using the default configs within the app (except the inputs.conf), I am unable to get the app to parse any fields. Data is flowing into the index nicely, it will display the data with syntax highlighted, but no fields are parsed.

What additional changes am I missing?

0 Karma
1 Solution

todd_miller
Communicator

And we're fixed!

Looks like the issue was due to a sourcetype of 'bit9' that we are using for CEF ingestion of logs via syslog. I moved the sourcetype over to bit9_test and it appears the props/transforms are working correctly. Once I can eliminate the CEF ingestion I can move back to bit9 and life shall be good.

View solution in original post

0 Karma

todd_miller
Communicator

And we're fixed!

Looks like the issue was due to a sourcetype of 'bit9' that we are using for CEF ingestion of logs via syslog. I moved the sourcetype over to bit9_test and it appears the props/transforms are working correctly. Once I can eliminate the CEF ingestion I can move back to bit9 and life shall be good.

View solution in original post

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

can you post your inputs config and a couple of sample records? No extractions like this indicate a conf file error.

0 Karma

todd_miller
Communicator

I was leaning towards this being a config file issue as well.

[monitor://D:\Bit9\LogFiles\*.bt9]
disabled = false
followTail = 0
index = bit9_test

Here's an example of an event:

{ [-] 
    ABId: 
    ABState: 
    BanName: 
    Bit9Server:  <redacted>
    CLVersion: 
    EventParam1:  381 
    EventParam2:  Dec 22 2015 12:00AM 
    EventParam3: 
    EventSubType:  Old events were deleted 
    EventSubTypeId:  107 
    EventType:  Server Management 
    EventTypeId:  0 
    FileHash: 
    FileHashType: 
    FileName: 
    FileThreat: 
    FileTrust: 
    HostIP: 
    HostId: 
    HostName:  System 
    IndicatorName: 
    InstallerHash: 
    InstallerHashType: 
    LocStringId:  247 
    Message:  Deleting 381 events older than Dec 22 2015 12:00AM. 
    MessageTime:  1/19/2016 8:00:51 AM 
    PathName: 
    Platform: 
    Policy: 
    PolicyId: 
    Priority:  Notice 
    ProcessFileName: 
    ProcessHash: 
    ProcessHashType: 
    ProcessKey: 
    ProcessPathName: 
    ProcessThreat: 
    ProcessTrust: 
    ProcessUsageCounter: 
    RootName: 
    RuleName: 
    RuleType: 
    Timestamp:  1/19/2016 8:00:51 AM 
    UpdaterName: 
    UsageCounter: 
    UserName:  System 
    UserSid:  2 
}

Same thing in raw text:

{ "Timestamp": "1/19/2016 8:00:51 AM", "MessageTime": "1/19/2016 8:00:51 AM", "Bit9Server": "<redacted>", "EventType": "Server Management", "EventSubType": "Old events were deleted", "EventTypeId": "0", "EventSubTypeId": "107", "Message": "Deleting 381 events older than Dec 22 2015 12:00AM.", "HostName": "System", "PathName": "", "FileName": "", "ProcessPathName": "", "ProcessFileName": "", "FileHash": "", "FileHashType": "", "InstallerHash": "", "InstallerHashType": "", "HostIP": "", "Policy": "", "Platform": "", "RuleName": "", "BanName": "", "UpdaterName": "", "Priority": "Notice", "UserName": "System", "ProcessHash": "", "ProcessHashType": "", "RootName": "", "RuleType": "", "FileTrust": "", "FileThreat": "", "UsageCounter": "", "ProcessTrust": "", "ProcessThreat": "", "ProcessUsageCounter": "", "CLVersion": "", "EventParam1": "381", "EventParam2": "Dec 22 2015 12:00AM", "EventParam3": "", "HostId": "", "PolicyId": "", "UserSid": "2", "ABId": "", "ABState": "", "LocStringId": "247", "ProcessKey": "", "IndicatorName": "" }
0 Karma

bit9
Path Finder

Hello, and thank you for your interest in the Bit9 Security Platform app.

I don't quite understand your question. What do you mean by "no fields are parsed?" Can you please give an example of a search you are trying to do, along with (a) what type of answers you would expect, and (b) what type of answers you are getting instead?

Thank you very much.

0 Karma

bit9
Path Finder

And you get no additional fields when you click the "All Fields" link?

And you are certain that there are items of those types within your selected time frame?

0 Karma

todd_miller
Communicator

There's one additional field. But definitely no field extraction is occurring with the Metadata or Event traces.

0 Karma

bit9
Path Finder

So you can't do searches like:

eventtype=bit9_event | top EventSubType

or

eventtype=bit9_fileCatalog | top PathName

The reason I didn't quite understand your question was that you indicated that the data shows up correctly syntax-highlighted, which is an indication that the fields are being parsed. It's a reasonably simple JSON input - we do add some additional "color" to it through the app's config properties, but the main fields that are in the data proper should be easily discoverable by Splunk.

So you get no results from the above searches?

0 Karma

bit9
Path Finder

And there are definitely entries from Metadata and Event within the selected time frame?

The idea that Splunk could syntax-highlight the content and not extract the fields would appear to be contradictory. I've never seen this happen, so that's why I'm asking so many questions.

0 Karma

todd_miller
Communicator

Questions are free my friend. Feel free to ask away.

This is just a search on Metadata and Event trace sources. Looks like "host" and "user" are extracting:

Selected Fields
a host 1
a index 1
a source 2
a sourcetype 1

Interesting Fields

date_hour 1

date_mday 1

date_minute 2

a date_month 1

date_second 3

a date_wday 1

date_year 1

date_zone 1

a eventtype 1

linecount 1

a splunk_server 4
a src_nt_domain 1

timeendpos 1

timestartpos 1

user 1

0 Karma

todd_miller
Communicator

Oh, and to make things even more complicated, parsing seemed to work when I dumped it into the "main" index inadvertently. When I moved it to my bit9_test index, parsing died.

0 Karma

bit9
Path Finder

The inputs.conf file on the Splunk forwarder has to be pointing to whatever index you are sending the data. And then after you change the inputs.conf file on the Splunk forwarder, you have to restart the forwarder. Did you do those things after changing the index?

(Also, the entries in eventtypes.conf are dependent on the index name, so that might be causing issues as well. You'll probably want to modify that file on the Splunk server to reflect the actual index name.)

0 Karma

todd_miller
Communicator

I updated the eventtypes for the correct index. I've updated the inputs.conf on the forwarder and restarted the forwarder.

0 Karma

bit9
Path Finder

So do the eventtype searches below return anything, now that they've been updated?

0 Karma

todd_miller
Communicator

The base searches work but the "top" doesn't because no fields are available.

0 Karma

bit9
Path Finder

This, combined with your comment that it worked in one index but not in another one, makes me wonder if there's a permissions issue somewhere along the way.

0 Karma

todd_miller
Communicator

It's quite simple actually. I'm looking at the events coming into the bit9 index (index=bit9, last 60 minutes). I see the event data (i.e. there are events in the index ). Where I'm having a problem is with the data sourcing from the "Metadata Trace". No parsing is occurring on the events (i.e. no key-value pairs are generated). The field extractions aren't working.

This is the same for "Metadata Trace" sources and "Event Trace" sources.

"Net Trace" sources don't seem to have this issue. This is Splunk 6.3.2.

Here are the fields we're seeing:
Selected Fields
ahost
aindex
asource
asourcetype

Interesting Fields
date_hour
date_mday
date_minute
adate_month
date_second
adate_wday
date_year
date_zone
linecount
aProcessFileName
aProcessPathName
aProcessPathNameX
asplunk_server
asrc_nt_domain
timeendpos
timestartpos
auser

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!