Getting Data In

How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797
Path Finder

I have single event looking like below and trying to figure the best way for Splunk to recognize the key-value pairs. Ideally would have each line as a separate event.

{
  "compsModelObjectName": "Desktop",
  "compsObjectList": [
    {
      "buildGUID": "8D36EF88-3319-4770-BDD3-DCDA614C40DB",
      "buildType": "ONEDESK - FULLBUILD",
      "buildVersion": "2.22.080214-1002",
      "description": "MY TEXT IN HERE",
      "purpose": "Normal",
      "lastScanDate": "Apr 29, 2010",
      "assetName": "WLDNETSBWGS41J",
      "dateModified": "Mar 27, 2013",
      "dateInstalled": "Dec 17, 2009",
      "invNo": "DIMS-1268745",
      "serialNo": "BWGS41J",
      "manufacturer": "UNKNOWN",
      "model": "UNKNOWN PC",
      "assetTag": "Z00880152",
      "status": "INAC",
      "productClass": "UNKNOWN PC",
      "productType": "UNKNOWN",
      "owner": "X1111111",
      "subStatus": "Disposal",
      "compsIdentifier": "DIMS-1268745"
    },
    {
      "buildGUID": "JENYX1111111XP",
      "buildType": "JENY",
      "description": "Unknown Class",
      "purpose": "Normal",
      "lastScanDate": "Nov 1, 2010",
      "assetName": "JENYX1111111XP",
      "dateModified": "Mar 31, 2011",
      "dateInstalled": "Jan 1, 1970",
      "invNo": "TEXTTEXT",
      "serialNo": "JENYX1111111XP",
      "manufacturer": "JENY",
      "model": "JENY",
      "assetTag": "D04936865",
      "status": "INAC",
      "productClass": "JENY",
      "productType": "JENY",
      "owner": "X1111111",
      "subStatus": "Disposal",
      "compsIdentifier": "DIMS-4182421"
    },
    {
      "buildGUID": "JENYX1111111",
      "buildType": "JENY",
      "description": "Unknown Class",
      "purpose": "Normal",
      "lastScanDate": "Nov 21, 2011",
      "assetName": "JENYX1111111",
      "dateModified": "Nov 20, 2011",
      "dateInstalled": "Jan 1, 1970",
      "invNo": "DIMS-4827747",
      "serialNo": "JENYX1111111",
      "manufacturer": "JENY",
      "model": "JENY",
      "assetTag": "D06722795",
      "status": "INAC",
      "productClass": "JENY",
      "productType": "JENY",
      "owner": "X1111111",
      "subStatus": "Disposal",
      "compsIdentifier": "DIMS-4827747"
    },
    {
      "buildGUID": "2DB77FB4-C1D2-4AD4-9453-4A06D4017076",
      "buildType": "xSPACE - FULLBUILD",
      "buildVersion": "4.12",
      "description": "Business Basic PC",
      "domain": "EMEA",
      "purpose": "Normal",
      "lastScanDate": "Aug 30, 2016",
      "assetName": "WC2291Y7F",
      "dateModified": "Aug 31, 2016",
      "dateInstalled": "Jun 24, 2013",
      "invNo": "DIMS-5916063",
      "serialNo": "CZC2291Y7F",
      "manufacturer": "DP",
      "model": "Z611",
      "assetTag": "08192",
      "status": "AC",
      "productClass": "Desktop",
      "productType": "DESKTOP",
      "owner": "X1111111",
      "subStatus": "CONFIGURED",
      "compsIdentifier": "DIMS-5916063"
    }
  ],
  "statusCode": 200
}
0 Karma

smudge797
Path Finder

Not sure why i cant add to thread but this works..

This is not working. I have tried adding as a flat file and changing sourcetype to _json but still the events dont break as they do in data preview.

0 Karma

justinatpnnl
Communicator

@smudge797 I am getting email notifications that you are replying, but nothing is displaying on the page here. Are you seeing the same thing?

0 Karma

bmacias84
Champion

When treads get to long they dont display correctly start a new thread.

0 Karma

bmacias84
Champion

Depending on what you want do to you can use mvexpand.

...| mvexpand compsObjectList 

OR

...| mvexpand compsObjectList | spath

Hope this helps

0 Karma

justinatpnnl
Communicator

That looks like JSON format. What is your sourcetype set to? Setting it to _json seems like it would do the trick. You can test this with a sample of the data using Settings > Add Data and uploading a sample. You don't have to go through the entire process, but it will show you how the extraction would look when the sourcetype is set to _json.

0 Karma

smudge797
Path Finder

Looks much better in the previewer but do i have to use _json sourcetype name? Thanks!

0 Karma

justinatpnnl
Communicator

If you want Splunk to understand how to work with it, the best option is to mark it with the correct sourcetype. Are you wanting to configure your own sourcetype to automatically extract these fields? Or do you want to do something manually at search time?

0 Karma

smudge797
Path Finder

Yes i want to use custom sourcetype to extract.

0 Karma

justinatpnnl
Communicator

Ok, if you want to keep your custom sourcetype name, then you'll need to add a props.conf entry for it. You can add it to $SPLUNK_HOME/etc/system/local/props.conf, but be sure it isn't going to conflict with an existing sourcetype:

[your_custom_sourcetype_name]
INDEXED_EXTRACTIONS = json
KV_MODE = none

Restart Splunk after adding it, and you should be good to go.

0 Karma

smudge797
Path Finder

Using props on indexers the events don't look the same as preview and are breaking on the [ ...]

0 Karma

justinatpnnl
Communicator

If each event is not on a single line, you will need to work on your line breaking configuration so it breaks where you expect. How is this data coming into Splunk?

0 Karma

smudge797
Path Finder

a single event is rendering like this:

{
"compsModelObjectName": "Desktop",
"compsObjectList": [
{
"buildGUID": "8D36EF88-3319-4770-BDD3-DCDA614C40DB",
"buildType": "ONEDESK - FULLBUILD",
"buildVersion": "2.22.080214-1002",

....

"subStatus": "CONFIGURED",
"compsIdentifier": "DIMS-5916063"
}
],
"statusCode": 200
}

0 Karma

justinatpnnl
Communicator

Are these being written to a file or being ingested another way?

0 Karma

smudge797
Path Finder

inserted via curl, that make a difference?

0 Karma

justinatpnnl
Communicator

Yes, but just in the way you tell Splunk how to break events. So right now it is breaking them on the opening of an array [ so we need to tell it not to.

As a test, have you tried sending an event using the sourcetype _json to see if that performs as expected? If it does, you might try adding this to your custom sourcetype stanza:

BREAK_ONLY_BEFORE = ^{
DATETIME_CONFIG = CURRENT
0 Karma

smudge797
Path Finder

I tried adding as a flat file and still not breaking as it looks in previewer?

0 Karma

smudge797
Path Finder

Tried changing to monitoring a flat file and still no change.

0 Karma

smudge797
Path Finder

The BREAK_ONLY_BEFORE made no difference..?

0 Karma

smudge797
Path Finder

I set that in props.conf system/local on indexer and still not working?

0 Karma

justinatpnnl
Communicator

What is the name of your custom sourcetype? Have you tried setting it for a test to _json and see if it works?

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...