topic Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data? in Getting Data In

How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Thu, 01 Sep 2016 21:35:19 GMT

I have single event looking like below and trying to figure the best way for Splunk to recognize the key-value pairs. Ideally would have each line as a separate event.

{
  "compsModelObjectName": "Desktop",
  "compsObjectList": [
    {
      "buildGUID": "8D36EF88-3319-4770-BDD3-DCDA614C40DB",
      "buildType": "ONEDESK - FULLBUILD",
      "buildVersion": "2.22.080214-1002",
      "description": "MY TEXT IN HERE",
      "purpose": "Normal",
      "lastScanDate": "Apr 29, 2010",
      "assetName": "WLDNETSBWGS41J",
      "dateModified": "Mar 27, 2013",
      "dateInstalled": "Dec 17, 2009",
      "invNo": "DIMS-1268745",
      "serialNo": "BWGS41J",
      "manufacturer": "UNKNOWN",
      "model": "UNKNOWN PC",
      "assetTag": "Z00880152",
      "status": "INAC",
      "productClass": "UNKNOWN PC",
      "productType": "UNKNOWN",
      "owner": "X1111111",
      "subStatus": "Disposal",
      "compsIdentifier": "DIMS-1268745"
    },
    {
      "buildGUID": "JENYX1111111XP",
      "buildType": "JENY",
      "description": "Unknown Class",
      "purpose": "Normal",
      "lastScanDate": "Nov 1, 2010",
      "assetName": "JENYX1111111XP",
      "dateModified": "Mar 31, 2011",
      "dateInstalled": "Jan 1, 1970",
      "invNo": "TEXTTEXT",
      "serialNo": "JENYX1111111XP",
      "manufacturer": "JENY",
      "model": "JENY",
      "assetTag": "D04936865",
      "status": "INAC",
      "productClass": "JENY",
      "productType": "JENY",
      "owner": "X1111111",
      "subStatus": "Disposal",
      "compsIdentifier": "DIMS-4182421"
    },
    {
      "buildGUID": "JENYX1111111",
      "buildType": "JENY",
      "description": "Unknown Class",
      "purpose": "Normal",
      "lastScanDate": "Nov 21, 2011",
      "assetName": "JENYX1111111",
      "dateModified": "Nov 20, 2011",
      "dateInstalled": "Jan 1, 1970",
      "invNo": "DIMS-4827747",
      "serialNo": "JENYX1111111",
      "manufacturer": "JENY",
      "model": "JENY",
      "assetTag": "D06722795",
      "status": "INAC",
      "productClass": "JENY",
      "productType": "JENY",
      "owner": "X1111111",
      "subStatus": "Disposal",
      "compsIdentifier": "DIMS-4827747"
    },
    {
      "buildGUID": "2DB77FB4-C1D2-4AD4-9453-4A06D4017076",
      "buildType": "xSPACE - FULLBUILD",
      "buildVersion": "4.12",
      "description": "Business Basic PC",
      "domain": "EMEA",
      "purpose": "Normal",
      "lastScanDate": "Aug 30, 2016",
      "assetName": "WC2291Y7F",
      "dateModified": "Aug 31, 2016",
      "dateInstalled": "Jun 24, 2013",
      "invNo": "DIMS-5916063",
      "serialNo": "CZC2291Y7F",
      "manufacturer": "DP",
      "model": "Z611",
      "assetTag": "08192",
      "status": "AC",
      "productClass": "Desktop",
      "productType": "DESKTOP",
      "owner": "X1111111",
      "subStatus": "CONFIGURED",
      "compsIdentifier": "DIMS-5916063"
    }
  ],
  "statusCode": 200
}

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Thu, 01 Sep 2016 23:13:29 GMT

That looks like JSON format. What is your sourcetype set to? Setting it to _json seems like it would do the trick. You can test this with a sample of the data using Settings > Add Data and uploading a sample. You don't have to go through the entire process, but it will show you how the extraction would look when the sourcetype is set to _json.

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Thu, 01 Sep 2016 23:41:07 GMT

Looks much better in the previewer but do i have to use _json sourcetype name? Thanks!

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Thu, 01 Sep 2016 23:47:41 GMT

If you want Splunk to understand how to work with it, the best option is to mark it with the correct sourcetype. Are you wanting to configure your own sourcetype to automatically extract these fields? Or do you want to do something manually at search time?

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

sundareshr — Thu, 01 Sep 2016 23:50:32 GMT

If you are using a custom sourcetype, you can try adding INDEXED_EXTRACTION = JSON to your props.conf for your sourcetype.

https://docs.splunk.com/Documentation/Splunk/6.4.3/Admin/Propsconf

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Thu, 01 Sep 2016 23:53:46 GMT

Yes i want to use custom sourcetype to extract.

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Fri, 02 Sep 2016 01:35:17 GMT

Ok, if you want to keep your custom sourcetype name, then you'll need to add a props.conf entry for it. You can add it to $SPLUNK_HOME/etc/system/local/props.conf, but be sure it isn't going to conflict with an existing sourcetype:

[your_custom_sourcetype_name]
INDEXED_EXTRACTIONS = json
KV_MODE = none

Restart Splunk after adding it, and you should be good to go.

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Fri, 02 Sep 2016 05:29:35 GMT

Using props on indexers the events don't look the same as preview and are breaking on the [ ...]

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Fri, 02 Sep 2016 14:40:44 GMT

If each event is not on a single line, you will need to work on your line breaking configuration so it breaks where you expect. How is this data coming into Splunk?

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Fri, 02 Sep 2016 16:41:10 GMT

a single event is rendering like this:

{
"compsModelObjectName": "Desktop",
"compsObjectList": [
{
"buildGUID": "8D36EF88-3319-4770-BDD3-DCDA614C40DB",
"buildType": "ONEDESK - FULLBUILD",
"buildVersion": "2.22.080214-1002",

....

"subStatus": "CONFIGURED",
"compsIdentifier": "DIMS-5916063"
}
],
"statusCode": 200
}

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Fri, 02 Sep 2016 16:42:33 GMT

Are these being written to a file or being ingested another way?

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Fri, 02 Sep 2016 16:56:40 GMT

inserted via curl, that make a difference?

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Fri, 02 Sep 2016 17:05:49 GMT

Yes, but just in the way you tell Splunk how to break events. So right now it is breaking them on the opening of an array [ so we need to tell it not to.

As a test, have you tried sending an event using the sourcetype _json to see if that performs as expected? If it does, you might try adding this to your custom sourcetype stanza:

BREAK_ONLY_BEFORE = ^{
DATETIME_CONFIG = CURRENT

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Fri, 02 Sep 2016 17:19:50 GMT

Set in system local in indexers and still no change. 😞

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Fri, 02 Sep 2016 17:24:11 GMT

What is the name of your custom sourcetype? Have you tried setting it for a test to _json and see if it works?

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Fri, 02 Sep 2016 17:47:32 GMT

I set that in props.conf system/local on indexer and still not working?

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

bmacias84 — Fri, 02 Sep 2016 19:04:07 GMT

Depending on what you want do to you can use mvexpand.

...| mvexpand compsObjectList 

OR

...| mvexpand compsObjectList | spath

Hope this helps

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Tue, 29 Sep 2020 10:49:30 GMT

The BREAK_ONLY_BEFORE made no difference..?

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

smudge797 — Wed, 07 Sep 2016 12:30:49 GMT

Tried changing to monitoring a flat file and still no change.

Re: How to configure Splunk to parse and recognize key value pairs with brackets in my sample data?

justinatpnnl — Wed, 07 Sep 2016 13:23:09 GMT

@smudge797 I am getting email notifications that you are replying, but nothing is displaying on the page here. Are you seeing the same thing?