Getting Data In

Displaying JSON Output

jedatt01
Builder

I have a scripted input that outputs in JSON format. Splunk is splitting up the records in the wrong place (At the timestamp, which makes sense). Can someone help me with writing the props.conf and transforms.conf to get it to split correctly and also to display the json hierarchical view? Example output below, the records should split where { begins. I put a dotted line between the records so its easy to see where the split should be.

{
    "interaction":{
        "author":{
            "avatar":"http://a0.twimg.com/profile_images/3022743022/da52025a93a30ecd80a741b85bd0d6a2_normal.jpeg",
            "id":370839918,
            "link":"http://twitter.com/LowKickSanghera",
            "name":"Ranjote Sanghera",
            "username":"LowKickSanghera"
        },
        "content":"Wish I could go to one more football game and support the homies",
        "created_at":"Sat, 29 Dec 2012 19:46:25 +0000",
        "id":"1e251f06dc27ae80e074940d994ae656",
        "link":"http://twitter.com/LowKickSanghera/statuses/285109538347180032",
        "schema":{
            "version":3
        },
        "source":"Twitter for Android",
        "type":"twitter"
    },
    "klout":{
        "score":42
    },
    "language":{
        "confidence":100,
        "tag":"en"
    },
    "salience":{
        "content":{
            "sentiment":0
        }
    },
    "twitter":{
        "created_at":"Sat, 29 Dec 2012 19:46:25 +0000",
        "id":"285109538347180032",
        "source":"<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>",
        "text":"Wish I could go to one more football game and support the homies",
        "user":{
            "created_at":"Fri, 09 Sep 2011 18:31:58 +0000",
            "description":"fast cars gs3 gt3rs. If size really matters then the elephant would be the king of the jungle",
            "followers_count":162,
            "friends_count":160,
            "geo_enabled":true,
            "id":370839918,
            "id_str":"370839918",
            "lang":"en",
            "name":"Ranjote Sanghera",
            "screen_name":"LowKickSanghera",
            "statuses_count":5084,
            "time_zone":"Pacific Time (US & Canada)",
            "utc_offset":-28800
        }
    }
}

{
    "interaction":{
        "author":{
            "avatar":"http://a0.twimg.com/profile_images/3033756072/22ab1cb455a230785ee7070afffca209_normal.jpeg",
            "id":107542067,
            "link":"http://twitter.com/samkattenhorn",
            "name":"samkattenhorn",
            "username":"samkattenhorn"
        },
        "content":"What a game of football that turned out to be #renewwalcottscontract",
        "created_at":"Sat, 29 Dec 2012 19:46:26 +0000",
        "id":"1e251f06e5b1a500e0742157048bac6e",
        "link":"http://twitter.com/samkattenhorn/statuses/285109539995537408",
        "schema":{
            "version":3
        },
        "source":"Twitter for iPhone",
        "type":"twitter"
    },
    "klout":{
        "score":40
    },
    "language":{
        "confidence":100,
        "tag":"en"
    },
    "salience":{
        "content":{
            "sentiment":0
        }
    },
    "twitter":{
        "created_at":"Sat, 29 Dec 2012 19:46:26 +0000",
        "id":"285109539995537408",
        "source":"<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>",
        "text":"What a game of football that turned out to be #renewwalcottscontract",
        "user":{
            "created_at":"Fri, 22 Jan 2010 22:45:35 +0000",
            "description":"abcd make it a double",
            "followers_count":165,
            "friends_count":165,
            "geo_enabled":true,
            "id":107542067,
            "id_str":"107542067",
            "lang":"en",
            "name":"samkattenhorn",
            "screen_name":"samkattenhorn",
            "statuses_count":1842,
            "time_zone":"Hawaii",
            "utc_offset":-36000
        }
    }
}

{
    "interaction":{
        "author":{
            "avatar":"http://a0.twimg.com/profile_images/1719254711/image_normal.jpg",
            "id":264354064,
            "link":"http://twitter.com/Malik9071",
            "name":"AbdulMalik Yaghmour",
            "username":"Malik9071"
        },
        "content":"RT @Omiurshotinstar: I think we should be 1st in the League as the best entertainers of Football #Arsenal",
        "created_at":"Sat, 29 Dec 2012 19:46:26 +0000",
        "id":"1e251f06e5b1a500e07440078b1ecd22",
        "link":"http://twitter.com/Malik9071/statuses/285109540930859008",
        "schema":{
            "version":3
        },
        "source":"web",
        "type":"twitter"
    },
    "language":{
        "confidence":100,
        "tag":"en"
    },
    "salience":{
        "content":{
            "sentiment":4
        }
    },
    "twitter":{
        "id":"285109540930859008",
        "retweet":{
            "count":2,
            "created_at":"Sat, 29 Dec 2012 19:46:26 +0000",
            "id":"285109540930859008",
            "source":"web",
            "text":"I think we should be 1st in the League as the best entertainers of Football #Arsenal",
            "user":{
                "created_at":"Fri, 11 Mar 2011 19:28:18 +0000",
                "description":"Medical Student at KAU || A massive Arsenal FC fan || I Love travelling & playing/watching football || My tweets are mostly about Arsenal ||",
                "followers_count":187,
                "friends_count":437,
                "geo_enabled":true,
                "id":264354064,
                "id_str":"264354064",
                "lang":"en",
                "listed_count":1,
                "name":"AbdulMalik Yaghmour",
                "screen_name":"Malik9071",
                "statuses_count":3574,
                "time_zone":"Quito",
                "utc_offset":-18000
            }
        },
        "retweeted":{
            "created_at":"Sat, 29 Dec 2012 19:44:40 +0000",
            "id":"285109097873932289",
            "source":"<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Mobile Web (M2)</a>",
            "user":{
                "created_at":"Wed, 20 Jan 2010 15:55:47 +0000",
                "description":"Just like any other Shooting star i dont grant wishes either 😛 what i can do is give u a Follow Back 🙂 Love The Mighty Arsenal and the things in my Wall",
                "followers_count":4372,
                "friends_count":4245,
                "id":106753666,
                "id_str":"106753666",
                "lang":"en",
                "listed_count":17,
                "location":"Follow Me",
                "name":"Romeo AFC",
                "screen_name":"Omiurshotinstar",
                "statuses_count":9755,
                "time_zone":"Chennai",
                "utc_offset":19800
            }
        }
    }
}

 

Labels (1)
0 Karma

lguinn2
Legend

See if this helps!

props.conf

BREAK_ONLY_BEFORE="interaction":
SHOULD_LINEMERGE=false
TIME_PREFIX ="created_at":"
MAX_TIMESTAMP_LOOKAHEAD =40
KV_MODE = json
MAX_EVENTS = 5000

Note that MAX_EVENTS refers to the number of lines per event, not the maximum number of evetnts.

0 Karma

yannK
Splunk Employee
Splunk Employee

double check that you put the props on the indexer (not only on the forwarder)

0 Karma

jedatt01
Builder

I've tried a ton of different regex patterns but none of them seem to work. Please help!

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...