HTTP Event Collector examples - Splunk Documentation need troubleshooting suggestion if possible/ available with user access ... View more

Hi @yuanliu this is truly excellent work, thank you so much for your time an determination on finding the root cause of this behaviour ... View more

Hi @yuanliu  I think I have the solution, I have pasted my guess work as a response to the original post I have uploaded a dashboard with a working method, albeit perhaps not an optional one: https://gist.github.com/niksheridan/d8377778e4c5f1ff3e2e49b0b9899185 I will try and "collapsing" the first and second steps of the chain in order to actually get a pipeline working that can be further extended. I'd be very grateful in your feelback, if you could be so kind as to review this, if you have the time - thank you again for you help    thanks nik ... View more

Think I have the solution Given the base search is in fact not really a search, but a reference to and index: index=_internal It seems that as no pipeline exists, this seems to break the chain, as perhaps arguably there is no real search to begin with.   By changing my base (base2) search as follows (which combines base0 and base0chain1a) this creates a pipeline index=_internal | search useTypeahead=true This then allows be to extend it further with an additional chained search successfully: | stats count This provides me with the expected behaviour, shown in the bottom row of graphs. ... View more

Is this what you mean?  Please let me know if I have misunderstood and thank you again   { "visualizations": { "viz_glNXouAy": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_gREZNTgj" }, "context": {}, "showProgressBar": false, "showLastUpdated": false }, "viz_1ibEKiXT": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_PozPBYIA_ds_gREZNTgj" }, "context": {}, "showProgressBar": false, "showLastUpdated": false }, "viz_rYhOWilO": { "type": "splunk.events", "options": {}, "dataSources": { "primary": "ds_Aoy6m25x_ds_PozPBYIA_ds_gREZNTgj" }, "context": {}, "showProgressBar": false, "showLastUpdated": false }, "viz_HS70GboS": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_n4Q7l7oK" }, "context": {}, "showProgressBar": false, "showLastUpdated": false }, "viz_cFoVJm3n": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_BgOJ54ak" }, "context": {}, "showProgressBar": false, "showLastUpdated": false }, "viz_Oh0reZaV": { "type": "splunk.markdown", "options": { "markdown": "base0\n\nsearch base0\n```\nindex=_internal\n```" } }, "viz_5jz1AX2v": { "type": "splunk.markdown", "options": { "markdown": "base0chain1a\n\nsearch base0\n```\nindex=_internal\n```\n\nsearch chain1a\n```\n| search useTypeahead=true\n```" } }, "viz_A5Kcf02B": { "type": "splunk.markdown", "options": { "markdown": "base0chain1achain2a\n\nsearch base0\n```\nindex=_internal\n```\n\nsearch chain1a\n```\n| search useTypeahead=true\n```\n\nsearch chain2a\n```\n| stats count\n```" } }, "viz_ymCZMl2z": { "type": "splunk.markdown", "options": { "markdown": "combined non-chained search\n```\nindex=_internal \n| search useTypeahead=true \n| stats count\n```\n" } }, "viz_rjA0XgMd": { "type": "splunk.markdown", "options": { "markdown": "base0\n\nexpected" } }, "viz_c4r59ekz": { "type": "splunk.markdown", "options": { "markdown": "base0chain1a\n\n**unexpected**" } }, "viz_hFoP7IsM": { "type": "splunk.markdown", "options": { "markdown": "base0chain1achain2a\n\n**unexpected**" } }, "viz_7lRWKreL": { "type": "splunk.markdown", "options": { "markdown": "combined non-chained search\n\nexpected" } } }, "dataSources": { "ds_n4Q7l7oK": { "type": "ds.search", "options": { "query": "index=_internal", "queryParameters": { "earliest": "$global_time.earliest$", "latest": "$global_time.latest$" } }, "name": "base0" }, "ds_gQjuR7jY": { "type": "ds.search", "options": { "query": "index=_internal\n| search useTypeahead=true", "queryParameters": { "earliest": "$global_time.earliest$", "latest": "$global_time.latest$" } }, "name": "base1" }, "ds_gREZNTgj": { "type": "ds.chain", "options": { "extend": "ds_gQjuR7jY", "query": "| stats count" }, "name": "base1chain2" }, "ds_PozPBYIA_ds_gREZNTgj": { "type": "ds.chain", "options": { "extend": "ds_Aoy6m25x_ds_PozPBYIA_ds_gREZNTgj", "query": "| stats count" }, "name": "base0chain1achain2a" }, "ds_Aoy6m25x_ds_PozPBYIA_ds_gREZNTgj": { "type": "ds.chain", "options": { "extend": "ds_n4Q7l7oK", "query": "| search useTypeahead=true" }, "name": "base0chain1a" }, "ds_BgOJ54ak": { "type": "ds.search", "options": { "query": "index=_internal \n| search useTypeahead=true \n| stats count" }, "name": "base0chain1achain2aFull" } }, "defaults": { "dataSources": { "ds.search": { "options": { "queryParameters": { "latest": "$global_time.latest$", "earliest": "$global_time.earliest$" } } } } }, "inputs": { "input_global_trp": { "type": "input.timerange", "options": { "token": "global_time", "defaultValue": "-24h@h,now" }, "title": "Global Time Range" } }, "layout": { "type": "absolute", "options": { "width": 1440, "height": 1200, "display": "auto" }, "structure": [ { "item": "viz_glNXouAy", "type": "block", "position": { "x": 0, "y": 940, "w": 270, "h": 300 } }, { "item": "viz_1ibEKiXT", "type": "block", "position": { "x": 580, "y": 90, "w": 270, "h": 300 } }, { "item": "viz_rYhOWilO", "type": "block", "position": { "x": 290, "y": 90, "w": 270, "h": 300 } }, { "item": "viz_HS70GboS", "type": "block", "position": { "x": 0, "y": 90, "w": 270, "h": 300 } }, { "item": "viz_cFoVJm3n", "type": "block", "position": { "x": 1170, "y": 90, "w": 270, "h": 300 } }, { "item": "viz_Oh0reZaV", "type": "block", "position": { "x": 0, "y": 390, "w": 290, "h": 300 } }, { "item": "viz_5jz1AX2v", "type": "block", "position": { "x": 280, "y": 390, "w": 290, "h": 300 } }, { "item": "viz_A5Kcf02B", "type": "block", "position": { "x": 580, "y": 390, "w": 290, "h": 300 } }, { "item": "viz_ymCZMl2z", "type": "block", "position": { "x": 1170, "y": 390, "w": 290, "h": 300 } }, { "item": "viz_rjA0XgMd", "type": "block", "position": { "x": 0, "y": 0, "w": 290, "h": 80 } }, { "item": "viz_c4r59ekz", "type": "block", "position": { "x": 290, "y": 0, "w": 290, "h": 80 } }, { "item": "viz_hFoP7IsM", "type": "block", "position": { "x": 580, "y": 0, "w": 290, "h": 80 } }, { "item": "viz_7lRWKreL", "type": "block", "position": { "x": 1150, "y": 10, "w": 290, "h": 80 } } ], "globalInputs": [ "input_global_trp" ] }, "description": "", "title": "Chain Test" } ... View more

I normally use requests when handling http or rest calls with python, as I find it really useful Requests has methods for deserialising json as well and options to configure proxies as well you might find this helpful? https://stackoverflow.com/questions/8287628/proxies-with-python-requests-module you can then handle your own exceptions based on the http response - the useless facts api is handy for testing if you want to avoid prod if you need to handle authentication on the proxy that may be more complex e.g. import requests def get_content(base, route, proxies): url = f'{base}/{route}' headers = { 'Content-Type': 'application/json' } response = requests.get(url, headers=headers, proxies=proxies) return response.json() def main(): base = "https://uselessfacts.jsph.pl" route = "api/v2/facts/random" http_proxy = "http://10.10.1.10:3128" https_proxy = "https://10.10.1.11:1080" ftp_proxy = "ftp://10.10.1.10:3128" proxies = { "http": http_proxy, "https": https_proxy, "ftp": ftp_proxy } response = get_content(base, route, proxies) print(response) if __name__ == "__main__": main()    ... View more

Im sure I am missing a fundamental point   ... View more

Hi thanks again for your attention I have reproduced this in my lab (using _internal) and I have noticed differences in the chaining of searching and how it is chained, despite the fact that when you click on the link it effectively concatenates and displays. I'll structure the findings into what I think is a more coherent way and post the results up later today.   ... View more

OK - that is a really good call - frustratingly I do get a working search even on the charts that show no data  I'll need to touch base with our cyber team to get a review done before I post anything, sorry I did create a test dashboard with  chart 1 = index=my_index    (shows data) chart 2 =   index=my_index (base)  timechart span=30m count(eventClass) by severity (chained search) (NO DATA) Really appreciate the time and effort spent here - I have used chained searches elsewhere, I'll check the docs again. ... View more

When looking at the job inspector there seems to be a massive difference (I am a novice at debugging this)  the normalizedSearch looks very different (due to chaining?).  Im unable to progress this further due to prestats command not being recognised - was hoping to recreate the search step by step to understand where this breaks so I can ask our splunk gurus a focused question  ... View more

Thanks for responding - when I run the search chained I get NULL, whereas when I run it in a single block, i get separation by severity field in severity.  (I've obfuscated the search a bit) Expected behaviour index=my_index | spath eventClass | search eventClass="my.event" | timechart count(eventClass) by severity Unexpected behaviour (displays graph, but without field separation showing "NULL" Chained Parent index=my_index Chained child  | spath eventClass | search eventClass="my.event" | timechart count(eventClass) by severity  What even more confusing is the graph on the dashboard, when view and clicked on to forward to the standard search and reporting, works.  So the division of the events seems to fail as it something has been lost, passing from parent to child. Thanks again for any time or attention given to this.  Each event is a JSON document logged via HEC, if that's important know. ... View more

if only it was supported in dashboard studio!  will use classic for the time being - thank you! ... View more

wow - I like that - looks perfect thank you for the ideas 😀 ... View more

I do agree with you and that was a first practical choice I was thinking of. There is a risk I may be over engineering an answer, but the goal would be to  get a simple view of "where the gaps are".   I find the more elaborate visualisations particularly helpful as they allow me to take dry technical topics and present to non technical people, who often control budgets, and I can automate the creation of data and feed into existing automated workflows, alerting, ticketing and so on; which for me results in the approach running itself so I can move on to something else.  This is really the my goal in practice. Really appreciate the quick feedback - starting simple sound very sensible! ... View more

Hi thanks - I have had to basically make this up, so the shape of the data below is broadly the same, but utterly fabricated to protect my client. 2019-07-29T05:29:26.762Z DEBUG <foo-5-thread-2> [BarAgent] The asynchronousNotification message is: <TERMINALRESPONSE> <SERVERDN>cn=00e5934f-12be-4cbf-8608-a75905dadbe7,ou=foo,dc=bar,dc=hyper,dc=out</SERVERDN> <SERVERPOOLDN>cn=foo-bar-agent,ou=server groups,dc=bar,dc=hyper,dc=out</SERVERPOOLDN> <SERVERDNSNAME>FOO-FB02Z-BAR019</SERVERDNSNAME> What is interesting it the data doesn't seem "clean" - that there is no closing tag for <foo-5-thread-2> (i.e. no </foo-5-thread> ; and in other instances there is - if you can provide an example, I can take this away and work on it, much appreciated. ... View more

couldnt see where to use that so I created two drop downs and put      Time Range Currently using Start Time input $earliest_time$ - $latest_time$   in the data definition I can link these two variables into all graphs - couldnt find the bog standard time picker  ... View more