Thanks @ITWhisperer  ... View more

Hi @ITWhisperer ,   Thanks for the SPL but this actually changes the entire query results. Doesn't work as intended.   Regards, Pravin ... View more

Hi @ITWhisperer ,   Thanks for the response. The main objective was to remove the below line in the query | search h = hp742srv OR dell970srv OR dell428srv OR hp548srv OR dell429srv OR dell477srv OR dell433srv  I tried a query similar to your query earlier but couldn't get the desired results as the original query. Is there some other query technique I can use?   Regards, Pravin ... View more

Hi @richgalloway ,   Do you mean to say that I can directly delete the data from the DB and still retain all the data from the indexers? Won't this affect the data in the indexers in any way?   Regards, Pravin ... View more

Hi @gcusello ,   Thanks for the response. I could use this solution to remove data older than 8 months and then revert back to the old configurations. Even I have the same doubt as to why the search head has data when all the logs have been forwarded to the indexers.   Regards, Pravin ... View more

Hi @sylim_splunk ,   Won't this delete the entire data from the index in the cluster? I want to delete only the data present in the search head.   Regards, Pravin ... View more

Hi @ITWhisperer ,   Thanks for the help. I added the default global_time input and it worked.   Thanks, Pravin ... View more

Hi @ITWhisperer ,   I do have the default dataSources but didn't add them due to the restrictions on the number of characters. { "visualizations": { "viz_rEigUEIR": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_RsGANSf0" } }, "viz_zmabNRJy": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_zljEkcGY" } }, "viz_AOddcf4l": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_EIcGt3S9" } }, "viz_sAOkMNEW": { "type": "splunk.singlevalue", "options": {}, "dataSources": { "primary": "ds_Qac6WE2F" } }, "viz_EvhBsJbp": { "type": "splunk.column", "options": {}, "dataSources": { "primary": "ds_a0sVgSvn" } }, "viz_YSosdreZ": { "type": "splunk.fillergauge", "options": {}, "dataSources": { "primary": "ds_EmopXHob" } }, "viz_4hVtGw0o": { "type": "splunk.fillergauge", "options": {}, "dataSources": { "primary": "ds_by7ATPra" } }, "viz_eRVtRn07": { "type": "splunk.pie", "options": {}, "dataSources": { "primary": "ds_SqaTby3h" } }, "viz_KROrPVIl": { "type": "splunk.table", "options": {} } }, "dataSources": { "ds_RsGANSf0": { "type": "ds.search", "options": { "query": "| stats max(requestDurationMs) \n | appendpipe \n [ stats count \n | eval \"NoResults\"=\"No Requests Found\" \n | where count=0 \n | table \"NoResults\"]" }, "name": "Max Request Duration" }, "ds_zljEkcGY": { "type": "ds.search", "options": { "query": "| eval NEW_TIME = now() - _time \n| where NEW_TIME < 10 \n| stats count(requestDurationMs) \n| appendpipe \n [ stats count \n | eval \"NoResults\"=\"No Requests Found\" \n | where count=0 \n | table \"NoResults\"]" }, "name": "Requests in the last 10s" }, "ds_TzAuc20h_ds_RsGANSf0": { "type": "ds.search", "options": { "query": "| stats max(requestDurationMs) \n | appendpipe \n [ stats count \n | eval \"NoResults\"=\"No Requests Found\" \n | where count=0 \n | table \"NoResults\"]" }, "name": "Max Request Duration copy 1" }, "ds_EIcGt3S9": { "type": "ds.search", "options": { "query": "| stats count(requestDurationMs) as count \n| eval count=count \n| appendpipe \n [| eval \"NoResults\"=\"No Requests Found\" \n | where count=0 \n | table \"NoResults\"]" }, "name": "Request Frequency" }, "ds_Qac6WE2F": { "type": "ds.search", "options": { "query": "| search (returnCode=\"4*\" OR returnCode=\"5*\") \n| stats count as failedRequestNb \n| appendpipe \n [| eval \"NoResults\"=\"No Requests Found\" \n | where count=0 \n | table \"NoResults\"]" }, "name": "Search_1" }, "ds_a0sVgSvn": { "type": "ds.search", "options": { "query": "| table _time requestDurationMs" }, "name": "Search_2" }, "ds_EmopXHob": { "type": "ds.search", "options": { "query": "| eval NEW_TIME = now() - _time \n| where NEW_TIME < 10 \n| stats count(requestDurationMs) \n| appendpipe \n [ stats count \n | eval \"NoResults\"=\"No Requests Found\" \n | where count=0 \n | table \"NoResults\"]" }, "name": "Search_3" }, "ds_by7ATPra": { "type": "ds.search", "options": { "query": "| stats count(requestDurationMs) as count \n| eval count=count \n| appendpipe \n [| eval \"NoResults\"=\"No Requests Found\" \n | where count=0 \n | table \"NoResults\"]" }, "name": "Search_4" }, "ds_SqaTby3h": { "type": "ds.search", "options": { "query": "| stats count by returnCode" }, "name": "Search_5" }, "ds_ljNWYr7J": { "type": "ds.search", "options": { "query": "| mstats avg(\"mx.process.cpu.utilization\") as X WHERE index=\"murex_metrics\" span=10s BY \"mx.env\" | dedup mx.env | table mx.env" }, "name": "search_6" } }, "defaults": { "dataSources": { "ds.search": { "options": { "queryParameters": { "latest": "$global_time.latest$", "earliest": "$global_time.earliest$" } } } } }, "inputs": { "input_H05frgOO": { "options": { "items": [], "token": "host_token", "defaultValue": "" }, "title": "HOST", "type": "input.dropdown", "dataSources": { "primary": "ds_ljNWYr7J" } }, "input_ricfJKyq": { "options": { "items": [ { "label": "All", "value": "*" } ], "defaultValue": "*", "token": "dd_mJPWNpbV" }, "title": "Dropdown Input Title", "type": "input.dropdown" } }, "layout": { "type": "absolute", "options": { "display": "auto-scale" }, "structure": [ { "item": "viz_zmabNRJy", "type": "block", "position": { "x": 300, "y": 0, "w": 300, "h": 250 } }, { "item": "viz_rEigUEIR", "type": "block", "position": { "x": 0, "y": 0, "w": 300, "h": 250 } }, { "item": "viz_AOddcf4l", "type": "block", "position": { "x": 600, "y": 0, "w": 300, "h": 250 } }, { "item": "viz_sAOkMNEW", "type": "block", "position": { "x": 900, "y": 0, "w": 300, "h": 250 } }, { "item": "viz_EvhBsJbp", "type": "block", "position": { "x": 0, "y": 250, "w": 300, "h": 300 } }, { "item": "viz_YSosdreZ", "type": "block", "position": { "x": 300, "y": 250, "w": 300, "h": 300 } }, { "item": "viz_4hVtGw0o", "type": "block", "position": { "x": 600, "y": 250, "w": 300, "h": 300 } }, { "item": "viz_eRVtRn07", "type": "block", "position": { "x": 900, "y": 250, "w": 300, "h": 300 } }, { "item": "viz_KROrPVIl", "type": "block", "position": { "x": 0, "y": 550, "w": 300, "h": 300 } } ], "globalInputs": [ "input_H05frgOO", "input_ricfJKyq" ] }, "description": "", "title": "MX.3 HTTP REQUESTS" }   All I am trying is to change the HOST title to accept dynamic values from a query. Let me know if this helps you to figure out where I am going wrong.   Regards, Pravin     ... View more

Hi @PickleRick ,   I have checked for all the below conditions. We have a proper network setup with no internet issues and limits.conf has default settings. I doubted that the issue might be because of the skewed clock since we work in a different time zone than my original location. But all the servers had a common time zone so this is also not the case here. Since the forwarder reads data from different log files across folders, the lag that we find is the maximum lag for a particular sourcetype. For example: In the above image, the event was generated at 9:00 in the morning but it was indexed only at 13:00 which is almost 4 hours after the event was generated. For the same source, when the run the SPL after a few minutes for the same source as earlier, I notice a delay of only a few minutes. I don't understand if this is because of the load on the forwarder or some other issue. For the same source, two different events have different indexing times. This is the one that really confuses me. Could you please throw some light on this?   Regards, Pravin   ... View more

Thank you @PickleRick. I will check the different reasons listed by you and update this thread. ... View more

Both the times are available from the data.       ... View more

Hi @ITWhisperer ,   One is the index time and another one is the event time. | eval lag_sec=_indextime-_time The difference is calculated as mentioned above.   Regards, Pravin ... View more

Thanks, @isoutamo  for the link to the previous answer. But I tried an alternative where I developed a script to delete the existing file and then have it replaced with the file of the same name. The forwarder was now able to read the updated file and also the changes within it.   Regards, Pravin ... View more

Hi @VatsalJagani ,   The script is being managed by input.conf from the internal forwarder. There are a few more scripts and files being managed by the same forwarder which are working as usual but only this particular script doesn't work. Also, the command doesn't produce any results and shows 0 results found. | tstats count where index=_internal host="<your forwarder host>" by _indextime | eval _time=_indextime | timechart span=1h sum(count)   Thanks, Pravin ... View more