All Topics

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Topics

Hi all, I am running splunk enterprise 8.1.2 and have a real headscratcher of a memory leak issue on dashboards running DataTables 1st Dashboard has a memory leak that causes the 10g memory of a p... See more...
Hi all, I am running splunk enterprise 8.1.2 and have a real headscratcher of a memory leak issue on dashboards running DataTables 1st Dashboard has a memory leak that causes the 10g memory of a pc to be used up in about 3 days 2nd Dashboard appears to have no leakage. If I swap the xml between the dashboards the leak doesn't move with the xml but stays with the 1st dashboard and dashboard 2 stays leak free. The xml for both dashboards except for very minor differences is identical and both dashboards use the same js scripts to generate the DataTables. If I create a brand new dashboard and copy in the "leaky" xml and save it the dashboard leaks memory but if I create another brand new dashboard and copy in the "good" xml originally from dashboard 2 there is no memory leak  "Good xml"   <form script="index.js" stylesheet="style.css" hideFilters="true"> <label>Alert Overview</label> <search id="dataTableSearch"> <query>| inputlookup noc_alerts_kv |search State=New OR State=Closing $OpenState$ | search $datasource$ | search $severity$ | eval AlertText = if(isnull(AlertText), Object, Hostname + " " + AlertText) | search AlertText = $textsearch$ | eval Key = _key | table Key DateTime Hostname IP_Address SnowID Severity Datasource State CustomerID AlertText Object |$sort$</query> <earliest>-24h@h</earliest> <latest>now</latest> <refresh>$refreshrate$</refresh> <sampleRatio>1</sampleRatio> </search> <fieldset autoRun="true" submitButton="true"> <input type="dropdown" token="font"> <label>Fontsize</label> <choice value="x-small">x-small</choice> <choice value="small">small</choice> <choice value="medium">medium</choice> <choice value="large">large</choice> <choice value="x-large">x-large</choice> <choice value="xx-large">xx-large</choice> <choice value="xxx-large">xxx-large</choice> <default>medium</default> <initialValue>medium</initialValue> </input> <input type="multiselect" token="fields"> <label>Show Columns</label> <default>AlertText,CustomerID,Datasource,DateTime,Hostname,IP_Address,Object,Severity,State,Key,SnowID</default> <initialValue>AlertText,CustomerID,Datasource,DateTime,Hostname,IP_Address,Object,State,Severity,Key,SnowID</initialValue> <fieldForLabel>Fields</fieldForLabel> <fieldForValue>Fields</fieldForValue> <search> <query>| inputlookup AlertFields.csv</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <delimiter> </delimiter> </input> <input type="dropdown" token="sort" searchWhenChanged="true"> <label>Default Sort</label> <default>sort - DateTime</default> <initialValue>sort - DateTime</initialValue> <fieldForLabel>SortField</fieldForLabel> <fieldForValue>SortQuery</fieldForValue> <search> <query>| inputlookup SortFields.csv</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> </input> <input type="dropdown" token="refreshrate"> <label>Refresh Rate</label> <choice value="10">10 secs</choice> <choice value="15">15 secs</choice> <choice value="20">20 secs</choice> <choice value="30">30 secs</choice> <choice value="45">45 secs</choice> <choice value="60">1 min</choice> <choice value="120">2 mins</choice> <choice value="300">5 mins</choice> <choice value="3600">Hourly</choice> <choice value="86400">Daily</choice> <default>60</default> <initialValue>60</initialValue> </input> <input type="multiselect" token="severity"> <label>Severity</label> <valuePrefix>Severity =</valuePrefix> <delimiter> OR </delimiter> <fieldForLabel>DropdownSev</fieldForLabel> <fieldForValue>DropdownSev</fieldForValue> <search> <query>| inputlookup DropdownSev.csv | dedup DropdownSev</query> <earliest>-15m</earliest> <latest>now</latest> </search> <default>*</default> <initialValue>*</initialValue> </input> <input type="multiselect" token="datasource"> <label>Sources</label> <valuePrefix>Datasource =</valuePrefix> <delimiter> OR </delimiter> <fieldForLabel>DTSource</fieldForLabel> <fieldForValue>DTSource</fieldForValue> <search> <query>| inputlookup DatasourceNOC.csv | dedup DTSource | sort DTSource</query> <earliest>-15m</earliest> <latest>now</latest> </search> <default>*</default> <initialValue>*</initialValue> </input> <input type="checkbox" token="OpenState"> <label></label> <choice value="*">Include Acknowledged</choice> <change> <condition label="Include Acknowledged"> <set token="OpenState">OR State=Acknowledged</set> </condition> <condition> <set token="OpenState"></set> </condition> </change> </input> <input type="text" token="textsearch" searchWhenChanged="false"> <label>Freeform Text</label> <default>*</default> <initialValue>*</initialValue> </input> </fieldset> <row> <panel> <html> <img src="/static/app/noc/images/xxx.jpg" style="width:359px; height:85px;"/> </html> </panel> </row> <row> <panel> <html> <div id="selectedrowcount"/> <button id="selectall" class="btn btn-primary">Select All</button> <button id="unselectall" class="btn btn-primary">Unselect All</button> <button id="acknow" class="btn btn-primary">Acknowledge</button> <button id="close" class="btn btn-primary">Close</button> <button id="submitInc" class="btn btn-primary">Add SNOW No.</button> <button id="raiseInc" class="btn btn-primary">Raise SNOW</button> <button id="getEventQuery" class="btn btn-primary">Get Event Data</button> <button id="clearEventQuery" class="btn btn-primary">Clear Event Data</button> <button id="openIPBrowser" class="btn btn-primary">Open in Browser</button> <button id="pingIP" class="btn btn-primary">Ping IP</button> <div id="tableWrapper"/> </html> </panel> </row> <row depends="$htmlevent$"> <panel> <html id="htmlevent"> <label>eventdate Stuff do not delete</label> <div id="eventdata"/> </html> </panel> </row> <row> <panel> <html id="snowfields"> <div id="snowcont"> <br/> <input type="radio" id="optinc" name="snowtype" value="Incident"/> <label for="incident">Incident</label> <input type="radio" id="optcase" name="snowtype" value="Case"/> <label for="case">Case</label> <label for="fsev">Severity:</label> <input type="text" id="sevfield" name="fsev"/> <label for="fshortdesc">Short Description:</label> <input type="text" id="shortdesc" name="fshortdesc"/> <label for="fdescription">Description:</label> <input type="text" id="descriptiontext" name="fdescription"/> <label for="fgroup">Group:</label> <select name="fgroup" id="assigngroup"> <option value="noc">Telecoms Network Operations Centre</option> <option value="support">Telecoms Network Operations Centre Support Admin</option> </select> <label for="fcat">Category:</label> <select name="fcat" id="cat"> <option value="none"/> <option value="datanetwork">Data Network</option> <option value="transmission">Transmission Network</option> <option value="infrastructure">Infrastructure</option> <option value="networkmanagement">Network Management</option> <option value="thirdparty">Third Party</option> <option value="itsystems">IT Systems</option> <option value="circuit">Circuit</option> <option value="security">Security</option> <option value="sites">Sites</option> <option value="controlsystems">Control Systems</option> <option value="mobile">Mobile</option> <option value="video">Video</option> <option value="voicerec">Voice Recording</option> <option value="voice">Voice</option> <option value="cti">CTI</option> <option value="reporting">Reporting</option> </select> <label for="fsubcat">Sub Category:</label> <select name="fsubcat" id="subcat"> <option value="none"/> </select> <label for="fcasetype">Case Fault Type:</label> <select name="fcasetype" id="casefaulttype"> <option value="none">--None--</option> <option value="servicedown">Service Down</option> <option value="degradedserv">Degraded Service</option> <option value="rfo">RFO</option> <option value="doa">DOA</option> <option value="telephony">Telephony</option> </select> <label for="fsla">SLA:</label> <select name="fsla" id="sla"> </select> <label for="fcmdb">CMDB CI:</label> <input type="text" id="cmdbci" name="fcmdb"/> <br/> <button id="submitsnow" class="btn btn-primary">Submit</button> <button id="cancelsnow" class="btn btn-primary">Cancel</button> </div> </html> </panel> </row> </form>       "Leaky xml"   <form script="index.js" stylesheet="style.css" hideFilters="true"> <label>NOC Radio New Alarms</label> <search id="dataTableSearch"> <query>| inputlookup noc_alerts_kv | search State=New OR State=Closing $OpenState$ | search $datasource$ | search $severity$ | eval AlertText = if(isnull(AlertText), Object, Hostname + " " + AlertText) | search AlertText = $textsearch$ | eval Key = _key | table Key DateTime Hostname IP_Address SnowID Severity Datasource State CustomerID AlertText Object | $sort$</query> <earliest>-24h@h</earliest> <latest>now</latest> <refresh>$refreshrate$</refresh> <sampleRatio>1</sampleRatio> </search> <fieldset autoRun="true" submitButton="true"> <input type="dropdown" token="font"> <label>Fontsize</label> <choice value="x-small">x-small</choice> <choice value="small">small</choice> <choice value="medium">medium</choice> <choice value="large">large</choice> <choice value="x-large">x-large</choice> <choice value="xx-large">xx-large</choice> <choice value="xxx-large">xxx-large</choice> <default>medium</default> <initialValue>medium</initialValue> </input> <input type="multiselect" token="fields"> <label>Show Columns</label> <default>AlertText,Datasource,DateTime,SnowID</default> <initialValue>AlertText,Datasource,DateTime,SnowID</initialValue> <fieldForLabel>Fields</fieldForLabel> <fieldForValue>Fields</fieldForValue> <search> <query>| inputlookup AlertFields.csv</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <delimiter> </delimiter> </input> <input type="dropdown" token="sort" searchWhenChanged="true"> <label>Default Sort</label> <default>sort - DateTime</default> <initialValue>sort - DateTime</initialValue> <fieldForLabel>SortField</fieldForLabel> <fieldForValue>SortQuery</fieldForValue> <search> <query>| inputlookup SortFields.csv</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> </input> <input type="dropdown" token="refreshrate"> <label>Refresh Rate</label> <choice value="10">10 secs</choice> <choice value="15">15 secs</choice> <choice value="20">20 secs</choice> <choice value="30">30 secs</choice> <choice value="45">45 secs</choice> <choice value="60">1 min</choice> <choice value="120">2 mins</choice> <choice value="300">5 mins</choice> <choice value="3600">Hourly</choice> <choice value="86400">Daily</choice> <default>30</default> <initialValue>30</initialValue> </input> <input type="multiselect" token="severity"> <label>Severity</label> <valuePrefix>Severity =</valuePrefix> <delimiter> OR </delimiter> <fieldForLabel>DropdownSev</fieldForLabel> <fieldForValue>DropdownSev</fieldForValue> <search> <query>| inputlookup DropdownSev.csv | dedup DropdownSev</query> <earliest>-15m</earliest> <latest>now</latest> </search> <default>*</default> <initialValue>*</initialValue> </input> <input type="multiselect" token="datasource"> <label>Sources</label> <valuePrefix>Datasource =</valuePrefix> <delimiter> OR </delimiter> <fieldForLabel>DTSource</fieldForLabel> <fieldForValue>DTSource</fieldForValue> <search> <query>| inputlookup DatasourceNOC.csv | dedup DTSource | sort DTSource</query> <earliest>-15m</earliest> <latest>now</latest> </search> <default>MDSRadio,Siaeradio</default> <initialValue>Siaeradio,MDSRadio</initialValue> </input> <input type="checkbox" token="OpenState"> <label></label> <choice value="*">Include Acknowledged</choice> <change> <condition label="Include Acknowledged"> <set token="OpenState">OR State=Acknowledged</set> </condition> <condition> <set token="OpenState"></set> </condition> </change> <delimiter> </delimiter> </input> <input type="text" token="textsearch" searchWhenChanged="false"> <label>Freeform Text</label> <default>*</default> <initialValue>*</initialValue> </input> </fieldset> <row> <panel> <html> <img src="/static/app/noc/images/xxx.jpg" style="width:359px; height:85px;"/> </html> </panel> </row> <row> <panel> <html> <div id="tableWrapper"/> <div id="selectedrowcount"/> <button id="selectall" class="btn btn-primary">Select All</button> <button id="unselectall" class="btn btn-primary">Unselect All</button> <button id="acknow" class="btn btn-primary">Acknowledge</button> <button id="close" class="btn btn-primary">Close</button> <button id="submitInc" class="btn btn-primary">Add SNOW No.</button> <button id="raiseInc" class="btn btn-primary">Raise SNOW</button> <button id="getEventQuery" class="btn btn-primary">Get Event Data</button> <button id="clearEventQuery" class="btn btn-primary">Clear Event Data</button> <button id="openIPBrowser" class="btn btn-primary">Open in Browser</button> <button id="pingIP" class="btn btn-primary">Ping IP</button> </html> </panel> </row> <row> <panel> <html id="htmlevent"> <div id="eventdata"/> </html> </panel> </row> <row> <panel> <html id="snowfields"> <div id="snowcont"><br /> <input type="radio" id="optinc" name="snowtype" value="Incident" /> <label for="incident">Incident</label> <input type="radio" id="optcase" name="snowtype" value="Case" /> <label for="case">Case</label> <label for="fsev">Severity:</label> <input type="text" id="sevfield" name="fsev" /> <label for="fshortdesc">Short Description:</label> <input type="text" id="shortdesc" name="fshortdesc" /> <label for="fdescription">Description:</label> <input type="text" id="descriptiontext" name="fdescription" /> <label for="fgroup">Group:</label> <select name="fgroup" id="assigngroup"> <option value="noc">Telecoms Network Operations Centre</option> <option value="support">Telecoms Network Operations Centre Support Admin</option> </select> <label for="fcat">Category:</label> <select name="fcat" id="cat"> <option value="none"></option> <option value="datanetwork">Data Network</option> <option value="transmission">Transmission Network</option> <option value="infrastructure">Infrastructure</option> <option value="networkmanagement">Network Management</option> <option value="thirdparty">Third Party</option> <option value="itsystems">IT Systems</option> <option value="circuit">Circuit</option> <option value="security">Security</option> <option value="sites">Sites</option> <option value="controlsystems">Control Systems</option> <option value="mobile">Mobile</option> <option value="video">Video</option> <option value="voicerec">Voice Recording</option> <option value="voice">Voice</option> <option value="cti">CTI</option> <option value="reporting">Reporting</option> </select> <label for="fsubcat">Sub Category:</label> <select name="fsubcat" id="subcat"> <option value="none"></option> </select> <label for="fcasetype">Case Fault Type:</label> <select name="fcasetype" id="casefaulttype"> <option value="none">--None--</option> <option value="servicedown">Service Down</option> <option value="degradedserv">Degraded Service</option> <option value="rfo">RFO</option> <option value="doa">DOA</option> <option value="telephony">Telephony</option> </select> <label for="fsla">SLA:</label> <select name="fsla" id="sla"> </select> <label for="fcmdb">CMDB CI:</label> <input type="text" id="cmdbci" name="fcmdb" /><br /> <button id="submitsnow" class="btn btn-primary">Submit</button> <button id="cancelsnow" class="btn btn-primary">Cancel</button> </div> </html> </panel> </row> </form>   Anyone got any ideas?
Hi, I have Splunk DB Connect App with 2.4.0 version .I am upgrading my python to 3 version.  i want to upgrade it to 3.4.2 to make it compatible with python 3 and splunk enterprise version 7.3 and ... See more...
Hi, I have Splunk DB Connect App with 2.4.0 version .I am upgrading my python to 3 version.  i want to upgrade it to 3.4.2 to make it compatible with python 3 and splunk enterprise version 7.3 and above Do i need to upgrade the existing version 2.4.0 to latest or reinstall Splunk DB Connect App with 3.4.2 version. could you'll please suggest on how i proceed and what are the steps and precaution to be taken  
Ended of hard disk space in SHC-01. is it possible to send a piece of data to SHC-02 to free up space in SHC-01
Hello, Suppose I have raw records like this:   user=blabla,org_L1=12345,org_L2=777,department=7890 user=testtt,org_L1=34567,org_L2=999,department=8910 ...   And I would like to extract the recor... See more...
Hello, Suppose I have raw records like this:   user=blabla,org_L1=12345,org_L2=777,department=7890 user=testtt,org_L1=34567,org_L2=999,department=8910 ...   And I would like to extract the records based on the rules defined in a lookup:   where_condition,role org_L1=12345 AND org_L2=777,superuser org_L1=34567 OR org_L2=999,normaluser   Is it feasible in some way to introduce into the SPL statement a "where condition" based on the where_condition field defined in the lookup? searchmatch command could be used but it does not accept fields, it only accepts strings. Thanks a lot, Edoardo
How do I get the data re-indexed to same sourcetype which I deleted using the delete command. for eg.         lets say I used this query: index=demo sourcetype=db_demo| delete  now here correct me... See more...
How do I get the data re-indexed to same sourcetype which I deleted using the delete command. for eg.         lets say I used this query: index=demo sourcetype=db_demo| delete  now here correct me If i am wrong, my "db_demo" data is marked as deleted that it is unsearchable but it is not deleted from disk space. now my question is without cleaning my index, how can I re-index or you can say monitor again my " db_demo" without changing the sourcetype. I don't want to change sourcetype "db_demo" to something else. is there a way ?
Hi I am monitoring dir paths on a syslog server with a UF. I have a few sources with different formats under the same sourcetype. I decided to use a regex to differentiate sources and set their un... See more...
Hi I am monitoring dir paths on a syslog server with a UF. I have a few sources with different formats under the same sourcetype. I decided to use a regex to differentiate sources and set their unique formats.   For example I have these two sources... source 1 = /splunkdata/foobar/serverName/2021-04-06-bar.log source 2 =/splunkdata/foobar/serverName/2021-04-06-foo-bar.log I cannot use a "*" wildcard because it is not specific enough,  for example *-bar.log will pickup -foo-bar.log too. The date prefix changes with the log names so I tried this... source 1 = /splunkdata/foobar/serverName/\d{4}-\d{2}-\d{2}-bar.log source 2 =/splunkdata/foobar/serverName/\d{4}-\d{2}-\d{2}-foo-bar.log but no luck...     IDK if I messed the regex or this is not possible...   Any ideas or examples appreciated. Thank you  
Hi, i have the free version of splunk.  I am being asked to forward the splunk logs to an enterprise siem.  I believe there is a forwarder license, is that true, and is that what is needed to be able... See more...
Hi, i have the free version of splunk.  I am being asked to forward the splunk logs to an enterprise siem.  I believe there is a forwarder license, is that true, and is that what is needed to be able to forward the collected logs?  No way to just forward with just the free version?
I have created 3 simple Splunk reports that I embed into a Confluence page using an iframe tag.  They render perfectly, but I noticed that each report has a 'splunk>' label at the bottom corner and I... See more...
I have created 3 simple Splunk reports that I embed into a Confluence page using an iframe tag.  They render perfectly, but I noticed that each report has a 'splunk>' label at the bottom corner and I wish to remove that if possible.   I have searched the community in numerous ways and have not yet uncovered a solution.  Does one exist? TIA
Hi,  I have: index=............|stats avg(test) by OrderNr Sub_OrderNr But I want to something like this: OrderNr       Sub_OrderNr_1   Sub_OrderNr_2 xxxxxx           avg(test)              ... See more...
Hi,  I have: index=............|stats avg(test) by OrderNr Sub_OrderNr But I want to something like this: OrderNr       Sub_OrderNr_1   Sub_OrderNr_2 xxxxxx           avg(test)                avg(test)   How shall I change the search?  Thank you for your help!    
Hi all,  I have got this SPL to perform what I was looking for but want to know if there is any more elegant way of achieving the same, possibly using just one stats command: index=index_1 | sta... See more...
Hi all,  I have got this SPL to perform what I was looking for but want to know if there is any more elegant way of achieving the same, possibly using just one stats command: index=index_1 | stats count(App) AS ACNT by Department | stats sum(ACNT) as "Application Count" by Department | sort Department | appendcols [search index=index_1 | dedup App | stats list(App) as Applications by Department | sort Department] | appendcols [search index=index_1 | dedup Developer | stats count(Developer) as "Developer Count" by Department | sort Department] | rename Department as Department | table Department "Application Count" Applications "Developer Count" Expected output in attach Thanks  
I am quite new to the Splunk currently Working on getting data from S3 file into Splunk. File Constraints -> 1) File will be replaced daily with updated file having previous and new data. 2) there... See more...
I am quite new to the Splunk currently Working on getting data from S3 file into Splunk. File Constraints -> 1) File will be replaced daily with updated file having previous and new data. 2) there will be field with - timestamp that can be used to find out which rows are new or updated.   Is it possible to configure splunk to get only new data from that file on daily basis. What configuration needs to be updated.  
I'm tasked with auditing syslog messages from some network devices for suspicious activity. I can use the IN operator to extract the significant messages by message code. Since some of these message... See more...
I'm tasked with auditing syslog messages from some network devices for suspicious activity. I can use the IN operator to extract the significant messages by message code. Since some of these messages are legitimately generated by several management  servers, I want to remove those messages from the report. Blocking all the messages created by the management servers is excessive and could potentially create a security risk. I'm trying to restrict the filtering for these management server to the messages that are legitimate part of their operation. I can select the syslog messages of interest using the message code field and a list of message numbers using the "IN" operator. I wanted to use the same logic to exclude the management servers using the logic in the form of a "NOT  DEVICE_IP IN (192.168.1.10, 192.168.1.20, 192.168.1.30)" Unfortunately, it doesn't work. What am I missing? Is there an equivalent function that will allow me to exclude a list of IP's? 
Hi I would like to add an additional Threat Intelligence Feed to the collection of the Intelligence Downloads in Enterprise Security.  The Service-URL needs to have an authorization header to allow... See more...
Hi I would like to add an additional Threat Intelligence Feed to the collection of the Intelligence Downloads in Enterprise Security.  The Service-URL needs to have an authorization header to allow the HTTPS call. So far i can't find any options within Intelligence Download Settings to add a custom HTTP-header with authorization Information like an API key. Is there any option in the Standard configuration of Intelligence Downloads to add a custom HTTP-Header to the requests? Or is there another extended configuration possibility? Thx for feedback.
i can obtain my base search kpi's via the following rest search : | rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/kpi_base_search report_as=text but is it possible to list an... See more...
i can obtain my base search kpi's via the following rest search : | rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/kpi_base_search report_as=text but is it possible to list any adhoc searches in itsi ?
We have different levels of data flow coordinated by a set of saved searches. We divide them into three tiers where the data after 3rd tier is displayed on the dashboard. Everyday for 30-40 minutes... See more...
We have different levels of data flow coordinated by a set of saved searches. We divide them into three tiers where the data after 3rd tier is displayed on the dashboard. Everyday for 30-40 minutes, this data flow is interrupted. The internal logs show nothing out of the ordinary and data flow is stuck at tier 2 saved searches. There are no skipped saved searches as well during this time and the server health is at top notch as well.  The trend of time range of the issue increases daily i.e. first day issue happened between 4-4:45, next day 5-5:45 and so on. Yesterday this happened between 10PM -10:45PM IST. There is no network connectivity issue as well as data is coming in tier 1 index. Please check the attached image.  Thanks in advance.  
Hi, I have 2 servers with the same names and I have installed universal forwarder on both servers. In forwarder management I have an issue that sometimes I can see one server another time I can see ... See more...
Hi, I have 2 servers with the same names and I have installed universal forwarder on both servers. In forwarder management I have an issue that sometimes I can see one server another time I can see the other one. How Can I solve this problem to see both of them ? Can I change name of one of the hosts before sending logs to indexers? Thanks, Shohre
I have a log of the form "Associated integration for customer AAA is Integration{id=1865, clientID}, carrying out deactivate call while processing message success" I wanted to extract the AAA w... See more...
I have a log of the form "Associated integration for customer AAA is Integration{id=1865, clientID}, carrying out deactivate call while processing message success" I wanted to extract the AAA which is the customer name from this log 
I want to put a filter box for all columns in a table. I know I can put a text box for all columns separately but that is not a good option. I want to put a text box besides/down each column name whe... See more...
I want to put a filter box for all columns in a table. I know I can put a text box for all columns separately but that is not a good option. I want to put a text box besides/down each column name where if I enter a value then all rows gets filtered on that particular value. The default value will be "*" for all text boxes.
Hi,   I'm currently running Splunk 7.3.0 and have 32 indexes running in a single cluster with 2 peers. Indexes are being replicated across both peers.   Everything was working fine until we expe... See more...
Hi,   I'm currently running Splunk 7.3.0 and have 32 indexes running in a single cluster with 2 peers. Indexes are being replicated across both peers.   Everything was working fine until we experienced a network blip 12 days ago, now I've noticed that the Replication Factor is not being met because there are some buckets from this time period which don't match, an average of about 3 buckets.   I've tried to Roll, Resync and Delete these buckets via the GUI but each step fails.  When I check splunkd.log, it appears as if Splunk is automatically trying to recover from these Fix Up tasks but it keeps reporting that the bucket is still in flight so can't. 04-06-2021 08:07:39.618 +0100 INFO CMSlave - truncate request bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 current bid status=Complete 04-06-2021 08:07:39.618 +0100 INFO CMSlave - bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=Complete to=PendingDiscard for reason="schedule delete bucket" 04-06-2021 08:07:39.618 +0100 WARN CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.618 +0100 ERROR CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.618 +0100 INFO CMSlave - bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=PendingDiscard to=Complete for reason="failed to schedule delete bucket" 04-06-2021 08:07:39.618 +0100 ERROR ClusterSlaveBucketHandler - truncate bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 earliest=0 latest=0 err='bucket already in flight' 04-06-2021 08:07:39.618 +0100 INFO CMSlave - truncate request bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 current bid status=Complete 04-06-2021 08:07:39.619 +0100 INFO CMSlave - bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=Complete to=PendingDiscard for reason="schedule delete bucket" 04-06-2021 08:07:39.619 +0100 WARN CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.619 +0100 ERROR CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.619 +0100 INFO CMSlave - bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=PendingDiscard to=Complete for reason="failed to schedule delete bucket" 04-06-2021 08:07:39.619 +0100 ERROR ClusterSlaveBucketHandler - truncate bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 earliest=0 latest=0 err='bucket already in flight' 04-06-2021 08:07:39.620 +0100 INFO CMSlave - Received resync bucket request for bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bucketExists=1 04-06-2021 08:07:39.620 +0100 INFO CMSlave - Received resync bucket request for bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bucketExists=1 Because of this, the Generation ID is also increasing quite rapidly.  The status for all the buckets in question is stuck on 'PendingDiscard'. The same messages are appearing on the second node but with different bucket IDs.  The same ID's keep repeating every few seconds on both peers. Should I restart each peer one at a time in hope that the bucket status is released and the fix up jobs can run as normal? Do I need to restart the cluster master? Any advice is appreciated.   Thank you  
I have a lookup file with 3 fields - source, status, timestamp.  Timestamp is saved as per below: eval timestamp=strftime(_time,"%d%m%y %H:%M:%S") Sample data: ABC, 1, 20/03/21 04:45:46 ABC, 0, 2... See more...
I have a lookup file with 3 fields - source, status, timestamp.  Timestamp is saved as per below: eval timestamp=strftime(_time,"%d%m%y %H:%M:%S") Sample data: ABC, 1, 20/03/21 04:45:46 ABC, 0, 27/03/21 11:17:31 ABC, 1, 29/03/21 14:33:06 ABC, 0, 01/04/21 12:56:41 Search query I am using is - | inputlookup test.csv | sort -TIMESTAMP result as below: ABC, 1, 29/03/21 14:33:06 ABC, 0, 27/03/21 11:17:31 ABC, 1, 20/03/21 04:45:46 ABC, 0, 01/04/21 12:56:41 and when I use query - |inputlookup test.csv | sort TIMESTAMP ABC, 0, 01/04/21 12:56:41 ABC, 1, 20/03/21 04:45:46 ABC, 0, 27/03/21 11:17:31 ABC, 1, 29/03/21 14:33:06   This is weird because sort is happening just based on date! I am not even able to use eval on TIMESTAMP field(result is always empty). Have tried addinfo, where timestamp>now-xxx with no luck.