While rex command in the search query will work, you should consider creating a field extraction so that the same regular expression need not to be written every time you need to fetch the same data from these events. Through field extractions the fields will be available at search time when you select specific index/sourcetype. Splunk can automatically generates generic regular expression in the interactive mode. However, you can also provide your own regular expression (which would be same as what you run in rex command). Interactive mode will allow allow you to review whether there are any unmatched events in the 1000 events listed by your search query. That way you test your regular expression across actual indexed data. You can also use regex101 website to test your regular expression for some of your sample log snippets. ... View more

You need to calculate delta for current bucket and previous one (provided your results are sorted by time). Logic: attached value for current row minus diffCount for current row will give you the attached value of previous row. Your base search | eval X=0.01 | delta attached as diffCount | eval forecast = (attached-diffCount)*X + (attached-diffCount) | table _time, attached, forecast I dont think you need a forecast value for the first row. However, if you need you can pipe the following code after delta command to set diffCount to 0 for the first row where it is null. eval diffCount=if(isnull(diffCount),0,diffCount) PS: Assuming your base search calculates value of factor X, I have hard-coded above as 0.01. Hopefully your base search will calculate the same. ... View more

You can definitely set token within selection event of the Pie chart. However, since the value is always going to be , I am not sure as to why you can not have a panel with hard-coded [fieldname]= and only set the $earliest$ and $latest$ tokens from the pie chart, which I believe you are already doing right now. <selection> <set token="allValues">*</set> </selection> ... View more

Since you want tokens to be available after zooming in a chart, I have edited my answer for the same. Please refer to Splunk 6.x Dashboard Examples App if you need more details or complete example. ... View more

One of the options could be to add data.Timestamp and TransactionName as new field using eval, then create transaction and print new field name. Provided data.Timestamp is string time and not epoc. If it is epoc then use strftime command after your base search. <your base search> | eval data.Timestamp=strftime(data.TimeStamp, "%Y-%m-%d %H:%M:%s.%3N) | eval TransactionTimeAndName= data.TimeStamp + " - " + TransactionName | fields - data.Timestamp TransactionName | transaction Device SessionID maxpause=300s nullstr="~" mvlist=t delim="','" | <your remaining search> Ideally you should use stats instead of transaction. Following is a sample of stats: stats count as eventcount values(TransactionTimeAndName) as "Transaction Time and Name" min(_time) as MinTime max(_time) as MaxTime by Device SessionID | eval duration=MaxTime-MinTime | ... View more

Splunk sends email message as plain-text. The content_type=html is for formatting of other contents set by Splunk like link to alerts, inline table with tabular formatting etc. This will in fact prevent malicious content to be embedded via message content as html. Having said that please try to send message with email address and web URL which should get auto formatted. Also \n should get formatted as new line. This might work in Outlook with Auto Formatting turned on, however, behavior for Desktop App and Web App may vary and would need testing. Try changing content_type between html and plain for triggering alerts and check in both Desktop and Web App. | sendemail content_type=html [email protected] subject="TEST" message="This is a test. \n Email To: [email protected] Link: http:\\localhost:8000 " Also ideally footer should be modified to add signature. However, directly no formatting can be done on footer and you will have to rely on modifying sendemail.py for complete control of the email. ... View more

Since Splunk sends email message as plain text, it can not have HTML content. However, if your email client has Auto Formatting turned on (for example in Microsoft Outlook Desktop App with Autoformat option turned on), then you can directly say your base search | sendemail content_type=html message="Splunk Alert \n Test Link: http:\\localhost:8000 .\n My Test Alert from email: [email protected] ." ... PS: Outlook Web App I had to change content_type from html to plain, so that the Web App auto formats the content and identify @ for email and http:// for hyperlink. Having said this, it is better to modify sendemail.py for custom formatting of email message. ... View more

By hostname do you imply Splunk extracted field host? If yes can you give some sample data with field names? ... View more

@martin_mueller. I also think above is the optimal search query for this particular scenario. Running dedup command on entire index for a year will be expensive as compared to use of stats command to get the oldest values of Transport and Time using earliest command. However, I also feel that base search should have TRANSPORT="*" AND device="*" added to filter out the events which do not have these two fields upfront as out further filters work on these two fields. Further since the query might run for one year period, it is an ideal candidate for Summary Indexing. Following should also work but it will be expensive query because of reverse and dedup command when compared to the answer suggested by @martin_mueller. PS: reverse command is still better than sort -_time when the search returns too many results. index=foo TRANSPORT="*" device="*" | reverse | dedup device | timechart span=1mon count by TRANSPORT On the same lines I would also like to add that the reason why Splunk returns the latest results first and then moves towards older logs is because the way data is indexed in various buckets. Most recent data is written in HOT bucket which Splunk reads first and fastest. Older data rolls over to WARM, COLD, FROZEN and eventually to THAWED buckets based on age of data. While FROZEN data is not searchable, the search query looks for WARM and eventually COLD buckets only for older data which is why they are returned afterwards. Refer to the following Splunk documentation of Splunk indexing: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/HowSplunkstoresindexes Having said this, in case you want to improve performance of your search running for a 1 year period, you should consider creation of summary index or using sistats instead of stats command which will run faster. ... View more

There are multiple ways of this correlation: transaction, append, appendcols, join, sub search and stats etc. Refer to following Splunk documentation for which Event Correlation to use under various circumstances: http://docs.splunk.com/Documentation/Splunk/latest/Search/Abouteventcorrelation Using stats command would be optimal for this scenario. Following is what the stats query might look like. index="index" OR index="index2" ip_adresses="*"| stats values(hostname) by ip_adresses If the IP address field names are different then you can use either eval or rename SPL command or create alias for index/sourcetype so that the field names for IP Address become same in both indexes and above search may work. For more accurate Splunk search field names for both indexes i.e. Index name (and sourcetypes), ip address field, host name field and some of their sample values will be required. ... View more

Please give an example so that I can make any correction if required. My intent was to state the fact that search command can be used to look through multivalued fields and equal to (=) and not equal to (!=) operators can be used for comparison with expected results. ... View more

Have you looked at Custom Layout Example: Dark in Splunk 6.x Dashboard Examples app? Following is a snippet from the example that they have used. /* EDIT PANELS */ .dashboard-panel input { background-color: #444; } .drag-handle a.delete-panel { background: #000 !important; } ... View more

Just before the timechart command have a table command to print the required values like <your base search> | table _time, id, status | timechart span=1h count(id) as Count by status This way when you click on timechart the drilldown will search for the required details in table. If you click on any event detail in the table it will drill down to the raw event. Edited to add tokens available on Zooming in a timechart Since you want to Drill Down the details of Selection made in chart when it is zoomed in you would need to code the selection event in the Simple XML. Following code from the timechart will extract the epoch start time and end time from the selection. Refer to Pan and Zoom chart Controls example in the Splunk 6.x Dashboard Examples App for details <selection> <set token="selection.earliest">$start$</set> <set token="selection.latest">$end$</set> <set token="start.count">$start.count$</set> <set token="end.count">$end.count$</set> </selection> ... View more

Assuming Field is the field extracted by your search which contains both ComputerName and IPAddress values separated by newline character. You can use replace command to add your own delimeter (in the following example \n is replaced with #. Finally you can use split and mvindex command to read the two fields as needed. <Your base Search> | eval Field= replace (Field,"\n","#") | eval Fields=split(Field,"#")| eval ComputerName=mvindex(Fields,0) | eval IPAddress=mvindex(Fields,1) | table Field ComputerName IPAddress ... View more

One of the crude options in our case would be to overwrite _time with field_time. Provided field_time is time stored in string format. PS: The time format below is assuming string date time string is in YYYY/MM/DD HH:MM:SS format. You can use your own time formatting based on your exiisting field_time values. | eval _time= strptime(field_time,"%Y/%m/%d %H:%M:S") | timechart ... If field_time contains epoch time and not string time then direct assignment should work: ** | eval _time=field_time | timechart **... Since identification of exact time for various event is most crucial for Splunk, ideally, _time should be parsed and identified directly during data ingestion for optimal performance and accurate results. Any modifications to _time field afterwards may lead to unwanted results and issues. ... View more

Can you give example from some sample data to explain? Like sample ThreadID and fieldX? If you have to group fields together and then work on the values of another field after joining, then stats might work better than transaction. ... View more

Trim function is ideally meant to remove spaces, that is when you just provide the input string to the trim function. In Splunk trim, ltrim and rtrim can accept two input parameters i.e. input string and trim character/s. However, you should understand that trim removes the fields only from beginning and/or the end of the input string and not from middle. This behavior of Trim is not specific to Splunk, it is expected across all languages/technologies. The output of the following trim command will be a##bc. eval x=trim("####a##bc###", "#") | table x Refer to trim eval function on Splunk documentation: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/CommonEvalFunctions ... View more

If your app name is myapp then following should be your image attribute in dashboard HTML panel <img src="/static/app/myapp/imagefilename.png" /> Refer to Image Overlay with Single Values example on the Splunk 6.x Dashboard Examples app. Adding a static file requires Splunk restart and also refresh of browser's history from cache. ... View more

If your Splunk App name is TestApp then following should be the path for static folder: /opt/splunk/etc/apps/TestApp/appserver/static Then you can access the contents through Splunk HTML panel elements, for eg: <html src="/static/app/TestApp/html_tags.html">Test HTML Link</html> Refer to HTML example on Splunk 6.x Dashboard Examples app for details on the same. ... View more

Can you perform the following on action field as well? eval finalCharacter=substr(action , 57,58) Also see if rex can be changed to the following: "\<Sql_Text\>(?<a_action>.*)\<\/Sql_Text\>" ... View more