About lguinn2

lguinn2 · ‎02-05-2018

I am not sure how to include a name in the title, but there probably is a way... However, if you are creating a scheduled dashboard, you could have a panel in your dashboard like this: | makeresults | eval previous_month_begin = relative_time(now(),"-1mon@mon") | eval "Previous Month Start" = strftime(previous_month_begin, "%B %d, %Y") | eval previous_month_end = relative_time(now(),"@mon-1m") | eval "Previous Month End" = strftime(previous_month_end, "%B %d, %Y") | eval heading = 'Previous Month Start' " to " 'Previous Month End' | fields heading which makes a beautiful heading. You might want to show it as a single value panel at the top of the dashboard.

lguinn2 · ‎02-05-2018

All of the dashboards, saved searches, tags, eventtypes, etc. are collectively referred to as "knowledge objects" in Splunk. They are stored in various directories under $SPLUNK_HOME/etc on the search head(s) system - for system-level configurations, which should not normally include knowledge objects (but could) apps - contains a directory for each app, with its associated knowledge objects (which are generally shared) users - contains a directory for each user, containing the users' private knowledge objects If you are moving from/to a search head cluster, you need to carefully follow the directions in the Distributed Search manual. If you are using independent search heads (not clustered), then you should be able to copy the apps and users directories from one search head to another. Don't copy the system directory across, but do check to see what is in it.

lguinn2 · ‎02-05-2018

I think this error message comes up when one of the indexers does not have a receiving port set. In order for indexer discovery to work, all of the indexers must have a receiving port set - the master node collects this information and then supplies it to the forwarder. It looks like the forwarder connected to the master node correctly (your config files look fine) - but the master couldn't supply the requested information. The "hostport":"?" part of the message is what makes me believe that this is the problem.

lguinn2 · ‎02-05-2018

What kind of information are you putting into Splunk? is it Windows or Linux or...? Are both logins and logouts recorded? Can you give an example of what each event looks like? This sort of search is pretty easy to do, but the community needs a little more information in order to help you...

lguinn2 · ‎02-05-2018

By default, clicking on a chart will open a browser tab that displays the underlying events. If that is not what you want, you can customize the drill-down behavior by adding the chart to a dashboard. Within a dashboard, there are a number of ways that you can customize. The Splunk Dashboards Example app is free and shows excellent examples of drill-downs. I would install this app on a test machine or your personal copy of Splunk. It won't hurt anything, but it really doesn't belong in your production environment. You can download it here http://splunkbase.splunk.com/app/1603/ If you prefer to read the manual, the section on drill-downs appears here http://docs.splunk.com/Documentation/Splunk/latest/Viz/DrilldownIntro

lguinn2 · ‎02-05-2018

Try this; it may not be the most efficient search, but it should work host=server* "Catering" txnType=order | timechart span=1d dc(txnId) as "totalTxns" | append [ search host=server* "Catering" txnType=order issuerResponse="authorized" NOT ("avsMismatch" OR "cvvMismatch") | timechart span=1d dc(txnId) by Decision | rename APPROVE as "Accept", REJECT as "Decline", ERROR as "Error" ] | stats first(*) as * by _time | foreach Accept Decline Error [ eval <<FIELD>>_percentOfTotal=(round((<<FIELD>>/totalTxns)*100)."%") ] | rename _time as Date, Accept_percentOfTotal as "Accept %", Decline_percentOfTotal as "Decline %", Error_percentOfTotal as "Error %", totalTxns as "Total Txn" | table Date "Total Txn" Accept "Accept %" Decline "Decline %" Error "Error %"

lguinn2 · ‎02-05-2018

New answer: what if you want to send some information to Splunk, but not everything? Maybe you don't want the metrics, but you would like the errors, etc. from the splunkd.log In $SPLUNK/HOME/etc/system/local/inputs.conf , only disable the metrics log [monitor://$SPLUNK_HOME/var/log/splunk/metrics.log] disabled = true You can set the "log levels" on the forwarder by copying $SPLUNK_HOME/etc/log.cfg to $SPLUNK_HOME/etc/log-local.cfg Edit $SPLUNK_HOME/etc/log-local.cfg to customize the logging, but remember that these logs are a primary source for Splunk Monitoring Console. These edits will mostly affect the splunkd.log There are many log channels, and you don't need to reset all of them. Just change "INFO" to "WARN" on any categories where you want to reduce the messages. You can delete any lines that you want to leave at INFO level. The following channels should always be left at INFO level: category.TailingProcessor=INFO category.loader=INFO

lguinn2 · ‎02-05-2018

Never edit the files in the default directories. Even if it works, your changes will be overwritten when you update Splunk. The files in the corresponding local directories always override the default directories. This should still work in Splunk 7, but you are in the wrong directory. Do the same thing, but put it in $SPLUNK_HOME/etc/system/local (which is probably /opt/splunkforwarder/etc/system/local on a Linux box).

lguinn2 · ‎01-31-2018

@davpx has a point - you are trying to run way more searches than your Splunk environment can probably handle. But you don't have to throw out the app! Just turn off report acceleration. (And do fix the num_cpus as @davpx suggested.) Go to Settings >> Reports... and look for any searches that have a lightning bolt icon (because they are accelerated). Edit the search settings so the searches are no longer accelerated. And you might want to look to see if you have accelerated searches in other apps as well... Turning off acceleration doesn't break anything, it just stops all this background processing that speeds up the searches. In your environment, the background processing was a bad trade-off. Letting the searches run a little slower should be much better. (And you could leave acceleration on for any searches that you use very frequently.) Finally, if you have data model acceleration, you can turn that off as well. But if you are using the Splunk Enterprise Security app (ES), you need to be very careful about turning off any data model acceleration.

lguinn2 · ‎01-31-2018

It looks like you are not comparing the same searches. The splunk-system-user account is used internally by Splunk to accomplish a lot of background work, so it is doing a lot of searches (and possibly alerts) that are more complex than most user searches. Ad-hoc searches run by users, typically using the Splunk GUI, have a higher priority than background or scheduled searches. They are also typically less complex. If this doesn't answer the question, then can you post a specific search that was run by both users, and give the execution time statistics for both searches?

lguinn2 · ‎01-31-2018

I understand your question, but the search that you ran isn't really helpful. First, take a look at the actual source file on the Universal Forwarder - does the log file (or whatever it is) have repeated data? Second, take a look at the splunkd.log from the Universal Forwarder. You can do this by searching the _internal index ( index=_internal host=ufname sourcetype=splunkd ) or by simply browsing the SPLUNK_HOME/var/log/splunk/splunkd.log on the Universal Forwarder. The best place to look in the log is during the Splunk startup process; what does Splunk do when it starts to read the files? Finally, Splunk tracks its status for each file that it is monitoring in the internal "fishbucket." Splunk must be able to maintain this state information. If log files are not appended, but have random updates throughout their records, this will confuse Splunk and it may duplicate data. There are a few other reasons that Splunk might think a file has changed when it has not, but you should get hints from the splunkd.log - when Splunk starts reading at the beginning of a monitored file (for whatever reason), it will log that fact. Once you look at all this, you could post back with any specific messages that you find in the Splunk logs.

lguinn2 · ‎01-29-2018

In @somesoni2 's answer, he gives a transform that will look for key-value pairs in the data: REGEX = \"([^\"]+)\":\"([^\"]+)\" FORMAT = $1::$2 This is a heckuva regular expression, but Splunk interprets it as "for every data pair that is found, where the two values are surrounded by double-quotes and separated by a colon, take the first item as the field name and the second item as the field value. You could change his regular expression to add "and there could be an optional backslash in front of the double-quote" by changing the regular expression to: REGEX = \"([^\"]+)\\?\":\\?\"([^\"]+)\\?\" (I think!) Try it and see what happens. I like this better than the other regexes, because it can handle events that have different fields - it doesn't require that all events have the same fields or that the fields are named consistently; it just extracts what it finds in each event. The good thing is that field extractions happen at search time, so you can change them as often as you like without having to re-index the data...

lguinn2 · ‎01-29-2018

It is tough to write a regular expression for JSON or XML files, as the field names are built-in. Here are a couple of choices that might be easier than writing regexes: Option 1: Look at the great answer provided by @somesoni2 here https://answers.splunk.com/answers/489517/what-is-the-best-approach-for-a-search-time-extrac.html Option 2: Copy a sample of the data onto a test box. Bring the data into Splunk using the Add Data wizard so that you can use the Data Preview. In the preview, choose "json_no_timestamp" for the sourcetype. Then click the Save As button next to the sourcetype name - and create your own sourcetype. For this example, name it "detector." Tell Splunk where to save this sourcetype - this defines where you will find the resulting props.conf entries. For example, if you choose the app "Searching and Reporting," you will find your new sourcetype in SPLUNK_HOME/etc/apps/search/local/props.conf. It will be located in a stanza named "[detector]" Now, Splunk should be able to identify all the fields in your data properly. As long as you are in the Data Preview, you can open some of the menus on the left and tell Splunk where to find the timestamp, etc. These settings will also be stored in the "[detector]" stanza of props.conf. When you are happy with the results, copy the stanza into production. Option 3: Like option 2, but use "json" instead of "json_no_timestamp" for the base sourcetype. In some ways, this is the easiest method BUT: it creates "index time fields." This means that the data will take up more space on disk and may be slower to retrieve (how much disk and how much slower depends on the volume of data). So I would avoid this option in favor of the other ones. HTH

lguinn2 · ‎01-18-2018

To update the list of names in the Monitoring Console, you need to go to Monitoring Console >> Settings >> General Setup. You should see your "new" server in the list. You may need to edit its roles, but you might not need to edit anything. But ALWAYS click on Apply Changes. This will ensure that the Monitoring Console (MC) updates its configuration to the same settings that you see on the screen. When you enter the General Setup windows, the MC will show you all the servers that it finds, but the new servers (or settings) are not actually configured in the MC until you click Apply Changes.

lguinn2 · ‎01-18-2018

To update the list of names in the Monitoring Console, you need to go to Monitoring Console >> Settings >> General Setup. You should see your "new" server in the list. You may need to edit its roles, but you might not need to edit anything. But ALWAYS click on Apply Changes. This will ensure that the Monitoring Console (MC) updates its configuration to the same settings that you see on the screen. When you enter the General Setup windows, the MC will show you all the servers that it finds, but the new servers (or settings) are not actually configured in the MC until you click Apply Changes.

lguinn2 · ‎01-16-2018

This worked for my problem: Option 1: with a subsearch (same as before) index=web sourcetype=access_combined status<400 [ search index=web sourcetype=access_combined status>=400 | dedup clientip | fields clientip ] | stats sum(bytes) as bytes by clientip | stats avg(bytes) as avg_bytes, median(bytes) as median_bytes Option 2: using an eval to replace the subsearch index=web sourcetype=access_combined | eventstats count(eval(status>=400)) as Bad by clientip | where Bad > 0 AND status < 400 | chart sum(bytes) as bytes by clientip | stats avg(bytes) as avg_bytes, median(bytes) as median_bytes Thank you to both @DalJeanis and @sideview - you got me to rethink the problem and my solution...

lguinn2 · ‎01-16-2018

I think your comment is good - but don't both searches summarize by clientip and then average?

lguinn2 · ‎01-16-2018

Here are two searches, which I think are logically equivalent, yet they return different results in Splunk. Option 1: with a subsearch index=web sourcetype=access_combined status<400 [ search index=web sourcetype=access_combined status>=400 | dedup clientip | fields clientip ] | stats sum(bytes) as bytes by clientip | stats avg(bytes) as avg_bytes, median(bytes) as median_bytes Option 2: using an eval to replace the subsearch index=web sourcetype=access_combined | eval check=if(status>=400,"Bad","Okay") | chart sum(bytes) as bytes by clientip check | where Bad > 0 | stats avg(Okay) as avg_bytes, median(Okay) as median_bytes The concept of both searches is the same: Identify IPs that have had HTTP errors in the previous week, and summarize the number of bytes of "successful" traffic, average and median during that timeframe. (My real search is slightly different, but this illustrates the problem perfectly.) Successful traffic is defined as status<400, and HTTP errors are status >=400. I am using the standard access_combined sourcetype for this example, so clientip is the IP address that is connecting to the Apache server, status is the HTTP status code, and bytes is the number of bytes in the HTTP request. But the searches give slightly different results. For example, the second search gave an average of 251923.11538461538 while the first search gave an average of 42823.32638888889. I am sure this is something simple that I have overlooked, but I don't see it! I've even looked at the Search Job Inspector, but nothing shows up there either. The subsearch is not hitting any limits on execution time or number of results; the overall data set is fairly small. What have I missed? [Update: fixed a couple of typos, which of course I didn't see until I posted... and I had messed up the second search in a big way!]

lguinn2 · ‎01-03-2018

Check the permissions for the app. Users must have at least read permissions in order to see/use an app.

lguinn2 · ‎01-03-2018

The master node must be a license slave. It must also be in the same pool as the indexers in the cluster.

lguinn2 · ‎01-03-2018

Does this script run properly when used independently of Splunk? Does this script run properly when using the same account as Splunk? Splunk has its own copy of python which it will use to execute scripts. Does the Splunk python have the libraries that you need? Did you look at the search.log that was created when you tried to use this command?

lguinn2 · ‎01-03-2018

Here is the documentation for starting/stopping Splunk. http://docs.splunk.com/Documentation/Splunk/latest/Admin/StartSplunk

lguinn2 · ‎01-03-2018

There is no "communication" between data in Splunk. If you have a lookup table, you can use the inputlookup command (along with other commands) to combine the lookup table data with the events retrieved from indexes. However, this is a very general statement. There is no way to answer your question without more details. What are you trying to accomplish? Do you want to see events from your indexes that match the ip addresses in the lookup table? What data is in the lookup table? Is it just a list of ip addresses? What data is in the indexes? If you can answer these questions, and provide a bit of sample data, the community can help.

lguinn2 · ‎01-03-2018

This message means that you are attempting to run more searches at once than your Splunk server can handle. It isn't an error; eventually all the necessary searches will run - they just can't all run simultaneously. The maximum number of historical searches that you can run is determined by 2 things: the settings for your role, and the server maximum limit. You can change the maximum for your role, although since you are running as the admin, your max is probably already unlimited. It is more likely than you are hitting the maximum for your server. This maximum is set in limits.conf based on the number of cores on your Splunk server. It reflects the fact that each simultaneous search requires a core. While you could change the limit, that ultimately does not change the hardware resources of the server. In other words, that isn't going to actually help. I would just accept this for what it is: a warning that you are running a lot of searches at once. If that is truly problematic, then you need to add more hardware to your master node.

lguinn2 · ‎01-03-2018

You should input this file on a test machine somewhere using the Add Data wizard. The wizard will show you a preview of how the data will be parsed, and allow you to experiment with various props.conf settings, such as linebreaking and timestamping. For JSON inputs, you can use INDEXED_EXTRACTIONS if you want, or simply parse the file as plain text. INDEXED_EXTRACTIONS will consume more disk space and potentially lower the search performance, but it may be your best choice when the data format is variable. Option 1: no indexed extractions (recommended if it works) props.conf [yoursourcetypehere] TRUNCATE = 0 CHARSET = UTF-8 KV_MODE=JSON SHOULD_LINEMERGE=true BREAK_ONLY_BEFORE={"time":" TIME_PREFIX={"time":" Option 2: indexed extractions props.conf [yoursourcetypehere] TRUNCATE = 0 CHARSET = UTF-8 KV_MODE = none INDEXED_EXTRACTIONS=JSON TIMESTAMP_FIELDS = time Finally, you may need to make sure that each new event starts on a new line in the log file.

Posts	3879
Solutions	853
Karma Given	3
Karma Received	3212
Member Since	‎10-20-2011

Online Status	Offline
Date Last Visited	‎10-14-2025 07:47 AM

Subsearch vs. stats = different results. Why?

Manually rebalancing primaries on indexer cluster ...

How can I check for events from a host in a list o...

How do I remove a search head from an Indexer Clus...

What should be the maximum disk capacity per index...

What does this error message mean: "something ... ...

Best/optimum log file size?

Accuracy of metadata command in large environments...

Things to do when you first install Splunk?

Splunk 6 - Customize Search App Views

Re: How to display previous month's name in the su...

Re: How can we migrate dashboards and alerts from ...

Re: Why am I unable to get index discovery working...

Re: How to show the number of logins and time leng...

Re: How to view logs in a pie chart in each indivi...

Re: How to create two searches combined into one c...

Re: How do I configure Universal Forwarder to not ...

Re: How do I configure Universal Forwarder to not ...

Re: The maximum number of concurrent auto-summariz...

Re: Why is there a big difference in performance o...

Re: How to find the log duplicate reasons?

Re: How to get the type extracted in regex?

Re: How to get the type extracted in regex?

Re: Why does the Monitoring Console not update wit...

Re: Why does the Monitoring Console not update wit...

Re: Subsearch vs. stats = different results. Why?

Re: Subsearch vs. stats = different results. Why?

Subsearch vs. stats = different results. Why?

Re: Why is my default app not loading when Users l...

Re: Should the master node be a license slave?

Re: External search command exited unexpectedly wi...

Re: how to kill defunct pid without rebooting the ...

Re: How to search comunication between a inputlook...

Re: The maximum number of concurrent historical se...

Re: Need help extracting timestamp from unstructur...

Join the Conversation