About emiller42

emiller42 · ‎09-26-2012

maxspan isn't a field, it's an argument for the transaction command. You're saying "I want to group events as a single transaction that starts with "FOO" and ends with "BAR" but only if the time difference between the two is one minute or less." Since there is no 'maxspan' field, it's not something you can reference later. However, when you transaction, it DOES create a new field called 'duration' which you can use later for evaluations. | transaction username maxspan=1m startswith=eval(request="type1") endswith=eval(request="type2") | timechart count should give you a chart of the number of transactions over time. If you want to group them by the transaction duration, I would recommend doing some bucketing: | transaction username maxspan=1m startswith=eval(request="type1") endswith=eval(request="type2") | bucket duration span=30 | timechart count by duration

emiller42 · ‎09-26-2012

streamstats might actually help here. Add the following to your search (Change fields as needed) | streamstats last(orderNumber) as last_order This will create a new field called last_order for each event which will have the last seen orderNumber up to the point of that event occurring. So something like this: _raw=orderNumber=123 last_order=123 _raw=some other message last_order=123 _raw=orderNumber=456 last_order=456 _raw=another message last_order=456 _raw=ERROR Message! last_order=456 Which will associate the last seen order number with the error message even though there's no explicit correlation. (Note, you can add a 'by' clause to the streamstats if you need to restrict the aggregation somehow, say by host) Streamstats Documentation

emiller42 · ‎09-26-2012

So two things you'll want to do: First, create a stanza in transforms.conf that uses regex to parse out your fields. (Below is an example, which is based on the one line you posted. It may need to be tweaked) [sourcetype_extraction] REGEX = (\w+)\s+(\d+)\s+(\w+)\s+(\d+)\s+(\w+)\s+(\d+/\d+/\d+\s\d+:\d+:\d+\.\d+)\s(.*) FORMAT = deviceId::$1 moduleId::$2 level::$3 id::$4 moduleName::$5 time::$6 message::$7 Then you'll want to create a stanza in your props.conf that applies the transform to your sourcetype. [sourcetype] REPORT-sourcetype = sourcetype_extraction All of this is applied at search time, so will apply to anything you've already indexed, and can be changed without losing anything. It's also worth noting that the timestamp should be getting extracted on index into the _time field, so you shouldn't have to explicitly pull it out. But it may be a good idea to do so anyway via the TIMEFORMAT setting in props.conf.

emiller42 · ‎09-19-2012

You might want to check the splunkd logs with: index=_internal sourcetype="splunkd" databasePartitionPolicy "Moving db with id" This shows when buckets are moved and why.

emiller42 · ‎09-17-2012

then you would need to filter down to just the 'email sent to' events before the pipe to stats/timechart using something like: sourcetype=FOO "email sent to" | ...

emiller42 · ‎09-17-2012

So for the first one, you simply need to do: sourcetype=FOO | stats count and restrict the time range to a single day. For the second, you can use: sourcetype=FOO | timechart span=1d count Which will make sure you get the zero values on days with no events. You can rename columns using count as "No of emails" or something similar.

emiller42 · ‎09-13-2012

I would change the way you're using whitelists/blacklists as your config is very permissive by default. For example, the global stanza whitelists everything that isn't explicitly blacklisted in a more specific stanza. In your serverclass stanzas, the filtertype=blacklist means that the default state is to match unless told otherwise. (So mistakes in your filters mean the app goes out to a wider than intended audience) I would use the following for your serverclass.conf [global] blacklist.0 = * [serverClass:testMachine1] filterType = whitelist whitelist.0 = client20_* stateOnClient = enabled restartSplunkd = true [serverClass:testMachine1:app:deploy_input1] [serverClass:testMachine1:app:deploy_output1] [serverClass:testMachine1:app:deploy_perfmon1] [serverClass:testMachine2] filterType = whitelist whitelist.0 = client23_* stateOnClient = enabled restartSplunkd = true [serverClass:testMachine2:app:deploy_input2] [serverClass:testMachine2:app:deploy_output2] [serverClass:testMachine2:app:deploy_perfmon2] This means that anything that talks to the deployment server starts out as blacklisted. Then the serverClass stanzas also say the default state is blacklisted unless they match the whitelist.0 filter. I also moved the stateOnClient and restartSplunkd lines to the serverClass as they're all defined the same. But you can keep them under the app stanzas if you want to specify them individually.

emiller42 · ‎08-14-2012

Good info, thanks!

emiller42 · ‎08-13-2012

Answering this one myself in case it helps someone else. I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template. Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host. Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

emiller42 · ‎08-13-2012

We recently added several hosts that would be forwarding data to our indexers. Since all the hosts were going to be cloned off of a template, we thought that to save time, we would simply install the forwarder on the template, give it the proper deploymentclient.conf. That way, when we cloned it, the clones would simply poll the deployment server and be up and running without any manual intervention. This seemed to work, as the clones did pull the appropriate application bundles from the deployment server. But we aren't seeing forwarded data on the indexer. Checking splunkd.log on the forwarders shows errors where the indexer would refuse the connection, but we can't see any reason why. All configs are correct, and there are no connectivity blocks between the forwarder and the indexer. Any ideas of what to check next?

emiller42 · ‎07-09-2012

I solved this by uninstalling and reinstalling the light forwarder. This didn't really shed any light on what the issue was, but it is functional now.

emiller42 · ‎07-09-2012

Doesn't look like that's the issue. That appears to limit the rate of event processing on the indexer side of things. Even if it could be a cause, it's defaulted to no limit.

emiller42 · ‎07-06-2012

Ran into something today that I can't seem to find much information on. I've got a single light forwarder instance that has stopped forwarding data. Checking metrics.log shows the following queueus are blocked: 07-06-2012 09:24:23.412 -0500 INFO Metrics - group=queue, name=aggqueue, blocked=true, max_size_kb=1024, current_size_kb=1023, current_size=2635, largest_size=2635, smallest_size=2635 07-06-2012 09:24:23.412 -0500 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1283, largest_size=1283, smallest_size=1283 07-06-2012 09:24:23.412 -0500 INFO Metrics - group=queue, name=typingqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1235, largest_size=1235, smallest_size=1235 I have several other identical forwarders that are not seeing these issues, and the indexer is showing no signs of strain. There aren't any connectivity issues between the forwarder and the indexer, nor any recent config changes. All the info I've found here regarding queue blockages at the indexer, so I get the impression this isn't a common issue. Any ideas on what to look for here? I'm at a loss as to why this particular instance is failing. Thanks!

emiller42 · ‎06-15-2012

What you're trying to do is essentially summary indexing. Basically, you take your daily scheduled search and instead of sending the output to display, you send it to a separate index. Then in three months, you run your output search against the summary index so it only has to deal with 90 datapoints. This is all configurable in the UI. Details are here

emiller42 · ‎06-15-2012

You can add whitelists/blacklists to your inputs.conf to filter out unwanted files: blacklist = \.(gz)$ Should filter out anything in the folder with a .gz extension. (Or you could just whitelist .log files to get the same result. Depends on what else is in there) http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Whitelistorblacklistspecificincomingdata

emiller42 · ‎06-12-2012

I would try this: Instead of stats, use timechart. That will retain the buckets with no results, which is the root of your problem. Then you can apply the rangemap, eval date_hour, and apply your final eval. Like the following: source="tcp:5543" Requester="uka*" hostname="L05236" earliest=@h latest=now | timechart span=1h count | rangemap field=count severe=0-0 elevated=1-50 default=low | eval date_hour=strftime(_time, "%H") | eval range=if(date_hour > 00 AND date_hour < 13 AND count > 11, "low",if(date_hour >= 14 AND date_hour < 24 AND count > 11, "severe", range))

emiller42 · ‎06-07-2012

Glad I could help! Do you mind accepting the answer so it shows up as such for others searching for this in the future? (Click the checkmark)

emiller42 · ‎06-06-2012

If you pipe your search above to | addtotals it will add a column with a summation of the row. I'm assuming this is what you're looking for?

emiller42 · ‎06-05-2012

The problem is that the data you're outputting isn't something that can be represented as a pie chart. A pie chart expects pairs of label:value, while you're giving it triads. (label_a, label_b, value) To make this work, you could just create a new field that's a concatenation of the 'Rule' and 'Rule_Compliance' fields, and then chart with that. Something like: source="hpna_source.csv" Rule="bootp" | eval new_field=Rule + " - " + Rule_Compliance | stats count by new_field Of course, since you're already narrowing results down to one Rule in the initial search, the above isn't really necessary, and you can just do the following: source="hpna_source.csv" Rule="bootp" | stats count by Rule_Compliance

emiller42 · ‎02-10-2012

I'm attempting to index the gc.log coming from a tomcat installation, and I can't seem to get it to linemerge properly. Here is a snippet from the log: 517045.370: [GC [PSYoungGen: 516944K->3097K(519296K)] 828767K->315076K(1830016K), 0.0065230 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 517076.460: [GC [PSYoungGen: 517337K->2116K(519232K)] 829316K->315460K(1829952K), 0.0184770 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] and my props.conf for this sourcetype: DATETIME_CONFIG = CURRENT LINE_BREAKER = ([\r\n]+)(?=\d+\.\d+:) SHOULD_LINEMERGE = FALSE I've also tried: DATETIME_CONFIG = CURRENT SHOULD_LINEMERGE = TRUE BREAK_ONLY_BEFORE = \d+\.\d+: MUST_NOT_BREAK_BEFORE = \[ Which did not help. I verified that the above props were being applied, and not overridden by any other configurations. The output is displaying as: 2/10/12 11:41:06.000 AM [Times: user=0.01 sys=0.00, real=0.01 secs] 2/10/12 11:40:50.000 AM 517076.460: [GC [PSYoungGen: 517337K->2116K(519232K)] 829316K->315460K(1829952K), 0.0184770 secs] So the second line of each entry is being recorded as a newline despite my props.conf config. My hunch here is that because the two lines are not being written to the log at the same time (There appears to be some delay between the two according to Splunk) and that I've set DATETIME_CONFIG = CURRENT it's not waiting for the second line to be written before committing the first as an event. This theory is strengthened by the fact that there is the occasional entry that is merged properly. Is there an appropriate attribute that can control the time Splunk will wait for more lines before considering the event 'done'? Thanks!

emiller42 · ‎01-31-2012

Actually answered this myself, but posting it all here in case someone else runs into this: As you can see from the code above, there are line breaks to aid readability. They did not cause any problems when running the search through the UI. If I scheduled the search, and used REST to access the results, there was no issue as well. However they did cause the search to fail if a new job using it was kicked off via REST. Removing the line breaks resolved the issue, and the search jobs now run appropriately when invoked via REST.

emiller42 · ‎01-31-2012

I have a rather complex saved search that functions perfectly when accessed via the UI. But when a job is kicked off via REST, it fails, giving the following error: "Error in 'eval' command: Failed to parse the provided arguments. Usage: eval dest_key = expression" Here is the search string itself: (sourcetype="livecycle:webserver:server" OR sourcetype="alfresco_cms:app:ds") tag::host="dev" nodeUUID=* | regex nodeUUID="[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}" | eval status=case(DS_PDFG=="doc_status_change",state, DS_PDFG!="doc_status_change",DS_PDFG, searchmatch("error OR fail"),"Error") | rex "(?msi)Conversion failed\s*:\s*(?<conversionFailed>.*?)nodeUUID" | rex "com.adobe.livecycle.assembler.client.ProcessingException message:(?<processingException>.*)nodeUUID" | rex "javax.resource.ResourceException\s*:\s*(?<resourceException>.*)" | rex "javax.naming.NameNotFoundException\s*:\s*(?<nameNotFound>.*)" | rex "\{\"exceptionType\":(?<exceptionType>.*)\}" | rex "\'Error\':\s*(?<error>.*)nodeUUID" | stats first(_time) as time, first(status) as status, first(conversionFailed) as conversionFailed, first(processingException) as processingException, first(resourceException) as resourceException, first(nameNotFound) as nameNotFound, first(exceptionType) as exceptionType, first(error) as error by nodeUUID | eval reason=coalesce(conversionFailed,resourceException,processingException, nameNotFound, exceptionType,error) | table time nodeUUID status reason | where status="Error" | sort -time | convert ctime(time)

emiller42 · ‎12-28-2011

Updated my response with proper casing. Glad it helped!

emiller42 · ‎12-28-2011

Try something like this inside your hidden search module. <module name="HiddenChartFormatter"> <param name="charting.chart">radialGauge</param> <param name="charting.chart.rangeValues">[0,5000.10000.15000.20000]</param> <module name="FlashChart" /> </module>

emiller42 · ‎11-08-2011

Hello! I'm trying to export a subset of logs indexed on one indexer, and then import them into another. I'm attempting to use the cli export tool to do this, and am running into issues. If I run the following: ./splunk export eventdata -index main -dir /tmp/export then I get a successful export of everything that has been indexed by the server. Unfortunately, this is far more data than I actually want to export. To try and narrow it down, I'm using further export flags, but they don't appear to be working at all. I'm trying to get a specific set of log files from specific hosts. Using commands like the following: ./splunk export eventdata -index main -dir /tmp/export -host HOSTNAME ./splunk export eventdata -index main -dir /tmp/export -source LOGFILEPATH I simply get nothing exported. I've verified that the host name and logfile info is correct, so I'm at a loss as to what is causing it to return nothing. I am assuming that the -host flag is used to denote the forwarder that the logs originated from, and that the -source is the full path of the logfile. (Ex: 'D:\apache-tomcat-6.0.32\bin\server.log'. I have tried it both escaped and not) Has anyone else run into this issue? Thanks!

Posts	275
Solutions	58
Karma Given	102
Karma Received	331
Member Since	‎08-11-2011

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

Why does my Search Head Cluster Captain start dele...

Why do scheduled searches randomly stop running in...

Changes to forwarder->indexer connectivity in 6.3....

Why does the Splunk Add-on for Cisco UCS not respe...

How to disable the schedule_rtsearch capability?

Fire Brigade Install on Distributed Environment

Summary Index -nolocal

Problem using multikv to parse tabular data

streamstats is reversed?

Inputs.conf - wildcard monitor stanzas on Windows

Re: Find requests hits within span of time

Re: Correlate field from earlier event with error

Re: Custom parsing

Re: How to verify that data is retained for 200 da...

Re: Logging number of events based on log file

Re: Logging number of events based on log file

Re: Question about Serverclass.conf ?

Re: Issue with light forwarder on cloned hosts

Re: Issue with light forwarder on cloned hosts

Issue with light forwarder on cloned hosts

Re: queue blocks on light forwarder

Re: queue blocks on light forwarder

queue blocks on light forwarder

Re: Graphing scheduled saved search results

Re: Log rotation for compressed files

Re: search : no result / single

Re: chart : Total

Re: chart : Total

Re: pie chart problems

Unable to eliminate line break in gc.log

Re: Saved Search working in UI, but errors when ca...

Saved Search working in UI, but errors when called...

Re: Advanced XML context for radialGauge formattin...

Re: Advanced XML context for radialGauge formattin...

Unable to filter CLI export

Join the Conversation