Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
- Find Answers
- :
- About emiller42
emiller42
Motivator
Member since:
08-11-2011
06-05-2020
Community Statistics
| Posts | 275 |
| Solutions | 58 |
| Karma Given | 102 |
| Karma Received | 331 |
| Member Since | 08-11-2011 |
Activity Feed
- Got Karma for Re: Dedup within a MV field. 07-08-2025 08:48 AM
- Got Karma for Re: Dedup within a MV field. 10-03-2024 12:17 PM
- Got Karma for Re: Part 1: How to extract a json portion of an event then use spath to extract key=value pairs. 07-25-2023 01:43 AM
- Got Karma for Re: Find last value of multivalue field (Split and mvindex). 05-19-2023 07:37 AM
- Got Karma for Re: Dedup within a MV field. 05-19-2023 06:30 AM
- Got Karma for Re: Find last value of multivalue field (Split and mvindex). 01-11-2023 12:10 AM
- Got Karma for Re: Find last value of multivalue field (Split and mvindex). 09-30-2022 04:07 PM
- Got Karma for Re: How do I create an alert to trigger when a transaction with 2 events is not complete?. 07-08-2022 03:31 PM
- Got Karma for Re: How do I create an alert to trigger when a transaction with 2 events is not complete?. 02-15-2022 02:27 AM
- Got Karma for Re: How to parse JSON log data?. 01-27-2022 11:35 PM
- Got Karma for Re: Find last value of multivalue field (Split and mvindex). 01-27-2022 07:05 PM
- Got Karma for Re: Custom parsing. 02-03-2021 10:11 AM
- Got Karma for Re: Dedup within a MV field. 01-22-2021 09:16 AM
- Got Karma for Re: Dedup within a MV field. 07-08-2020 10:44 PM
- Got Karma for Re: Union in Splunk like in MySQL. 06-24-2020 08:22 AM
- Karma Re: What is the purpose of the file conf.conf found in .../etc/system/default ? for dshpritz. 06-05-2020 12:48 AM
- Karma Re: What is the purpose of the file conf.conf found in .../etc/system/default ? for ddrillic. 06-05-2020 12:48 AM
- Karma Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0? for vliggio. 06-05-2020 12:48 AM
- Got Karma for Re: What does Splunk do when one index in an indexer has reached maximum capacity?. 06-05-2020 12:48 AM
- Got Karma for Re: What does Splunk do when one index in an indexer has reached maximum capacity?. 06-05-2020 12:48 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 10 | |||
| 15 | |||
| 1 | |||
| 0 | |||
| 5 | |||
| 1 | |||
| 0 | |||
| 4 | |||
| 1 | |||
| 2 |
02-13-2013
11:24 AM
You should be using == instead of =.
foo=something | REX "start(?<"name">.*)end" | EVAL NameColor=case(name==1,"red",name==2,"blue") | table _time NameColor
... View more
02-12-2013
01:26 PM
I did not find the above to work in my experience. Full regex is NOT compatible with [source::] stanzas in props.conf. According to the documentation, the only regex-like syntax valid is:
... recurses through directories until the match is met.
* matches anything but / 0 or more times.
| is equivalent to 'or'
( ) are used to limit scope of |.
see the following for details:
http://splunk-base.splunk.com/answers/71035/using-regex-in-source-stanzas-propsconf
... View more
02-12-2013
01:25 PM
5 Karma
After working with Splunk support and trying a whole lot of things, I have found a solution: whitelists in inputs. This allowed me to have monitor stanzas that technically overlapped, but the whitelisting would then narrow the targetting as needed. This meant I didn't have to muck with [source::...] stanzas at all.
Example:
[monitor://c:\foo\paapps\log\stp1pfsprf*.xml]
sourcetype = sourcetype_a
whitelist = .+?[/\\]stp1pfsprf\d+_\d+-\d+.xml
[monitor://c:\foo\paapps\log\*p.xml]
sourcetype = sourcetype_b
whitelist = .+?[/\\]\d+-\d+p.xml
[monitor://c:\foo\paapps\log\*.xml]
sourcetype = sourcetype_c
whitelist = .+?[/\\]\d+-\d+.xml
So even though the third monitor technically would capture the same files as the previous two, the whitelists clears up the collisions so that each monitor stanza only applies to the files it's intended to.
It also appears that the information in this answer is not correct. [source::...] stanzas can not use full regex, and are limited to the following:
When setting a [spec] stanza, you can use the following regex-type syntax:
... recurses through directories until the match is met.
* matches anything but / 0 or more times.
| is equivalent to 'or'
( ) are used to limit scope of |.
... View more
02-08-2013
03:52 PM
1 Karma
There are a couple ways to go about this:
1) Use the bucket command to get your time ranges instead of a time chart:
| bucket _time span=1h | stats count by _time field1 field2
2) join your two distinct fields into a single value you can timechart over
| eval combined_field=field1 + "_" + field2 | timechart count by combined_field
Both get you the data you want, just in different formats. Try them both and see what works best for you.
... View more
02-05-2013
05:48 PM
Have you looked at the fieldformat command?
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Fieldformat
... View more
02-05-2013
04:29 PM
those may not be real transactions if they only contain the repeated content. By default, a transaction will only span 1000 events. (maxevents= parameter) So if you have a transaction that has 3000 events between the start and end, it will display as three transactions. (And each will be truncated in the display to 500 lines)
When you remove the junk lines, then the transaction is no longer split because of the event count, and you end up with fewer transactions.
This is further supported by your observation that the transactions that disappeared do not show the beginning or end of the transaction. If the display truncates down to 500 lines, it'll be the first 500 so you should see a legitimate start to the transaction.
To make sure you're not losing anything, you can do a search of JUST your start/end criteria and see if there are indeed legitimate transactions being dropped. Another option is to run the transaction with keeporphans=true to retain any records that aren't getting collected into a transaction for some reason.
... View more
01-10-2013
01:19 PM
see my comment above for the explanation to your error. You can't have two TRANSFORMS lines in a single props.conf stanza.
... View more
01-10-2013
11:53 AM
A Linux instance of Splunk can't monitor Windows data directly. If you want to monitor those metrics (WMI, perfmon) you will need to install a forwarder on the Windows system that will capture the data and then forward it to the Linux Splunk server.
... View more
01-10-2013
11:45 AM
The reason for the typo is you have two TRANSFORMS configurations when you should only have one. Instead of the two lines you have there, use:
TRANSFORMS-changesourcetype = riverbed_steelhead, sourcetype_cisco_asa
If you need to add more transforms.conf stanzas to this source, you just append them to the line, comma separated.
... View more
01-08-2013
09:16 AM
1 Karma
The problem is I can't assign sourcetypes in inputs.conf due to the filenames. The wildcard filtering on inputs is much more restrictive, and prevents me from doing it there.
... View more
01-08-2013
08:08 AM
To accomplish this the second host in the chain needs to be a full version of splunk, not a Universal Forwarder. This is because the intermediary will be acting as an indexer to collect any data forwarded to it.
So on your application server, in it's Universal forwarder instance, you will want an outputs.conf with something like:
[tcpout]
server=logforwarder
Then, on the logforwarder machine, you will have a full splunk install, but it will also have a outputs.conf indicating where it should send it's data to:
[tcpout]
server=webpanel
logforwarder does not need an inputs.conf, as it's not monitoring any logs directly. It's simply accepting incoming data much like an indexer would. You would also want to have any other props.conf stanzas present that are relevant at index-time. (line breaking, timestamps, etc)
... View more
01-08-2013
07:41 AM
Can you be more specific in the roles each of these servers plays? Why isn't appserver forwarding directly to webpanel?
... View more
01-08-2013
07:23 AM
1 Karma
Are the multi-line events being broken into multiple events in Splunk?
Event 1:
2013-01-07 13:28:27,325 INFO (LoggerName) Something is wrong with Website http://foo.com/
Event 2:
Now follow a few important details.
Event 3:
A few more details.
Event 4:
2013-01-07 13:28:27,325 INFO (LoggerName) Another log entry
If so, you can fix that where it keeps the whole thing as one event.
In your props.conf, you want to add the following config to the appropriate stanza for this sourcetype:
LINE_BREAKER = ([\r\n]+)\d+-\d+-\d+\s\d+:\d+:\d+,\d+
SHOULD_LINEMERGE = false
This tells it not to automatically break events, and instead only break when it encounters a new line starting with a timestamp. (What the regex matches)
Then when new log lines are indexed, multi-line events will be kept as a single event, keeping the context you want:
Event 1:
2013-01-07 13:28:27,325 INFO (LoggerName) Something is wrong with Website http://foo.com/
Now follow a few important details.
A few more details.
Event 2:
2013-01-07 13:28:27,325 INFO (LoggerName) Another log entry
... View more
01-08-2013
06:55 AM
Whoops, transcriptions typo there. Those are actually .xml files, I just mistyped the filenames when providing examples. The conf stanzas are correct. Good eye! I fixed the original question.
... View more
01-07-2013
10:17 PM
Indexer is RHEL, forwarders are Windows. I did check for ^M characters in the props.conf, and found none. running btool shows all the appropriate configs as well.
Good idea though!
... View more
01-07-2013
12:13 PM
see the following link for an example of using transforms on syslog data:
http://splunk-base.splunk.com/answers/60972/split-syslog-input-into-multiple-indexes
... View more
01-07-2013
12:12 PM
That makes things more complicated. You're going to want to use a transform to get the appropriate host values out of the individual log lines, then host:: stanzas with the relevant TZ settings. However, I'm not sure if this can be done in one props.conf, or if a heavy forwarder is needed. (HF does the transform portion before forwarding to the indexer)
Can't syslog-ng include timezone in the timestamp of the logs? That would take care of things too. (Although that might not be something you control)
... View more
01-07-2013
11:50 AM
1 Karma
Were you trying to use a wildcard in the tag? (%2A = *)
[host=server1*]
test = enabled
That's not something you can do in tags. Each host=value pair must be explicitly defined. So if you have server10, server11, server12, then you need three separate tags defined.
[host=server10]
test = enabled
[host=server11]
test = enabled
[host=server12]
test = enabled
Yes, this sucks when you have a lot of servers.
... View more
01-07-2013
11:45 AM
1 Karma
So I'm interpreting this as two separate configurations:
Configuration 1:
Forwarder on external host, forwarding to Splunk indexer.
Splunk indexer has props.conf stanza for that host name with a TZ setting.
Configuration 2
Syslog on external host sending lines to indexer.
Indexer is monitoring syslog locally to index lines.
Same props.conf stanza as above.
In the second configuration, the props.conf stanza won't apply as it's specifying the remote host, while the syslog data is being monitored locally (and thus would have the indexer as the host)
In the second scenario, I would recommend using sourcetype or source stanzas to apply the TZ setting, not host. As setting it for the indexer would apply the TZ setting to everything indexed locally. (Including _internal logs) Instead, create a source stanza or sourcetype stanza. (Whichever is most appropriate) and use that.
[source::syslog]
TZ = UTC
if your source name is different, then change accordingly.
... View more
01-07-2013
11:29 AM
What are you seeing in tags.conf? Look in both the etc/apps/ /local folder and etc/users/ / /local.
... View more
01-07-2013
11:20 AM
1 Karma
You may need to add a MAX_DAYS_AGO line to your props.conf stanza. By default, Splunk will throw out any extracted timestamps more than 2000 days old. Your example is past that threshold, which is why it used index time.
from the props.conf spec
MAX_DAYS_AGO = integer
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.
... View more
01-07-2013
11:11 AM
1 Karma
I'm assuming you're using the GUI to set up data inputs. I've noticed that when indicating Select the Sourcetype "From List" that not all existing sourcetypes are available in the drop-down list. However, you can change the "From List" to "Manual" and then fill in the desired sourcetype in the provided field.
... View more
01-07-2013
11:03 AM
1 Karma
Hello!
I have some log files with dynamic naming that I'm having trouble matching with props.conf stanzas. Here are examples of the filenames:
c:\Foo\pa.log
c:\Foo\PaApps\log\11-2012.xml
(changes for each month)
c:\Foo\PaApps\log\11-2012p.xml
(changes for each month)
c:\Foo\PaApps\log\1e85358e-11a3-44c2-8f62-ffe88c035478.log
(This is an error log. New file generated with each error, and has a unique GUID filename)
c:\Foo\PaApps\log\ProofServerManager_11-2012.log
(changes for each month)
c:\Foo\PaApps\log\ProofServerMonitor_11-2012.log
(changes for each month)
c:\Foo\PaApps\stp1pfsprf001_11-2012.xml
(That host_month-year. All files are collected on one host, but come from several. Hence the naming)
Now, given the variability of the filenames, I don't think I have any hope of specifically targetting them in my monitor stanzas. So I'm not assigning sourcetype on the forwarders. Monitoring Stanza names are as follows:
[monitor://c:\Foo\PaApps\log\*.log]
[monitor://c:\Foo\PaApps\log\*.xml]
[monitor://c:\Foo\pa.log]
Now so far, so good. All appropriate logs are being monitored and sent to the indexer as expected. So the next step is to use [source::] stanzas in the props.conf file on the indexer to sort them out, apply sourcetypes, etc. This is where things go wrong.
Here are my source:: stanzas, and what they should match:
[source::...[/\\]\d+-\d+p.xml]
(Should match c:\Foo\PaApps\log\11-2012p.xml)
[source::...[/\\]\d+-\d+.xml]
(Should match c:\Foo\PaApps\log\11-2012.xml)
[source::...[/\\]ProofServerManager_\d+-\d+.log]
(Should match c:\Foo\PaApps\log\ProofServerManager_11-2012.log)
[source::...[/\\]ProofServerMonitor_\d+-\d+.log]
(Should match c:\Foo\PaApps\log\ProofServerMonitor_11-2012.log)
[source::...[/\\]stp1pfsprf\d+_\d+-\d+.xml]
(Should match c:\Foo\PaApps\stp1pfsprf001_11-2012.xml)
[source::...[/\\]\w+-\w+-\w+-\w+-\w+.log]
(Should match c:\Foo\PaApps\log\1e85358e-11a3-44c2-8f62-ffe88c035478.log)
[source::...[/\\]pa.log]
(Should match c:\Foo\pa.log)
See this answer for details on where I got the syntax for these stanzas.
Now, when I first set this up, I did so on a local instance of Splunk running on my desktop. I recreated the log file path, and populated it with sample files. (So it was functionally identical to the servers I would be monitoring) Then using the above inputs.conf and props.conf, I was able to properly index the files, and all props.conf stanzas applied appropriately.
However, when I push the configuration out to my indexer and forwarders, the props.conf stanzas are not being applied, and the indexer is essentially 'learning' the sources. Resulting in sourcetypes of 'xml-1', 'xml-2', etc.
Any ideas why this worked locally, but not on the server?
Thank you!
... View more
- Tags:
- props.conf
- regex
12-12-2012
10:11 AM
Can you provide sample events and the actual search you're running?
... View more
11-07-2012
03:59 PM
The field extraction UI just builds the transforms/props.conf stanzas for you. So it really depends on which is easier for you. I find this easier, as I bundle it into an app that is pushed to all my search heads via the deployment server.
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
06-05-2020
02:03 AM
|
Karma from
