About Rob

Rob · ‎10-13-2011

This is an issue specific to AIX causing the fsck utility to fail. If you do not wish to see this error you may wish to delete the meta.dirty file that is located in the $SPLUNK_HOME/var/lib/splunk/defaultdb/db/ directory. Keep in mind that this will only delete the file that was generated on an unclean shutdown and does not actually repair the buckets. To do so you may wish to run the fsck utility manually with the following command: $SPLUNK_HOME/bin/splunk cmd splunkd fsck --all --mode metadata --repair Otherwise, have a look at the following answer which describes how to validate the metadata files as well as the bucket tsidx files. http://splunk-base.splunk.com/answers/5374/how-to-quickly-validate-the-metadata-files-of-a-given-index-and-of-all-its-buckets

Rob · ‎04-13-2011

I am not quite sure what you are looking to do. But if you are looking to configure a Universal Forwarder to forward all data to 3 different indexers for specific ports then you will want to create a stanza for each indexer in your outputs.conf file like this: [tcpout] defaultGroup=* [tcpout:Nix] server = xxx.xxx.xxx.xxx:7700 [tcpout:inputA] server = xxx.xxx.xxx.xxx:7701 [tcpout:inputB] server = xxx.xxx.xxx.xxx:7702 If you want some additional information from the Splunk documentation, here is a link for cloning data across indexes and here is a link for forwarding data to indexes.

Rob · ‎04-13-2011

Hi I-Man Have you tried the following stanza in outputs.conf? [tcpout-server://splunk02...:9998] server = 10.255.4.213:514 sendCookedData=false I am just basing that off the documentation for forwarding data found here. I am not entirely certain how you want to send data but if its a subset of syslog data then you may be interested in the section near the end (found here)

Rob · ‎04-04-2011

You will probably want to route those events to the null queue. This means that you will want to configure Splunk to take specific events such as what you have posted, and it will not index these events. Here is another answer link that is similar: http://answers.splunk.com/questions/11617/route-unwanted-logs-to-a-null-queue Also, the Splunk documentation contains some great examples to get you started with filtering events out that you do not want. http://www.splunk.com/base/Documentation/4.2/Deploy/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

Rob · ‎04-04-2011

Hi Josh, You may want to create a log-local.cfg file in $SPLUNK_HOME/etc/ which includes a custom path for the following log configuration setting: appender.A1.fileName=/path/to/your/custom/file.log For more information, the following documentation link has some useful configuration settings for your log files. http://www.splunk.com/base/Documentation/4.2/Admin/SplunkLogFiles#log.cfg

Rob · ‎03-24-2011

Verified fixed in 4.2

Rob · ‎03-16-2011

What I am looking for is to determine what the maximum amount of virtual memory is that could be used if every queue were full in the universal forwarder. As I understand it for 4.1.x, that number would be 256MB on a 64bit system as there are 4 queues which can hold 1000 x 64KB chunks of event data. Only one queue can have the queue size be modified by outputs.conf (tcpout) whereas the rest can not be modified. I imagine there is a ceiling for the amount of virtual memory that the forwarder will request from the OS as otherwise it would be considered a memory leak.

Rob · ‎03-15-2011

How much memory can I expect the Universal Forwarder to require on a machine? Is there a hard ceiling for the virtual memory that a Universal forwarder will ask to be allocated from the OS? If so, can this value be configured?

Rob · ‎02-18-2011

Is your file named 'input.conf' or 'inputs.conf'?

Rob · ‎02-11-2011

Have you tried taking a look at using either the "uniq" function with your search? http://www.splunk.com/base/Documentation/4.1.7/SearchReference/Uniq You could do something such as: log_level="ERROR" | uniq | cluster field=error showcount=true countfield=count | table, _time, count, error | sort _time| outputcsv errorhistory.csv

Rob · ‎01-25-2011

I can't be sure with a Solaris setup but I think that if you tar a file on the same system you want to deploy to and keep permissions then it should keep the permissions intact when extracting to the same type of system as long as that tar file had not been repackaged on another system. It would be great to find out how that works on Solaris though 🙂

Rob · ‎01-21-2011

The problem with file permissions and ownership is based on the interpreting file system. This means that if you transfer files from one file system to another that the receiving file system will usually not be able to read the "properties" of the file such ownership and permissions. To break it down further, if a file has specific permissions and ownership properties on a machine with FAT or NTFS files system and it is transferred to a machine with ext/reiser/hfs/(other *nix variants), then the file properties are typically lost. This same rule applies for a file transfer in the opposite direction (e.g. *nix to Windows) as well. This file system interpretation issue can sometimes be worked around using tar to keep permissions of files within a tar package intact. However, this would only allow for the files to keep the permissions intact if both the endpoints use a file system that can interpret the same file permission properties. This means that a tar package containing files with permissions from a *nix system will be maintained as long as the files are un-packaged on a *nix system. It doesn't matter if the tar file had been transferred from another file system. Here is an excerpt from the tar man page: -p (x mode only) Preserve file permissions. Attempt to restore the full permissions, including owner, file modes, fileflags and ACLs, if available, for each item extracted from the archive. By default, newly-created files are owned by the user running tar, the file mode is restored for newly-created regular files, and all other types of entries receive default permissions. If tar is being run by root, the default is to restore the owner unless the -o option is also specified. Theoretically, if you were to create a deployment package on Windows using tar to preserve the file permissions, then you could then possibly deploy that package from a *NIX machine as long as the files had not been extracted from the package and re-packaged. The same could be said for a *NIX package created on a *NIX machine but stored on a Windows deployment server. Then it would simply be a matter of deploying the correct package based on the target OS.

Rob · ‎01-17-2011

However, in reference to the EXTRACT syntax being used, here is an excerpt from the Splunk documentation found at http://www.splunk.com/base/Documentation/4.1.6/SearchReference/Extract Syntax: Description: A stanza that can be found in transforms.conf. This is used when props.conf did not explicitly cause an extraction for this source, sourcetype, or host.

Rob · ‎01-17-2011

Paolo, you are correct with the caret immediately after the bracket. However, since there is no closing quote, it means that everything after the beginning quote will be matched to the field that is to be mapped. Without an event line to customize this to, we are making more of a guess as to what the specific regex should be.

Rob · ‎01-14-2011

He won't need the '=' as he is not using any options for the extract command and apparently making a call to a stanza labeled 'snort_signature' in his transforms.conf. However, in case the extract syntax needs to be checked, here is the documentation reference http://www.splunk.com/base/Documentation/4.1.6/SearchReference/Extract

Rob · ‎01-14-2011

From the line you have listed your are going to match something like the following: plugin_id="1001" unrelatedInfoThatWillBeMatched somelog=" snortSignature anythingElse The parenthetical value will be: snortSignature anythingElse The [^,]+ part will match anything anything after your first match as it is essentially looking for anything that could be considered the start of a line. I don't know what your events look like but you might want to try the following regex, which I based on your own regex, to hopefully match what I think the event may contain. plugin_id="10\d[1,3,4-9]"|"11[0-2]\d"|"113[0-1]".*log="\s*(?P<snort_signature>.*)" This will the plugin_id match an event that looks somewhat like this: plugin_id="1001" irrelevantInfo somelog=" snort123" more irrelevantInfo The extracted value for snort_signature will be: snort123 Furthermore, the extraction is going to depend on what you have defined in your snort_signature stanza in your transforms.conf file. If you can provide an example line that you are trying to extract fields from then we can see about fine tuning the extraction regex to match what you are looking for.

Rob · ‎01-14-2011

If you could provide a sample event line for us and what timestamp splunk gives it versus what timestamp it should get then that may help us figure out a way to get the correct timestamp in there.

Rob · ‎01-13-2011

can you see which date is being used? Is it using the system time?

Rob · ‎01-12-2011

If you are looking to see a common subject across several IP address then you just need to include the subject as you would like to search for it. Taking your search example, have you tried the following? index=smtp "subject" | transaction qid keepevicted=true | table subject,ip If you want to search for multiple subjects with the IP addresses shown, you could separate the subjects by using the OR operator within the search string. index=smtp "subject1" OR "subject2" | transaction qid keepevicted=true | table subject,ip

Rob · ‎01-12-2011

I believe you may want to use something similar to the following search. host="hostName" | rex "(?P<IDLetter>[A-Z]).RECV..(?P<Requests>\\d+),.(?P<ErrorCount>\\d+)" | eval time=_time| transaction fields=IDLetter maxspan=60m | stats range(ErrorCount) range(Requests) range(time) by IDLetter The search will first break up the string by using regex to extract the fields we want to pay attention to followed by getting the UTC time stamp and then piping that to only show the events for the last 60min. Then its just a final pipe to the stats in order to work the math to find the differences between the error count, the requests and the time.

Rob · ‎01-12-2011

The following example events are indexed by Splunk: Dec 1 00:47:58 serverName data-collector[1234]: #A_RECV# 1234, 5 Dec 1 00:47:58 serverName data-collector[1234]: #B_RECV# 5678, 10 Dec 1 00:47:58 serverName data-collector[1234]: #C_RECV# 2587, 100 Dec 1 00:47:58 serverName data-collector[1234]: #D_RECV# 9852, 150 What I would like to do is for a search that can show a table including the host name (A_RECV, B_RECV, etc.) the number of requests (first number) and the number of errors (second number). Both the number of requests and the number of errors are incremental so therefore the search will need to show the difference between the number of requests from the previous event to the last event if there has been a change. An example of the events from host A might be: Dec 1 00:47:58 serverName data-collector[1234]: #A_RECV# 1234, 5 Dec 1 00:53:58 serverName data-collector[1234]: #A_RECV# 1240, 7 If so, then the table should show that Host A had 6 requests and 2 errors.

Rob · ‎01-12-2011

You might want to take a look at the following documentation links: http://www.splunk.com/base/Documentation/4.1.6/Admin/ConfigureTimestampRecognition http://www.splunk.com/base/Documentation/4.1.6/admin/ConfigurePositionalTimestampExtraction For the event you are showing above, you might want to add something similar such as the following stanza: [<source|sourcetype|host>] MAX_TIMESTAMP_LOOKAHEAD = 90 TIME_PREFIX = .+(?=\w{3} \d{2}, \d{4} \d{2}:\d{2}:\d{2}) TIME_FORMAT = %b %d, %Y %H:%M:%S The MAX_TIMESTAMP_LOOKAHEAD is the number of characters that Splunk should "skip" before it starts looking for a timestamp. 90 is the number I used above as your time stamp starts after 92 characters. This is something that could be different for different events so you may want to change that value accordingly. The regex for the TIME_PREFIX will essentially match anything before that time stamp format that is at the end of the string which should pretty much guarantee that the correct timestamp is found according to the event you provided.

Rob · ‎12-06-2010

I am assuming that you do not want to use the Splunk time extractor which extracts the logged timestamp and "normalizes" it according to time zones at all? In which case you will want to edit/create the props.conf file in $SPLUNK_HOME/etc/system/local/ with the following stanza: [default] DATETIME_CONFIG = NONE This will turn off all the time adjustments. You can also set the option to "CURRENT" which will insert the current system time or you can take a look at the datetime.xml file which Splunk uses for the conversion. There are some other options for configure time zones which you can find here. For information about Splunk timestamp recognition, you can check out the docs here.

Rob · ‎11-29-2010

You may want to take a look at the answer posted here. More specifically, you will want to take a look at the serverclass.conf file so that you can set up the source directory of the config files you want to deploy as well as the target directory. Lastly, you will need to configure the client to be sure to grab the config files. The documentation illustrates this quite nicely in the terms of apps but config files can be deployed in a similar manner.

Rob · ‎11-29-2010

There are several formats in which IPv6 can be displayed in your event log. You will want to use transforms.conf to find and parse these addresses. Here is a list of regex that matches the different forms. (The IPv4 address converted to IPv6 used in the examples below is 192.168.10.100 with a net mask of 255.255.255.0) Full IPv6 address: fe80:0000:0000:0000:0000:0000:c0a8:a64 Regex to match and return full address as $1: ([0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}) IPv6 drop leading zero's: fe80:0:0:0:0:0:c0a8:a64 Regex to match and return full address as $1 (yes, its the same as the above): ([0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}) IPv6 collapse multiple zero's: fe80::c0a8:a64 Regex to match collapsed zero groups. This will also work with collapsed zeros at the beginning of the address but not for single group addresses(e.g. '::1') and does not check for illegal IPv6 addresses (e.g. fe80::c0a8::a64): (:?:?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[[::]?[0-9A-Fa-f]{1,4}]?) To account for mixed IPv4 and IPv6 addresses, IPv6 allows for changing the last 4 bits to include the IPv4 address fe80:0000:0000:0000:0000:0000:c0a8:a64 would then be noted with the quad address at the end and become 'fe80:0000:0000:0000:0000:0000:192.168.10.100'. Full IPv6 with IPv4 quad: fe80:0000:0000:0000:0000:0000:192.168.10.100 Regex to match: ([0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:(?:\d{1,3}.){3}\d{1,3}) IPv6 dropping leading zero's with IPv4 quad: fe80:0:0:0:0:0:192.168.10.100 Regex to match (same as above): ([0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:(?:\d{1,3}.){3}\d{1,3}) IPv6 with collapsed zero's and IPv4 quad: fe80::192.168.10.100 Regex to match: (:?:?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[::]?[0-9A-Fa-f]{1,4}:?[[::]?[0-9A-Fa-f]{1,4}]?(?:\d{1,3}.){3}\d{1,3}) Depending on the IPv6 address type that you are seeing in your events, you may want to tailor the regex to fit your IPv6 addresses more specifically.

Posts	156
Solutions	31
Karma Given	211
Karma Received	192
Member Since	‎11-20-2010

Online Status	Offline
Date Last Visited	‎09-30-2020 01:01 PM

How to use EVAL in props.conf with a Multivalue fi...

when using ntpdate I see ERROR ProcessDispatchedSe...

Indexing performance has slowed to a stop

Windows app indicates "not set up" after upgrading...

Eval can not concatenate fields where there is a n...

How can I not index commented lines from my files?

What is the virtual memory footprint for Universal...

How to count the difference in a rolling number in...

Re: During startup recovery, error reading from pr...

Re: universal forwarder with more than one outputs...

Re: How do I Configure the Universal Forwarder to ...

Re: Audit logs added to Windows security events

Re: Universal Forwarder Log DIR

Re: LDAP Authentication error: Splunk failed to fe...

Re: What is the virtual memory footprint for Unive...

What is the virtual memory footprint for Universal...

Re: Host name can not be change

Re: Unique Errors for Last Week over All Time

Re: Application scripts not executable when deploy...

Re: Application scripts not executable when deploy...

Re: Help with extracting fields using numeric rang...

Re: Help with extracting fields using numeric rang...

Re: Help with extracting fields using numeric rang...

Re: Help with extracting fields using numeric rang...

Re: How to debug timestamp assignment issues?

Re: How to debug timestamp assignment issues?

Re: Difficult SMTP log search. Help!

Re: How to count the difference in a rolling numbe...

How to count the difference in a rolling number in...

Re: How to define a time format using regex?

Re: How can we use original timestamps with distri...

Re: Deployment server for new Indexers

Re: IPv6 addresses parsed properly?

Are you a member of the Splunk Community?