Hey everyone, I am using a script (running via cron) to pull files from a remote server once a day. On that remote server, there is an export which runs once a day, and is then zipped. This is stored in a repository directory of all files ever created. I run this command to pull down the files: rsync -avz --exclude 'callhistoryexport_L*' [email protected]:FOLDER/callhistoryexport*.gz FOLDER/ Now, this should conserve all file information, including the modtime of the files. The script is set up because more devices will be added in the future, and they will all store their exports in this directory. The universal forwarder installed on the machine running this script is configured to monitor that directory. Now, after rsync runs, the splunk forwarder is re-sending everything to the indexers, and I can't figure out why. Rsync is preserving all timestamps, so that shouldn't be it. It's not redownloading anything, only the new file (verified in the logs from the script). So why is the forwarder resending everything? Is anyone doing anything similar to this? If so, how did you resolve this problem? Or is there any way to get the forwarder to stop forwarding this data? ... View more

Hey everyone, I am trying to figure out the most efficient way to get polled SNMP data into splunk. Strangely while there appears to be a JMX extension for Splunk, there is nothing for SNMP which is far more prevalent. I was considering writing a script to poll numerous servers using net-snmp, and then write to a flat text file, letting splunk sort out the results but that causes some issues with handling table data. Anyone who has tackled this, I would love to hear how you resolved it. Edit: Just to provide some information - the SNMP queries we are executing are looking, in some cases, at several hundred different OID's. There is no realistic way to manually script out every single thing I'd like to query. By far the preference would be some sort of app I could drop the MIB into place and go from there. ... View more

Greetings everyone. Right now I am working with a filetype which contains a compilation of events from 4 different sources. Three of those original sources have fairly precise timestamps (to milliseconds), whereas one does not - it is precise to the second level. So what you could see in the timestamp field is: Source 1: 20110101235959999 Source 2: 20110101235959999 Source 3: 20110101235959999 Source 4: 20110101235959 In my props.conf here are the settings: TIME_FORMAT=%Y%m%d%H%M%S%3N TIME_PREFIX=^([^,]*,){24} TZ=UTC MAX_TIMESTAMP_LOOKAHEAD=20 This data will be going into a heavy forwarder and I had hoped to send everything into the same index and same sourcetype. However when I run through the timestamp preview (using 4.3) it's saying that, for source 4, strptime() is unable to obtain the timestamp. Despite this message, it looks like it's still able to ascertain the timestamp. How is splunk doing that? Is splunk still using strptime up to the point where it hits %3N? Or is it failing back to trying to figure out the timestamp on its own, and guessing correctly? So, from here there are a few ways I could go - I could forego the TIME_FORMAT and let splunk figure it out, since it appears to be doing a pretty fine job of that - at least with my test data. I could also set up routing on the heavy forwarder to send the less precise source to another sourcetype. And finally I could just keep everyting the way it is, which covers 3 of the sources. For the sources where we have precision I need to keep it, since this is to help calculate latency and every ms is important. ... View more

I noticed that in the capacity planning guide, there is no mention of the capacity of a forwarder. Right now I am looking at sending a significant amount of data to two different forwarders. How much data can the forwarder handle? These are heavy forwarders, I know the guideline for an indexer is 100GB/day, but I can't find anything similar for forwarders. ... View more

Greetings everyone. I am trying to build a large search into a dashboard and getting the error shown in the title. Any ideas? It works just fine from the command line (other than the regex capture groups using the greater than/less than escape codes instead of symbols). ... | append [search (index=prod_msp sourcetype=msp-prod Call_ID_1!=unknown Call_ID_1!=NULL AND ($TN$ OR 1$TN$ OR +1$TN$)) OR (index=prodcorr sourcetype=as-cdr AS_Direction=Terminating Call_ID_1!=NULL Call_ID_2!=NULL AND ($TN$ OR 1$TN$ OR +1$TN$)) OR (index=premed_pbts PBTS_ORIG_SIP_CALL_ID!=NULL PBTS_TERM_SIP_CALL_ID!=NULL AND ($TN$ OR 1$TN$ OR +1$TN$)) OR (index=premed_sbc SBC_ORIG_SIP_CALL_ID!=NULL AND ($TN$ OR 1$TN$ OR +1$TN$))|eval CID1=Call_ID_1|eval CID2=coalesce(Call_ID_2,PBTS_TERM_SIP_CALL_ID)|eval CID3=coalesce(PBTS_ORIG_SIP_CALL_ID,SBC_TERM_SIP_CALL_ID)] ... As you can see, the last part of the append statement IS a subsearch, so I don't understand why splunk is complaining. ... View more

Greetings everyone. This is hopefully a pretty simple question - is there a way to "flatten" transactions? After it runs, you end up with a transaction which is a multiline event. What I'd like to do is flatten the lines into a single line of text. For items where there are multiple matching lines, compress them into a single line which is semicolon delimited or something. Any ideas? ... View more

Hey everyone. One of my sources has a field which repeats occasionally. I want to filter out any events where there is a repeat of the particular field - so basically only return unique values. Really all I'm trying to get is a count of the non-repeated fields. I've tried using stats to subtract the distinct count from the normal count, but that still includes the repeated items. Any ideas would really be appreciated. Thanks! To clarify, suppose your data set has a field called MYFIELD. Across 16 events, it has the following values: 1 1 1 1 1 2 3 4 5 5 6 7 8 9 9 9 Count would return 16. dcount would return 9. What I want is something that will return 6, counting only the fields which have no repeats. ... View more

Greetings everyone. Is there any way to modify _time's value for the sake of a single search? One of our sources has the time set 2 hours behind where it should be. We have to present data tomorrow, and it will take at least a week to re-index everything. Any ideas would be appreciated. ... View more

Hey everyone. Right now I am working with a transaction. I currently have two sources which I am trying to correlate based on a single field. The issue is that the single field's value occasionally repeats in one of the sources. So, after the transaction takes place, there are a number of transactions only showing events from one of the sources. We honestly don't care about those transactions - we're only looking for transactions where both source types are present. So is there a way to say something like this: sourcetype=AS_CDR OR sourcetype=MSP-PROD|dedup _raw|eval CID1=coalesce(AS_Call_ID,MSP_Call_ID)|transaction fields=CID1 maxspan=1m keepevicted=true|where eventcount>1 AND contains(AS_CDR) AND contains(MSP-PROD) We could do this with a join, but when we're correlating 4 different sources for millions of records, its simply too expensive. Any help would be greatly appreciated. ... View more

Hey everyone. I am trying to add a time-format to my props.conf file. The source is a csv file containing multiple timestamps for each event. I'm only interested in the first one. The timestamp looks like this in the event: 15:09:55.128 UTC AUG 31 2011 So in my props.conf I was planning to add something like this: %H:%M:%S.%q%Z%tUTC%t%b%t%d%t%Y Is there a more elegant way to handle the time zone being embedded in the middle? I can't find any reference in strptime() to dealing with time zones. Any help would be appreciated. I've tried letting splunk recognize the timestamp but it seems to be screwing up a bit. Thanks! ... View more