Hey everyone, I am working on an issue right now and I'm running into a problem with my understanding of how splunk works. I have multiple sourcetypes, and each is going into its own index (the times when the individual sources are received is extremely staggered, and this is the recommendation from Splunk to take care of this). I need to correlate these four sources together. So, when I begin a query, it starts like this: index=prodcorr OR index=premed_bts OR index=premed_pbts|rename MSP_vqmcallid AS Call_ID_1 |rename PBTS_ORIG_SIP_CALL_ID AS Call_ID_2 |rename PBTS_TERM_SIP_CALL_ID AS Call_ID_3 |rename BTS_ORIG_SIP_CALL_ID AS Call_ID_3 |transaction Call_ID_1,Call_ID_2,Call_ID_3 maxspan=1d keepevicted=true The issue comes in with some of my sources. They may have multiple records with the same Call ID field, so I need to dedup them prior to the transaction command. Will running the dedup command on one sourcetype (call it sourcetype A) out of the group of different sourcetypes (sourcetype B, C and D) get rid of sourcetype B, C, and D? Right now I am having a heck of a time getting any type of correlation to work because of the multiple events for each of my sources. Sadly the call ID is the only way to reliably correlate the records. Further, is there a way to dedup by specifying a field, and then excluding all events where a certain field is null? For instance, event1 has a call id field of abc123 and a Call_Type field of Network. Event2 and event3 have a call id field of abc123, and a Call_Type field that is null. I'd want to keep event1, and ignore event2 and event3. I want to do this for each of the sourcetypes included in my search (so that's a total of four sourcetypes). Any suggestions would be appreciated. Hopefully I explained this well enough. ... View more

Hey everyone. I am trying to put together an application and need some ideas. Right now my situation involves taking in data from a variety of sources across an enterprise network. The problem is a number of the sources only have the ability to export data to a single location, and that has to be dedicated to billing. So, before I can get the data, I have to wait for the billing department to do their thing with it, and then export it so I can use it. I have one source feeding to me directly utilizing the universal forwarder. So in total I have one source in near real time, one source once every 24 hours, and the final source as late as 72-96 hours. To have a complete record of an event, I will need all of the sources. So for each of the sources, as they come in, I will periodically run a scheduled search to toss the contents I'm interested in into a summary index specific to that sourcetype. Then, whenever a user searches against our dashboard, it will correlate all of the records in the different summary indexes in real time. The fields from data sources we currently have will be filled in. The fields we don't have will be blank. This will be updated as more stuff comes in. I would love for there to be an automated way to do this, however I've been told that a scheduled search that outputs to a summary index can't backfill in the missing data; it just creates new events in the summary index. Is this correct? The issue is finding a way for this to be done via an API. Suppose I have a saved search which returns the most current data in real time. Can you take arguments via the REST API like you can using a dashboard??? ... View more

Hey everyone, I am trying to convert two different date fields into unix time to do some mathematical operations. One of my timestamps doesn't want to convert for some reason. Here it is: 20110527162327.013 The time format is YYYYMMDDHHmmss.sss. I pass it into convert like so: convert timeformat="%Y%m%d%H%M%S.%q" mktime(AS_Start_Time) But it still outputs the exact same input time. Anyone have any ideas? ... View more

Hey everyone. We have a number of different sources. Each needs to be correlated with the others using either an ingress or egress ID. For example, in our first source, we have an egress ID since its where the process starts. The next server has an ingress ID and an egress ID, and so on until the final device. All of our data sources are CSV files. First, what is the best way to correlate each of these sources? Right now I'm looking at doing something like this: search index=correlation sourcetype=source1 |eval ingress_id_for_source2=egress_call_id_from_source1 |join ingress_id_for_source2 [search index=correlation sourcetype=source2] Something like this. I know this isn't optimum (and its probably not correct). Any suggestions? Please note this will happen multiple times. Stats won't work since we actually need the data from individual events, not aggregations of it. I want to chain all of the fields together to get a single record of the user's action across all of the systems. ... View more