ah sorry typo there in my search. Should be index=main auditSource=for-frontend auditType=FormSaved | eval epochevent=strptime(generatedAt, "%Y/%m/%d") | eval epoch1hago=relative_time(now(), "-1h@h" ) | table detail.referenceNumber epochevent epoch1hago But given you're getting no values epochevent my guess is that the generatedAt field is not actually UNIX time. Can you post some sample data? ... View more

If you're not getting events then either 1. The epochtime is being calculated incorrectly 2. They are getting excluded by the subsearch index=main auditSource=for-frontend auditType=FormSaved | eval epochevent=strptime(generatedAt, "%Y/%m/%d") | eval epoch1hago=relative_time(now(), "-1h@h" ) | table detail.referenceNumber epochevent epoch30daysago This search should tell you whats happening with the epochdates - if it looks correct then your search at the end is filtering them out. ... View more

~3 trillion events and counting 🙂 Our sourcetype naming is fairly good, but its fairly domain specific to the individual services being monitored, many of which are in house and so bewildering to new staff. This is a big company problem rather than a specific Splunk issue - I'm just wondering how other people have solved/failed to solve the problem ... View more

Assuming your events contain zip codes your search might look something like this: <event search> | stats <some aggregator> by zipcode | lookup ... | geostats ... (I have elided the lookup and geostats commands, as they are totally dependent on what your tables look like) By default geostats will group locations into buckets. You can change the resolution of with the binspanlong and binspanlat options to geostats. See http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/Geostats. There are also a few examples already in splunk answers. I you're using Splunk 6.3 you can make choropleths which are even better. You'll need a KMZ (a zipped KML) file with the boundaries of the area you're interested in, rather than a straight lat/long table. See the geom command http://docs.splunk.com/Documentation/Splunk/6.3.1/SearchReference/Geom ... View more

Thanks for that, but as I said the specific metadata is not something we have an issue with. What we're trying to create are more generalized catalog browsing tools. For example the command you suggestgenerates almost half a million rows, and there's only so much information you can have in a sourcetype name. Cheers, JP ... View more

Hi Chris, where epoch30daysago>=epochevent will only return events that are more than 30 days old. If you change the time picker to 1 hour but don't change this expression you will still only see events that are older than 30 days, so your events from the last hour will not display. If you want to test why not change the epoch30daysago eval to use -1h@h ? Also your subsearch is very broad. You might want to restrict the fields that get returned eg search NOT[search index=main auditSource=for-frontend auditType=FormSubmission | fields detail.referenceNumber] Remember you can always check what a subsearch does by looking aty the remoteSearch field in the job inspector ... View more

I don't know the official method, but you could try a hacky way. Put eval description="My description" into the search string, then in your script get the description out with echo $SPLUNK_ARG_2 | sed -e 's/.*description=\(.*\)/\1/' Assuming the search string comes out in ARG_2. See http://docs.splunk.com/Documentation/Splunk/6.3.1/Alert/Configuringscriptedalerts ... View more

I'm not sure of the differences with SplunkCloud. I would definitely get in touch with Splunk. ... View more

The official advice from splunk is to simply Use the auditd daemon on *nix systems and monitor output from the daemon. (See: http://docs.splunk.com/Documentation/Splunk/6.3.1/Data/Monitorchangestoyourfilesystem) Have a look a google for auditd, of have a look on splunk answers for how to monitor file system changes. It's a very common use case so there will doubtless be a common method. It should save you a lot of grief! ... View more

Have a look at the config options for maxDataSize (http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Indexesconf). You should have it set to auto_high_volume. Not 100% sure if thats the heart of the issue but that is probably something you should change. Have a read of this too: http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs ... View more

Cool. So check those splunkd logs (hopefully you have them available in index=_internal). Also I don't suppose you have any licensing issues processing that much data? ... View more

Can you put some sample data here? The closer to your original data the better ... View more

No problem mate. If that's working for you do you mind accepting the answer? Cheers ... View more

Have you seen: http://docs.splunk.com/Documentation/Splunk/6.3.0/Viz/ChartDisplayissues Your code looks right, can you post the full XML? it may be that your tag is in the wrong spot. Also put your search string if possible ... View more

Hard to now what the exact problem is, but I would start by checking the splunkd logs for the forwarder and indexer to see if you have hit your limits. Also when you say you have [thruput]=0, do you mean you're limits.conf is [thruput] maxKBps = 0 ? ... View more

The diff field is in seconds. The _indextime and _time fields are in unix epoch time format, the number of seconds since January 1970. When you subtract one from the other the result is a value expressed in seconds ... View more

Also you may want to look at the docs for coalesce +1 javiergn. After learning to use stats and chart correctly, I have never needed to use join. Remember there are no tables involved, so you shouldn't need joins. Your initial search should already contain all the data you need (The union? A SQL expert may be able to describe this better). If you don't know the full result set in advance (ie you're relying on field values) try a subsearch instead of join. ... View more

It looks ok to me - have you run $SPLUNK_HOME/bin/splunk cmd btool transforms list --debug and $SPLUNK_HOME/bin/splunk cmd btool props list --debug to make sure that the stanzas are being read correctly? ... View more

Ah yes I think the whitelist stanza doesn't support backslashes in windows - so no files were getting indexed. See http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Specifyinputpathswithwildcards and http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Whitelistorblacklistspecificincomingdata You should be able to put the whitelist in the monitor part. EG: [monitor://c:\program files\app\appserver\logs\*\communication\*.txt and drop the whitelist entry altogether ... View more

Just a note on #4. For your deployer, which doesn't do a whole lot resource wise, there is no problem having two splunk instances on one server. eg /opt/splunk-ES and /opt/splunk-non-ES. This may save you paying for two servers, but you will still have two separate deployer instances ... View more

run $SPLUNK_HOME/bin/splunk cmd btool props list --debug on your indexer, to make sure that the changes have been applied correctly. Also have you confirmed that the regex breaks your events properly using a regex checker as I suggested? The text in your files may not be identical to what you posted here, due to format changes. ... View more

If you want to have a common set of dashboards and saved searches, why not use a deployment server? See: http://docs.splunk.com/Documentation/Splunk/6.3.1511/Updating/Deploymentserverarchitecture If you want to replicate user created knowledge objects between splunk servers, you're better off using Search Head Clustering See: http://docs.splunk.com/Documentation/Splunk/6.3.1511/DistSearch/AboutSHC ... View more

It should indeed! You could use TIME_FORMAT and TIME_PREFIX instead of DATETIME_CONFIG. And as the timestamp will always be at the start of the line you can drop MAX_TIMESTAMP_LOOKAHEAD ... View more