Hi, I have the same problem with rotated logfiles. I'm using Universal Forwarder in version 6.4.5 to monitor a log file and it's rotated versions. There was a network outage and the UF was not able to send it's data for some time. In the meanwhile the logs were rotated and zipped. The files it never began to read were read fine after the Network problem was resolved - even the zipped ones. But the file it was reading at the beginning of the outage was only unzipped and then commented with "already read, so skipped". When I manually unpacked the file and put it in place the UF started reading where it stopped because of the outage. So I think UF is skipping the check of the seekCRC at seekAdress as mentioned here: https://docs.splunk.com/Documentation/Splunk/6.4.5/Data/HowLogFileRotationIsHandled Does anyone know, if this is resolved in any Version? ... View more

The "root directory of the index" is e.g. $SPLUNK_DB/defaultdb/db/ ($SPLUNK_DB/defaultdb/ will NOT work). With Splunk 7, meta.dirty is deleted from db/ upon restart but the index is not rebuilt. I found the following method on https://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html (dating back to 2013): 1) # splunk stop 2) # splunk clean eventdata -index main This sort of worked, except older data did not get re-indexed. My horizon shrunk from several days to about 5 hours. It ended up easier to remove the data sources (which were directories under surveillance anyway) and add them back in. ... View more

if i upgrade to 4.2.2, do I still need to run the rebuild/repair operations? ... View more

OK, I will keep it simple and pre-parse. ... View more

The multikv command is likely occurring prior to the lookup. Therefore, you can manually specify the lookup to occur after the multikv: source=top | multikv | lookup ... View more

If you have a user that exists in both the scripted authentication system and the Splunk authentication system, the Splunk authenticated user/password will take priority. An example of this situation might be the "admin" user. Note that in 4.1.x and later, Splunk will authenticate users in both systems. Similarly, if you are running LDAP auth and Splunk auth, the LDAP user's credentials will be used instead of Splunk auth. ... View more

I signed up just to be able to upvote this question, since it has been so helpful to me! ... View more

Chart generated nicely! Thanks for the help as I missed a few minor details as usual! ... View more

Recover padding is simply a line in the metadata file that gets created when Splunk needs to fix/recover metadata. Host values are an example of metadata, which are contained in a file called Hosts.data. You can simply ignore these names, although they can be a bit annoying when looking at your summary page. To fix the summary page, you can modify the default summary dashboard to include the following additional query terms for the metadata search: | search host!=*recover-padding* So, any time you use a metadata search you will need to append the above, similar to this complete search: | metadata type=hosts | search host!=*recover-padding* ... View more

Much obliged - nbc ... View more

A few new examples... Asynchronous search: $ curl -u admin:changeit -k https://localhost:8089/services/search/jobs -d search="search index=_internal" <?xml version="1.0" encoding="UTF-8"?> <response> <sid>1520569635.358</sid> </response> Fetching results: $ curl -G -u admin:changeit -k https://localhost:8089/services/search/jobs/1520569635.358/results -d output_mode=csv Synchronous search: $ curl -u admin:changeit -k https://localhost:8089/services/search/jobs/export -d output_mode=csv -d search="search index=_internal |head 10" Getting authentication token: $ curl -k https://localhost:8089/services/auth/login --data-urlencode username=admin --data-urlencode password=changeit <response> <sessionKey>lTsi0Gyhadou77kplKboa8_4DBsMbRB1gpu6sCEvIXIFotnMqNLOJyXQgCLdwM^uhDSRgxpfg_dG0gSbtRIkObpkWrbF2TisTo</sessionKey> </response> Running synchronous search with authentication token: $ curl -k -H "Authorization: Splunk lTsi0Gyhadou77kplKboa8_4DBsMbRB1gpu6sCEvIXIFotnMqNLOJyXQgCLdwM^uhDSRgxpfg_dG0gSbtRIkObpkWrbF2TisTo" \ https://localhost:8089/services/search/jobs/export \ -d output_mode=csv \ -d search="search index=_internal |head 10" ... View more

I think that's due to Splunk trying to match the 12 against the indexed words rather than the raw event: the _raw contains 12ms which is not segmented in two blocks, it has been indexed as a single term, being it a single word without any major/minor breaking character into it (ref: segmenters.conf) This instead could work (but would return more results than expected): sourcetype=ruby ruby_call_completed=12* because Splunk shoud try to find indexed tokens starting with 12 (so 12ms, 123ms, ... would be found in the index) On the other side, your second example: sourcetype=ruby | search ruby_call_completed=12 first acts on the indexed data matching sourcetype=ruby, then fields are extracted, THEN the secondary search is executed. I think this is due to the map-reduce paradigm: "map" is executed on the distributed servers, and it is just a search for matching events based on the precomputed index of the logs "reduce" extracts fields and applyes the secondary search, but this is only executed on the node where the search was first launched. However, that's only my two cents... Paolo ... View more

You can do this via configuration files or search-time "kv" (aka extract command). Specifically, for your situation you want to delimit based on the "=>" and ", ". You can use the extract command as follows: ... | extract pairdelim=", }{", kvdelim="=>", auto=f This will turn off auto extraction, break the key value pairs based on the =>, and break the pairs based on the "," whitespace, or either curly bracket. So your extracted fields would be: item1=food item2=drink item3=water ... View more

You can set frozenTimePeriodInSecs , but it does not necessarily guarantee that data will be removed when it reaches that age. What it does specify is that data may be rolled out when it hits that age, provided that everything in its data bucket is also aged enough to be rolled out. You can control to some degree by setting maxHotSpanSecs , but this setting can have significant impact on search performance and changing it probably requires changing other index configurations, and generally should not be done without official Splunk recommendations. ... View more

This is in the log. How do I get my data back? 10-25-2010 10:10:08.452 INFO databasePartitionPolicy - Moving db with id of 43: /opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_43 to warm: size exceeded: maxDataSize=104857600 bytes, bucketSize=106525084 bytes 10-25-2010 10:10:08.452 WARN databasePartitionPolicy - About to move db at /opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_43 to warm ... View more

Correct - https://docs.splunk.com/Documentation/Splunk/7.2.3/ReleaseNotes/RunningSplunkalongsideWindowsantivirusproducts ... View more

A simple and direct answer for this is to use "Format Visualization". I found this in the Search Tutorial manual (after looking in several other manuals, experimenting unsuccessfully in my case using the log() function with the timechart command; also found only way in Splunk Light Cloud to edit the formatting xml to be in the Dashboard feature, not right in Visualization). Working in Splunk Light Cloud, display results with Visualization tab; in the upper left there are three links - click the Format one: In that dialog, pick the Y-Axis button on the left and change the Scale from Linear to Log: ... View more

Thanks for the reply Simeon. I figured out the problem. Somehow my inputs.conf file got poplulated with a bunch of things that shouldn't have been in there, and missing what should have been in there. Once I got that fixed, the licensing information was OK. ... View more

For me worked, renaming the file locate in $Splunk_Home\etc\licenses\download-trial\enttrial.lic and restart Splunk services. ... View more

'randomly' is a little unfair. The OS will choose an ephemeral port number and use that. How the OS determines the ephemeral port is OS dependent and also is related to how many ephemeral ports have been used so far. On Linux the ephemeral port range is controlled by the sysctl net.ipv4.ip_local_port_range, and on Windows it's a registry setting. ... View more

the c_ip field contains the external IP addresses of the client upon connection. I would rather not post exact examples since they contain secure data. I can say however that I'm not getting any fields that contain lat,long for the ip addresses when doing: host=" " | geoip I do get client_lat,client_lon when doing: host=" " | lookup geoip clientip as c_ip | geonormalize This does not show any results on the map when in the Google Maps search. ... View more

Splunk will track the top 10 inputs based on source and host. To retrieve that information, you could run the following search: index=_internal source=*metrics.log* per_host_thruput | timechart sum(kb) by series To increase the number of tracked inputs, you can set that in your limits.conf file for metrics tracking. ... View more

Assuming your list of events is in chronological order and belongs to a single user, you can try this: *| delta _time as timeSpentOnPreviousPage | accum timeSpentOnPreviousPage as totalTime From your 2nd event on you will get for each event a timeSpentOnPreviousPage and totalTime field containing running time difference between events, and running total time, respectively. ... View more