Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About ahartge
ahartge
Path Finder
Member since:
12-28-2011
06-08-2022
Community Statistics
| Posts | 30 |
| Solutions | 1 |
| Karma Given | 1 |
| Karma Received | 13 |
| Member Since | 12-28-2011 |
Activity Feed
- Got Karma for Local lookup CSV to Threat Intel KV timestamp - age out indicators. 02-10-2021 09:21 AM
- Got Karma for Local lookup CSV to Threat Intel KV timestamp - age out indicators. 06-05-2020 12:50 AM
- Got Karma for SHC - failed on handle async replicate request. 06-05-2020 12:50 AM
- Got Karma for SHC - failed on handle async replicate request. 06-05-2020 12:50 AM
- Karma Re: Some problems with TA_Demisto configuration for 206103593. 06-05-2020 12:49 AM
- Got Karma for How to capture the RSS Timestamp for the Syndication Input app?. 06-05-2020 12:47 AM
- Got Karma for Finding random() events. 06-05-2020 12:47 AM
- Got Karma for Finding random() events. 06-05-2020 12:47 AM
- Got Karma for python eventgen.py process not killed when stopping splunk ?. 06-05-2020 12:47 AM
- Got Karma for python eventgen.py process not killed when stopping splunk ?. 06-05-2020 12:47 AM
- Got Karma for How to extract the timestamp from a filename at index-time to use as _time?. 06-05-2020 12:47 AM
- Got Karma for How to extract the timestamp from a filename at index-time to use as _time?. 06-05-2020 12:47 AM
- Got Karma for Re: How to extract the timestamp from a filename at index-time to use as _time?. 06-05-2020 12:47 AM
- Got Karma for Re: How to extract the timestamp from a filename at index-time to use as _time?. 06-05-2020 12:47 AM
- Posted Re: Horizon Chart - How can I change the axis based on user's timezone? on All Apps and Add-ons. 11-18-2019 03:45 PM
- Posted SHC - failed on handle async replicate request on Deployment Architecture. 10-15-2019 06:33 PM
- Tagged SHC - failed on handle async replicate request on Deployment Architecture. 10-15-2019 06:33 PM
- Tagged SHC - failed on handle async replicate request on Deployment Architecture. 10-15-2019 06:33 PM
- Tagged SHC - failed on handle async replicate request on Deployment Architecture. 10-15-2019 06:33 PM
- Posted Re: Some problems with TA_Demisto configuration on All Apps and Add-ons. 08-14-2019 10:52 PM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 2 | |||
| 2 | |||
| 0 | |||
| 2 | |||
| 1 | |||
| 2 | |||
| 2 |
11-18-2019
03:45 PM
I know its over a year later - but did you resolve this ? I would like to know why Horizon Chart doesnt use the localised _time generated by | timestamp but instead uses the UTC timestamp from the event
... View more
10-15-2019
06:33 PM
2 Karma
I have noticed something odd in a SHC deployment. Im consistently seeing "SHCMasterArtifactHandler - failed on handle async replicate request" errors, these report to be caused by the reason "active replication count >= max_peer_rep_load"
While these errors dont appear to be causing any actual impact on executing scheduled searches or anything else, I would like to get to the bottom of what is causing them. It doesnt seem to be a particular node or search or user that these occur for.
See the end of the post for the relevant error messages.
There are 4x nodes in the multi-site search head cluster (2 in each site)
The [shclustering] stanza of server.conf has replication_factor=2 configured
Where I suspect the problem is occuring is the search heads not implimenting the replication_factor setting, because when I run the command "splunk list shcluster-members" I see the replication_count numbers vary between 3 - 5 - I should expect to see 2 here shouldnt I ?
If the default max_peer_rep_load = 5 and the replication_count of at least one of those search heads is showing 5 when I check them - then I assume this is what is causing the excess replication to occur ?
When I run the command "splunk list shcluster-config" against all the nodes - they correctly show [max_peer_rep_load = 5] and [replication_factor = 2]
Has anyone seen this in a search head cluster before ? - I have read https://answers.splunk.com/answers/242905/shc-troubleshooting-configurations-under-search-he.html and looked into the settings from the last comment from SplunkIT - but want to verify before I go tweaking configs before finding the root cause of the issue.
Thanks in advance for anyone's insight into what may be causing it and how to further troubleshoot / resolve.
10-15-2019 09:50:15.153 +0000 ERROR SHCMasterArtifactHandler - failed on handle async replicate request sid=scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B err='srcPeer="<SHC_SRC_PEER>", srcGuid="4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B" cannot be valid source for artifactId=scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B targetPeer="<SHC_TARGET_PEER>", targetGuid="7A3F5991-2373-40BA-998A-79193A40CF27" reason="active replication count >= max_peer_rep_load"'
10-15-2019 09:50:15.159 +0000 ERROR SHCRepJob - job=SHPAsyncReplicationJob aid=scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B srcGuid=4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B tgtGuid=7A3F5991-2373-40BA-998A-79193A40CF27 failed. reason failed method=POST path=/services/shcluster/captain/artifacts/scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B/async_replicate captain=<SHC_MASTER>:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line="Internal Server Error" transaction_error="<response>\n <messages>\n <msg type="ERROR">failed on handle async replicate request sid=scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B err='srcPeer="<SHC_SRC_PEER>", srcGuid="4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B" cannot be valid source for artifactId=scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B targetPeer="<SHC_TARGET_PEER>", targetGuid="7A3F5991-2373-40BA-998A-79193A40CF27" reason="active replication count >= max_peer_rep_load"'</msg>\n </messages>\n</response>\n"
10-15-2019 09:50:15.159 +0000 WARN SHCMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/artifacts/scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B/async_replicate captain=<SHC_MASTER>:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line="Internal Server Error" transaction_error="<response>\n <messages>\n <msg type="ERROR">failed on handle async replicate request sid=scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B err='srcPeer="<SHC_SRC_PEER>", srcGuid="4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B" cannot be valid source for artifactId=scheduler__<USERNAME>_aGFpZ3NfZ2VuZXJhbA__RMD5f433e0e54b7570e0_at_1571133000_9209_4B87E9AA-3400-4FC0-ADFC-9C98728A2D1B targetPeer="<SHC_TARGET_PEER>", targetGuid="7A3F5991-2373-40BA-998A-79193A40CF27" reason="active replication count >= max_peer_rep_load"'</msg>\n </messages>\n</response>\n"
... View more
08-14-2019
10:52 PM
thanks - this worked for me !
... View more
03-12-2019
03:44 PM
no, unfortunately I havent found a fully working solution yet. Incrementally updating the lookup table might work as the timestamps in the KV shouldnt change, but pulling the entire list each day refreshes the timestamps when the KV is populated - hence it will never seem to reach the 7 day age / expiry time.
... View more
02-18-2019
03:34 PM
2 Karma
Has anyone tackled IOC expiry / timestamp issues between a local lookup and the Splunk ES Threat Intel KV store ?
I currently have a scheduled process that populates a local lookup with a list of indicators (IP / url / domain etc).
This local lookup is added as a threat intel download which collects the indicators and stores them within the Splunk ES Threat Intel KV stores (ip_intel / http_intel etc). This process works well, deduplication with existing indicators etc works, and I can see the indicators become available in the Threat Artifacts dashboard, and see matches from the Threat Gen searches etc etc.
Unfortunately every time the Threat Intel scripts collect the indicators from the local lookup and store them in the Threat Intel KV - they add a timestamp when the process runs. This means that each time it runs, its effectively a "new" indicator. I have set a maximum age of -7d on this list however this will never match as the timestamp is refreshed each time.
Populating the local lookup, I can only use the few fields available (e.g. - description,ip,weight) - it doesnt seem to support the addition of a timestamp which could be carried across to the KV store.
https://docs.splunk.com/Documentation/ES/5.2.2/Admin/Supportedthreatinteltypes
The macros responsible for cleaning up expired intel from the lists ("filter_threatintel_collection(1)" and "filter_by_threat_age") use the timestamp in the KV - which means the indicators will never match as they are updated every hour, and a new timestamp is added.
... View more
05-07-2018
04:27 AM
example of this is 2001:0000:85A3:0000:0000:7334:0000:0001
| makeresults | eval src_ip="2001:0000:85A3:0000:0000:7334:0000:0001"
| eval src_ip=lower(src_ip)
| rex mode=sed field=src_ip "s/(:0{4}){2,}/:/g"
| rex mode=sed field=src_ip "s/:(0){1,3}/:/g"
2001:0:85a3::7334:0:1
... View more
05-07-2018
04:16 AM
this is one reason why I love splunk answers - feeling a little silly now lol - thanks for the link.
... View more
05-07-2018
04:05 AM
HI Frank,
Thanks so much for your continued help with this. One way to lessen the chances might be to modify the regex to match only two or more instances together, leaving single instances of 0000 alone.
| rex mode=sed field=src_ip "s/(:0{4}){2,}/:/g"
not 100% but closer.
... View more
05-02-2018
09:50 PM
I receive logs from a device with the full form IPv6 address, as well as using capital letters.
Example: 2001:0DB8:85A3:0000:0000:8A2E:0370:7334
I have a lookup table with summarised IPv6 addresses that I want to match against.
src_ip,bunit
2001:db8:85a3::8a2e:370:7334 ,bobs_burgers
Using SPL-fu I can do massage the addresses so the lookup match works
Run Anywhere Example:
| makeresults | eval src_ip="2001:0DB8:85A3:0000:0000:8A2E:0370:7334"
| eval src_ip=lower(src_ip)
| rex mode=sed field=src_ip "s/(:0{4})+/:/g"
| rex mode=sed field=src_ip "s/:(0){1,3}/:/g"
My question is how should these be applied without users having to manually add the SPL. I cannot seem to find how to add SED or lowercase transforms at search time. There is many references of SEDCMD at index time to mask / replace etc, but I want to store the original data and do this at search time.
Is there a solution to create a transform that can perform this at search time, or am I resigned to adding the 3 eval/rex lines to all searches / dashboards that use this data.
Thanks,
Ash
... View more
11-04-2015
10:24 PM
2 Karma
So I logged a support case [283416] for this problem and unfortunately for me & everyone following along here - extracting a full timestamp from a filename is not currently supported in Splunk.
You can capture the date from the filename, but not the time - so in essence each line in the file needs to have a time record.
There apparently is several other customers that have requested this feature and I have been added to the bottom of that enhancement request.
I was hoping that I wouldnt have to modify the data before indexing it, but the way it looks like im going to have to get around it is with some pre-processing of the log.
This will append the timestamp of the filename to the front of each line of the file, with a pipe "|" seperator - at least this will index with automatic timestamp extraction, without having to define any time format strings.
find . -name '*201510210345.txt' -type f -print | xargs sed -i 's/^/201510210345\|/'
... View more
10-28-2015
04:12 PM
Thanks again for your help woodcock - but this still failed
Here are all the details if you could reproduce the issue.
The Data
root@testbox:/opt/data# cat myfile_201510210345.txt
IP|10.0.0.1/32|proxy|65|||2015/10/19|proxy server
root@testbox:/opt/data#
props.conf and datetime.xml
root@testbox:/opt/splunk/etc/apps/myapp/local# cat props.conf
[mysource]
DATETIME_CONFIG = /etc/apps/myapp/local/datetime.xml
FIELD_DELIMITER = |
FIELD_NAMES = F1,F2,F3,F4,F5,F6,F7,F8
TZ = UTC
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
root@testbox:/opt/splunk/etc/apps/myapp/local#
root@testbox:/opt/splunk/etc/apps/myapp/local# cat datetime.xml
<!-- Version 4.0 -->
<!-- datetime.xml -->
<!-- This file contains the general formulas for parsing date/time formats. -->
<datetime>
<define name="_mydatetime" extract="year, month, day, hour, minute">
<text><![CDATA[source::.*?_(\d{4})(\d{2})(\d{2})(\d{2})(\d{2}).txt]]></text>
</define>
<timePatterns>
<use name="_mydatetime"/>
</timePatterns>
<datePatterns>
<use name="_mydatetime"/>
</datePatterns>
</datetime>
root@testbox:/opt/splunk/etc/apps/myapp/local#
btool output showing full config
root@ashlubuntu:/opt/splunk/etc/apps/myapp/local# /opt/splunk/bin/splunk btool props list mysource
[mysource]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/apps/myapp/local/datetime.xml
FIELD_DELIMITER = |
FIELD_NAMES = F1,F2,F3,F4,F5,F6,F7,F8
HEADER_MODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = false
TRANSFORMS =
TRUNCATE = 10000
TZ = UTC
detect_trailing_nulls = false
maxDist = 100
priority =
sourcetype =
root@ashlubuntu:/opt/splunk/etc/apps/myapp/local#
oneshot import
root@ashlubuntu:/opt/data# /opt/splunk/bin/splunk add oneshot myfile_201510210345.txt -sourcetype "mysource" -index "testindex" -host "myhost"
Your session is invalid. Please login.
Splunk username: admin
Password:
Oneshot '/opt/data/myfile_201510210345.txt' added
root@ashlubuntu:/opt/data#
splunkd.log DEBUG output - showing it recognises the config, but fails to parse the timestamp
10-28-2015 22:56:39.543 DEBUG REST_Calls - app=search POST data/inputs/oneshot/ id=/opt/data/myfile_201510210345.txt: host -> [myhost], index -> [testindex], sourcetype -> [mysource]
10-28-2015 22:56:39.543 DEBUG AdminManager - Validating argument values...
10-28-2015 22:56:39.543 DEBUG AdminManager - Validating rule='validate(len(name) < 1024, 'Parameter "name" must be less than 1024 characters.')' for arg='name'.
10-28-2015 22:56:39.589 DEBUG FilesystemFilter - Testing path=/opt/data/myfile_201510210345.txt(real=/opt/data/myfile_201510210345.txt) with global blacklisted paths
10-28-2015 22:56:39.590 INFO AdminManager - feedName=oneshotinput, atomUrl=services
10-28-2015 22:56:39.590 INFO UserManager - Unwound user context: admin -> NULL
10-28-2015 22:56:39.590 DEBUG InThreadActor - this=0x7f8454016b50 waitForActorToComplete start actor=0x7f844a3fcdf0
10-28-2015 22:56:39.592 DEBUG InThreadActor - this=0x7f8454016b50 waitForActorToComplete end actor=0x7f844a3fcdf0
10-28-2015 22:56:39.593 DEBUG ArchiveContext - /opt/data/myfile_201510210345.txt is NOT an archive file.
10-28-2015 22:56:39.593 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/opt/data/myfile_201510210345.txt
10-28-2015 22:56:39.593 DEBUG OneShotWriter - Got new entry in the archive: /opt/data/myfile_201510210345.txt
10-28-2015 22:56:39.593 DEBUG OneShotWriter - Will call classifier with given_type="mysource".
10-28-2015 22:56:39.593 DEBUG FileClassifierManager - Finding type for file: /opt/data/myfile_201510210345.txt
10-28-2015 22:56:39.593 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/opt/data/myfile_201510210345.txt
10-28-2015 22:56:39.593 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/opt/data/myfile_201510210345.txt|mysource
10-28-2015 22:56:39.593 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-28-2015 22:56:39.593 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-28-2015 22:56:39.593 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/opt/data/myfile_201510210345.txt|host::myhost|mysource|
10-28-2015 22:56:39.594 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-28-2015 22:56:39.594 DEBUG OneShotWriter - Setting sourcetype="sourcetype::mysource"
10-28-2015 22:56:39.594 DEBUG OneShotWriter - Setting channelKey="2"
10-28-2015 22:56:39.594 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/opt/data/myfile_201510210345.txt|host::myhost|mysource|2
10-28-2015 22:56:39.594 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-28-2015 22:56:39.594 DEBUG StructuredDataHeaderExtractor - Read configuration: configured=1 mode=6 HEADER_FIELD_LINE_NUMBER=0 HEADER_FIELD_DELIMITER='|' HEADER_FIELD_QUOTE='"' FIELD_DELIMITER='|' FIELD_QUOTE='"'.
10-28-2015 22:56:39.594 DEBUG OneShotWriter - Structured data configurations loaded
10-28-2015 22:56:39.594 INFO UTF8Processor - Converting using CHARSET="UTF-8" for conf "source::/opt/data/myfile_201510210345.txt|host::myhost|mysource|2"
10-28-2015 22:56:39.594 INFO LineBreakingProcessor - Using truncation length 10000 for conf "source::/opt/data/myfile_201510210345.txt|host::myhost|mysource|2"
10-28-2015 22:56:39.594 INFO LineBreakingProcessor - Using lookbehind 100 for conf "source::/opt/data/myfile_201510210345.txt|host::myhost|mysource|2"
10-28-2015 22:56:39.594 DEBUG StructuredDataHeaderExtractor - Read configuration: configured=1 mode=6 HEADER_FIELD_LINE_NUMBER=0 HEADER_FIELD_DELIMITER='|' HEADER_FIELD_QUOTE='"' FIELD_DELIMITER='|' FIELD_QUOTE='"'.
10-28-2015 22:56:39.594 INFO AggregatorMiningProcessor - Setting up line merging apparatus for: source::/opt/data/myfile_201510210345.txt|host::myhost|mysource|2
10-28-2015 22:56:39.595 DEBUG LoadDateParserRegexes - put _mydatetime regex=source::.*?_(\d{4})(\d{2})(\d{2})(\d{2})(\d{2}).txt
10-28-2015 22:56:39.595 DEBUG LoadDateParserRegexes - * year
10-28-2015 22:56:39.595 DEBUG LoadDateParserRegexes - * month
10-28-2015 22:56:39.595 DEBUG LoadDateParserRegexes - * day
10-28-2015 22:56:39.595 DEBUG LoadDateParserRegexes - * hour
10-28-2015 22:56:39.595 DEBUG LoadDateParserRegexes - * minute
10-28-2015 22:56:39.595 INFO DateParser - Set timezone to: UTC
10-28-2015 22:56:39.595 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/opt/data/myfile_201510210345.txt", data_host="myhost", data_sourcetype="mysource"
Here is the data in splunk - you can see the same time for _time and _indextime. Splunk didnt even use the file time, it reverted to the indextime.
Im wondering if this needs to be a support case, I cannot seem to get the timestamp from a filename.
hxxps://www.dropbox.com/s/f40xx78v5zctpon/mysource.PNG?dl=0
... View more
10-22-2015
09:11 PM
an ugly search time hack makes the results appear correct when returning searches, but as the indexed timestamp is incorrect, I cant actually search on the specific day, just have to search for All Time.
Adding the following to props.conf overwrites _time at search time - as I said - U.G.L.Y .....
EXTRACT-timestamp = \w_(?<srctimestamp>\d+) in source
EVAL-_time = strptime(srctimestamp, "%Y%m%d%H%M")
... View more
10-22-2015
07:53 PM
One thought also sprang to mind - are we all trying to perform something out of the processing order ?
When I re-read http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Overviewofeventprocessing
When Splunk Enterprise indexes events, it:
Configures character set encoding.
Configures linebreaking for multi-line events.
Identifies event timestamps (and applies timestamps to events if they do not exist).
Extracts a set of useful standard fields such as host, source, and sourcetype.
Segments events.
Dynamically assigns metadata to events, if specified.
Anonymizes data, if specified.
To me - this indicates that splunk will apply timestamps BEFORE extracting the source field ........ so if im trying to get the timestamp FROM source before it has been extracted ...... am I trying to tear a hole in the space-time continuum ?
... View more
10-22-2015
06:50 PM
I ran the import while splunk was running in debug mode & it appears to pull out the config ok (I updated the regex based off some other answers questions) - confirmed it matches on regex101
10-23-2015 12:36:49.101 DEBUG FilesystemFilter - Testing path=/sourcedata/mydatafiles/myfile_201510212245.txt(real=/sourcedata/mydatafiles/myfile_201510212245.txt) with global blacklisted paths
10-23-2015 12:36:49.101 INFO AdminManager - feedName=oneshotinput, atomUrl=services
10-23-2015 12:36:49.101 INFO UserManager - Unwound user context: admin -> NULL
10-23-2015 12:36:49.101 DEBUG InThreadActor - this=0x7f472c417150 waitForActorToComplete start actor=0x7f4723bfcdf0
10-23-2015 12:36:49.103 DEBUG InThreadActor - this=0x7f472c417150 waitForActorToComplete end actor=0x7f4723bfcdf0
10-23-2015 12:36:49.104 DEBUG ArchiveContext - /sourcedata/mydatafiles/myfile_201510212245.txt is NOT an archive file.
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/sourcedata/mydatafiles/myfile_201510212245.txt
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)' matches with lowest priority
10-23-2015 12:36:49.104 DEBUG OneShotWriter - Got new entry in the archive: /sourcedata/mydatafiles/myfile_201510212245.txt
10-23-2015 12:36:49.104 DEBUG OneShotWriter - Will call classifier with given_type="mysource".
10-23-2015 12:36:49.104 DEBUG FileClassifierManager - Finding type for file: /sourcedata/mydatafiles/myfile_201510212245.txt
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/sourcedata/mydatafiles/myfile_201510212245.txt
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)' matches with lowest priority
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/sourcedata/mydatafiles/myfile_201510212245.txt|mysource
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)' matches with lowest priority
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/sourcedata/mydatafiles/myfile_201510212245.txt|host::server|mysource|
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)' matches with lowest priority
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-23-2015 12:36:49.104 DEBUG OneShotWriter - Setting sourcetype="sourcetype::mysource"
10-23-2015 12:36:49.104 DEBUG OneShotWriter - Setting channelKey="2"
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/sourcedata/mydatafiles/myfile_201510212245.txt|host::server|mysource|2
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)' matches with lowest priority
10-23-2015 12:36:49.104 DEBUG PropertiesMapConfig - Pattern 'mysource' matches with priority 100
10-23-2015 12:36:49.104 DEBUG StructuredDataHeaderExtractor - Read configuration: configured=1 mode=6 HEADER_FIELD_LINE_NUMBER=0 HEADER_FIELD_DELIMITER='|' HEADER_FIELD_QUOTE='"' FIELD_DELIMITER='|' FIELD_QUOTE='"'.
10-23-2015 12:36:49.104 DEBUG OneShotWriter - Structured data configurations loaded
10-23-2015 12:36:49.104 INFO UTF8Processor - Converting using CHARSET="UTF-8" for conf "source::/sourcedata/mydatafiles/myfile_201510212245.txt|host::server|mysource|2"
10-23-2015 12:36:49.104 INFO LineBreakingProcessor - Using truncation length 10000 for conf "source::/sourcedata/mydatafiles/myfile_201510212245.txt|host::server|mysource|2"
10-23-2015 12:36:49.104 INFO LineBreakingProcessor - Using lookbehind 100 for conf "source::/sourcedata/mydatafiles/myfile_201510212245.txt|host::server|mysource|2"
10-23-2015 12:36:49.104 DEBUG StructuredDataHeaderExtractor - Read configuration: configured=1 mode=6 HEADER_FIELD_LINE_NUMBER=0 HEADER_FIELD_DELIMITER='|' HEADER_FIELD_QUOTE='"' FIELD_DELIMITER='|' FIELD_QUOTE='"'.
10-23-2015 12:36:49.105 INFO AggregatorMiningProcessor - Setting up line merging apparatus for: source::/sourcedata/mydatafiles/myfile_201510212245.txt|host::server|mysource|2
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - put _mydate regex=source::.*?_(\d{4})(\d{2})(\d{2})
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * year
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * month
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * day
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - put _mytime regex=source::.*?_\d{8}(\d{2})(\d{2})
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * hour
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * minute
10-23-2015 12:36:49.105 INFO DateParser - Set timezone to: UTC
10-23-2015 12:36:49.105 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/sourcedata/mydatafiles/myfile_201510212245.txt", data_host="server", data_sourcetype="mysource"
10-23-2015 12:36:49.105 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/sourcedata/mydatafiles/myfile_201510212245.txt", data_host="server", data_sourcetype="mysource"
10-23-2015 12:36:49.105 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/sourcedata/mydatafiles/myfile_201510212245.txt", data_host="server", data_sourcetype="mysource"
10-23-2015 12:36:49.105 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/sourcedata/mydatafiles/myfile_201510212245.txt", data_host="server", data_sourcetype="mysource"
10-23-2015 12:36:49.105 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/sourcedata/mydatafiles/myfile_201510212245.txt", data_host="server", data_sourcetype="mysource"
10-23-2015 12:36:49.105 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/sourcedata/mydatafiles/myfile_201510212245.txt", data_host="server", data_sourcetype="mysource"
To me - the following indicate that the config was successfully loaded from datetime.xml
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - put _mydate regex=source::.*?_(\d{4})(\d{2})(\d{2})
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * year
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * month
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * day
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - put _mytime regex=source::.*?_\d{8}(\d{2})(\d{2})
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * hour
10-23-2015 12:36:49.105 DEBUG LoadDateParserRegexes - * minute
The regex listed does match & return the fields for the following source:
source::/sourcedata/mydatafiles/myfile_201510212245.txt|host::server|mysource|2
So im really at a bit of a loss at this point 😞
... View more
10-22-2015
06:27 PM
I have also tried this on 6.2.3 & 6.3 with no success in either version.
... View more
10-22-2015
05:06 PM
Hi tmeader - did you get an answer to this ? did you go through support ?
I am having a very similar problem https://answers.splunk.com/answers/320978/how-to-extract-the-timestamp-from-a-filename-at-in.html and wondering if you found a solution ?
Thanks,
Ash
... View more
10-22-2015
04:08 PM
2 Karma
I have searched answers high & low to try and extract the timestamp from my filename at index-time, but I'm still unable to get the timestamp from the filename used as _time.
Summary:
- The filename contains a timetamp in %Y%m%d%H%M format ( myfile_201510210345.txt )
- While the events in the file do contain a date, this is not the date I want to use for the timestamp.
- Sample data: [IP|1.2.3.4/32|proxy|75|||2015/10/19|some server hostname: server]
I have read the blog post as well as numerous other answers regarding custom datetime.xml usage, but I still cannot seem to crack this nut - each time it indexes using the server time (last resort in the timestamp handling). I have tried adding these definitions to a copy of the original datetime.conf as well as creating a blank datetime.conf with only these definitions in it - neither worked.
My props.conf
[mysource]
DATETIME_CONFIG = /etc/apps/myapp/local/datetime.xml
FIELD_DELIMITER = |
FIELD_NAMES = F1,F2,F3,F4,F5,F6,F7,F8
TIME_FORMAT = %Y%m%d%H%M
TZ = UTC
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
My datetime.xml modifications (attempt 1 - extracting date & time together)
<define name="_mydatetime" extract="year, month, day, hour, minute">
<text><![CDATA[(?:^|source:).*?.*?(20\d\d)(0\d|1[012])([012]\d|3[01])([01]\d|2[0123])([0-6]\d)]]></text>
</define>
<timePatterns>
<use name="_mydatetime"/>
</timePatterns>
<datePatterns>
<use name="_mydatetime"/>
</datePatterns>
My datetime.xml modifications (attempt 2 - extracting date & time separately after reading another answers example)
<define name="_mydate" extract="year, month, day">
<text><![CDATA[(?:^|source:).*?.*?(20\d\d)(0\d|1[012])([012]\d|3[01])(?:[01]\d|2[0123])(?:[0-6]\d)]]></text>
</define>
<define name="_mytime" extract="hour, minute">
<text><![CDATA[(?:^|source:).*?.*?(?:20\d\d)(?:0\d|1[012])(?:[012]\d|3[01])([01]\d|2[0123])([0-6]\d)]]></text>
</define>
<timePatterns>
<use name="_mydate"/>
</timePatterns>
<datePatterns>
<use name="_mytime"/>
</datePatterns>
There has been a lot of answers questions about datetime.xml and also about date / time from filenames - but there does not appear to be a lot of definitive answers. Can anyone who is successfully extracting date / time stamp from the filename provide a working example of using the source at index time for this timestamp, I have been trying to get this working on & off for a long time now.
Thanks,
Ash
... View more
07-24-2015
06:14 AM
The workaround I did was to run 6.1.8 for testing with SplunkIT
There is potentially a Splunk Aptitude Challenge app that may fit the bill, but I havent investigated further than just seeing it: http://challengepost.com/software/gladiator-osvf97
... View more
07-09-2015
08:30 PM
Support ticket has been closed & documentation updated: http://docs.splunk.com/Documentation/Splunk/6.2.4/Installation/InstallonLinux#Default_shell
Bottom line - dont use dash 😄
... View more
07-08-2015
03:30 AM
I am marking this as correct as it resolved the problem - the question now remains as to "why" this is happening.
You can see below that indeed dash/sh was set to true, then changing it to false, I no longer get the issue.
root@splunkbox:~# debconf-show dash
* dash/sh: true
root@splunkbox:~# debconf-set-selections <<< "dash dash/sh string false"
root@splunkbox:~# debconf-show dash
* dash/sh: false
root@splunkbox:~# dpkg-reconfigure -f noninteractive dash
Removing 'diversion of /bin/sh to /bin/sh.distrib by dash'
Adding 'diversion of /bin/sh to /bin/sh.distrib by bash'
Removing 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
Adding 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by bash'
root@splunkbox:~# debconf-show dash
* dash/sh: false
root@splunkbox:~#
Python is no longer called with /bin/sh -c
root@splunkbox:~# ps -ef | grep python
root 5571 5514 0 20:25 ? 00:00:00 python /opt/splunk/etc/apps/file_meta_data/bin/file_meta_data.py
root 5572 5514 1 20:25 ? 00:00:01 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
root 5640 5514 0 20:25 ? 00:00:00 python /opt/splunk/etc/apps/syndication/bin/syndication.py
root 5645 5514 0 20:25 ? 00:00:00 python /opt/splunk/etc/apps/website_input/bin/web_input.py
root 6479 1918 0 20:28 pts/0 00:00:00 grep --color=auto python
root@splunkbox:~# /opt/splunk/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
..
Stopping splunk helpers...
Done.
root@splunkbox:~# ps -ef | grep python
root 6843 1918 0 20:29 pts/0 00:00:00 grep --color=auto python
root@splunkbox:~#
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
06-08-2022
12:03 AM
|
