timestamp assignment not working ELB logs Splunk Add-on for AWS

[nmohammed]

We're sending AWS ELB Access logs (Classic ELB, NLB and ALB) using Lambda to HEC.  I have installed the Splunk add-on for AWS on SH and HEC . The add-on has regexes to parse the access logs and all the fields extractions from REGEX for access logs seems to be working fine.

However, we're having issues with the timestamp of the event, which is also extracted as "timestamp" field and the _time is getting assigned as ingestion time instead of actual time from the event.  I tried to add timestamp PREFIX in the props.conf in Splunk_TA_AWS for the aws:elb:access logs sourcetype, however, it doesn't work.  

Sample events - 

NLB - 

tls 2.0 2025-01-15T23:59:54 net/loadbalancerName/guid 10.xxx.xxx.1:32582 10.xxx.x.xx:443 1140251 85 3546 571 - arn:aws:acm:us-west-2:026921344628:certificate/guid - ECDHE-RSA-XXXX-GCMXXX tlsv12 - example.io - - - 2025-01-15T23:40:54

ALB - 

https 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 10.0.0.1:80 0.086 0.048 0.037 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.46.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337281-1d84f3d73c47ec4e58577259" "www.example.com" "arn:aws:acm:us-east-2:123456789012:certificate/12345678-1234-1234-1234-123456789012" 1 2018-07-02T22:22:48.364000Z "authenticate,forward" "-" "-" "10.0.0.1:80" "200" "-" "-" TID_123456

ELB - 

2018-12-31T00:08:01.715269Z loadbalancerName 187.xx.xx.xx:48364 - -1 -1 -1 503 0 0 0 "GET http://52.x.xxx.xxx:80/ HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" - -

props.conf

## Classic Load Balancer ##
[source::http:lblogs]
EXTRACT-elb = ^\s*(?P<timestamp>\S+)(\s+(?P<elb>\S+))(\s+(?P<client_ip>[\d.]+):(?P<client_port>\d+))(\s+(?P<backend>\S+))(\s+(?P<request_processing_time>\S+))(\s+(?P<backend_processing_time>\S+))(\s+(?P<response_processing_time>\S+))(\s+(?P<elb_status_code>\S+))(\s+(?P<backend_status_code>\S+))(\s+(?P<received_bytes>\d+))(\s+(?P<sent_bytes>\d+))(\s+"(?P<request>[^"]+)")(\s+"(?P<user_agent>[^"]+)")(\s+(?P<ssl_cipher>\S+))(\s+(?P<ssl_protocol>\S+))
EVAL-rtt = request_processing_time + backend_processing_time + response_processing_time
sourcetype = aws:elb:accesslogs

## Application Load Balancer ##
[source::http:aws-lblogs]
EXTRACT-elb = ^\s*(?P<type>\S+)(\s+(?P<timestamp>\S+))(\s+(?P<elb>\S+))(\s+(?P<client_ip>[\d.]+):(?P<client_port>\d+))(\s+(?P<target>\S+))(\s+(?P<request_processing_time>\S+))(\s+(?P<target_processing_time>\S+))(\s+(?P<response_processing_time>\S+))(\s+(?P<elb_status_code>\S+))(\s+(?P<target_status_code>\S+))(\s+(?P<received_bytes>\d+))(\s+(?P<sent_bytes>\d+))(\s+"(?P<request>[^"]+)")(\s+"(?P<user_agent>[^"]+)")(\s+(?P<ssl_cipher>\S+))(\s+(?P<ssl_protocol>\S+))(\s+(?P<target_group_arn>\S+))(\s+"(?P<trace_id>[^"]+)")(\s+"(?P<domain_name>[^"]+)")?(\s+"(?P<chosen_cert_arn>[^"]+)")?(\s+(?P<matched_rule_priority>\S+))?(\s+(?P<request_creation_time>\S+))?(\s+"(?P<actions_executed>[^"]+)")?(\s+"(?P<redirect_url>[^"]+)")?(\s+"(?P<error_reason>[^"]+)")?
EVAL-rtt = request_processing_time + target_processing_time + response_processing_time
priority = 1
sourcetype = aws:elb:accesslogs

## Network Load Balancer ##
[source::http:lblogs]
EXTRACT-elb-nlb = ^\s*(?P<type>\S+)(\s+(?P<log_version>\S+))(\s+(?P<timestamp>\S+))(\s+(?P<elb>\S+))(\s+(?P<listener>\S+))(\s+(?P<client_ip>[\d.]+):(?P<client_port>\d+))(\s+(?P<destination_ip>[\d.]+):(?P<destination_port>\d+))(\s+(?P<connection_time>\S+))(\s+(?P<tls_handshake_time>\S+))(\s+(?P<received_bytes>\d+))(\s+(?P<sent_bytes>\d+))(\s+(?P<incoming_tls_alert>\S+))(\s+(?P<chosen_cert_arn>\S+))(\s+(?P<chosen_cert_serial>\S+))(\s+(?P<tls_cipher>\S+))(\s+(?P<tls_protocol_version>\S+))(\s+(?P<tls_named_group>\S+))(\s+(?P<domain_name>\S+))(\s+(?P<alpn_fe_protocol>\S+))(\s+(?P<alpn_be_protocol>\S+))(\s+(?P<alpn_client_preference_list>\S+))
sourcetype = aws:elb:accesslogs

[aws:elb:accesslogs]
TIME_PREFIX = ^.*?(?=20\d\d-\d\d)
TIME_FORMAT =
MAX_TIME_LOOKAHEAD 

[isoutamo]

Hi

I think that your issue is setting sourcetype inside source stanza. As splunk has only one linear data pipeline and now it has taken those events based on source:: definition it will apply only those values on indexing phase. You cannot put events back into the start of this pipeline again and start same event manipulation with sourcetype stanza. 

Your aws:elb:accesslog definitions are used, but only in search time, not in index time. And as those definitions are affecting only in index time it's obviously that nothing happen for you _time value.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline

Have you try to add those same definitions under every source:: stanzas?

Of course as you are using HEC it also mater which endpoint you are using. There are differences what manipulations you can do with props.conf based on endpoint.

r. Ismo

r. Ismo

[nmohammed]

Thanks @isoutamo ,

As I understand ; Since these definitions are used only at search time , then I only need the add-on installed on the search Head. On the HEC I will put the props.conf with the TIME PREFIX related regex , so time will be extracted from the incoming logs and sent to the indexers. 

[isoutamo]

I suppose that you should try to move those timestamp extractions under each source:: definitions. Then those should work.

Anyhow those definitions which you have put on that new sourcetype definitions are working on search time if those can apply on search time. But example those _time settings are working only in indexing phase.

[nmohammed]

on HEC - I tried the following by moving the TIME definitions under the source (for all 3 sources) in props.conf and removed them from sourcetype.  Restarted Splunk, but still did not work.

[source::http:aws-lblogs]
EXTRACT-elb = ^\s*(?P<type>\S+)(\s+(?P<timestamp>\S+))(\s+(?P<elb>\S+))(\s+(?P<client_ip>[\d.]+):(?P<client_port>\d+))(\s+(?P<target>\S+))(\s+(?P<request_processing_time>\S+))(\s+(?P<target_processing_time>\S+))(\s+(?P<response_processing_time>\S+))(\s+(?P<elb_status_code>\S+))(\s+(?P<target_status_code>\S+))(\s+(?P<received_bytes>\d+))(\s+(?P<sent_bytes>\d+))(\s+"(?P<request>[^"]+)")(\s+"(?P<user_agent>[^"]+)")(\s+(?P<ssl_cipher>\S+))(\s+(?P<ssl_protocol>\S+))(\s+(?P<target_group_arn>\S+))(\s+"(?P<trace_id>[^"]+)")(\s+"(?P<domain_name>[^"]+)")?(\s+"(?P<chosen_cert_arn>[^"]+)")?(\s+(?P<matched_rule_priority>\S+))?(\s+(?P<request_creation_time>\S+))?(\s+"(?P<actions_executed>[^"]+)")?(\s+"(?P<redirect_url>[^"]+)")?(\s+"(?P<error_reason>[^"]+)")?
EVAL-rtt = request_processing_time + target_processing_time + response_processing_time
priority = 1
SHOULD_LINEMERGE = false
TIME_PREFIX = ^.*?(?=20\d\d-\d\d)
TIME_FORMAT =    
MAX_TIMESTAMP_LOOKAHEAD = 28


[aws:elb:accesslogs]

[isoutamo]

Have you check that this REGEX and also your TIME_PREFIX is working? Try those e.g. regex101.com. I’m not sure what all you could have in TIME_PREFIX?

This seems to be working at least on regex101.com, but check that it works also with splunk rex command.

Btw which HEC endpoint you are using? Some of those are not extracting timestamp!