topic Re: need to help extract the timestamp value from eventlast seen field from aws. in Getting Data In

need to help extract the timestamp value from eventlast seen field from aws.

lksridhar — Tue, 29 Sep 2020 19:44:30 GMT

Hi Folks,

we have ingested the aws logs using aws add on and able to see the logs. now we are trying to extract the timestamp value from eventlastseen field and that value in eventlastseen field but it is not working while using the below method. could you please help me to extract timestamp value from eventlastseen field.

sample event:

[
{
"schemaVersion": "2.0",
"accountId": "893386011857",
"region": "us-east-2",
"partition": "aws",
"id": "42b1b691434e33399b13f013f9740757",
"arn": "arn:aws:guardduty:us-east-2:893386011857:detector/a8b16945bc4a8767dcac8d7edbf84e03/finding/42b1b691434e33399b13f013f9740757",
"type": "Recon:EC2/PortProbeUnprotectedPort",
"resource": {
"resourceType": "Instance",
"instanceDetails": {
"instanceId": "i-048dfd648094a1fdc",
"instanceType": "t2.micro",
"launchTime": "2018-05-17T07:06:43Z",
"platform": "windows",
"productCodes": [],
"iamInstanceProfile": null,
"networkInterfaces": [
{
"ipv6Addresses": [],
"networkInterfaceId": "eni-0b1a9d3e70ae5c43b",
"privateDnsName": "ip-172-31-22-179.us-east-2.compute.internal",
"privateIpAddress": "172.31.22.179",
"privateIpAddresses": [
{
"privateDnsName": "ip-172-31-22-179.us-east-2.compute.internal",
"privateIpAddress": "172.31.22.179"
}
],
"subnetId": "subnet-2076a65a",
"vpcId": "vpc-1d741675",
"securityGroups": [
{
"groupName": "launch-wizard-2",
"groupId": "sg-007141e05a28f56e8"
}
],
"publicDnsName": "ec2-18-191-78-236.us-east-2.compute.amazonaws.com",
"publicIp": "18.191.78.236"
}
],
"tags": [],
"instanceState": "running",
"availabilityZone": "us-east-2b",
"imageId": "ami-fd0f3298",
"imageDescription": "Microsoft Windows Server 2016 with Desktop Experience Locale English AMI provided by Amazon"
}
},
"service": {
"serviceName": "guardduty",
"detectorId": "a8b16945bc4a8767dcac8d7edbf84e03",
"action": {
"actionType": "PORT_PROBE",
"portProbeAction": {
"portProbeDetails": [
{
"localPortDetails": {
"port": 3389,
"portName": "RDP"
},
"remoteIpDetails": {
"ipAddressV4": "103.89.91.215",
"organization": {
"asn": "135905",
"asnOrg": "VIETNAM POSTS AND TELECOMMUNICATIONS GROUP",
"isp": "Vietnam Posts And Telecommunications Group",
"org": "Vietnam Posts And Telecommunications Group"
},
"country": {
"countryName": "Vietnam"
},
"city": {
"cityName": "Hanoi"
},
"geoLocation": {
"lat": 21.0333,
"lon": 105.85
}
}
}
],
"blocked": false
}
},
"resourceRole": "TARGET",
"additionalInfo": {
"threatName": "Scanner",
"threatListName": "ProofPoint"
},
"eventFirstSeen": "2018-05-17T08:02:12Z",
"eventLastSeen": "2018-05-17T08:55:47Z",
"archived": false,
"count": 2
},
"severity": 2,
"createdAt": "2018-05-17T08:06:09.308Z",
"updatedAt": "2018-05-17T09:06:09.505Z",
"title": "Unprotected port on EC2 instance i-048dfd648094a1fdc is being probed.",
"description": "EC2 instance has an unprotected port which is being probed by a known malicious host."
}
]

props.conf

TIME_PREFIX="eventLastSeen":\s"
TIME_FORMAT=%Y-%m-%dT%H:%M:%SZ
MAX_TIMESTAMP_LOOKAHEAD=20

Re: need to help extract the timestamp value from eventlast seen field from aws.

adonio — Thu, 31 May 2018 14:19:10 GMT

hello there,

seems like a nice JSON format event you got there.
why not use the oootb configurations for it from splunk?
try this in props.conf (on your first full splunk instance, HF or Indexer) rename the stanza as you wish

[ CHANGE_ME_json ]
CHARSET=AUTO
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%SZ
TIMESTAMP_FIELDS=service.eventLastSeen

hope it helps

Re: need to help extract the timestamp value from eventlast seen field from aws.

lksridhar — Tue, 29 Sep 2020 19:48:09 GMT

Thanks adonio for your input, i have tried above method it is not working and still it is not extract the value from eventlastseen field.

[ aws:kiness ]
CHARSET=AUTO
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%SZ
TIMESTAMP_FIELDS=service.eventLastSeen

Re: need to help extract the timestamp value from eventlast seen field from aws.

lksridhar — Mon, 04 Jun 2018 09:47:12 GMT

Folks,

Could you please check above post and let me know your thought as still it is not resolved.

Re: need to help extract the timestamp value from eventlast seen field from aws.

adonio — Mon, 04 Jun 2018 12:59:29 GMT

where did you place the props.conf
it works nice on my single instance lab
are you using the app for AWS?

Re: need to help extract the timestamp value from eventlast seen field from aws.

lksridhar — Tue, 05 Jun 2018 05:08:11 GMT

i have configured the props.conf in HF, also we are using the AWS app.

Re: need to help extract the timestamp value from eventlast seen field from aws.

adonio — Tue, 05 Jun 2018 14:39:42 GMT

not sure, this props.conf works well for me.
try to remove the spaces in your [ aws:kiness ]
so it will look like that: [aws:kiness]
also, double check there is no conflict props in your aws app / ta

Re: need to help extract the timestamp value from eventlast seen field from aws.

lksridhar — Tue, 29 Sep 2020 19:50:34 GMT

i have checked and found that the aws kiness input configuration is created in aws_kinesis_task.conf in TA.

look like it is not communicating with props.conf to extract the time_prefix and other index time opertation

is there any separate file do we need to use to do index operation for aws:kinesis sourcetype.

Re: need to help extract the timestamp value from eventlast seen field from aws.

ameyap16 — Thu, 14 Mar 2019 03:01:07 GMT

Did you find a solution to this issue?