Hello all,
I've been trying to get some stats from JSON data that I've been receiving in Splunk.
See:
I think I'm not getting the stats because the events are multifield and it is all repeated.
See the search here in case is needed:
sourcetype=api_in_tasks to_team=TEAM-177 "TK-EU-28209292.1.1" | dedup task consecutive=true | rex field=task "(?<ticket>\w[\w\-]*)" | join type=inner ticket [search sourcetype=api_in_tasks from_team=TEAM-210| rex field=task "(?<ticket>\w[\w\-]*)" | dedup task consecutive=true | rename created_at as created_at_ticket | fields ticket,created_at_ticket ] | eval ticket_created=strftime(created_at_ticket,"%m/%d/%y %H:%M:%S") | eval diff=created_at-created_at_ticket | eval diff_time=strftime(diff,"%m/%d/%y %H:%M:%S") | eval created_task=strftime(created_at,"%m/%d/%y %H:%M:%S") | table task,ticket,created_task,ticket_created,diff_time, created_at_ticket
Here another photo from the data:
Now, I've tried to reduce it to one line adding these options (two separate times, not at the same time) in the props.conf.
First try:
LINE_BREAKER = {(.*?)}
SHOULD_LINEMERGE = false
Second try:
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
But none of this has worked. Any idea?
Thank you in advance.
Looking back at some of your comments, I noticed the following:
INDEXED_EXTRACTIONS = json
If on top of these indexed extractions KV_Mode is left to the default auto setting, would Splunk run KV extractions again on searchtime, adding on to the values extracted as indexed extractions?
Can you try either removing that INDEXED_EXTRACTIONS = json, or adding KV_Mode=none in props.conf?
Looking back at some of your comments, I noticed the following:
INDEXED_EXTRACTIONS = json
If on top of these indexed extractions KV_Mode is left to the default auto setting, would Splunk run KV extractions again on searchtime, adding on to the values extracted as indexed extractions?
Can you try either removing that INDEXED_EXTRACTIONS = json, or adding KV_Mode=none in props.conf?
I've tried both options with two different indexes (incoming tasks and out coming ones) and here you have the results:
For the in ones, I've added KV_MODE=none
It is working now!
I can see all the fields with calculations filled with results and the events only contain one line.
created_at
1516970893.18886
cti
CTI-EU-00000085
customer
GLASS-EU:5701
In the out ones, I've deleted INDEXED_EXTRACTIONS = json
And the events are empty, they are just these characters "},{"
So, the right one is to include both,
KV_Mode=none
INDEXED_EXTRACTIONS = json
Can you give more information or explain me a little more about the KV_MODE?
Why was it happening? So if I find the problem again I will understand it better. Also, post an anwser so I can click as right answer an upvote it!
Thank you a lot!
Hi marina,
try the following configuration.
BREAK_ONLY_BEFORE = (?=\{")
SHOULD_LINEMERGE = false
Do I have to re read everything again? or just restarting splunk?
Hi marina,
unfortunately you have to re-read everything again.
same 😞 don't I need another " character? Or does it undertand this well?
What I don't get is why it just doesn't work from the start... when I manually add the data over the web-gui splunk automatically notices the format and does the linebreaks in a correct format...
Can you show me some lines of the file you are trying to index?
And your props.conf configs with the stanza.
This data is coming from an API, so I'm using REST API app for getting it.
I can modify a sourcetype but I haven't done any far as setting that the data is coming with jason format.
The props is simple, is the basic + the lines we talked about above.
[angora_api_redirection_in_tasks]
CHARSET = ISO-8859-1
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = created_at
TIME_FORMAT = %s
category = Custom
pulldown_type = 1
LINE_BREAKER = (?=\{")
SHOULD_LINEMERGE = false
Should I change any of these parameters?
Thank you
From the screenshot in your question, the event seems to be a single event, right? It is just the field extractions that are duplicated somehow?
Did you write custom field extractions and also left KV_Mode to its default setting (auto)?
You're right for the first question.
Did you write custom field extractions and also left KV_Mode to its default setting (auto)? -> mmmm no custom field extractions and everything is by default except the thigs in the sourcetype below.
Should I? left KV_Mode, what do you mean?
KV_Mode (configurable in props.conf) instructs Splunk how to automatically extract key value pairs from events. In this case Splunk will detect this as JSON data and automatically extract the fields and their values.
I'm trying to understand why Splunk in your case duplicates the values for each of the fields...
He already knows it is jason data, and I'm trying to understand it too.
The thing I don't know is that, with the app, I don't know how to tell him to not read everything again and just add the newly data every 10 min aprox.
So now is reading it all I guess, maybe this is causing this behavior.
Not sure about this.... but I don't understand either.
Let me know if you come up with something please.
can you give sample data? in text
{"owned_by":null,"created_at":"1513607755.69364","priority":"PRI-444","ticket_source":"Email (Customer)","task_state":"Complete","cti":"CTI-EU-00001610","customer":null,"from_team":"TEAM-EU-00000210","ticket_state":"Closed","due_date":null,"to_team":"TEAM-EU-00000177","last_updated":"1515425723.04014","redirection_type":"Delegate","task":"TK-EU-28209292.1.1","impact":"General Request","ticket_type":"Internal","urgency":"Low"}