Indexed only Partial Contents of log file

[dhirendra761]

Hi,

We are monitoring whole file in index. As file is in huge in size. which indexed all the content of files.

But we require only specific part of files to be indexed.

SAMPLE DATA:

{"quiz": { "sport": { "q1": { "question": "Which one is correct team name in NBA?", "options": [ "New York Bulls", "Los Angeles Kings", "Golden State Warriros", "Huston Rocket" ], "answer": "Huston Rocket" } }, "maths": { "q1": { "question": "5 + 7 = ?", "options": [ "10", "11", "12", "13" ], "answer": "12" }, "q2": { "question": "12 - 8 = ?", "options": [ "1", "2", "3", "4" ], "answer": "4" } } } }

Sample SPL:  

index="test" "answer"|<further spl>

How to indexed partial data of file for answer string, Not to be indexed whole file.

Thank you in advance for your help! 

[gcusello]

Hi @dhirendra761,

it's possible to truncate a log event defining the lenght of each event, but, having a json format, in this way you loose the json format and the choice to use spath command to extract fields, so you have to manually extract all the fields, so I hint to avoid.

Maybe (I'm not sure) it's possible to identify a part of the log event that can be removed (using the SEDCMD command in props.conf) maintaining the json structure, but it isn't so easy! 

Ciao.

Giuseppe

[dhirendra761]

Hi @gcusello ,

Thank you for responce. In fact, file content are mixed-syntax. some of lines are json format and log-info-type format.

2024-02-08 | 23.118 | <hostname> | DEBUG | QueryForSuccess

 we run the specify content with different search string . I agree defining SEDCMD is not easy.

Any other way where we can prevent unused data and indexed only wanted data.?

[gcusello]

Hi @dhirendra761,

sorry no: the only ways to remove part of events are TRUNCATE or SEDCMD or transforms.

You can also remove the full event before indexing.

Ciao.

Giuseppe