topic Re: Indexed only Partial Contents of log file in Getting Data In

Indexed only Partial Contents of log file

dhirendra761 — Tue, 05 Mar 2024 15:51:54 GMT

Hi,

We are monitoring whole file in index. As file is in huge in size. which indexed all the content of files.

But we require only specific part of files to be indexed.

SAMPLE DATA:

{"quiz": { "sport": { "q1": { "question": "Which one is correct team name in NBA?", "options": [ "New York Bulls", "Los Angeles Kings", "Golden State Warriros", "Huston Rocket" ], "answer": "Huston Rocket" } }, "maths": { "q1": { "question": "5 + 7 = ?", "options": [ "10", "11", "12", "13" ], "answer": "12" }, "q2": { "question": "12 - 8 = ?", "options": [ "1", "2", "3", "4" ], "answer": "4" } } } }

Sample SPL:

index="test" "answer"|<further spl>

How to indexed partial data of file for answer string, Not to be indexed whole file.

Thank you in advance for your help!

Re: Indexed only Partial Contents of log file

gcusello — Tue, 05 Mar 2024 16:02:58 GMT

Hi @dhirendra761,

it's possible to truncate a log event defining the lenght of each event, but, having a json format, in this way you loose the json format and the choice to use spath command to extract fields, so you have to manually extract all the fields, so I hint to avoid.

Maybe (I'm not sure) it's possible to identify a part of the log event that can be removed (using the SEDCMD command in props.conf) maintaining the json structure, but it isn't so easy!

Ciao.

Giuseppe

Re: Indexed only Partial Contents of log file

dhirendra761 — Tue, 05 Mar 2024 16:19:44 GMT

Hi @gcusello ,

Thank you for responce. In fact, file content are mixed-syntax. some of lines are json format and log-info-type format.

2024-02-08 | 23.118 | <hostname> | DEBUG | QueryForSuccess

we run the specify content with different search string . I agree defining SEDCMD is not easy.

Any other way where we can prevent unused data and indexed only wanted data.?

Re: Indexed only Partial Contents of log file

gcusello — Tue, 05 Mar 2024 16:29:42 GMT

Hi @dhirendra761,

sorry no: the only ways to remove part of events are TRUNCATE or SEDCMD or transforms.

You can also remove the full event before indexing.

Ciao.

Giuseppe