topic What configuration is required to index a single log with one event only, transforms.conf or props.conf? in Getting Data In

What configuration is required to index a single log with one event only, transforms.conf or props.conf?

AdsicSplunk — Wed, 31 Jan 2018 12:22:54 GMT

Hi,
My query is that Splunk indexer is indexing a single log with two separate events whereas it should be one event only.
The issue is that I am receiving two timestamps in a single log and I need Splunk to index it as a single event only.

Full Event Expected:-

[2018-01-31T15:23:25.470+04:00]...abc.....def...........ghi...........
......................................................................
......................................................................
  <ABC xmlns="http://tempuri.org/">
    <A>
      <ID>1234567</ID>
      <tickets>
        <DEF>
          <ticketNumber>12345</ticketNumber>
          <paidAmount>100</paidAmount>
          <paymentDateTime>2015-02-10T15:25:19Z</paymentDateTime>
          <receiptNumber>987654321</receiptNumber>
        </DEF>
      </tickets>
    </A>
  </ABC>

Received Event 1:-

[2018-01-31T15:23:25.470+04:00]...abc.....def...........ghi...........
......................................................................
......................................................................
  <ABC xmlns="http://tempuri.org/">
    <A>
      <ID>1234567</ID>
      <tickets>
        <DEF>
          <ticketNumber>12345</ticketNumber>
          <paidAmount>100</paidAmount>

Received Event 2:-

      <paymentDateTime>2015-02-10T15:25:19Z</paymentDateTime>
      <receiptNumber>987654321</receiptNumber>
    </DEF>
  </tickets>
</A>

Could anyone please suggest me how to proceed with this and what parameters to use for configuring props.conf or ?transforms.conf(if required)?

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

harsmarvania57 — Wed, 31 Jan 2018 12:39:30 GMT

Hi @AdsicSplunk,

Please try below configuration in props.conf on Indexer/Heavy Forwarder whichever comes first.

props.conf

[yoursourcetype]
TIME_PREFIX = ^\[
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%:z
MAX_TIMESTAMP_LOOKAHEAD = 29

Restart splunk on Indexer/Heavy Forwarder.

I hope this helps.

Thanks,
Harshil

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

adonio — Wed, 31 Jan 2018 12:46:55 GMT

hello there:

in inputs.conf:

[monitor://path.to.file]
index = index
sourcetype = your_sourcetype

in props.conf on indexer or heavy forwarder:

 [your_sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m%dT%H:%M%s.%3N:%z
TIME_PREFIX=[
MAX_TIMESTAMP_LOOKAHEAD=30
BREAK_ONLY_BEFORE=\

further reading regarding where to place files and which configurations goes in each file here:
https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

hope it helps

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

AdsicSplunk — Thu, 01 Feb 2018 05:37:35 GMT

Hi Harsmarvania,

Thank you for your support!!

I tried this config in props.conf but it got worse for me. Now, my indexer is creating even more events breaking each line and putting each line in a separate event. My question is that my event should not break into 2 events but should create one event only ignoring the second timestamp coming inside the event. Please read my questions, if you need some clarifications on this. please feel free to ask me questions.

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

harsmarvania57 — Thu, 01 Feb 2018 05:58:09 GMT

Hi @AdsicSplunk,

I tried with below configuration in splunk

props.conf

[mysourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N%:z
TIME_PREFIX=^\[
MAX_TIMESTAMP_LOOKAHEAD=29

And it is working perfectly fine.

Please refer screenshot https://imgur.com/a/BnQJ9

If this does not work for you, can you please let us know whether do you have any whitespace before [2018-01-31T15:23:25.470+04:00] ?

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

AdsicSplunk — Thu, 01 Feb 2018 06:00:31 GMT

Hi Adonio,

Thank you for your support!!
i tried this config but now the events are created not based on [2018-01-31T15:23:25.470+04:00] which was being picked without props.conf config. Now, this config has scattered the events and displaying the results which are not at all good. My requirement is to create events based on [2018-01-31T15:23:25.470+04:00] not based on 2015-02-10T15:25:19Z. Please provide your inputs.

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

AdsicSplunk — Thu, 01 Feb 2018 06:15:26 GMT

Hi @harsmarvania57,

This is the same config that Adonio provided.
I tried this config but now the events are created not based on [2018-01-31T15:23:25.470+04:00] which was being picked without props.conf config earlier. Now, this config has scattered the events and displaying the results which are not at all good. My requirement is to create events based on [2018-01-31T15:23:25.470+04:00] not based on 2015-02-10T15:25:19Z. Please provide your inputs.

Note:- There is no space in the timestamp. The data begins with "[" only.

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

AdsicSplunk — Thu, 01 Feb 2018 06:15:56 GMT

The screenshot is not accessible. Could you please share again?

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

harsmarvania57 — Thu, 01 Feb 2018 06:31:22 GMT

If you look at TIME_FORMAT parameter closely in config which is provided by @adonio and config which I have provided then there are difference . You can see screenshot here as well https://prnt.sc/i8i709

Re: What configuration is required to index a single log with one event only, transforms.conf or props.conf?

AdsicSplunk — Thu, 01 Feb 2018 06:59:01 GMT

I tried with both configs, still I am not getting what is required. Anyway, Thank you for your support, I will try again to get to the desired requirement.

Really appreciate your support.