How can we get cribl to keep the time within the log?
How can we get cribl to keep the time within the log?
Answers
-
Is your data coming into Splunk without a time field?
0 -
Yeah, I also don't really get the question
0 -
sorry, I was going to add more detail but got called away. The time in the event, is the local time for the system where the log is generated. We do not want Splunk or Cribl to adjust the time
0 -
We thought by using auto timestamp and setting it to local would keep the local tme.
0 -
What is the timestamp you are expecting from the original event in Cribl? I can think of a few ways to maintain that is its in the Cribl _raw
0 -
There are no other timestamps in the event, other than the first one. Here is a snip of the raw event. 2023/03/07 10:52:49.815 stAgentSvc p581c t34f0 info tunnel.cpp:841 nsTunnel TLS [sessId 1] Tunneling flow
0 -
Local means the timezone on the Cribl worker itself. If you want to adjust the time to UTC, then you'll need to specify UTC in the "Default Timezone" box. I would also encourage you to adjust this on your Event Breaker/Source. For Syslog, this is under Advanced Settings.
0 -
Are Cribl and that source running on the same timezone?
0 -
If not, how should Cribl (or Splunk) crystalbally guess the timezone on the remote box?
0 -
Also, the display in Splunk is based on the Splunk users timezone setting
0 -
The logs are from systems all over the world, and it looks like the log entry uses the system time.
0 -
Yeah, then that's the issue you have
0 -
Syslog is an old and crappy standard that never included timezones because, to be fair, when it was invented nobody thought about worldwide log collection and stuff
0 -
My $0.02 here... use a lookup to determine the sender's timezone. Use this to then feed to the `C.Time.adjustTz()` function. I do this in the PAN pack. https://github.com/criblpacks/cribl-palo-alto-networks/blob/master/default/pipelines/pan_traffic/conf.yml#L227-L228
0 -
these are not syslog, these are collected via the Splunk UF installed on the system
0 -
Ah, you're right, my concentration was disturbed by a 4 year old gremlin
0 -
Mh, UFs should transport the system time zone but I vaguely remember that that doesn't work super well and/or wasn't supported by Cribl
0 -
yep, that is what is odd. I do not remember really ever running into this before
0 -
I assume most of the time your logs had timezone info, or the sender and Cribl where in the same time zone
0 -
Default Time Zone for the Splunk TCP source is also `Local` . <@UQA16GHUN> can you run the command `date` on your Stream worker to see the configured time zone?
0 -
You can use C.Time.strftime and grab the Cribl raw _time and convert all of it to UTC if you're looking for a singular time zone use. The last parameter of True/False is the setting you'll want to look at
0 -
The issue here is that these logs come from a timezone different than the Cribl timezone and that zone information is missing from the event
0 -
So you effectively have to keep a lookup or something else to derive the timezone from hostname etc
0 -
I see what you're saying
0 -
Or, what you can do, if your logs have no delay, determine difference between recognized timestamp and current time. That delay is likely very close to a multiple of 1 hour. If that's the case, use that multiple and add/subtract it from the timestamp
0 -
Brendan, I am having the linux engineer run that when thye get back, but I am almost 100% positive that it is in UTC. All servers/appliances are set to UTC
0 -
Same for Cribl?
0 -
I think what I am going to do is just use index time. It is not ideal, but to keep up with all the 1000's of desktops is not sustainable.
0 -
The latest release of Cribl supports S2S v4 which would accept the OS time when the UF connects to Cribl.
0 -
Does that help at least for the Splunk side of things?
0
