syslog output to ALB showing TCP endpoint errors
I'm working with our security team who's wanting me to send events to his AWS ALB syslog destination which has multiple syslog nodes behind it.
He's getting events, but the cribl UI is consistantly showing the destination as experiencing issues. It's going through a loop of successfully establishing a connection, 1 minute later the sender disconnects it, 1 minute later it restablishes
I do not experience this same thing when I use a non-loadbalanced syslog destination.
Does anyone have thoughts on how to resolve or even troubleshoot this? Would using TLS help?
Answers
-
Hey @Jason Rehm, AWS ALBs are for Layer 7 (HTTP) traffic. I'd recommend you look into using an NLB instead which is designed for Layer 4 (TCP) traffic.
0 -
blah - also it looks like my eyes were deceiving me yesterday. The original destination is an ELB (not an ALB)
(aws resource ends in 'elb.us-east-1.amazonaws.com"0 -
Thanks @Brendan Dalpe, I just tested an NLB and the result is indeed a bit different.
I still see errors in the Destinations "Log" tab, but they are not constant and the destination is not consistantly marked as "Red:Having Severe Issues".
Are these logs marked as level=error normal then and just telling us that the connection went idle and we needed to start a new connection?
0 -
@Jason Rehm is this a syslog-ng box you're sending to from Cribl? I see port 601 in use here.
0 -
If you're indeed using syslog-ng, note that by default, port 601 expects RFC5424 formatted messages with "octet count framing" enabled.
In Cribl Stream, you'll need to enable the Octet Count Framing toggle under Advanced Settings for your Syslog destination. Make sure you select RFC5424 for the Message Format on the General Settings tab.
0
