Would this be a max active requests settings issue?
We're seeing a lot of these 500 errors to our HEC endpoint from Stream. And customers are reporting receiving 503 status codes. Would this be a max active requests settings issue? Maybe something else?
Answers
-
The HEC source will return a 503 if there are too many open connections. I'd recommend turning debug for the source to get more detail. This message is logged if w are closing because of too many connections: ```this.dLogger.debug('Dropping request because max active connections has been reached', {```
0 -
If this turns out to be the issue, this parameter can be modified to allow more concurrent connections.
0 -
I dont see where to turn on debug for the source. Is that one of the options under settings>logging?levels ?
0 -
We are thinking it may be the 'max active requests' but im unsure what the repercussions of raising that are. Would it be server performance? Would it be smarter to raise that up or to add more workers to the group?
0 -
To enable debug go to Group Settings -> Logging -> Levels and search for the source there.
0 -
Then commit and deploy the changes.
0 -
ahh those settings, ok thanks.
0 -
> Would it be server performance? Would it be smarter to raise that up or to add more workers to the group? I think the answer depends on how the current workers are performing, if they are over taxed then adding more workers would make sense. I'll let <@U012ZP93EER> <@U01C35EMQ01> or someone else chime in here.
0 -
Repercussions would be additional data being processed. How much? Depends on how much is rejected vs what's being accepted. So if you don't have any spare capacity then you may need to add a bit more if the amount of rejected requests is a non-trivial percentage of the total hitting the nodes. The limit is a throttling mechanism so depending on how much extra capacity was provisioned on your nodes will dictate whether that application level limit can be raised w/o changing anything else.
0 -
You might want to consider this setting, if you are sending high volumes:
0 -
Interested to know the use case for sending to the `raw` endpoint vs `event` as the latter batches and is more performant. :slightly_smiling_face:
0 -
<@UUP82DJHE> does the full debug message list the number of active connections?
0 -
Im wondering if there is a way to monitor and alert when we get close to the limit.
0 -
<@U04NPSXKJKV> been set like this for a while. We think we must have tried the event endpoint initially and it wasnt working but dont recall. We're going to test in dev if event will work or has any issues.
0 -
Yes, here's the full log message: ```debug('Dropping request because max active connections has been reached', { numActive: numInProgressConn, maxAllowed: numMaxActiveReq, statusCode: 503, });```
0 -
If the message is logged it means you hit the limit for that worker process configured for the worker group.
0 -
ok so it says its been reached, but there isnt an informational saying heres the current amount being used.
0 -
the `numMaxActiveReq` is a variable that contains an integer representing the current max
0
