Cribl supports receiving Windows events via Windows Event Forwarding (WEF). One method of authentication is done by Kerberos. This article lists some common issues and troubleshooting tips for Kerberos authentication.
Helpful Error Logs
Most of the relevant errors will not be logged on the Cribl side in Stream or in the supporting diagnostic files. Start with these Windows Event logs on the machine(s) trying to connect to the Stream WEF Source:
- Generic errors for the WEF plug-in): Application and Services Logs → Microsoft → Windows → Eventlog-ForwardingPlugin → Operational
- Root errors that may have caused the plug-in errors: Application and Services Logs → Microsoft → Windows → Windows Remote Management → Operational
Troubleshooting Kerberos and Keytab on the Worker Node
The below commands may be useful in troubleshooting the Worker node further:
Command: klist
Purpose: Displays the contents of a Kerberos credentials cache or key table.
Example 1: klist -k /path/to/keytab
List the tickets in a credentials cache. This is the default if neither the -c nor the -k flag is specified. This flag is mutually exclusive with the -k flag.
Example 2: klist -k -e /path/to/keytab
Display the encryption and SPN for the keytab.
*****
Command: kinit
Purpose: Obtains or renews the Kerberos ticket-granting ticket.
Example: kinit -kt /path/to/keytab http/<host>@DOMAIN
-k: Specifies to obtain the key for the ticket principal from a key table. If you do not specify this flag, you are prompted to enter the password for the ticket principal.
-t keytab_file: Specifies the key table name. The default key table is used if this flag is not specified and the -k flag is specified. The -t flag implies the -k flag.
*****
Command: xxd
Purpose: Creates a hex dump of a given file or standard input. It can also convert a hex dump back to its original binary form. Like uuencode and uudecode, it allows the transmission of binary data in a ‘mail-safe’ ASCII representation, but has the advantage of decoding to standard output. Moreover, it can be used to perform binary file patching.
Example: xxd /path/to/keytab
*****
Command: kdestroy
Purpose: The kdestroy utility destroys the user’s active Kerberos authorization tickets by overwriting and deleting the credentials cache that contains them. If the credentials cache is not specified, the default credentials cache is destroyed.
Example: kdestroy -p <principle name>
-p princ_name: If a cache collection is available, destroy the cache for princ_name instead of the primary cache. May be used with the -c option to specify the collection to be searched.
Troubleshooting Kerberos and Keytab on the KDC
Command: klist
Purpose: The klist command displays the contents of a Kerberos credentials cache or key table. List all Kerberos tickets.
Example: klist
*****
Command: klist purge
Purpose: Allows you to delete a specific ticket. Purging tickets destroys all tickets that you have cached, so use this attribute with caution. It might stop you from being able to authenticate to resources. If this happens, you'll have to log off and log on again
Example: klist -li <LogonID> purge
LogonID: Identified in hexadecimal.
*****
For additional klist commands, parameters, and examples, see the Microsoft documentation.
Additional Troubleshooting and Tips
- To enable kerberos event logging on a specific computer:
- Open Registry Editor
- Add the following registry value:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters
Registry Value: LogLevel
Value Type: REG_DWORD
Value Data: 0x1
- If a group policy is used, you may want to double check that the correct encryption types are set to support AES 128 and AES 256 encryption. See here for more.
- For deeper troubleshooting, a packet capture tool like Wireshark may help on the KDC to inspect kerberos handshaking. Run a 'kinit' on the Cribl Worker node calling the keytab with the SPN to force the handshake.
- Note on Load Balancing: The load balancer keep-alive/idle socket timeout must be set higher than the subscription batch timeout (or the lowest batch timeout if multiple subscriptions are configured). In the Stream WEF Source, check that the WEF Advanced → Keep-alive timeout setting is higher than the smallest subscription batch timeout.
Working with Cribl Support
If you need to submit a ticket to get assistance from Cribl, along with a diagnostic bundle, it can be helpful to provide an output from the resultant set of policies (RSOP). To do so, run the following command from an administrator PowerShell window (NOTE: you may need to adjust the output location to a directory that exists; this uses 'temp' as an example):
Get-GPResultantSetOfPolicy -ReportType Html -Path "c:\temp\LocalUserAndComputerReport.Html"
