Skip to content

Troubleshooting Guide

This guide helps you diagnose and resolve common issues with ZeroTrustKerberosLink. Each section focuses on a specific area where problems might occur.

Table of Contents

  1. Authentication Issues
  2. AWS Integration Issues
  3. Configuration Issues
  4. Performance Issues
  5. Security Issues
  6. Logging and Monitoring Issues
  7. Deployment Issues

Authentication Issues

Kerberos Authentication Failures

Symptoms: - Error message: "Kerberos authentication failed" - HTTP 401 Unauthorized responses - Client unable to obtain service tickets

Common Causes and Solutions:

  1. Invalid Keytab File
  2. Verify keytab file exists and has correct permissions: 
    ls -la /etc/zerotrustkerberos/krb5.keytab
# Should show: -r-------- 1 zerotrustkerberos zerotrustkerberos

  3. Check keytab content: 

    klist -kt /etc/zerotrustkerberos/krb5.keytab

  4. Clock Skew

  5. Verify time synchronization: 
    chronyc tracking
# or
ntpq -p

  6. Ensure time difference is less than 5 minutes between client, KDC, and ZeroTrustKerberosLink

  7. Incorrect Service Principal

  8. Confirm service principal in configuration matches keytab: 

    grep service_principal /etc/zerotrustkerberos/config.yaml
klist -kt /etc/zerotrustkerberos/krb5.keytab | grep HTTP

  9. Network Connectivity Issues

  10. Test connectivity to Kerberos KDC: 
    telnet kdc.example.com 88

User Authentication Failures

Symptoms: - Users can't authenticate despite valid Kerberos tickets - Error message: "User not authorized"

Common Causes and Solutions:

  1. Missing Role Mapping
  2. Check role mappings in configuration: 
    grep -A 20 role_mappings /etc/zerotrustkerberos/config.yaml

  3. Add appropriate mapping for user principal

  4. Principal Format Mismatch

  5. Ensure principal format matches exactly: 

    # Configuration: admin@EXAMPLE.COM
# User ticket: Admin@EXAMPLE.COM (case mismatch)

  6. Context Evaluation Failure

  7. Check context evaluation logs: 
    grep "context evaluation" /var/log/zerotrustkerberos/app.log
  8. Verify user meets all context requirements (IP, time, etc.)

AWS Integration Issues

Role Assumption Failures

Symptoms: - Error message: "Failed to assume role" - HTTP 403 Forbidden responses

Common Causes and Solutions:

  1. Invalid IAM Role ARN
  2. Verify role exists: 
    aws iam get-role --role-name AdminRole

  3. Check role ARN format in configuration

  4. Trust Relationship Issues

  5. Examine trust policy: 
    aws iam get-role --role-name AdminRole --query 'Role.AssumeRolePolicyDocument'

  6. Update trust policy to allow ZeroTrustKerberosLink to assume the role

  7. Permission Issues

  8. Verify ZeroTrustKerberosLink has sts:AssumeRole permission: 
    aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/ZeroTrustKerberosLinkRole \
  --action-names sts:AssumeRole

AWS Credential Issues

Symptoms: - Error message: "Unable to retrieve AWS credentials" - AWS API calls fail after authentication

Common Causes and Solutions:

  1. AWS Region Configuration
  2. Verify AWS region in configuration: 
    grep region /etc/zerotrustkerberos/config.yaml

  3. Ensure region is valid and accessible

  4. AWS API Throttling

  5. Check for throttling errors in logs: 
    grep "throttling" /var/log/zerotrustkerberos/app.log

  6. Implement exponential backoff for API calls

  7. Temporary Credential Expiration

  8. Verify session duration settings: 
    grep -A 5 session /etc/zerotrustkerberos/config.yaml
  9. Adjust session duration as needed (max 12 hours)

Configuration Issues

Syntax Errors

Symptoms: - Application fails to start - Error message: "Invalid configuration"

Common Causes and Solutions:

  1. YAML Syntax Errors
  2. Validate YAML syntax: 
    yamllint /etc/zerotrustkerberos/config.yaml

  3. Fix indentation and formatting issues

  4. Missing Required Fields

  5. Check for all required configuration fields: 
    zerotrustkerberos-cli validate-config --config /etc/zerotrustkerberos/config.yaml

  6. Add missing required fields

  7. Invalid Values

  8. Verify all configuration values are valid: 
    zerotrustkerberos-cli validate-config --strict --config /etc/zerotrustkerberos/config.yaml
  9. Correct any invalid values

Environment Variable Issues

Symptoms: - Configuration doesn't match expected values - Overrides not taking effect

Common Causes and Solutions:

  1. Case Sensitivity

  2. Environment variables are case-sensitive: 

    # Correct: ZEROTRUST_SERVER_PORT=8443
# Incorrect: zerotrust_server_port=8443

  3. Variable Format

  4. Check variable format: 

    # Correct: ZEROTRUST_REDIS_HOST=redis.example.com
# Incorrect: ZEROTRUST_REDIS.HOST=redis.example.com

  5. Variable Precedence

  6. Understand precedence order:
    1. Command-line arguments
    2. Environment variables
    3. Configuration file
    4. Default values

Performance Issues

High Latency

Symptoms: - Authentication takes more than 1 second - AWS role assumption is slow

Common Causes and Solutions:

  1. Insufficient Resources
  2. Check CPU and memory usage: 
    top -p $(pgrep -f zerotrustkerberos)

  3. Increase allocated resources

  4. Network Latency

  5. Test network latency to AWS and KDC: 
    ping kdc.example.com
ping sts.amazonaws.com

  6. Consider deploying closer to AWS region

  7. Inefficient Caching

  8. Enable or optimize Redis caching: 
    cache:
  enabled: true
  ttl: 300  # seconds

Connection Pooling Issues

Symptoms: - Increasing latency under load - Connection-related errors

Common Causes and Solutions:

  1. Insufficient Connection Pool Size

  2. Increase connection pool size: 

    aws:
  connection_pool_size: 20

  3. Connection Leaks

  4. Check for connection leaks in logs: 
    grep "connection" /var/log/zerotrustkerberos/app.log

  5. Implement proper connection closing

  6. Connection Timeouts

  7. Adjust timeout settings: 
    aws:
  timeout: 5  # seconds

Security Issues

TLS Configuration Issues

Symptoms: - TLS handshake failures - Certificate validation errors

Common Causes and Solutions:

  1. Invalid Certificate
  2. Verify certificate validity: 
    openssl x509 -in /etc/zerotrustkerberos/tls/certificate.crt -text -noout

  3. Ensure certificate is not expired

  4. Cipher Suite Compatibility

  5. Check cipher suite configuration: 
    grep -A 5 tls /etc/zerotrustkerberos/config.yaml

  6. Update cipher suites for better compatibility

  7. TLS Version Issues

  8. Verify minimum TLS version: 
    server:
  tls:
    min_version: "TLSv1.2"
  9. Update clients to support required TLS version

Security Header Issues

Symptoms: - Security scanners report missing headers - Browser security warnings

Common Causes and Solutions:

  1. Missing Headers
  2. Check header configuration: 
    grep -A 10 headers /etc/zerotrustkerberos/config.yaml

  3. Add missing security headers

  4. Invalid Header Values

  5. Verify header values: 
    curl -I https://zerotrustkerberos.example.com/auth/test

  6. Correct invalid header values

  7. CSP Configuration

  8. Test Content Security Policy: 
    curl -I https://zerotrustkerberos.example.com/auth/test | grep Content-Security-Policy
  9. Adjust CSP for required functionality

Logging and Monitoring Issues

Missing Logs

Symptoms: - Events not appearing in logs - Incomplete audit trail

Common Causes and Solutions:

  1. Log Level Too High
  2. Check log level configuration: 
    grep level /etc/zerotrustkerberos/config.yaml

  3. Set appropriate log level (debug, info, warn, error)

  4. Log Destination Issues

  5. Verify log destination: 
    grep output /etc/zerotrustkerberos/config.yaml

  6. Ensure log destination is writable

  7. Log Rotation Issues

  8. Check log rotation configuration: 
    cat /etc/logrotate.d/zerotrustkerberos
  9. Configure proper log rotation

Monitoring Integration Issues

Symptoms: - Metrics not appearing in monitoring system - Alerts not triggering

Common Causes and Solutions:

  1. Prometheus Endpoint Issues
  2. Verify Prometheus endpoint: 
    curl http://localhost:8080/metrics

  3. Check Prometheus scrape configuration

  4. Metric Format Issues

  5. Validate metric format: 
    curl http://localhost:8080/metrics | grep auth_requests

  6. Correct any metric format issues

  7. Alert Configuration Issues

  8. Review alert configuration: 
    grep -A 10 alerts /etc/zerotrustkerberos/config.yaml
  9. Adjust alert thresholds and conditions

Deployment Issues

Docker Deployment Issues

Symptoms: - Container fails to start - Container exits unexpectedly

Common Causes and Solutions:

  1. Volume Mount Issues
  2. Check volume mounts: 
    docker inspect zerotrustkerberos | grep -A 10 Mounts

  3. Correct volume mount paths and permissions

  4. Environment Variable Issues

  5. Verify environment variables: 
    docker inspect zerotrustkerberos | grep -A 20 Env

  6. Set missing or correct invalid environment variables

  7. Container Resource Limits

  8. Check resource limits: 
    docker stats zerotrustkerberos
  9. Adjust CPU and memory limits as needed

Kubernetes Deployment Issues

Symptoms: - Pods fail to start - Pods crash loop

Common Causes and Solutions:

  1. ConfigMap Issues
  2. Verify ConfigMap: 
    kubectl describe configmap zerotrustkerberos-config -n zerotrustkerberos

  3. Update ConfigMap with correct configuration

  4. Secret Issues

  5. Check Secret: 
    kubectl describe secret zerotrustkerberos-secrets -n zerotrustkerberos

  6. Ensure Secret contains required data

  7. Resource Constraints

  8. Examine pod resources: 
    kubectl describe pod -l app=zerotrustkerberos -n zerotrustkerberos
  9. Adjust resource requests and limits

Diagnostic Tools

ZeroTrustKerberosLink includes several built-in diagnostic tools:

Health Check

curl http://localhost:8080/health

The health check returns detailed information about: - Service status - Component health - Resource utilization - Recent errors

Configuration Validator

zerotrustkerberos-cli validate-config --config /etc/zerotrustkerberos/config.yaml

The validator checks: - Configuration syntax - Required fields - Value constraints - Security best practices

Connectivity Tester

zerotrustkerberos-cli test-connectivity

Tests connectivity to: - Kerberos KDC - AWS services - Redis cache - Monitoring endpoints

Getting Help

If you're unable to resolve an issue using this guide:

  1. Check Documentation
  2. Review the Implementation Guide

  3. Check the FAQ

  4. Search Known Issues

  5. Visit the GitHub Issues

  6. Search for similar problems

  7. Contact Support

  8. Email: support@zerotrustkerberos.com
  9. Include:
    • Detailed description of the issue
    • Configuration files (with sensitive data removed)
    • Relevant log excerpts
    • Steps to reproduce the issue