Python Logging Basics for Developers

Logging stands as one of the most undervalued yet critical skills in a developer's toolkit. When applications fail in production, when performance degrades mysteriously, or when security incidents occur, your logging strategy becomes the difference between quick resolution and hours of painful debugging. Poor logging practices lead to blind spots in your application's behavior, making troubleshooting feel like searching for a needle in a haystack while blindfolded.

Python's logging framework provides a sophisticated, flexible system for tracking events, errors, and application flow. Rather than relying on scattered print statements that disappear in production or clutter your codebase, proper logging creates a structured record of your application's lifecycle. This comprehensive guide explores logging from multiple angles: the technical implementation, best practices for different environments, performance considerations, and real-world patterns that separate amateur codebases from production-ready systems.

You'll discover how to implement logging configurations that scale from simple scripts to complex distributed systems, understand the hierarchy of loggers and handlers, master formatting techniques that make logs actionable, and learn strategies for different deployment scenarios. Whether you're debugging a local script or managing logs across microservices, this guide provides the practical knowledge to make logging work for you rather than against you.

Understanding Python's Logging Architecture

Python's logging module follows a sophisticated architecture that separates concerns and provides remarkable flexibility. At its core, the system consists of four main components that work together: loggers, handlers, filters, and formatters. Understanding how these pieces interact transforms logging from a mysterious black box into a powerful tool you can shape to your exact needs.

The logger serves as the entry point for your application code. When you call logger.info() or logger.error(), you're interacting with a logger object. Loggers form a hierarchy based on their names, using dot notation similar to Python modules. A logger named "myapp.database" is a child of "myapp", and this relationship affects how log messages propagate through the system.

Handlers determine where log messages go—whether to console output, files, network sockets, or external services. A single logger can have multiple handlers, allowing you to simultaneously write errors to a file while displaying warnings on the console. This separation means you can adjust output destinations without touching your application code.

"The biggest mistake developers make is treating logging as an afterthought. By the time you need detailed logs, it's already too late to add them."

The Logger Hierarchy and Propagation

Logger names create an implicit hierarchy that mirrors your application structure. When you create a logger named "myapp.authentication.oauth", it automatically becomes a child of "myapp.authentication" and a grandchild of "myapp". This hierarchy enables powerful configuration patterns where settings cascade from parent to child loggers.

Message propagation flows upward through this hierarchy by default. When a child logger emits a message, it passes through the child's handlers first, then propagates to parent loggers and their handlers. This behavior allows you to set up a root-level handler that catches all messages while adding specialized handlers for specific subsystems. You can disable propagation for any logger by setting its propagate attribute to False.

import logging

# Create hierarchical loggers
root_logger = logging.getLogger('myapp')
db_logger = logging.getLogger('myapp.database')
auth_logger = logging.getLogger('myapp.authentication')

# Configure root logger
root_logger.setLevel(logging.INFO)
console_handler = logging.StreamHandler()
root_logger.addHandler(console_handler)

# Child loggers inherit root configuration
db_logger.info("This message propagates to root logger")

# Disable propagation for specific logger
auth_logger.propagate = False
auth_logger.info("This message stays with auth_logger handlers only")

Log Levels and When to Use Each

Python defines five standard log levels, each representing a different severity of information. Choosing the appropriate level for each message creates a filtering system that allows you to adjust verbosity without code changes. The levels form a hierarchy where setting a logger to a particular level shows messages at that level and above.

• DEBUG (10) – Detailed diagnostic information useful during development and troubleshooting. Use for variable values, function entry/exit points, and detailed state information.
• INFO (20) – Confirmation that things are working as expected. Use for significant application events like service startup, configuration loading, or successful completion of major operations.
• WARNING (30) – Indication of potential problems or unexpected situations that don't prevent operation. Use for deprecated API usage, configuration issues with fallbacks, or resource constraints.
• ERROR (40) – Serious problems that prevented a specific operation from completing. Use for caught exceptions, failed external API calls, or database connection failures.
• CRITICAL (50) – Very serious errors that may cause the application to terminate. Use for unrecoverable system failures, exhausted resources, or critical security violations.

The numeric values matter because they determine filtering behavior. When you set a logger's level to WARNING (30), it will display WARNING, ERROR, and CRITICAL messages while suppressing DEBUG and INFO. This filtering happens before handlers process messages, making it computationally efficient.

"Log levels aren't just about severity—they're about audience. DEBUG is for developers, INFO is for operators, WARNING is for both, and ERROR is for everyone including automated alerting systems."

Strategic Level Selection in Practice

Effective logging requires thinking about who will read each message and under what circumstances. DEBUG messages should help a developer understand program flow without needing a debugger. Include loop iterations, conditional branch decisions, and intermediate calculation results. However, be judicious—excessive DEBUG logging can overwhelm and obscure important information.

INFO messages tell the story of your application's operation. They should allow someone unfamiliar with the code to understand what the application is doing. Log user actions, business process completions, and significant state transitions. In a web application, you might log each request with its endpoint and response status. In a data pipeline, log each processing stage completion with record counts.

WARNING messages deserve careful consideration because they represent the gray area between normal operation and problems. Use warnings for situations that might become errors under different conditions or that indicate suboptimal behavior. A missing optional configuration file warrants a warning. A slow database query that still completes might warrant a warning if it exceeds expected thresholds.

ERROR and CRITICAL messages should be actionable. Every error should represent something that needs investigation or fixing. Avoid logging expected error conditions at ERROR level—if your application regularly encounters a situation and handles it gracefully, it's probably a WARNING at most. Reserve CRITICAL for situations that threaten the entire application's viability.

Basic Logging Configuration

Python provides multiple approaches to configure logging, ranging from simple one-liners for scripts to sophisticated dictionary-based configurations for production systems. The simplest approach uses basicConfig(), which sets up a root logger with reasonable defaults. While quick for prototypes, this approach lacks the flexibility needed for production applications.

import logging

# Simple configuration for scripts
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

logger = logging.getLogger(__name__)
logger.info("Application started")
logger.debug("This won't appear because level is INFO")

The format parameter accepts a string containing LogRecord attributes as placeholders. Common attributes include asctime (formatted timestamp), name (logger name), levelname (log level), message (the log message), pathname (source file), lineno (line number), and funcName (function name). Choosing the right attributes balances information richness with log readability.

Programmatic Configuration for Flexibility

For applications requiring more control, programmatic configuration creates loggers, handlers, and formatters explicitly. This approach provides maximum flexibility and makes the logging setup clear and maintainable. You can adjust configurations based on runtime conditions, such as different settings for development versus production environments.

import logging
import sys

def setup_logging(log_level=logging.INFO):
    # Create logger
    logger = logging.getLogger('myapp')
    logger.setLevel(log_level)
    
    # Console handler for all messages
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setLevel(logging.DEBUG)
    console_formatter = logging.Formatter(
        '%(levelname)-8s %(name)s: %(message)s'
    )
    console_handler.setFormatter(console_formatter)
    
    # File handler for warnings and above
    file_handler = logging.FileHandler('app.log')
    file_handler.setLevel(logging.WARNING)
    file_formatter = logging.Formatter(
        '%(asctime)s %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s',
        datefmt='%Y-%m-%d %H:%M:%S'
    )
    file_handler.setFormatter(file_formatter)
    
    # Add handlers to logger
    logger.addHandler(console_handler)
    logger.addHandler(file_handler)
    
    return logger

# Use the configured logger
app_logger = setup_logging()
app_logger.info("Application initialized")
app_logger.warning("This appears in both console and file")
app_logger.debug("This appears only in console")

Dictionary-Based Configuration

For complex applications, dictionary-based configuration provides the best balance of power and maintainability. This approach separates configuration from code, allows loading from external files (YAML, JSON), and supports comprehensive logging setups in a structured format. The configuration dictionary follows a specific schema that defines loggers, handlers, formatters, and their relationships.

import logging.config
import yaml

# Configuration as dictionary
LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        },
        'detailed': {
            'format': '%(asctime)s [%(levelname)s] %(name)s.%(funcName)s:%(lineno)d - %(message)s'
        }
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'DEBUG',
            'formatter': 'standard',
            'stream': 'ext://sys.stdout'
        },
        'file': {
            'class': 'logging.handlers.RotatingFileHandler',
            'level': 'INFO',
            'formatter': 'detailed',
            'filename': 'app.log',
            'maxBytes': 10485760,  # 10MB
            'backupCount': 5
        }
    },
    'loggers': {
        'myapp': {
            'level': 'DEBUG',
            'handlers': ['console', 'file'],
            'propagate': False
        }
    },
    'root': {
        'level': 'INFO',
        'handlers': ['console']
    }
}

# Apply configuration
logging.config.dictConfig(LOGGING_CONFIG)
logger = logging.getLogger('myapp')
logger.info("Dictionary configuration loaded successfully")

Handlers for Different Output Destinations

Handlers determine where log messages ultimately end up, and Python's logging module includes numerous handler types for different scenarios. Choosing appropriate handlers based on your deployment environment, performance requirements, and operational needs significantly impacts your application's observability and debuggability.

🔧 StreamHandler writes to any file-like object, most commonly sys.stdout or sys.stderr. This handler works perfectly for containerized applications where logs should go to standard output for collection by orchestration platforms like Kubernetes. The simplicity of StreamHandler makes it ideal for development environments where immediate console feedback helps rapid iteration.

📁 FileHandler writes messages to a specified file on disk. While straightforward, basic FileHandler has a significant limitation: files grow indefinitely. For long-running applications, this eventually causes disk space problems. FileHandler works well for applications with short lifespans or when external log rotation tools manage file sizes.

🔄 RotatingFileHandler automatically manages log file sizes by rotating to a new file when the current file reaches a specified size. When rotation occurs, the current file gets renamed with a numeric suffix, and a new file starts. This handler maintains a specified number of backup files, automatically deleting the oldest when creating new backups. This approach prevents disk space exhaustion while preserving recent logs.

from logging.handlers import RotatingFileHandler

# Rotate after 10MB, keep 5 backup files
rotating_handler = RotatingFileHandler(
    'application.log',
    maxBytes=10*1024*1024,  # 10MB
    backupCount=5
)
rotating_handler.setLevel(logging.INFO)

# Results in files: application.log, application.log.1, application.log.2, etc.

📅 TimedRotatingFileHandler rotates logs based on time intervals rather than file size. You can configure rotation daily, weekly, at midnight, or on custom schedules. This handler proves particularly useful for compliance requirements that mandate daily log files or when correlating logs with time-based events. The handler automatically adds timestamps to rotated filenames.

"The right handler setup means the difference between having logs when you need them and drowning in gigabytes of unmanageable text files."

Network and System Handlers

🌐 SocketHandler sends log messages over TCP/IP to a remote logging server. This approach centralizes logs from distributed applications, making it easier to search and correlate events across multiple services. However, network handlers introduce dependencies—if the logging server becomes unavailable, your application must handle the failure gracefully.

📧 SMTPHandler sends log messages via email, typically used for ERROR and CRITICAL level messages that require immediate attention. Configure this handler with SMTP server details, recipient addresses, and authentication credentials. Be cautious with this handler—high error rates can flood email inboxes, and email delivery adds latency to your logging pipeline.

📋 SysLogHandler integrates with Unix syslog daemons, allowing Python applications to participate in system-wide logging infrastructure. This handler works well in traditional server environments where centralized syslog collection already exists. Modern containerized deployments often prefer StreamHandler with external log aggregation instead.

from logging.handlers import SysLogHandler, SMTPHandler

# Syslog integration
syslog_handler = SysLogHandler(address='/dev/log')
syslog_handler.setLevel(logging.WARNING)

# Email critical errors
smtp_handler = SMTPHandler(
    mailhost=('smtp.example.com', 587),
    fromaddr='app@example.com',
    toaddrs=['admin@example.com'],
    subject='Application Critical Error',
    credentials=('username', 'password'),
    secure=()
)
smtp_handler.setLevel(logging.CRITICAL)

Custom Handlers for Specialized Needs

When built-in handlers don't meet your requirements, creating custom handlers extends the logging system to integrate with any service or storage mechanism. Custom handlers inherit from logging.Handler and implement the emit() method, which receives LogRecord objects and processes them according to your needs.

import logging
import requests

class SlackHandler(logging.Handler):
    def __init__(self, webhook_url, channel):
        super().__init__()
        self.webhook_url = webhook_url
        self.channel = channel
    
    def emit(self, record):
        try:
            log_entry = self.format(record)
            payload = {
                'channel': self.channel,
                'text': log_entry,
                'username': 'AppLogger'
            }
            requests.post(self.webhook_url, json=payload, timeout=5)
        except Exception:
            self.handleError(record)

# Use custom handler
slack_handler = SlackHandler(
    webhook_url='https://hooks.slack.com/services/YOUR/WEBHOOK/URL',
    channel='#alerts'
)
slack_handler.setLevel(logging.ERROR)
logger.addHandler(slack_handler)

Formatting Log Messages for Readability

Log formatting transforms raw LogRecord objects into readable text that conveys necessary information without overwhelming readers. Effective formats balance completeness with readability, including enough context to understand what happened while remaining scannable when reviewing hundreds or thousands of log lines.

The Formatter class accepts a format string containing LogRecord attributes as placeholders. These attributes provide access to every aspect of the logging event: timestamp, logger name, level, message, source file location, thread information, process ID, and more. Selecting the right subset of attributes depends on your application's complexity and operational requirements.

import logging

# Minimal format for simple applications
simple_formatter = logging.Formatter('%(levelname)s: %(message)s')

# Standard format with timestamp and logger name
standard_formatter = logging.Formatter(
    '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Detailed format for debugging
detailed_formatter = logging.Formatter(
    '%(asctime)s [%(levelname)-8s] %(name)s.%(funcName)s:%(lineno)d - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

# Production format with process/thread info
production_formatter = logging.Formatter(
    '%(asctime)s [%(process)d:%(thread)d] %(levelname)-8s %(name)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

Structured Logging with JSON

Structured logging formats messages as JSON objects rather than plain text, making logs machine-parseable and enabling sophisticated analysis with log aggregation tools. Each log entry becomes a JSON object with fields for timestamp, level, message, and any additional context. This approach dramatically improves searchability and allows creating dashboards and alerts based on specific log attributes.

"Structured logging isn't just a technical choice—it's an investment in your future ability to understand what your application is doing in production."

import logging
import json
from datetime import datetime

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_object = {
            'timestamp': datetime.utcnow().isoformat(),
            'level': record.levelname,
            'logger': record.name,
            'message': record.getMessage(),
            'module': record.module,
            'function': record.funcName,
            'line': record.lineno
        }
        
        # Include exception info if present
        if record.exc_info:
            log_object['exception'] = self.formatException(record.exc_info)
        
        # Include extra fields if present
        if hasattr(record, 'user_id'):
            log_object['user_id'] = record.user_id
        if hasattr(record, 'request_id'):
            log_object['request_id'] = record.request_id
            
        return json.dumps(log_object)

# Use JSON formatter
json_handler = logging.StreamHandler()
json_handler.setFormatter(JSONFormatter())
logger = logging.getLogger('myapp')
logger.addHandler(json_handler)

# Log with extra context
logger.info('User login', extra={'user_id': 12345, 'request_id': 'abc-123'})

Custom Formatting Techniques

Beyond standard attributes, custom formatters can calculate derived values, mask sensitive information, or add environment-specific context. Subclassing Formatter and overriding the format() method provides complete control over log message structure and content.

import logging
import re

class SensitiveDataFormatter(logging.Formatter):
    """Formatter that masks sensitive information like credit cards and SSNs"""
    
    PATTERNS = {
        'credit_card': re.compile(r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b'),
        'ssn': re.compile(r'\b\d{3}-\d{2}-\d{4}\b'),
        'email': re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b')
    }
    
    def format(self, record):
        # Format the message normally
        original = super().format(record)
        
        # Mask sensitive patterns
        masked = original
        masked = self.PATTERNS['credit_card'].sub('****-****-****-****', masked)
        masked = self.PATTERNS['ssn'].sub('***-**-****', masked)
        masked = self.PATTERNS['email'].sub('***@***.***', masked)
        
        return masked

# Protects against accidentally logging sensitive data
secure_formatter = SensitiveDataFormatter(
    '%(asctime)s - %(levelname)s - %(message)s'
)
handler = logging.StreamHandler()
handler.setFormatter(secure_formatter)
logger = logging.getLogger('secure')
logger.addHandler(handler)

# Sensitive data gets masked automatically
logger.info("Processing payment for card 4532-1234-5678-9010")
# Output: "Processing payment for card ****-****-****-****"

Logging Best Practices for Production

Production logging requires different considerations than development logging. Performance impacts matter more, log volume affects storage costs, and log quality directly influences incident response times. Following established best practices prevents common pitfalls and ensures your logging system remains valuable rather than becoming a burden.

✨ Use appropriate log levels consistently across your codebase. Establish team conventions for what constitutes each level and document these decisions. Inconsistent level usage makes filtering unreliable and forces operators to read more logs than necessary. Review log levels during code review to maintain consistency.

🎯 Include contextual information that helps understand not just what happened, but why and under what circumstances. User IDs, request IDs, transaction IDs, and correlation IDs enable tracing events through complex systems. Use the extra parameter to add context without cluttering the message itself.

import logging

logger = logging.getLogger(__name__)

# Poor logging - lacks context
logger.error("Database connection failed")

# Better logging - includes context
logger.error(
    "Database connection failed",
    extra={
        'database': 'users_db',
        'host': 'db-primary.example.com',
        'attempt': 3,
        'user_id': 12345,
        'request_id': 'abc-123-def-456'
    }
)

⚡ Be mindful of performance impacts, especially in hot code paths. Logging isn't free—formatting strings, writing to disk, and network transmission all consume resources. Use appropriate log levels to avoid expensive operations in production, and consider lazy evaluation for complex log messages.

# Inefficient - always constructs complex string
logger.debug("Processing data: " + expensive_function())

# Efficient - only evaluates if DEBUG level is enabled
if logger.isEnabledFor(logging.DEBUG):
    logger.debug("Processing data: %s", expensive_function())

# Even better - use lazy formatting
logger.debug("Processing data: %s", expensive_function)

🔒 Never log sensitive information like passwords, API keys, credit card numbers, or personally identifiable information (PII) unless absolutely necessary and properly protected. Logs often have weaker access controls than databases and may be stored for extended periods. Implement formatters that automatically mask sensitive patterns, and train developers to recognize sensitive data.

"The best logging strategy is invisible during normal operation but provides exactly the information you need when things go wrong."

Exception Logging Strategies

Logging exceptions properly captures critical debugging information while avoiding common mistakes. Python's logging module provides special support for exceptions through the exc_info parameter and dedicated exception logging methods. Understanding these mechanisms ensures you capture full stack traces when needed without cluttering logs with redundant information.

import logging

logger = logging.getLogger(__name__)

try:
    result = risky_operation()
except ValueError as e:
    # Logs exception with full stack trace
    logger.exception("Failed to process data")
    
    # Alternative: explicit exc_info parameter
    logger.error("Failed to process data", exc_info=True)
    
    # Without stack trace (when you want just the message)
    logger.error(f"Failed to process data: {e}")
    
except Exception as e:
    # Catch-all with exception info
    logger.critical("Unexpected error occurred", exc_info=True)
    raise  # Re-raise after logging

The logger.exception() method automatically includes exception information and should be called from within an exception handler. It's equivalent to logger.error() with exc_info=True. This captures the exception type, message, and full traceback, providing complete context for debugging. Use logger.exception() for unexpected errors and logger.error() without exc_info for expected error conditions that don't require stack traces.

Log Rotation and Retention Policies

Unmanaged logs consume disk space indefinitely, eventually causing application failures or system outages. Implementing rotation and retention policies prevents this problem while preserving logs long enough for troubleshooting and compliance requirements. Different applications require different retention periods based on regulatory requirements, troubleshooting needs, and storage costs.

from logging.handlers import TimedRotatingFileHandler
import logging

# Rotate daily, keep 30 days of logs
daily_handler = TimedRotatingFileHandler(
    'application.log',
    when='midnight',
    interval=1,
    backupCount=30
)

# Rotate weekly, keep 12 weeks
weekly_handler = TimedRotatingFileHandler(
    'weekly-summary.log',
    when='W0',  # Monday
    interval=1,
    backupCount=12
)

# Size-based rotation with compression
from logging.handlers import RotatingFileHandler
import gzip
import os

class CompressingRotatingFileHandler(RotatingFileHandler):
    def doRollover(self):
        super().doRollover()
        # Compress the rotated file
        log_file = f"{self.baseFilename}.1"
        if os.path.exists(log_file):
            with open(log_file, 'rb') as f_in:
                with gzip.open(f"{log_file}.gz", 'wb') as f_out:
                    f_out.writelines(f_in)
            os.remove(log_file)

Logging in Different Environments

Effective logging strategies adapt to different environments—development, testing, staging, and production each have distinct requirements. Development environments prioritize immediate feedback and detailed information, while production environments balance information needs with performance and storage costs. Configuring logging appropriately for each environment prevents information overload in development and ensures adequate observability in production.

In development environments, verbose logging helps understand program behavior and debug issues quickly. Set the root logger to DEBUG level, use console output for immediate feedback, and include detailed formatting with file names and line numbers. The performance impact of verbose logging matters less than rapid iteration and problem identification.

import logging
import os

def configure_development_logging():
    logging.basicConfig(
        level=logging.DEBUG,
        format='%(asctime)s [%(levelname)-8s] %(name)s.%(funcName)s:%(lineno)d - %(message)s',
        datefmt='%H:%M:%S'
    )
    
    # Reduce noise from third-party libraries
    logging.getLogger('urllib3').setLevel(logging.WARNING)
    logging.getLogger('requests').setLevel(logging.WARNING)
    
    logger = logging.getLogger(__name__)
    logger.debug("Development logging configured")

Production environments require careful balance between observability and resource consumption. Set the default level to INFO or WARNING, use structured logging for machine parsing, implement log rotation to manage disk space, and consider centralized log aggregation for distributed systems. Production logging should provide enough information to diagnose issues without overwhelming storage or impacting performance.

import logging.config
import os

def configure_production_logging():
    config = {
        'version': 1,
        'disable_existing_loggers': False,
        'formatters': {
            'json': {
                'class': 'pythonjsonlogger.jsonlogger.JsonFormatter',
                'format': '%(asctime)s %(name)s %(levelname)s %(message)s'
            }
        },
        'handlers': {
            'console': {
                'class': 'logging.StreamHandler',
                'level': 'INFO',
                'formatter': 'json',
                'stream': 'ext://sys.stdout'
            },
            'error_file': {
                'class': 'logging.handlers.RotatingFileHandler',
                'level': 'ERROR',
                'formatter': 'json',
                'filename': '/var/log/app/errors.log',
                'maxBytes': 52428800,  # 50MB
                'backupCount': 10
            }
        },
        'root': {
            'level': 'INFO',
            'handlers': ['console', 'error_file']
        }
    }
    
    logging.config.dictConfig(config)

Environment-Specific Configuration Loading

Loading configuration based on environment variables or configuration files allows the same codebase to behave appropriately in different deployment contexts. This approach separates configuration from code, following twelve-factor app principles and simplifying deployment across environments.

import logging.config
import os
import yaml

def setup_logging():
    # Determine environment
    env = os.getenv('APP_ENV', 'development')
    
    # Load environment-specific configuration
    config_file = f'logging_{env}.yaml'
    
    if os.path.exists(config_file):
        with open(config_file, 'r') as f:
            config = yaml.safe_load(f)
            logging.config.dictConfig(config)
    else:
        # Fallback to basic configuration
        logging.basicConfig(
            level=logging.INFO if env == 'production' else logging.DEBUG,
            format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
        )
    
    logger = logging.getLogger(__name__)
    logger.info(f"Logging configured for {env} environment")

# Call during application startup
setup_logging()

Integration with Frameworks and Libraries

Popular Python frameworks like Django, Flask, and FastAPI include their own logging configurations and conventions. Understanding how to integrate your logging strategy with framework-specific logging ensures consistent log output and prevents conflicts between application and framework logs.

Flask applications automatically configure logging but allow customization through the app.logger object. Flask's default configuration logs to the console in development and requires explicit configuration for production deployments. You can add handlers to app.logger or configure logging before creating the Flask app instance.

from flask import Flask, request
import logging
from logging.handlers import RotatingFileHandler

app = Flask(__name__)

# Configure application logging
if not app.debug:
    file_handler = RotatingFileHandler(
        'flask_app.log',
        maxBytes=10485760,
        backupCount=10
    )
    file_handler.setFormatter(logging.Formatter(
        '%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'
    ))
    file_handler.setLevel(logging.INFO)
    app.logger.addHandler(file_handler)
    app.logger.setLevel(logging.INFO)
    app.logger.info('Application startup')

@app.before_request
def log_request_info():
    app.logger.debug('Headers: %s', request.headers)
    app.logger.debug('Body: %s', request.get_data())

@app.route('/')
def index():
    app.logger.info('Index page accessed')
    return 'Hello World'

Django projects use a sophisticated logging configuration defined in settings.py. Django's logging integrates with Python's logging module but adds Django-specific loggers for different components (requests, database, security). Customizing Django logging involves updating the LOGGING dictionary in settings.

"Framework integration isn't about fighting the framework's logging—it's about understanding its conventions and extending them to meet your needs."

# Django settings.py
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/var/log/django/app.log',
            'maxBytes': 1024 * 1024 * 15,  # 15MB
            'backupCount': 10,
            'formatter': 'verbose',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console', 'file'],
            'level': 'INFO',
            'propagate': False,
        },
        'myapp': {
            'handlers': ['console', 'file'],
            'level': 'DEBUG',
            'propagate': False,
        },
    },
}

Third-Party Library Log Management

Third-party libraries often create their own loggers, which can flood your logs with information you don't need. Managing these loggers prevents noise while preserving important messages. Set appropriate levels for library loggers to reduce verbosity without losing critical information.

import logging

# Reduce verbosity of noisy libraries
logging.getLogger('urllib3').setLevel(logging.WARNING)
logging.getLogger('requests').setLevel(logging.WARNING)
logging.getLogger('boto3').setLevel(logging.INFO)
logging.getLogger('botocore').setLevel(logging.INFO)

# Completely silence a library (use sparingly)
logging.getLogger('overly_verbose_library').setLevel(logging.CRITICAL)

# Redirect library logs to separate file
library_handler = logging.FileHandler('third_party.log')
library_handler.setLevel(logging.DEBUG)
logging.getLogger('some_library').addHandler(library_handler)
logging.getLogger('some_library').propagate = False

Advanced Logging Patterns

Beyond basic logging, advanced patterns address specific challenges in complex applications: contextual logging that maintains request-specific information, performance logging that tracks timing and resource usage, and audit logging that creates immutable records of important actions. These patterns build on logging fundamentals to solve real-world problems.

Contextual Logging with Context Managers

Maintaining context across function calls and modules challenges distributed systems and web applications handling multiple concurrent requests. Context managers provide an elegant solution for temporarily adding context to all log messages within a scope, then automatically removing that context when the scope exits.

import logging
from contextlib import contextmanager
import threading

# Thread-local storage for context
_context = threading.local()

class ContextFilter(logging.Filter):
    def filter(self, record):
        # Add context attributes to log record
        for key, value in getattr(_context, 'data', {}).items():
            setattr(record, key, value)
        return True

@contextmanager
def log_context(**kwargs):
    """Add contextual information to all logs within this context"""
    if not hasattr(_context, 'data'):
        _context.data = {}
    
    # Save previous context
    previous = _context.data.copy()
    
    # Add new context
    _context.data.update(kwargs)
    
    try:
        yield
    finally:
        # Restore previous context
        _context.data = previous

# Configure logger with context filter
logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.addFilter(ContextFilter())
formatter = logging.Formatter(
    '%(asctime)s [%(levelname)s] [request_id:%(request_id)s] [user_id:%(user_id)s] %(message)s',
    defaults={'request_id': 'none', 'user_id': 'none'}
)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Use context manager
def process_request(request_id, user_id):
    with log_context(request_id=request_id, user_id=user_id):
        logger.info("Processing request")
        do_work()
        logger.info("Request completed")

def do_work():
    # Context automatically available here
    logger.info("Performing work")

Performance Logging and Timing

Understanding performance characteristics requires logging execution times and resource usage. Decorators and context managers provide clean patterns for timing operations without cluttering business logic with timing code. This approach enables identifying performance bottlenecks and tracking performance trends over time.

import logging
import time
from functools import wraps
from contextlib import contextmanager

logger = logging.getLogger(__name__)

def log_execution_time(func):
    """Decorator to log function execution time"""
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.time()
        try:
            result = func(*args, **kwargs)
            return result
        finally:
            execution_time = time.time() - start_time
            logger.info(
                f"{func.__name__} executed in {execution_time:.4f} seconds",
                extra={
                    'function': func.__name__,
                    'execution_time': execution_time,
                    'module': func.__module__
                }
            )
    return wrapper

@contextmanager
def log_timing(operation_name):
    """Context manager for timing code blocks"""
    start_time = time.time()
    try:
        yield
    finally:
        execution_time = time.time() - start_time
        logger.info(
            f"{operation_name} completed in {execution_time:.4f} seconds",
            extra={
                'operation': operation_name,
                'execution_time': execution_time
            }
        )

# Usage examples
@log_execution_time
def expensive_operation():
    time.sleep(2)
    return "result"

def process_data():
    with log_timing("data_processing"):
        # Complex processing here
        time.sleep(1.5)

Audit Logging for Compliance

Audit logs create immutable records of important actions for security, compliance, and forensic purposes. Unlike operational logs that help debug issues, audit logs document who did what and when. Audit logging requires special handling: separate storage, stricter access controls, longer retention periods, and protection against tampering.

import logging
import hashlib
import json
from datetime import datetime

class AuditLogger:
    def __init__(self, log_file):
        self.logger = logging.getLogger('audit')
        self.logger.setLevel(logging.INFO)
        self.logger.propagate = False
        
        # Separate file for audit logs
        handler = logging.FileHandler(log_file)
        handler.setFormatter(logging.Formatter('%(message)s'))
        self.logger.addHandler(handler)
        
        self.previous_hash = None
    
    def log_action(self, user_id, action, resource, details=None):
        """Log an auditable action with integrity checking"""
        audit_entry = {
            'timestamp': datetime.utcnow().isoformat(),
            'user_id': user_id,
            'action': action,
            'resource': resource,
            'details': details or {},
            'previous_hash': self.previous_hash
        }
        
        # Calculate hash for integrity
        entry_json = json.dumps(audit_entry, sort_keys=True)
        current_hash = hashlib.sha256(entry_json.encode()).hexdigest()
        audit_entry['hash'] = current_hash
        
        # Log the entry
        self.logger.info(json.dumps(audit_entry))
        
        # Update previous hash for next entry
        self.previous_hash = current_hash

# Usage
audit = AuditLogger('audit.log')
audit.log_action(
    user_id=12345,
    action='user_login',
    resource='authentication_system',
    details={'ip_address': '192.168.1.100', 'method': 'password'}
)

audit.log_action(
    user_id=12345,
    action='data_access',
    resource='customer_records',
    details={'record_count': 50, 'query': 'SELECT * FROM customers LIMIT 50'}
)

How do I prevent logging from impacting application performance?

Use appropriate log levels to avoid expensive operations in production, implement lazy evaluation for complex log messages using the % formatting style rather than f-strings, consider asynchronous logging handlers for high-throughput applications, and profile your logging to identify bottlenecks. For extremely performance-sensitive code paths, check if logging is enabled before constructing log messages using logger.isEnabledFor().

What's the difference between print statements and proper logging?

Print statements write directly to stdout without levels, formatting control, or destination flexibility. They can't be disabled without code changes, don't include timestamps or context automatically, and don't integrate with log aggregation systems. Logging provides configurable levels, multiple output destinations, structured formatting, and the ability to adjust verbosity without code changes. Print statements work for quick debugging but should never appear in production code.

How should I handle logging in multi-threaded or asynchronous applications?

Python's logging module is thread-safe by default, so multiple threads can safely log to the same logger. For asynchronous applications using asyncio, use QueueHandler and QueueListener to avoid blocking the event loop during I/O operations. The QueueHandler sends log records to a queue, while a separate QueueListener thread processes them asynchronously, preventing logging from blocking async operations.

What information should I include in log messages for effective debugging?

Include contextual information that answers who, what, when, where, and why: user identifiers, request IDs for tracing, relevant data values (but never sensitive information), operation names, and error details. Use structured logging with the extra parameter to add context without cluttering messages. Include enough information that someone unfamiliar with the code can understand what happened, but avoid logging excessive data that makes finding relevant information difficult.

How do I manage logs in containerized applications and Kubernetes?

In containerized environments, write logs to stdout/stderr using StreamHandler rather than files. Container orchestration platforms collect stdout/stderr automatically and forward to centralized logging systems. Use structured JSON logging for better parsing and searchability. Include container metadata like pod name and namespace in log context. Avoid writing to files inside containers since containers are ephemeral and file-based logs disappear when containers restart.

Should I use different loggers for different modules or a single logger?

Use different loggers for different modules or components, creating a hierarchy that mirrors your application structure. This allows fine-grained control over logging levels and output for different parts of your application. You can debug database issues by setting the database logger to DEBUG while keeping other components at INFO. Follow the pattern of using logging.getLogger(__name__) in each module to automatically create appropriately named loggers based on the module hierarchy.