How to Create Log Files from Scripts

How to Create Log Files from Scripts

When scripts fail silently in production environments, the consequences can be devastating. Without proper logging mechanisms, developers find themselves blind to critical errors, performance bottlenecks, and security incidents that could have been prevented. The ability to track what happens during script execution isn't just a nice-to-have feature—it's an essential practice that separates amateur code from professional, maintainable systems.

Creating log files from scripts involves systematically recording events, errors, and informational messages generated during program execution. This documentation practice provides visibility into application behavior, enabling developers to debug issues, monitor performance, and maintain audit trails. Whether you're working with Python, Bash, JavaScript, or any other scripting language, implementing robust logging transforms opaque processes into transparent, traceable operations.

Throughout this comprehensive guide, you'll discover practical techniques for implementing logging across multiple scripting environments. You'll learn how to structure log entries for maximum usefulness, manage log file rotation to prevent storage issues, and implement different logging levels to balance detail with performance. We'll explore both simple approaches for quick scripts and sophisticated frameworks for enterprise applications, ensuring you have the right tools regardless of your project's complexity.

Understanding the Fundamentals of Script Logging

Logging serves as the nervous system of any application, transmitting signals about what's happening inside your code. At its core, logging captures temporal information about script execution—recording when events occur, what actions the script performs, and whether those actions succeed or fail. This information becomes invaluable when troubleshooting issues that only manifest in specific environments or under particular conditions.

The architecture of logging systems typically involves three primary components: the logger itself, which generates log messages; the handler, which determines where those messages go; and the formatter, which structures how messages appear. Understanding this separation allows you to create flexible logging solutions that can simultaneously write to files, send alerts, and display console output without duplicating code.

"The difference between a script that logs and one that doesn't is the difference between flying blind and having instrumentation in your cockpit."

Different logging levels provide granular control over what information gets recorded. These levels typically include DEBUG for detailed diagnostic information, INFO for general informational messages, WARNING for potentially problematic situations, ERROR for serious issues that don't stop execution, and CRITICAL for failures that might cause the script to terminate. Selecting appropriate levels for different messages ensures logs remain useful without becoming overwhelming.

Why Standard Output Isn't Enough

Many developers initially rely on print statements or console output for monitoring script behavior. While this approach works during development, it becomes inadequate in production environments where scripts run unattended or as background processes. Standard output disappears once the terminal closes, provides no historical record, and offers limited filtering capabilities.

Log files persist beyond script execution, creating permanent records that support post-mortem analysis. They enable correlation of events across time, making it possible to identify patterns that lead to failures. When a script crashes at 3 AM, log files provide the only witness to what happened, preserving context that would otherwise be lost forever.

Implementing Basic Logging in Python Scripts

Python's built-in logging module provides a robust foundation for creating log files without external dependencies. The module offers both simple and sophisticated approaches, allowing developers to start with basic implementations and evolve toward more complex configurations as needs grow.

import logging

# Basic configuration
logging.basicConfig(
    filename='script_execution.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

# Writing log messages
logging.info('Script started successfully')
logging.warning('Configuration file not found, using defaults')
logging.error('Failed to connect to database')

This basic setup creates a log file named 'script_execution.log' in the current directory, records messages at INFO level and above, and formats each entry with a timestamp, severity level, and the message itself. The format string uses placeholders that the logging module automatically populates with contextual information.

Advanced Python Logging Configurations

As scripts grow more complex, logging requirements typically expand beyond single-file output. Python's logging module supports multiple handlers, each directing messages to different destinations based on severity or other criteria. This capability enables sophisticated logging architectures where critical errors trigger immediate alerts while routine information accumulates in standard log files.

import logging
from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler

# Create logger
logger = logging.getLogger('MyScript')
logger.setLevel(logging.DEBUG)

# Create handlers
file_handler = RotatingFileHandler(
    'application.log',
    maxBytes=10485760,  # 10MB
    backupCount=5
)
file_handler.setLevel(logging.INFO)

error_handler = RotatingFileHandler(
    'errors.log',
    maxBytes=10485760,
    backupCount=3
)
error_handler.setLevel(logging.ERROR)

# Create formatters
detailed_formatter = logging.Formatter(
    '%(asctime)s - %(name)s - %(levelname)s - %(funcName)s:%(lineno)d - %(message)s'
)

# Attach formatters to handlers
file_handler.setFormatter(detailed_formatter)
error_handler.setFormatter(detailed_formatter)

# Add handlers to logger
logger.addHandler(file_handler)
logger.addHandler(error_handler)

# Usage
logger.debug('Detailed diagnostic information')
logger.info('General informational message')
logger.error('An error occurred during processing')

This configuration implements rotating file handlers that automatically manage log file sizes. When a log file reaches 10MB, the system renames it with a numbered suffix and starts a new file. The backupCount parameter limits how many old files to retain, preventing unlimited disk space consumption.

"Logs should tell a story about what your application did, not just what went wrong."

Creating Log Files in Bash Scripts

Shell scripts require different approaches to logging since Bash lacks built-in logging frameworks. However, the flexibility of Unix command-line tools provides multiple strategies for creating comprehensive log files. The simplest approach redirects output streams to files, while more sophisticated methods implement custom logging functions.

#!/bin/bash

# Define log file location
LOG_FILE="/var/log/myscript.log"

# Simple logging function
log_message() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

# Usage examples
log_message "Script execution started"
log_message "Processing data files"

# Capture command output and errors
if ! result=$(some_command 2>&1); then
    log_message "ERROR: Command failed - $result"
    exit 1
fi

log_message "Script completed successfully"

This basic implementation creates a logging function that prepends timestamps to messages and appends them to a designated log file. The double greater-than symbol (>>) ensures messages append rather than overwrite, preserving the complete execution history.

Advanced Bash Logging Techniques

Professional Bash scripts often implement more sophisticated logging that mirrors the level-based approach found in dedicated logging frameworks. By creating separate functions for different severity levels, scripts can conditionally log messages based on verbosity settings or direct critical errors to different destinations.

#!/bin/bash

# Configuration
LOG_FILE="/var/log/application.log"
ERROR_LOG="/var/log/application_error.log"
LOG_LEVEL="INFO"  # DEBUG, INFO, WARN, ERROR

# Logging functions
log_debug() {
    [[ "$LOG_LEVEL" == "DEBUG" ]] && echo "[$(date '+%Y-%m-%d %H:%M:%S')] [DEBUG] $1" >> "$LOG_FILE"
}

log_info() {
    [[ "$LOG_LEVEL" =~ ^(DEBUG|INFO)$ ]] && echo "[$(date '+%Y-%m-%d %H:%M:%S')] [INFO] $1" >> "$LOG_FILE"
}

log_warn() {
    [[ "$LOG_LEVEL" =~ ^(DEBUG|INFO|WARN)$ ]] && echo "[$(date '+%Y-%m-%d %H:%M:%S')] [WARN] $1" >> "$LOG_FILE"
}

log_error() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $1" >> "$LOG_FILE"
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $1" >> "$ERROR_LOG"
}

# Redirect all output to log file
exec 1> >(tee -a "$LOG_FILE")
exec 2> >(tee -a "$ERROR_LOG" >&2)

# Script logic with logging
log_info "Beginning data processing"
log_debug "Current working directory: $(pwd)"

if [[ ! -f "config.ini" ]]; then
    log_warn "Configuration file missing, using defaults"
fi

# Error handling with logging
if ! process_data; then
    log_error "Data processing failed"
    exit 1
fi

log_info "Processing completed successfully"

The exec commands redirect standard output and standard error streams through the tee command, which simultaneously writes to log files and displays output on the terminal. This approach captures all script output automatically, including messages from external commands, without requiring explicit logging calls for every operation.

JavaScript and Node.js Logging Strategies

JavaScript environments, particularly Node.js, offer numerous logging solutions ranging from simple console wrappers to enterprise-grade frameworks. The choice depends on application complexity, performance requirements, and integration needs with monitoring systems.

const fs = require('fs');
const path = require('path');

class SimpleLogger {
    constructor(logFilePath) {
        this.logFilePath = logFilePath;
        this.logStream = fs.createWriteStream(logFilePath, { flags: 'a' });
    }

    formatMessage(level, message) {
        const timestamp = new Date().toISOString();
        return `${timestamp} [${level}] ${message}\n`;
    }

    info(message) {
        const formatted = this.formatMessage('INFO', message);
        this.logStream.write(formatted);
        console.log(formatted.trim());
    }

    error(message) {
        const formatted = this.formatMessage('ERROR', message);
        this.logStream.write(formatted);
        console.error(formatted.trim());
    }

    warn(message) {
        const formatted = this.formatMessage('WARN', message);
        this.logStream.write(formatted);
        console.warn(formatted.trim());
    }

    close() {
        this.logStream.end();
    }
}

// Usage
const logger = new SimpleLogger('./application.log');
logger.info('Application started');
logger.warn('High memory usage detected');
logger.error('Database connection failed');

process.on('exit', () => logger.close());

This custom logger implementation uses Node.js streams for efficient file writing, automatically formatting messages with timestamps and severity levels. The class-based structure provides a clean interface while maintaining a persistent file stream that remains open throughout script execution.

Using Winston for Professional Node.js Logging

Winston represents the de facto standard for production Node.js logging, offering extensive features including multiple transports, log levels, formatting options, and exception handling. Its modular architecture supports everything from simple file logging to complex distributed logging systems.

const winston = require('winston');

const logger = winston.createLogger({
    level: 'info',
    format: winston.format.combine(
        winston.format.timestamp({
            format: 'YYYY-MM-DD HH:mm:ss'
        }),
        winston.format.errors({ stack: true }),
        winston.format.splat(),
        winston.format.json()
    ),
    defaultMeta: { service: 'my-script' },
    transports: [
        new winston.transports.File({ 
            filename: 'error.log', 
            level: 'error',
            maxsize: 5242880, // 5MB
            maxFiles: 5
        }),
        new winston.transports.File({ 
            filename: 'combined.log',
            maxsize: 5242880,
            maxFiles: 5
        })
    ]
});

// Add console output in development
if (process.env.NODE_ENV !== 'production') {
    logger.add(new winston.transports.Console({
        format: winston.format.combine(
            winston.format.colorize(),
            winston.format.simple()
        )
    }));
}

// Usage examples
logger.info('User authentication successful', { userId: 12345 });
logger.warn('API rate limit approaching', { current: 950, limit: 1000 });
logger.error('Payment processing failed', { 
    error: 'Connection timeout',
    orderId: 'ORD-789',
    amount: 99.99
});

Winston's JSON formatting makes logs machine-readable, facilitating integration with log aggregation platforms like ELK Stack or Splunk. The structured format allows querying logs based on specific fields, dramatically improving troubleshooting efficiency in complex systems.

"Good logging practices transform debugging from archaeology into science."

Log File Management and Rotation Strategies

Without proper management, log files grow indefinitely, eventually consuming all available disk space. Log rotation addresses this challenge by implementing policies that archive old logs and create fresh files, maintaining a balance between historical data retention and storage constraints.

Most logging frameworks include built-in rotation capabilities. Python's RotatingFileHandler and TimedRotatingFileHandler provide size-based and time-based rotation respectively. Node.js libraries like Winston offer similar features through transport configuration. For Bash scripts, external tools like logrotate typically handle rotation more efficiently than custom implementations.

Implementing Logrotate for System-Wide Management

On Linux systems, logrotate provides centralized log management across all applications. This system utility runs periodically (typically daily) via cron, processing log files according to configuration files that specify rotation frequency, compression, retention periods, and post-rotation actions.

# /etc/logrotate.d/myscript
/var/log/myscript/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0644 scriptuser scriptgroup
    postrotate
        systemctl reload myscript.service > /dev/null 2>&1 || true
    endscript
}

This configuration rotates logs daily, keeps seven days of history, compresses old logs (except the most recent), and doesn't fail if log files are missing or empty. The postrotate script reloads the application after rotation, ensuring it begins writing to the new log file rather than continuing with the renamed old file.

Structured Logging for Enhanced Analysis

Traditional plain-text logs work well for human reading but present challenges for automated analysis. Structured logging formats messages as key-value pairs or JSON objects, making logs machine-readable while remaining human-friendly. This approach dramatically improves the ability to search, filter, and analyze log data at scale.

"Structured logs are the difference between having data and having searchable, queryable information."

import json
import logging
from datetime import datetime

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_object = {
            'timestamp': datetime.utcnow().isoformat(),
            'level': record.levelname,
            'logger': record.name,
            'message': record.getMessage(),
            'module': record.module,
            'function': record.funcName,
            'line': record.lineno
        }
        
        if record.exc_info:
            log_object['exception'] = self.formatException(record.exc_info)
        
        if hasattr(record, 'user_id'):
            log_object['user_id'] = record.user_id
        
        if hasattr(record, 'request_id'):
            log_object['request_id'] = record.request_id
            
        return json.dumps(log_object)

# Setup
logger = logging.getLogger('structured_logger')
handler = logging.FileHandler('structured.log')
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Usage with additional context
logger.info('User login successful', extra={'user_id': 12345, 'request_id': 'req-789'})

Each log entry becomes a complete JSON object containing not just the message but also contextual metadata. This structure enables powerful queries like "show all ERROR logs from the payment module for user_id 12345 in the last hour" without complex text parsing.

Contextual Information in Log Entries

The most valuable logs capture not just what happened but the context in which it happened. Including relevant metadata—user identifiers, transaction IDs, request traces, session information—transforms individual log entries into interconnected narratives that reveal system behavior patterns.

• 🔍 Request Identifiers: Unique IDs that trace single operations across multiple log entries and services
• 👤 User Context: Information about who triggered the action, enabling user-specific issue investigation
• ⏱️ Timing Information: Execution duration, timestamps for performance analysis and bottleneck identification
• 🌐 Environment Details: Server names, IP addresses, deployment versions for multi-environment troubleshooting
• 📊 Business Metrics: Transaction amounts, item counts, conversion steps for operational intelligence

Performance Considerations for Logging

Logging introduces overhead that can impact application performance, particularly in high-throughput systems. Every log write consumes CPU cycles for formatting and I/O operations for file writing. Balancing comprehensive logging with acceptable performance requires strategic decisions about what to log, at what level, and how to handle the data.

Asynchronous logging represents one of the most effective performance optimizations. Rather than blocking script execution while writing to disk, asynchronous approaches queue log messages in memory and write them in separate threads or processes. This technique minimizes the performance impact of logging on critical code paths.

import logging
from logging.handlers import QueueHandler, QueueListener
import queue
import atexit

# Create queue and handlers
log_queue = queue.Queue(-1)
queue_handler = QueueHandler(log_queue)

# File handler for actual writing
file_handler = logging.FileHandler('async.log')
file_handler.setFormatter(logging.Formatter(
    '%(asctime)s - %(levelname)s - %(message)s'
))

# Queue listener writes asynchronously
listener = QueueListener(log_queue, file_handler)
listener.start()

# Ensure cleanup on exit
atexit.register(listener.stop)

# Configure logger to use queue
logger = logging.getLogger('async_logger')
logger.addHandler(queue_handler)
logger.setLevel(logging.INFO)

# Logging now happens asynchronously
for i in range(10000):
    logger.info(f'Processing item {i}')

The QueueListener runs in a separate thread, continuously pulling messages from the queue and writing them to the file handler. The main application thread only needs to add messages to the queue—a fast, memory-only operation—allowing it to continue execution without waiting for disk I/O.

Conditional Logging and Debug Levels

Not all logging statements should execute in production environments. Debug-level logs that provide detailed diagnostic information during development often generate excessive output and performance overhead in production. Using logging levels appropriately ensures detailed information remains available when needed without impacting normal operations.

"The best logging strategy is one that gives you the information you need without drowning you in data you don't."

Security and Privacy in Log Files

Log files often contain sensitive information—user credentials, personal data, financial details, security tokens. Improper handling of this data in logs creates security vulnerabilities and privacy violations. Implementing appropriate safeguards ensures logging provides operational benefits without exposing sensitive information.

Sensitive data should never appear in plain text within log files. Before logging, scripts should sanitize inputs by redacting or hashing sensitive values. This practice prevents credential leakage while maintaining enough information for troubleshooting. For example, logging "User authentication failed for user_id: 12345" provides useful information without exposing the attempted password.

import hashlib
import re

def sanitize_for_logging(data):
    """Remove or hash sensitive information before logging"""
    sanitized = data.copy() if isinstance(data, dict) else str(data)
    
    # Remove credit card numbers
    if isinstance(sanitized, str):
        sanitized = re.sub(r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b', 
                          'XXXX-XXXX-XXXX-XXXX', sanitized)
    
    # Hash email addresses
    if isinstance(sanitized, dict) and 'email' in sanitized:
        email = sanitized['email']
        hashed = hashlib.sha256(email.encode()).hexdigest()[:16]
        sanitized['email_hash'] = hashed
        del sanitized['email']
    
    # Remove passwords
    if isinstance(sanitized, dict):
        sensitive_keys = ['password', 'token', 'secret', 'api_key']
        for key in sensitive_keys:
            if key in sanitized:
                sanitized[key] = '***REDACTED***'
    
    return sanitized

# Usage
user_data = {
    'username': 'john_doe',
    'email': 'john@example.com',
    'password': 'secret123',
    'credit_card': '4532-1234-5678-9010'
}

logger.info(f'User registration: {sanitize_for_logging(user_data)}')

Log File Access Control

Restricting who can read log files is as important as sanitizing their contents. Log files should have appropriate file system permissions that limit access to authorized users and processes. On Unix-like systems, setting permissions to 600 or 640 ensures only the file owner (or owner and group) can read logs containing potentially sensitive operational data.

• 🔐 File Permissions: Set restrictive permissions (600 or 640) on log files immediately upon creation
• 📁 Dedicated Directories: Store logs in protected directories with limited access, separate from application code
• 🔄 Regular Audits: Periodically review log access patterns and permissions to detect unauthorized access
• 🗑️ Secure Deletion: Use secure deletion methods when removing old logs containing sensitive information
• 🔒 Encryption at Rest: Consider encrypting log files on disk for highly sensitive environments

Error Handling and Exception Logging

Exceptions and errors represent critical events that logging must capture comprehensively. When scripts encounter unexpected conditions, logs should record not just that an error occurred but complete context including stack traces, variable states, and the sequence of operations that led to the failure.

import logging
import traceback
import sys

logger = logging.getLogger(__name__)

def safe_execute(func, *args, **kwargs):
    """Execute function with comprehensive error logging"""
    try:
        logger.info(f'Executing {func.__name__} with args={args}, kwargs={kwargs}')
        result = func(*args, **kwargs)
        logger.info(f'{func.__name__} completed successfully')
        return result
    except Exception as e:
        logger.error(f'Exception in {func.__name__}: {str(e)}')
        logger.error(f'Exception type: {type(e).__name__}')
        logger.error(f'Stack trace:\n{traceback.format_exc()}')
        
        # Log local variables at time of exception
        frame = sys.exc_info()[2].tb_frame
        logger.error(f'Local variables: {frame.f_locals}')
        
        raise  # Re-raise after logging

# Usage
def process_data(filename):
    with open(filename, 'r') as f:
        data = f.read()
        return len(data)

try:
    result = safe_execute(process_data, 'nonexistent.txt')
except Exception:
    logger.critical('Script terminating due to unrecoverable error')
    sys.exit(1)

This approach wraps function execution in comprehensive error handling that captures the complete exception context. The stack trace reveals exactly where the error occurred, while logging local variables provides insight into the state that caused the failure. This information proves invaluable when debugging issues that only manifest in production environments.

"An exception without a logged stack trace is a mystery you'll spend hours solving."

Centralized Logging for Distributed Systems

When applications run across multiple servers or containers, aggregating logs from all sources becomes essential. Centralized logging collects log data from distributed systems into a single location where it can be searched, analyzed, and correlated. This approach transforms fragmented logs into a unified view of system behavior.

Modern logging architectures typically employ log shipping agents that monitor local log files and forward entries to central collection points. Tools like Filebeat, Fluentd, or Logstash read log files as they're written and transmit entries to centralized storage systems like Elasticsearch, where they become searchable through interfaces like Kibana.

// Node.js example: Logging to remote service
const winston = require('winston');
require('winston-syslog').Syslog;

const logger = winston.createLogger({
    level: 'info',
    format: winston.format.json(),
    defaultMeta: { 
        service: 'my-script',
        hostname: require('os').hostname()
    },
    transports: [
        // Local file logging
        new winston.transports.File({ filename: 'local.log' }),
        
        // Remote syslog server
        new winston.transports.Syslog({
            host: 'logs.example.com',
            port: 514,
            protocol: 'udp4',
            app_name: 'my-script'
        })
    ]
});

logger.info('Application started', { 
    version: '1.2.3',
    environment: process.env.NODE_ENV 
});

Cloud-Native Logging Approaches

Cloud environments introduce unique logging considerations. Container-based applications often write logs to standard output rather than files, relying on container orchestration platforms to capture and route log streams. This approach aligns with the twelve-factor app methodology, which treats logs as event streams rather than files.

• ☁️ Stdout/Stderr Logging: Write logs to standard output streams, letting container platforms handle collection
• 📡 Cloud Provider Integration: Use native logging services like CloudWatch, Stackdriver, or Azure Monitor
• 🏷️ Metadata Tagging: Include cloud-specific metadata (instance IDs, regions, zones) in log entries
• 💰 Cost Management: Monitor logging volume to control cloud logging service costs
• ⚡ Streaming Analytics: Leverage cloud services for real-time log analysis and alerting

Monitoring and Alerting Based on Logs

Logs serve not just as historical records but as real-time data sources for monitoring and alerting. By analyzing log patterns, systems can detect anomalies, trigger alerts for critical errors, and provide early warning of developing problems. This proactive approach transforms logging from passive documentation into active system protection.

Log-based alerting typically involves defining patterns or thresholds that indicate problems. For example, more than five authentication failures in a minute might indicate a brute-force attack, while a sudden spike in error-level logs suggests system degradation. Automated monitoring tools continuously analyze incoming logs and trigger notifications when these patterns emerge.

# Python example: Simple log monitoring with alerting
import re
import time
from collections import deque
from datetime import datetime, timedelta

class LogMonitor:
    def __init__(self, log_file, alert_callback):
        self.log_file = log_file
        self.alert_callback = alert_callback
        self.error_window = deque(maxlen=100)
        
    def check_error_rate(self):
        """Alert if error rate exceeds threshold"""
        now = datetime.now()
        recent_errors = [ts for ts in self.error_window 
                        if now - ts < timedelta(minutes=5)]
        
        if len(recent_errors) > 10:
            self.alert_callback(
                f'High error rate: {len(recent_errors)} errors in 5 minutes'
            )
    
    def monitor(self):
        """Monitor log file for patterns"""
        with open(self.log_file, 'r') as f:
            # Move to end of file
            f.seek(0, 2)
            
            while True:
                line = f.readline()
                if not line:
                    time.sleep(0.1)
                    continue
                
                # Check for error patterns
                if re.search(r'\[ERROR\]', line):
                    self.error_window.append(datetime.now())
                    self.check_error_rate()
                
                # Check for critical patterns
                if re.search(r'\[CRITICAL\]|database.*connection.*failed', line, re.I):
                    self.alert_callback(f'CRITICAL: {line.strip()}')

def send_alert(message):
    """Send alert notification"""
    print(f'ALERT: {message}')
    # Implement actual alerting: email, SMS, Slack, PagerDuty, etc.

# Usage
monitor = LogMonitor('application.log', send_alert)
monitor.monitor()

Debugging Techniques Using Log Files

Effective debugging relies on having the right information at the right time. Well-structured log files transform debugging from guesswork into systematic investigation. By strategically placing log statements and analyzing their output, developers can trace execution flow, identify where behavior deviates from expectations, and pinpoint root causes.

When debugging with logs, start by reproducing the issue while logging is set to DEBUG level. This captures maximum detail about script execution. Then, work backward from the error, examining log entries to understand the sequence of events that led to the failure. Look for unexpected values, missing operations, or timing anomalies that might explain the problem.

"Logs don't just record what happened—they reveal why it happened."

Correlation IDs for Request Tracing

In complex systems where single user actions trigger multiple script executions or service calls, correlation IDs enable tracing complete request paths. By assigning a unique identifier to each request and including it in all related log entries, developers can filter logs to see only entries relevant to a specific transaction, even when those entries span multiple scripts or services.

import logging
import uuid
from contextvars import ContextVar

# Context variable for correlation ID
correlation_id = ContextVar('correlation_id', default=None)

class CorrelationFilter(logging.Filter):
    def filter(self, record):
        record.correlation_id = correlation_id.get() or 'no-correlation-id'
        return True

# Setup logger with correlation ID
logger = logging.getLogger(__name__)
handler = logging.FileHandler('correlated.log')
handler.addFilter(CorrelationFilter())
formatter = logging.Formatter(
    '%(asctime)s [%(correlation_id)s] %(levelname)s - %(message)s'
)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

def process_request(user_id):
    """Process request with correlation tracking"""
    # Generate and set correlation ID
    correlation_id.set(str(uuid.uuid4()))
    
    logger.info(f'Processing request for user {user_id}')
    validate_user(user_id)
    fetch_data(user_id)
    logger.info('Request processing complete')

def validate_user(user_id):
    logger.debug(f'Validating user {user_id}')
    # Validation logic

def fetch_data(user_id):
    logger.debug(f'Fetching data for user {user_id}')
    # Data fetching logic

# All logs from this request will share the same correlation ID
process_request(12345)

Best Practices for Production Logging

Production logging requires balancing multiple concerns: capturing sufficient information for troubleshooting, maintaining acceptable performance, protecting sensitive data, and managing storage costs. Following established best practices ensures logging provides maximum value while minimizing risks and overhead.

• 📝 Log Meaningful Messages: Write clear, actionable messages that explain what happened and why it matters
• 🎯 Use Appropriate Levels: Reserve ERROR for actual errors, INFO for significant events, DEBUG for detailed diagnostics
• 🔢 Include Context: Add relevant identifiers, timestamps, and metadata to make logs searchable and correlatable
• 🚫 Never Log Secrets: Sanitize passwords, tokens, keys, and other sensitive data before logging
• ⚖️ Balance Detail and Volume: Provide enough information to troubleshoot without overwhelming storage or analysis tools

Standardizing log formats across applications simplifies analysis and enables shared tooling. Whether using JSON for structured logs or consistent text formatting, maintaining uniformity allows developers to apply the same parsing and searching techniques across all logs. This standardization pays dividends when troubleshooting issues that span multiple systems.

Testing Logging Implementation

Logging code requires testing just like any other functionality. Tests should verify that appropriate messages are logged at correct levels, that log formats match expectations, and that sensitive data is properly sanitized. Automated testing of logging ensures it continues functioning correctly as applications evolve.

import unittest
import logging
from io import StringIO

class TestLogging(unittest.TestCase):
    def setUp(self):
        self.log_stream = StringIO()
        self.handler = logging.StreamHandler(self.log_stream)
        self.logger = logging.getLogger('test_logger')
        self.logger.addHandler(self.handler)
        self.logger.setLevel(logging.DEBUG)
    
    def tearDown(self):
        self.logger.removeHandler(self.handler)
    
    def test_info_logging(self):
        self.logger.info('Test message')
        log_contents = self.log_stream.getvalue()
        self.assertIn('Test message', log_contents)
        self.assertIn('INFO', log_contents)
    
    def test_sensitive_data_sanitization(self):
        sensitive_data = {'password': 'secret123', 'user': 'john'}
        sanitized = sanitize_for_logging(sensitive_data)
        self.logger.info(f'User data: {sanitized}')
        log_contents = self.log_stream.getvalue()
        self.assertNotIn('secret123', log_contents)
        self.assertIn('REDACTED', log_contents)
    
    def test_error_with_exception(self):
        try:
            raise ValueError('Test exception')
        except ValueError:
            self.logger.exception('Error occurred')
        
        log_contents = self.log_stream.getvalue()
        self.assertIn('ValueError', log_contents)
        self.assertIn('Test exception', log_contents)

if __name__ == '__main__':
    unittest.main()

Log Analysis and Visualization Tools

Raw log files contain valuable information, but extracting insights requires appropriate analysis tools. Modern log management platforms provide search interfaces, visualization dashboards, and analytics capabilities that transform raw log data into actionable intelligence. These tools enable developers and operators to understand system behavior at scale.

The ELK Stack (Elasticsearch, Logstash, Kibana) represents one of the most popular open-source solutions for log analysis. Logstash collects and parses logs, Elasticsearch indexes them for fast searching, and Kibana provides visualization and dashboard capabilities. This combination enables real-time log analysis and historical trend identification.

• 🔍 Full-Text Search: Query logs using natural language or regular expressions to find specific events
• 📊 Visualization Dashboards: Create graphs and charts showing error rates, response times, and other metrics
• 🎯 Pattern Detection: Identify recurring issues or anomalous behavior through pattern matching
• ⏰ Time-Series Analysis: Track how metrics change over time to identify trends and predict issues
• 🔔 Automated Alerting: Configure rules that trigger notifications when specific patterns appear

Compliance and Audit Requirements

Many industries face regulatory requirements around logging and audit trails. Healthcare applications must comply with HIPAA, financial systems with SOX and PCI-DSS, and various industries with GDPR for data protection. These regulations often mandate specific logging practices, retention periods, and access controls.

Compliance-focused logging typically requires immutable audit trails that record who did what and when. These logs must be tamper-evident, meaning any attempts to modify or delete entries are detectable. Some regulations specify minimum retention periods, requiring logs to be preserved for months or years, while others mandate maximum retention to protect privacy.

# Example: Audit log with integrity checking
import hashlib
import json
from datetime import datetime

class AuditLogger:
    def __init__(self, log_file):
        self.log_file = log_file
        self.previous_hash = self.get_last_hash()
    
    def get_last_hash(self):
        """Get hash of last log entry"""
        try:
            with open(self.log_file, 'r') as f:
                lines = f.readlines()
                if lines:
                    last_entry = json.loads(lines[-1])
                    return last_entry.get('hash', '0')
        except FileNotFoundError:
            return '0'
        return '0'
    
    def log_audit_event(self, event_type, user_id, details):
        """Log audit event with integrity hash"""
        entry = {
            'timestamp': datetime.utcnow().isoformat(),
            'event_type': event_type,
            'user_id': user_id,
            'details': details,
            'previous_hash': self.previous_hash
        }
        
        # Calculate hash including previous hash (blockchain-style)
        entry_str = json.dumps(entry, sort_keys=True)
        entry['hash'] = hashlib.sha256(entry_str.encode()).hexdigest()
        
        # Write entry
        with open(self.log_file, 'a') as f:
            f.write(json.dumps(entry) + '\n')
        
        self.previous_hash = entry['hash']
    
    def verify_integrity(self):
        """Verify audit log hasn't been tampered with"""
        with open(self.log_file, 'r') as f:
            previous_hash = '0'
            for line_num, line in enumerate(f, 1):
                entry = json.loads(line)
                
                # Verify hash chain
                if entry['previous_hash'] != previous_hash:
                    return False, f'Hash chain broken at line {line_num}'
                
                # Verify entry hash
                stored_hash = entry.pop('hash')
                calculated_hash = hashlib.sha256(
                    json.dumps(entry, sort_keys=True).encode()
                ).hexdigest()
                
                if stored_hash != calculated_hash:
                    return False, f'Entry modified at line {line_num}'
                
                previous_hash = stored_hash
        
        return True, 'Audit log integrity verified'

# Usage
audit_logger = AuditLogger('audit.log')
audit_logger.log_audit_event('user_login', 12345, {'ip': '192.168.1.100'})
audit_logger.log_audit_event('data_access', 12345, {'record_id': 'REC-789'})

# Verify integrity
is_valid, message = audit_logger.verify_integrity()
print(message)

What's the difference between logging to files versus databases?

File-based logging offers simplicity, high performance, and works without external dependencies. Files integrate easily with standard Unix tools and log rotation utilities. Database logging provides structured storage with powerful querying capabilities and better support for concurrent access from multiple scripts. However, databases introduce additional complexity, dependencies, and potential performance bottlenecks. For most scripts, file-based logging provides the best balance of simplicity and functionality, while database logging makes sense for applications requiring complex queries or centralized log management across many systems.

How do I prevent log files from filling up disk space?

Implement log rotation using either built-in framework features or system tools like logrotate. Configure size-based rotation (create new file when current reaches specific size) or time-based rotation (daily, weekly, monthly). Set retention policies that automatically delete logs older than necessary. Compress archived logs to reduce storage requirements. Monitor disk usage and set up alerts when log directories approach capacity thresholds. For high-volume applications, consider shipping logs to centralized storage or implementing log sampling that captures representative data without recording every event.

Should I use synchronous or asynchronous logging?

Synchronous logging blocks script execution until log writes complete, ensuring messages are recorded before continuing. This approach guarantees logs capture all events even if the script crashes immediately after logging. Asynchronous logging queues messages and writes them in separate threads, minimizing performance impact but risking message loss if the script terminates before the queue flushes. Use synchronous logging for critical errors and audit events where guaranteed recording matters more than performance. Use asynchronous logging for high-frequency informational messages where slight message loss is acceptable in exchange for better performance.

What information should I include in error log messages?

Effective error logs include the error type and message, complete stack trace showing where the error occurred, relevant variable values at the time of failure, timestamp with timezone information, correlation or request IDs for tracing, user or session identifiers (sanitized), the operation being attempted when the error occurred, and any relevant environmental context like server names or deployment versions. Avoid logging sensitive data like passwords or personal information. Include enough context that someone unfamiliar with the code can understand what happened and begin troubleshooting without access to the source code.

How long should I retain log files?

Retention periods depend on multiple factors including regulatory requirements, storage capacity, and operational needs. Compliance regulations often mandate minimum retention (typically 30 days to 7 years depending on industry). Balance legal requirements against storage costs and practical utility. A common approach retains detailed logs for 30-90 days for troubleshooting, keeps compressed archives for 6-12 months for trend analysis, and maintains audit logs for legally required periods. Implement tiered storage moving older logs to cheaper storage solutions. Document retention policies clearly and automate enforcement through rotation and deletion scripts.

Can logging impact application performance significantly?

Yes, excessive or poorly implemented logging can substantially impact performance. Each log operation consumes CPU for formatting and I/O for writing. High-frequency logging in tight loops can become a bottleneck. Minimize impact by using appropriate log levels (disable DEBUG in production), implementing asynchronous logging for high-volume scenarios, batching log writes rather than flushing after each message, avoiding expensive operations in log messages (like serializing large objects), and sampling high-frequency events rather than logging every occurrence. Profile applications to identify logging bottlenecks and optimize accordingly. Well-implemented logging typically adds less than 5% overhead.