Handling File Exceptions and Errors in Python

Every developer encounters file-related errors at some point in their programming journey, and these moments can transform from frustrating roadblocks into opportunities for building resilient applications. When your Python script attempts to read a configuration file that doesn't exist, or tries to write to a directory without proper permissions, the difference between a graceful recovery and a catastrophic crash lies entirely in how you handle these exceptions. Understanding file error management isn't just about preventing your program from breaking—it's about creating software that responds intelligently to the unpredictable nature of file systems, user inputs, and operating environments.

File exception handling in Python encompasses the systematic approach to anticipating, catching, and responding to errors that occur during file operations such as opening, reading, writing, and closing files. This practice involves using Python's built-in exception handling mechanisms to intercept errors before they terminate your program, allowing you to implement alternative strategies, provide meaningful feedback to users, and maintain data integrity. From simple file-not-found scenarios to complex permission issues and encoding problems, proper exception handling addresses multiple perspectives: the developer's need for debuggable code, the user's expectation of clear error messages, and the system's requirement for resource management.

Throughout this exploration, you'll gain practical knowledge of Python's exception hierarchy related to file operations, learn proven patterns for handling common and uncommon file errors, discover techniques for implementing robust error recovery mechanisms, and understand how to write defensive code that anticipates failure points. You'll see concrete examples demonstrating context managers, custom exception handling strategies, and best practices that professional developers use to build production-ready applications that interact with file systems reliably and safely.

Understanding Python's File Exception Hierarchy

Python's exception system for file operations follows a well-organized hierarchy that allows developers to catch errors at different levels of specificity. At the foundation sits the BaseException class, from which all exceptions inherit, but for practical file handling purposes, you'll primarily work with exceptions that inherit from Exception and more specifically from OSError. This hierarchy matters because it determines how you structure your exception handling blocks and how granular your error responses can be.

The most common file-related exceptions you'll encounter include FileNotFoundError, raised when attempting to access a file that doesn't exist; PermissionError, triggered when the program lacks necessary access rights; IsADirectoryError, which occurs when you try to open a directory as if it were a file; and IOError, a general input/output exception that covers various file operation failures. Each of these exceptions provides specific information about what went wrong, allowing you to implement targeted recovery strategies rather than generic error handling.

"The key to robust file handling isn't preventing errors—it's anticipating them and responding appropriately when they inevitably occur."

Understanding this hierarchy enables you to write exception handlers that catch specific problems while allowing unexpected errors to propagate upward. For instance, you might catch FileNotFoundError to create a missing file automatically, handle PermissionError by requesting elevated privileges, but allow other exceptions to bubble up for logging or user notification. This selective approach creates more maintainable code because each handler addresses a specific, known failure scenario with an appropriate response.

Common File Exception Types

Beyond the basic exceptions, Python provides several specialized exception types that address specific file operation scenarios. UnicodeDecodeError and UnicodeEncodeError occur when reading or writing files with incorrect encoding specifications—a particularly common issue when dealing with internationalized content or legacy data files. These encoding-related exceptions require different handling strategies than simple file access errors because they often indicate data format problems rather than system-level issues.

Another important category includes BlockingIOError, which happens during non-blocking file operations, and TimeoutError, relevant when working with network file systems or remote storage. The FileExistsError exception prevents accidental overwrites when creating files with exclusive access flags, while NotADirectoryError catches attempts to treat files as directories. Each exception type carries contextual information through its attributes, including error codes, messages, and sometimes the filename that caused the problem.

Basic Exception Handling Patterns for File Operations

The foundational pattern for handling file exceptions in Python uses the try-except block structure, which separates the code that might fail from the code that handles the failure. This separation creates cleaner, more readable code because the "happy path" logic remains distinct from error handling logic. When working with files, you place the file operation code inside the try block and specify which exceptions to catch in one or more except clauses, each potentially handling different error types with appropriate responses.

A basic implementation might look straightforward, but several important considerations affect how you structure these blocks. First, you should catch specific exceptions rather than using a bare except clause, which would catch all exceptions including system exits and keyboard interrupts. Second, the order of except clauses matters because Python evaluates them sequentially and uses the first match, meaning more specific exceptions should appear before more general ones. Third, you can use the else clause to execute code only when no exceptions occurred, and the finally clause to ensure cleanup code runs regardless of whether exceptions were raised.

try:
    with open('data.txt', 'r') as file:
        content = file.read()
        process_data(content)
except FileNotFoundError:
    print("Data file not found, using default values")
    content = get_default_data()
except PermissionError:
    print("Cannot access file due to permissions")
    log_security_event()
except UnicodeDecodeError as e:
    print(f"Encoding error: {e}")
    content = read_with_fallback_encoding()
else:
    print("File processed successfully")
finally:
    cleanup_resources()

This pattern demonstrates several best practices: catching specific exceptions allows targeted responses, using the exception object (as e) provides access to error details, and the finally block ensures resource cleanup happens even if an exception occurs. The with statement (context manager) automatically handles file closing, which is particularly important because it ensures the file closes even if an exception occurs during processing, preventing resource leaks and file corruption.

Context Managers and Automatic Resource Management

Context managers represent Python's elegant solution to the resource management problem, particularly critical when working with files. The with statement creates a runtime context that guarantees cleanup operations execute regardless of how the block exits—whether normally, through an exception, or via a return statement. This guarantee is invaluable for file operations because it prevents common bugs like forgetting to close files, which can lead to data loss, resource exhaustion, and file locking issues that affect other processes.

"Context managers transform file handling from a manual, error-prone process into an automated, reliable pattern that handles both success and failure scenarios uniformly."

The beauty of context managers extends beyond simple file closing. When an exception occurs within a with block, Python ensures the file's __exit__ method executes before the exception propagates, allowing the file object to flush buffers, release locks, and perform other cleanup operations. This behavior means you can focus on handling the business logic exceptions without worrying about low-level resource management. Additionally, context managers can be nested or combined, allowing you to manage multiple files simultaneously while maintaining clean exception handling for each resource.

try:
    with open('input.txt', 'r') as infile, open('output.txt', 'w') as outfile:
        for line in infile:
            processed = transform_line(line)
            outfile.write(processed)
except FileNotFoundError as e:
    print(f"Missing file: {e.filename}")
except PermissionError:
    print("Access denied to one or more files")
except Exception as e:
    print(f"Unexpected error during file processing: {e}")
    raise  # Re-raise after logging

Advanced Exception Handling Techniques

Moving beyond basic try-except blocks, advanced exception handling incorporates strategies like exception chaining, custom exception classes, and exception groups (in Python 3.11+). Exception chaining using the "raise from" syntax preserves the original exception context while adding higher-level semantic meaning, creating a traceback that shows the complete error history. This technique proves invaluable when debugging complex file operations where a high-level failure (like "configuration loading failed") resulted from a low-level issue (like "permission denied on specific file").

Custom exception classes allow you to create domain-specific exceptions that carry additional context about file operation failures. Rather than catching generic OSError exceptions throughout your codebase, you might define ConfigurationFileError, DataValidationError, or BackupFailureError classes that inherit from appropriate base exceptions but add application-specific attributes and behaviors. These custom exceptions make your code more self-documenting and enable more precise error handling at different layers of your application.

class ConfigurationError(Exception):
    """Base exception for configuration-related errors"""
    def __init__(self, message, filename=None, line_number=None):
        super().__init__(message)
        self.filename = filename
        self.line_number = line_number

class ConfigurationFileNotFoundError(ConfigurationError, FileNotFoundError):
    """Configuration file doesn't exist"""
    pass

class ConfigurationParseError(ConfigurationError):
    """Configuration file exists but cannot be parsed"""
    pass

def load_configuration(config_path):
    try:
        with open(config_path, 'r') as config_file:
            return parse_config(config_file.read())
    except FileNotFoundError as e:
        raise ConfigurationFileNotFoundError(
            f"Configuration file not found",
            filename=config_path
        ) from e
    except ValueError as e:
        raise ConfigurationParseError(
            f"Invalid configuration format",
            filename=config_path
        ) from e

Retry Mechanisms and Fallback Strategies

Robust file handling often requires implementing retry logic for transient failures and fallback strategies for persistent errors. Transient failures—like temporary file locks, network hiccups when accessing remote files, or momentary permission issues—often resolve themselves within seconds. Implementing exponential backoff retry logic allows your application to automatically recover from these temporary conditions without user intervention, improving reliability and user experience.

Fallback strategies provide alternative paths when primary file operations fail. These might include using cached data when fresh data isn't accessible, falling back to default configuration values when custom configs are unavailable, or switching to alternative file formats or locations. The key to effective fallback implementation lies in maintaining a clear hierarchy of preferences and ensuring each fallback attempt is itself protected by appropriate exception handling to prevent cascading failures.

import time
from pathlib import Path

def read_file_with_retry(filepath, max_attempts=3, delay=1):
    """
    Attempts to read a file with exponential backoff retry logic
    """
    for attempt in range(max_attempts):
        try:
            with open(filepath, 'r') as file:
                return file.read()
        except PermissionError:
            if attempt < max_attempts - 1:
                wait_time = delay * (2 ** attempt)
                print(f"Access denied, retrying in {wait_time} seconds...")
                time.sleep(wait_time)
            else:
                raise
        except FileNotFoundError:
            # Don't retry for non-existent files
            raise
    
def load_config_with_fallback(primary_path, fallback_path, default_config):
    """
    Attempts to load configuration with multiple fallback levels
    """
    paths = [primary_path, fallback_path]
    
    for path in paths:
        try:
            return read_file_with_retry(path)
        except FileNotFoundError:
            print(f"Config not found at {path}, trying next option...")
            continue
        except PermissionError:
            print(f"Cannot access {path}, trying next option...")
            continue
        except Exception as e:
            print(f"Unexpected error with {path}: {e}")
            continue
    
    print("All config sources failed, using defaults")
    return default_config

"Effective error handling isn't about preventing all failures—it's about ensuring your application degrades gracefully and provides clear feedback when things go wrong."

Handling Encoding and Binary File Exceptions

Encoding-related exceptions represent a particularly challenging category of file errors because they often manifest unexpectedly when processing files from different sources, platforms, or locales. When Python encounters a byte sequence it cannot decode using the specified encoding, it raises UnicodeDecodeError, providing information about the problematic byte position and the encoding that failed. Similarly, UnicodeEncodeError occurs when writing text containing characters that cannot be represented in the target encoding, a common issue when working with international text or emoji characters.

Handling encoding exceptions effectively requires understanding Python's encoding error handlers: 'strict' (default, raises exceptions), 'ignore' (skips problematic characters), 'replace' (substitutes with replacement characters), 'backslashreplace' (uses Python string escape sequences), and 'xmlcharrefreplace' (uses XML character references). Each handler offers different trade-offs between data fidelity and processing robustness, and choosing the appropriate handler depends on your application's requirements for data integrity versus fault tolerance.

def read_file_with_encoding_fallback(filepath, encodings=['utf-8', 'latin-1', 'cp1252']):
    """
    Attempts to read a file trying multiple encodings
    """
    for encoding in encodings:
        try:
            with open(filepath, 'r', encoding=encoding) as file:
                content = file.read()
                print(f"Successfully read file using {encoding} encoding")
                return content
        except UnicodeDecodeError:
            print(f"Failed to decode with {encoding}, trying next...")
            continue
        except FileNotFoundError:
            raise
    
    # Last resort: read with error handling
    try:
        with open(filepath, 'r', encoding='utf-8', errors='replace') as file:
            print("Reading with UTF-8 and replacing invalid characters")
            return file.read()
    except Exception as e:
        raise IOError(f"Could not read file with any encoding: {e}")

def safe_write_with_encoding(filepath, content, encoding='utf-8'):
    """
    Writes content with encoding error handling
    """
    try:
        with open(filepath, 'w', encoding=encoding) as file:
            file.write(content)
    except UnicodeEncodeError as e:
        print(f"Encoding error at position {e.start}: {e.reason}")
        # Retry with error handling
        with open(filepath, 'w', encoding=encoding, errors='xmlcharrefreplace') as file:
            file.write(content)
            print("Written with XML character references for problematic characters")

Binary File Exception Handling

Binary file operations introduce their own exception considerations, particularly when dealing with structured binary formats, compressed files, or memory-mapped files. Unlike text files where encoding issues dominate, binary file exceptions often relate to format validation, corruption detection, and incomplete reads. When reading binary files, you must handle scenarios where the file appears valid initially but contains corrupted data structures, truncated content, or unexpected format variations.

Working with binary files requires more defensive programming because there's no encoding layer to validate data integrity. You should implement checksum verification, magic number validation, and size consistency checks as part of your exception handling strategy. Additionally, when working with memory-mapped files or large binary datasets, you need to handle MemoryError exceptions that might occur when system resources are exhausted, implementing chunked reading strategies as fallbacks.

Logging and Debugging File Exceptions

Effective exception handling extends beyond catching errors to include comprehensive logging that facilitates debugging and monitoring. Python's logging module integrates seamlessly with exception handling, allowing you to capture exception details, stack traces, and contextual information without cluttering your error handling code. When file exceptions occur, logging should capture not just the exception type and message, but also relevant context like file paths, operation types, user identities, and system states that might have contributed to the failure.

"Logging isn't just about recording failures—it's about creating a narrative that helps you understand the sequence of events leading to an error."

Structured logging approaches, where log entries include machine-readable fields alongside human-readable messages, enable powerful analysis of file operation patterns and failure modes. You can track which files fail most frequently, which exception types dominate your error logs, and whether failures correlate with specific times, users, or system conditions. This data-driven approach to exception handling transforms reactive debugging into proactive system improvement.

import logging
from pathlib import Path
from datetime import datetime

# Configure logging with detailed format
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('file_operations.log'),
        logging.StreamHandler()
    ]
)

logger = logging.getLogger(__name__)

def process_file_with_logging(filepath):
    """
    Process file with comprehensive exception logging
    """
    logger.info(f"Starting file processing: {filepath}")
    
    try:
        file_path = Path(filepath)
        
        # Log file metadata
        if file_path.exists():
            stats = file_path.stat()
            logger.debug(f"File size: {stats.st_size} bytes, "
                        f"Modified: {datetime.fromtimestamp(stats.st_mtime)}")
        
        with open(file_path, 'r') as file:
            content = file.read()
            result = process_content(content)
            logger.info(f"Successfully processed {filepath}")
            return result
            
    except FileNotFoundError:
        logger.error(f"File not found: {filepath}", exc_info=True)
        raise
    except PermissionError:
        logger.error(f"Permission denied: {filepath}", 
                    extra={'filepath': str(filepath), 'user': get_current_user()})
        raise
    except UnicodeDecodeError as e:
        logger.error(f"Encoding error in {filepath}: {e.reason} at position {e.start}",
                    exc_info=True)
        raise
    except Exception as e:
        logger.critical(f"Unexpected error processing {filepath}: {type(e).__name__}",
                       exc_info=True,
                       extra={'filepath': str(filepath), 'exception_type': type(e).__name__})
        raise

Platform-Specific Exception Handling

File system operations behave differently across operating systems, and robust exception handling must account for these platform-specific variations. Windows, Linux, and macOS implement different file locking mechanisms, permission models, and path conventions, each potentially raising distinct exceptions or requiring different recovery strategies. For example, Windows raises PermissionError when attempting to delete a file that's currently open, while Unix-like systems allow deletion but prevent reading until the file handle closes.

Path handling represents another platform-specific consideration. Windows uses backslashes and drive letters, Unix-like systems use forward slashes and mount points, and each platform has different rules for maximum path lengths, reserved filenames, and case sensitivity. Using Python's pathlib module abstracts many of these differences, but you still need to handle exceptions that arise from platform-specific limitations, such as OSError with errno values indicating path-too-long conditions or invalid filename characters.

Cross-Platform File Operation Patterns

Writing truly cross-platform file handling code requires abstracting platform differences behind consistent interfaces while still handling platform-specific exceptions appropriately. The pathlib.Path class provides an excellent foundation, automatically handling path separator differences and offering methods that work consistently across platforms. However, you must still implement exception handling that accounts for behaviors that cannot be abstracted, such as file locking semantics or permission checking mechanisms.

import os
import sys
from pathlib import Path
import platform

def create_file_safely(filepath, content):
    """
    Creates a file with platform-aware exception handling
    """
    path = Path(filepath)
    
    try:
        # Validate path length based on platform
        if platform.system() == 'Windows' and len(str(path.absolute())) > 260:
            # Try using extended path syntax on Windows
            path = Path('\\\\?\\' + str(path.absolute()))
        
        # Create parent directories if needed
        path.parent.mkdir(parents=True, exist_ok=True)
        
        # Write file with platform-appropriate handling
        with open(path, 'w') as file:
            file.write(content)
            
    except OSError as e:
        if e.errno == 36:  # ENAMETOOLONG on Unix
            raise ValueError(f"Path too long: {filepath}")
        elif e.errno == 63:  # ENAMETOOLONG on some systems
            raise ValueError(f"Filename too long: {path.name}")
        elif platform.system() == 'Windows' and 'reserved' in str(e).lower():
            raise ValueError(f"Reserved filename on Windows: {path.name}")
        else:
            raise
    except PermissionError:
        if platform.system() == 'Windows':
            # On Windows, check if file is locked
            raise IOError(f"File may be locked by another process: {filepath}")
        else:
            # On Unix, check permissions
            raise PermissionError(f"Insufficient permissions for: {filepath}")

def safe_file_delete(filepath):
    """
    Deletes file with platform-specific retry logic
    """
    path = Path(filepath)
    max_attempts = 3 if platform.system() == 'Windows' else 1
    
    for attempt in range(max_attempts):
        try:
            path.unlink()
            return True
        except PermissionError:
            if platform.system() == 'Windows' and attempt < max_attempts - 1:
                # File might be locked on Windows, wait and retry
                import time
                time.sleep(0.1)
                continue
            raise
        except FileNotFoundError:
            return False  # Already deleted

Testing Exception Handling Code

Comprehensive testing of file exception handling requires simulating various failure conditions that might be difficult or impossible to reproduce in normal development environments. Mock objects and pytest fixtures enable you to inject controlled failures into your file operations, verifying that your exception handlers respond correctly to each error type. Testing exception handling isn't just about confirming that exceptions get caught—it's about validating that your recovery logic works, cleanup operations execute, and error messages provide useful information.

"Untested exception handling code is technical debt waiting to manifest at the worst possible moment—usually in production when stakes are highest."

Effective exception handling tests should cover multiple scenarios: the happy path where no exceptions occur, each specific exception type your code handles, combinations of exceptions, and edge cases like exceptions during cleanup operations. Using context managers in your tests, such as pytest's raises context manager, allows you to assert that specific exceptions occur under expected conditions and verify exception attributes like error messages and attached context.

import pytest
from unittest.mock import mock_open, patch, MagicMock
from pathlib import Path

def test_file_not_found_handling():
    """Test that FileNotFoundError is handled correctly"""
    with pytest.raises(FileNotFoundError):
        with open('nonexistent_file.txt', 'r') as f:
            content = f.read()

def test_permission_error_with_retry():
    """Test retry logic for permission errors"""
    mock_file = mock_open()
    mock_file.side_effect = [
        PermissionError("Access denied"),
        PermissionError("Access denied"),
        mock_open(read_data="success").return_value
    ]
    
    with patch('builtins.open', mock_file):
        result = read_file_with_retry('test.txt', max_attempts=3)
        assert result == "success"
        assert mock_file.call_count == 3

def test_encoding_fallback_mechanism():
    """Test that encoding fallback works correctly"""
    # Create file with Latin-1 encoding
    test_content = "Café résumé"
    
    with patch('builtins.open', mock_open(read_data=test_content.encode('latin-1'))):
        result = read_file_with_encoding_fallback('test.txt')
        assert result is not None

def test_cleanup_on_exception():
    """Test that cleanup occurs even when exceptions happen"""
    cleanup_called = False
    
    def cleanup():
        nonlocal cleanup_called
        cleanup_called = True
    
    try:
        with open('test.txt', 'w') as f:
            f.write("test")
            raise ValueError("Simulated error")
    except ValueError:
        cleanup()
    
    assert cleanup_called

def test_custom_exception_attributes():
    """Test that custom exceptions carry correct context"""
    with pytest.raises(ConfigurationError) as exc_info:
        raise ConfigurationError(
            "Invalid config",
            filename="config.yaml",
            line_number=42
        )
    
    assert exc_info.value.filename == "config.yaml"
    assert exc_info.value.line_number == 42

@pytest.fixture
def temp_file_with_permissions(tmp_path):
    """Fixture that creates a file with specific permissions"""
    test_file = tmp_path / "test.txt"
    test_file.write_text("test content")
    test_file.chmod(0o000)  # Remove all permissions
    yield test_file
    test_file.chmod(0o644)  # Restore permissions for cleanup

def test_permission_error_handling(temp_file_with_permissions):
    """Test handling of actual permission errors"""
    with pytest.raises(PermissionError):
        with open(temp_file_with_permissions, 'r') as f:
            content = f.read()

Best Practices and Common Pitfalls

Professional file exception handling follows established patterns that balance robustness with maintainability. Always use context managers (with statements) for file operations to ensure proper resource cleanup, even when exceptions occur. Catch specific exceptions rather than using bare except clauses, which can mask unexpected errors and make debugging difficult. When re-raising exceptions, use raise without arguments to preserve the original traceback, or use raise from to chain exceptions while adding context.

Avoid common pitfalls like catching exceptions too broadly, silently swallowing errors without logging, or implementing retry logic without maximum attempt limits. Don't use exceptions for normal control flow—checking if a file exists with Path.exists() before opening is more appropriate than catching FileNotFoundError when existence is the expected case. However, when dealing with race conditions or concurrent access, exception handling might be more reliable than pre-checking conditions that could change before the actual operation.

• ✨ Always use context managers for file operations to guarantee resource cleanup regardless of exceptions
• 🔍 Log exceptions with full context including file paths, operation types, and relevant system state
• 🎯 Catch specific exceptions rather than generic Exception or bare except clauses
• 🔄 Implement retry logic with limits for transient failures but not for permanent errors like FileNotFoundError
• 📝 Provide meaningful error messages that help users understand what went wrong and how to fix it
• 🛡️ Validate inputs before operations to prevent predictable exceptions when possible
• 🔗 Use exception chaining to preserve error context when wrapping low-level exceptions in high-level ones
• 🧪 Test exception handling paths as thoroughly as success paths to ensure recovery logic works
• ⚡ Consider performance implications of exception handling in hot code paths

Performance Considerations

While Python's exception handling is reasonably efficient, exceptions do carry performance overhead compared to normal control flow. In performance-critical code that processes thousands of files, the difference between checking file existence before opening versus catching FileNotFoundError can be measurable. However, premature optimization often leads to less robust code—prioritize correctness and clarity first, then optimize based on profiling data if performance becomes an issue.

When performance matters, consider strategies like batching file operations, caching file metadata, and using asynchronous I/O for concurrent file access rather than trying to eliminate exception handling. In scenarios where exceptions are expected frequently (like processing user-uploaded files with unpredictable formats), structuring your code to validate inputs before expensive operations can improve performance while maintaining robust error handling for truly unexpected failures.

Real-World Application Patterns

Production applications often require sophisticated exception handling patterns that go beyond simple try-except blocks. Consider a data processing pipeline that reads configuration files, processes input data, and writes results—each stage needs appropriate exception handling with different recovery strategies. Configuration loading might fall back to defaults, data processing might skip corrupted records while logging errors, and output writing might retry with backoff or switch to alternative storage locations.

"Production-grade file handling isn't about preventing all errors—it's about ensuring your application remains functional and informative when the inevitable failures occur."

import json
import csv
from pathlib import Path
from typing import List, Dict, Any
import logging

logger = logging.getLogger(__name__)

class DataProcessor:
    """Production-ready data processor with comprehensive error handling"""
    
    def __init__(self, config_path: str):
        self.config = self._load_config(config_path)
        self.processed_count = 0
        self.error_count = 0
        
    def _load_config(self, config_path: str) -> Dict[str, Any]:
        """Load configuration with fallback to defaults"""
        default_config = {
            'input_dir': 'input',
            'output_dir': 'output',
            'error_dir': 'errors',
            'max_errors': 10
        }
        
        try:
            with open(config_path, 'r') as f:
                config = json.load(f)
                logger.info(f"Loaded configuration from {config_path}")
                return {**default_config, **config}
        except FileNotFoundError:
            logger.warning(f"Config file not found, using defaults")
            return default_config
        except json.JSONDecodeError as e:
            logger.error(f"Invalid JSON in config: {e}")
            return default_config
        except Exception as e:
            logger.error(f"Unexpected error loading config: {e}")
            return default_config
    
    def process_directory(self, input_dir: str = None) -> Dict[str, int]:
        """Process all files in directory with error tracking"""
        input_path = Path(input_dir or self.config['input_dir'])
        
        if not input_path.exists():
            logger.error(f"Input directory not found: {input_path}")
            return {'processed': 0, 'errors': 0, 'skipped': 0}
        
        results = {'processed': 0, 'errors': 0, 'skipped': 0}
        
        try:
            files = list(input_path.glob('*.csv'))
            logger.info(f"Found {len(files)} files to process")
            
            for file_path in files:
                try:
                    self._process_single_file(file_path)
                    results['processed'] += 1
                except Exception as e:
                    logger.error(f"Error processing {file_path}: {e}", exc_info=True)
                    self._move_to_error_dir(file_path)
                    results['errors'] += 1
                    
                    if results['errors'] >= self.config['max_errors']:
                        logger.critical("Maximum error count reached, stopping processing")
                        break
                        
        except Exception as e:
            logger.critical(f"Fatal error during directory processing: {e}", exc_info=True)
            
        return results
    
    def _process_single_file(self, file_path: Path):
        """Process individual file with specific error handling"""
        try:
            with open(file_path, 'r', encoding='utf-8') as f:
                reader = csv.DictReader(f)
                data = list(reader)
                
            processed_data = self._transform_data(data)
            self._write_output(file_path.stem, processed_data)
            
        except UnicodeDecodeError:
            # Try alternative encoding
            logger.warning(f"UTF-8 decode failed for {file_path}, trying latin-1")
            with open(file_path, 'r', encoding='latin-1') as f:
                reader = csv.DictReader(f)
                data = list(reader)
            processed_data = self._transform_data(data)
            self._write_output(file_path.stem, processed_data)
            
        except csv.Error as e:
            raise ValueError(f"CSV format error: {e}")
        except PermissionError:
            raise IOError(f"Cannot access file: {file_path}")
    
    def _transform_data(self, data: List[Dict]) -> List[Dict]:
        """Transform data with row-level error handling"""
        results = []
        for i, row in enumerate(data):
            try:
                transformed = self._transform_row(row)
                results.append(transformed)
            except (KeyError, ValueError) as e:
                logger.warning(f"Skipping row {i}: {e}")
                continue
        return results
    
    def _write_output(self, filename: str, data: List[Dict]):
        """Write output with retry and fallback"""
        output_dir = Path(self.config['output_dir'])
        output_dir.mkdir(parents=True, exist_ok=True)
        output_path = output_dir / f"{filename}_processed.json"
        
        max_attempts = 3
        for attempt in range(max_attempts):
            try:
                with open(output_path, 'w') as f:
                    json.dump(data, f, indent=2)
                logger.info(f"Wrote output to {output_path}")
                return
            except PermissionError:
                if attempt < max_attempts - 1:
                    import time
                    time.sleep(0.5 * (attempt + 1))
                else:
                    # Fallback to alternative location
                    fallback_path = Path.home() / 'output' / output_path.name
                    fallback_path.parent.mkdir(parents=True, exist_ok=True)
                    with open(fallback_path, 'w') as f:
                        json.dump(data, f, indent=2)
                    logger.warning(f"Wrote to fallback location: {fallback_path}")
    
    def _move_to_error_dir(self, file_path: Path):
        """Move problematic files to error directory"""
        try:
            error_dir = Path(self.config['error_dir'])
            error_dir.mkdir(parents=True, exist_ok=True)
            destination = error_dir / file_path.name
            file_path.rename(destination)
            logger.info(f"Moved error file to {destination}")
        except Exception as e:
            logger.error(f"Could not move error file: {e}")

What's the difference between catching Exception and catching specific exceptions like FileNotFoundError?

Catching specific exceptions allows you to handle different error conditions with appropriate responses, while catching generic Exception can mask unexpected errors that should propagate. Specific exception handling makes your code more maintainable because each handler addresses a known failure scenario with targeted recovery logic. Generic exception catching should be reserved for logging or cleanup operations where you truly need to catch all possible errors.

Should I check if a file exists before opening it, or just catch FileNotFoundError?

The best approach depends on your use case. If file existence is the expected case and absence is exceptional, checking with Path.exists() first makes intent clearer and performs better. However, in concurrent environments or when race conditions matter, catching FileNotFoundError is more reliable because the file's existence could change between your check and the actual open operation. For most single-threaded applications, pre-checking is fine and often more readable.

How do I handle encoding errors when I don't know the file's encoding in advance?

Implement a fallback strategy that tries multiple common encodings (UTF-8, Latin-1, CP1252) in order of likelihood, catching UnicodeDecodeError for each attempt. As a last resort, open the file with errors='replace' or errors='ignore' to handle problematic bytes. Consider using libraries like chardet to detect encoding automatically, though this adds dependencies and processing overhead. Always log which encoding succeeded to help diagnose patterns in your data sources.

What's the best way to ensure files are closed even when exceptions occur?

Always use context managers (with statements) for file operations. The with statement guarantees that the file's __exit__ method executes regardless of how the block exits—whether normally, through an exception, or via a return statement. This automatic cleanup prevents resource leaks, data corruption, and file locking issues. Even if you need to catch exceptions, wrap the with statement in try-except rather than manually managing file closing in finally blocks.

How can I test exception handling code effectively?

Use mocking frameworks like unittest.mock to simulate various failure conditions without relying on actual file system states. Create fixtures that set up specific error scenarios, use pytest.raises to assert that expected exceptions occur, and verify that your recovery logic executes correctly. Test not just that exceptions are caught, but that cleanup operations run, fallback strategies work, and error messages provide useful information. Include tests for exception combinations and edge cases like exceptions during cleanup operations.

When should I use custom exception classes instead of built-in ones?

Create custom exceptions when you need to add domain-specific context, group related errors under a common base class, or provide additional attributes beyond the standard message. Custom exceptions make your code more self-documenting and enable more precise error handling at different application layers. However, custom exceptions should inherit from appropriate built-in exceptions (like FileNotFoundError or ValueError) to maintain compatibility with existing exception handling patterns and allow catching by either specific or general exception types.