Streams – The Universal I/O Model

The stream is .NET's fundamental abstraction for sequential access to data. Whether the data source is a file on disk, bytes arriving over a network connection, a block of memory, or a compressed archive – all of them surface through the same System.IO.Stream base class. This uniformity isn't just convenient; it enables stream composition, the technique of wrapping one stream inside another to add capabilities like compression, encryption, or character encoding without changing the code that actually reads the data.

Every Stream exposes three core capabilities, each controlled by a corresponding boolean property: CanRead, CanWrite, and CanSeek. A FileStream supports all three. A NetworkStream supports reading and writing but not seeking – you can't jump backwards in a network byte stream. A MemoryStream supports all three. Code that only needs to read can depend on Stream directly; code that tests whether seeking is possible checks CanSeek before calling Seek.

Stream Composition in Practice

Stream composition is what makes the abstraction truly powerful for test infrastructure. Imagine loading test fixture data from a compressed file – a common approach when fixtures are large and stored in source control. Without composition, you'd need to decompress the file to disk first, then read from the decompressed copy. With composition, you wrap a GZipStream around a FileStream and wrap a StreamReader around that – each layer adds a capability, and the code that reads the data sees only text:

// Stream composition: each wrapper adds a capability.
// This chain reads a gzip-compressed fixture file as UTF-8 text.
await using var fileStream = File.OpenRead("fixtures/products.json.gz");
await using var gzipStream = new GZipStream(fileStream, CompressionMode.Decompress);
using         var reader    = new StreamReader(gzipStream, Encoding.UTF8);

// ReadToEndAsync sees plain UTF-8 text – the compression is completely invisible
var json = await reader.ReadToEndAsync();

// The same code reads uncompressed data if you remove the GZipStream wrapper.
// The reader doesn't care what's beneath it.

Disposal flows outward. When reader is disposed, it disposes gzipStream. When gzipStream is disposed, it disposes fileStream. Using nested await using declarations achieves the same result explicitly – each stream is disposed in declaration order (bottom up), matching the composition order.

MemoryStream for In-Memory Testing

In test contexts, MemoryStream is a valuable tool for unit-testing code that consumes or produces streams without touching the filesystem. Any code that accepts a Stream parameter can be tested by passing a MemoryStream pre-populated with known bytes, making assertions straightforward and cleanup automatic:

// Unit-testing a serialization helper without touching disk
[Test]
public async Task SerializeUser_ShouldProduceValidJson()
{
    var user = new User { UserId = 1, Email = "[email protected]" };

    // MemoryStream acts as an in-memory file
    using var stream = new MemoryStream();
    await JsonSerializer.SerializeAsync(stream, user);

    // Rewind to the beginning for reading
    stream.Position = 0;
    using var reader = new StreamReader(stream);
    var json = await reader.ReadToEndAsync();

    Assert.That(json, Does.Contain("[email protected]"));
}

The stream model is one of .NET's most elegant design decisions. The upfront investment of learning its composition model pays dividends every time test code needs to add encryption, compression, or encoding to an existing I/O pipeline – one wrapper class, no changes to the code that reads or writes data.

Reading and Writing Files

.NET provides three levels of abstraction for file access, each appropriate for different scenarios. Understanding where each fits prevents both under-engineering (reading a 50MB log file into a single string) and over-engineering (using FileStream raw byte operations for a 2KB JSON fixture).

• File static methods: The simplest option. File.ReadAllText, File.WriteAllText, File.ReadAllBytes. Load the entire file into memory in one call. Right for small files where simplicity matters more than memory efficiency.
• StreamReader / StreamWriter: Character-level wrappers around a FileStream. Support line-by-line reading, configurable encoding, and efficient handling of large text files without loading everything at once.
• FileStream directly: Raw byte access with full control over buffering, reading strategy, and file sharing options. Right for binary data and performance-critical scenarios.

Path Handling Across Platforms

Test suites frequently run on Windows development machines, Linux CI agents, and macOS developer workstations simultaneously. Path handling is the most common source of cross-platform breaks. The Path class handles platform differences correctly when used consistently:

// ALWAYS use Path.Combine for building paths – never string concatenation.
// Path.Combine handles the platform separator automatically.
var dataDirectory = Path.Combine(AppContext.BaseDirectory, "TestData");
var usersFile     = Path.Combine(dataDirectory, "users.json");
var artifactDir   = Path.Combine(AppContext.BaseDirectory, "Artifacts", "Screenshots");

// BAD – works on Windows, breaks on Linux (wrong separator)
// var badPath = AppContext.BaseDirectory + "\\TestData\\users.json";

// Directory.CreateDirectory is idempotent: creates the full path,
// including intermediate directories, and doesn't throw if they already exist.
Directory.CreateDirectory(artifactDir);

// Path utility methods for working with names and extensions
var directory  = Path.GetDirectoryName(usersFile);   // "...TestData"
var filename   = Path.GetFileName(usersFile);         // "users.json"
var nameNoExt  = Path.GetFileNameWithoutExtension(usersFile); // "users"
var extension  = Path.GetExtension(usersFile);        // ".json"

Reading Test Data Files

For small-to-medium fixture files – typically under 10MB – File.ReadAllTextAsync is the simplest correct choice. For large files where memory is a concern, StreamReader with ReadLineAsync processes the file one line at a time without loading all content simultaneously:

// Small fixture: load entirely into memory
var json = await File.ReadAllTextAsync(
    Path.Combine(dataDirectory, "orders.json"));

// Large log file: process line by line without loading everything at once
var failedOrderIds = new List<string>();
using var reader = new StreamReader(Path.Combine(logDirectory, "application.log"));

while (!reader.EndOfStream)
{
    var line = await reader.ReadLineAsync();
    if (line is not null && line.Contains("ORDER_FAILED"))
    {
        // Extract order ID from structured log line
        var orderId = ExtractOrderId(line);
        if (orderId is not null)
            failedOrderIds.Add(orderId);
    }
}

Writing Artifacts Safely

Writing test artifacts – screenshots, reports, response logs – requires a filename strategy that prevents conflicts between parallel test runs and makes artifacts navigable after the fact. A timestamp combined with a sanitised test name produces filenames that sort chronologically and identify their source:

// Generate a safe, unique artifact filename
private static string BuildArtifactPath(string directory, string testName, string extension)
{
    // Remove characters that are invalid in file/directory names on any platform
    var invalidChars = Path.GetInvalidFileNameChars();
    var safeName     = string.Concat(
        testName.Select(c => invalidChars.Contains(c) ? '_' : c));

    // Timestamp with milliseconds prevents collisions in parallel test runs
    var fileName = $"{safeName}_{DateTime.UtcNow:yyyyMMdd_HHmmss_fff}{extension}";
    return Path.Combine(directory, fileName);
}

// Writing a screenshot file
public static async Task<string> WriteScreenshotAsync(
    byte[] screenshotBytes,
    string testName,
    string outputDirectory)
{
    Directory.CreateDirectory(outputDirectory);
    var path = BuildArtifactPath(outputDirectory, testName, ".png");
    await File.WriteAllBytesAsync(path, screenshotBytes);
    return path;
}

File and path operations have a disproportionate failure rate in cross-platform test environments precisely because they seem straightforward but contain subtle platform assumptions. Committing to Path.Combine for every path, Directory.CreateDirectory before every write, and async variants for all I/O eliminates the entire class of problems – and does so with code that's no more complex than the naive approach.

JSON Serialization for Test Data

System.Text.Json is the modern, first-party JSON library in .NET and the right default for new test automation code. It's faster than Newtonsoft.Json, allocates less memory, and is included in the framework without adding a NuGet dependency. Its API is deliberately simple for common scenarios and extensible with custom converters when needed.

Configuration with JsonSerializerOptions

The default JsonSerializerOptions are deliberately strict – property name matching is case-sensitive, null values are serialized, and numbers aren't read from strings. For test data files, more permissive options are usually appropriate since the files may come from varied sources, be hand-edited, or originate from systems with different naming conventions:

// Standard options for test data – define once and reuse across the project
internal static class JsonOptions
{
    public static readonly JsonSerializerOptions TestData = new()
    {
        // Case-insensitive matching: "productId", "ProductId", "product_id" all work
        PropertyNameCaseInsensitive = true,

        // Compact output for stored test data files (saves space without GZip)
        WriteIndented = false,

        // Skip null properties when serializing expected results
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,

        // Read numbers from JSON strings ("42" parsed as 42) – common in export files
        NumberHandling = JsonNumberHandling.AllowReadingFromString
    };

    public static readonly JsonSerializerOptions HumanReadable = new()
    {
        PropertyNameCaseInsensitive = true,
        WriteIndented = true, // Pretty-print for report files and debugging output
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
    };
}

Stream-Based Serialization

The stream-based overloads of JsonSerializer – SerializeAsync and DeserializeAsync – are preferable to their string-based counterparts for file operations. They never materialise the entire JSON string in memory simultaneously, which matters for large fixture files and is a better fit for the async I/O model:

// Stream-based deserialization: reads and parses concurrently without loading all bytes first
public static async Task<T> LoadJsonAsync<T>(string filePath)
{
    if (!File.Exists(filePath))
        throw new FileNotFoundException($"Test data file not found: {filePath}", filePath);

    await using var stream = File.OpenRead(filePath);
    return await JsonSerializer.DeserializeAsync<T>(stream, JsonOptions.TestData)
        ?? throw new InvalidOperationException(
               $"File '{Path.GetFileName(filePath)}' deserialised to null.");
}

// Stream-based serialization: writes bytes directly to the file stream
public static async Task SaveJsonAsync<T>(T data, string filePath)
{
    Directory.CreateDirectory(Path.GetDirectoryName(filePath)!);
    await using var stream = File.Create(filePath);
    await JsonSerializer.SerializeAsync(stream, data, JsonOptions.HumanReadable);
}

A Caching Test Data Loader

A generic loader that caches deserialized fixtures prevents repeated disk reads when multiple tests use the same data file. The cache key is the filename; the first access pays the I/O cost, all subsequent accesses read from memory:

// Generic test data loader with in-process caching.
// Designed for fixture data shared across multiple tests in a suite.
public sealed class TestDataLoader<T>
{
    private readonly string _directory;
    private readonly Dictionary<string, T> _cache = new();

    public TestDataLoader(string directory)
    {
        _directory = directory;
    }

    public async Task<T> LoadAsync(string fileName)
    {
        // Cache hit: no file I/O, no allocation
        if (_cache.TryGetValue(fileName, out var cached))
            return cached;

        var path = Path.Combine(_directory, fileName);
        await using var stream = File.OpenRead(path);
        var data = await JsonSerializer.DeserializeAsync<T>(stream, JsonOptions.TestData)
            ?? throw new InvalidOperationException(
                   $"'{fileName}' deserialised to null – check the fixture file.");

        _cache[fileName] = data;
        return data;
    }

    // Explicit cache invalidation for tests that modify shared data
    public void Invalidate(string fileName) => _cache.Remove(fileName);
}

JSON serialization is the single most common file format in test automation. Test data fixtures, expected API responses, configuration overrides, and test run reports all tend toward JSON. Establishing a single JsonSerializerOptions constant shared across the project – rather than constructing new options at each call site – ensures consistent behaviour and avoids subtle differences between serialized and deserialized representations.

CSV Data in Test Automation

CSV is a natural format for test data that originates in spreadsheets: QA teams maintaining test cases in Excel, business analysts providing golden datasets, or product owners defining acceptance criteria in tabular form. Its simplicity is appealing. Its implementation is treacherous.

Why string.Split(',') Always Breaks

The naive parsing approach – split each line on commas – fails the moment real data appears. Product names contain commas. Addresses contain commas. Description fields wrap across lines. The CSV specification (RFC 4180) defines escaping rules for all of these: fields that contain commas are wrapped in quotes, quotes within quoted fields are doubled. A correct parser handles every case. string.Split(',') handles none of them:

// BROKEN – do not use. Fails on real-world data.
var records = File.ReadAllLines("products.csv")
    .Skip(1) // skip header row
    .Select(line => line.Split(','))
    .Select(parts => new TestProduct(
        int.Parse(parts[0]),
        parts[1],            // what if name is "Monitor, 27-inch HD"?
        decimal.Parse(parts[2]),
        int.Parse(parts[3])))
    .ToList();

// Input:  1,"Monitor, 27-inch HD",299.99,5
// Result: parts = ["1", "\"Monitor", " 27-inch HD\"", "299.99", "5"]
// int.Parse("\"Monitor") throws. Or worse: it silently uses the wrong value.

CsvHelper – The Standard Choice

CsvHelper is the community-standard CSV library for .NET. It handles the full CSV specification, maps rows to strongly-typed objects by header name, supports custom conversions, and reads large files efficiently without loading everything into memory. Installation is one NuGet package: CsvHelper.

// Define a record that maps to the CSV columns.
// CsvHelper maps by property name, case-insensitive by default.
public record TestProduct(
    int     ProductId,
    string  Name,
    decimal Price,
    int     CategoryId,
    int     StockQuantity);

// Load CSV test data into a list of strongly-typed records
public static IReadOnlyList<TestProduct> LoadProductTestData(string csvPath)
{
    using var reader = new StreamReader(csvPath);
    using var csv    = new CsvReader(reader, CultureInfo.InvariantCulture);

    // GetRecords lazily enumerates rows – ToList() materialises them all.
    // For large files, enumerate directly rather than calling ToList().
    return csv.GetRecords<TestProduct>().ToList();
}

// CSV file contents:
// ProductId,Name,Price,CategoryId,StockQuantity
// 1,"Monitor, 27-inch HD",299.99,5,42
// 2,Keyboard,79.99,5,150
// 3,"USB-C Hub (7-port)",49.99,5,89

Custom Column Mapping

When CSV column names don't match C# property names, CsvHelper's ClassMap provides the mapping without modifying the record definition. This is important when the CSV comes from an external system that uses different naming conventions:

// ClassMap: explicit column-to-property mapping for non-matching names
public sealed class TestProductMap : ClassMap<TestProduct>
{
    public TestProductMap()
    {
        Map(p => p.ProductId).Name("product_id");      // snake_case column headers
        Map(p => p.Name).Name("product_name");
        Map(p => p.Price).Name("unit_price");
        Map(p => p.CategoryId).Name("category_id");
        Map(p => p.StockQuantity).Name("stock");
    }
}

// Register the map when creating the CsvReader
public static IReadOnlyList<TestProduct> LoadWithMapping(string csvPath)
{
    using var reader = new StreamReader(csvPath);
    using var csv    = new CsvReader(reader, CultureInfo.InvariantCulture);

    csv.Context.RegisterClassMap<TestProductMap>();
    return csv.GetRecords<TestProduct>().ToList();
}

The investment in setting up CsvHelper is small – a NuGet package and a record class. The alternative is a parser that breaks silently on real data and produces intermittent, hard-to-diagnose test failures. In a professional context, the choice is clear.

Managing Test Artifacts

Test artifacts – screenshots on failure, HTTP response bodies, performance traces, test run summaries – are the evidence that makes failures diagnosable when they occur in environments you can't easily reproduce. Artifact management that works correctly under pressure (CI failures at peak usage, parallel test runs, limited disk space) requires decisions about directory structure, file naming, temporary file cleanup, and retention policy.

Directory Structure and File Naming

A well-designed artifact directory survives parallel test runs without conflicts and produces files that are navigable without a tool. Organising by date and test class produces a directory tree that's useful on its own:

// Structured artifact directory with safe, unique file names
public static class TestArtifacts
{
    private static readonly string RootDirectory =
        Path.Combine(AppContext.BaseDirectory, "TestArtifacts");

    // Build a path for a test artifact: Artifacts/Screenshots/TestClassName/testname_timestamp.png
    public static string BuildPath(string category, string testClass, string testName, string extension)
    {
        var dir      = Path.Combine(RootDirectory, category, SanitizeName(testClass));
        Directory.CreateDirectory(dir);

        // Include milliseconds to prevent collisions in parallel runs
        var fileName = $"{SanitizeName(testName)}_{DateTime.UtcNow:yyyyMMdd_HHmmss_fff}{extension}";
        return Path.Combine(dir, fileName);
    }

    // Screenshot helper: captures and writes in one call, returns the file path for logging
    public static async Task<string> SaveScreenshotAsync(
        byte[] screenshotBytes,
        string testClass,
        string testName)
    {
        var path = BuildPath("Screenshots", testClass, testName, ".png");
        await File.WriteAllBytesAsync(path, screenshotBytes);
        return path;
    }

    // Remove characters that are invalid in file system names (works cross-platform)
    private static string SanitizeName(string name)
    {
        var invalid = Path.GetInvalidFileNameChars();
        return string.Concat(name.Select(c => invalid.Contains(c) ? '_' : c));
    }
}

Temporary Files with Guaranteed Cleanup

Temporary files are appropriate when test code needs to produce a file to pass to a component under test, and that component can't accept a stream directly. The key requirement is that the temporary file is always deleted – even when the test throws. Combining Path.GetTempFileName with a finally block from the previous lesson achieves this:

// Path.GetTempFileName() creates a unique zero-byte file in the OS temp directory
// and returns its fully-qualified path. It doesn't generate a name only – the file exists.
public static async Task<T> WithTempFileAsync<T>(
    string content,
    Func<string, Task<T>> action)
{
    var path = Path.GetTempFileName();
    try
    {
        // Write the content the test component needs
        await File.WriteAllTextAsync(path, content);
        return await action(path);
    }
    finally
    {
        // Best-effort deletion: log failures but don't let cleanup hide the test result
        try   { File.Delete(path); }
        catch { /* best effort – OS will clean temp directory eventually */ }
    }
}

// Usage: test a component that reads configuration from a file path
[Test]
public async Task ConfigurationLoader_ShouldParseAllSettings()
{
    const string json = """
        {
          "baseUrl": "https://staging.example.com",
          "timeoutSeconds": 30
        }
        """;

    var config = await WithTempFileAsync(
        content: json,
        action:  path => _configLoader.LoadFromFileAsync(path));

    Assert.That(config.BaseUrl, Is.EqualTo("https://staging.example.com"));
    Assert.That(config.TimeoutSeconds, Is.EqualTo(30));
}

Test artifact management is one of those areas where a small amount of upfront design – a consistent directory structure, a safe naming helper, guaranteed temp file cleanup – eliminates an entire category of operational problems. The team that discovers a CI failure and immediately finds a screenshot, response body, and log entry for it is the team that actually fixes the bug quickly.

Async File I/O in Practice

File operations are I/O-bound work. When a test reads a fixture file or writes a screenshot, the CPU is idle while the operating system fetches bytes from disk. Synchronous file methods block the calling thread for that entire duration. In a thread-pool context – which includes NUnit and xUnit test runners – blocking threads unnecessarily reduces the parallelism available to the suite and can, under heavy load, exhaust the thread pool entirely. Async file methods release the thread back to the pool during the I/O wait and resume on any available thread when the I/O completes.

Async Equivalents for Common Operations

The naming convention is consistent: every synchronous file method has an async counterpart suffixed with Async. Prefer these in any method that's already marked async:

// Reading – use async variants in async contexts
string json        = await File.ReadAllTextAsync(path);
byte[] imageBytes  = await File.ReadAllBytesAsync(screenshotPath);
string[] lines     = await File.ReadAllLinesAsync(logPath);

// Writing – async for non-blocking I/O
await File.WriteAllTextAsync(reportPath, reportContent);
await File.WriteAllBytesAsync(screenshotPath, screenshotBytes);
await File.AppendAllTextAsync(logPath, $"[{DateTime.UtcNow:O}] {message}{Environment.NewLine}");

// StreamReader and StreamWriter also have async read/write methods
using var reader = new StreamReader(path);
string line = await reader.ReadLineAsync();
string all  = await reader.ReadToEndAsync();

Parallel Fixture Loading with Task.WhenAll

When test setup requires loading multiple independent fixture files, the same Task.WhenAll pattern from the async lesson applies directly. All file reads start simultaneously; the total wait time is the duration of the slowest individual read, not the sum of all reads:

// Load three fixture files in parallel – total time ≈ max(each read), not sum
public static async Task<TestSuiteData> LoadAllFixturesAsync(string dataDirectory)
{
    // Start all reads simultaneously
    var usersTask    = File.ReadAllTextAsync(Path.Combine(dataDirectory, "users.json"));
    var productsTask = File.ReadAllTextAsync(Path.Combine(dataDirectory, "products.json"));
    var ordersTask   = File.ReadAllTextAsync(Path.Combine(dataDirectory, "orders.json"));

    // Wait for all reads to complete
    await Task.WhenAll(usersTask, productsTask, ordersTask);

    // Deserialize from strings already in memory – CPU-bound, no more I/O waiting
    return new TestSuiteData(
        Users:    JsonSerializer.Deserialize<List<User>>(await usersTask,    JsonOptions.TestData)!,
        Products: JsonSerializer.Deserialize<List<Product>>(await productsTask, JsonOptions.TestData)!,
        Orders:   JsonSerializer.Deserialize<List<Order>>(await ordersTask,   JsonOptions.TestData)!);
}

// Alternatively: stream-based parallel deserialization avoids the intermediate strings
public static async Task<TestSuiteData> LoadAllFixturesStreamAsync(string dataDirectory)
{
    var usersTask = LoadJsonAsync<List<User>>(Path.Combine(dataDirectory, "users.json"));
    var productsTask = LoadJsonAsync<List<Product>>(Path.Combine(dataDirectory, "products.json"));
    var ordersTask = LoadJsonAsync<List<Order>>(Path.Combine(dataDirectory, "orders.json"));

    await Task.WhenAll(usersTask, productsTask, ordersTask);
    return new TestSuiteData(await usersTask, await productsTask, await ordersTask);
}

// The LoadJsonAsync helper from the serialization chapter wraps stream deserialization
private static async Task<T> LoadJsonAsync<T>(string path)
{
    await using var stream = File.OpenRead(path);
    return await JsonSerializer.DeserializeAsync<T>(stream, JsonOptions.TestData)
        ?? throw new InvalidOperationException($"'{path}' deserialised to null.");
}

Async file I/O isn't a performance micro-optimisation – it's the correct model for I/O-bound work in an async codebase. Every fixture load, every screenshot write, every log append is an opportunity to keep threads available for the rest of the test suite. Combined with Task.WhenAll for parallel fixture loading, async file operations are a genuine contributor to suite throughput, not just a stylistic choice.