Streaming Serialization

Learn how to efficiently serialize and deserialize large datasets using ToonNet's streaming API.

Overview

When working with millions of records, building one huge string or list can exhaust memory. ToonNet's streaming API writes one TOON document per item and reads them back one at a time:

Bounded Memory Usage: items are pulled from an IAsyncEnumerable<T> and only the current batch is buffered, so memory does not grow with the number of items
Batched I/O: serialized documents are buffered and written in batches (BatchSize, default 50)
Cancellation Support: the CancellationToken is checked for every item
Separator Modes: Choose between blank-line or explicit separator formats

When to Use Streaming

Use streaming serialization when:

Processing millions of records that don't fit in memory
Exporting database tables incrementally
Building ETL pipelines with large datasets
Processing multi-GB log files
Avoiding OutOfMemoryException in production

SerializeDocumentsAsync and DeserializeDocumentsAsync live in ToonNet.Core.Serialization. They write and read each item as a separate TOON document; before 3.2 they were named SerializeStreamAsync, DeserializeStreamAsync and SerializeCollectionToStreamAsync, names that still work but are obsolete.

Basic Streaming Serialization

Serialize Database Records

using ToonNet.Core.Serialization;

// Export millions of users without loading all into memory
await ToonSerializer.SerializeDocumentsAsync(
    items: dbContext.Users.AsAsyncEnumerable(),
    filePath: "users_export.toon",
    cancellationToken: cancellationToken
);

The file methods (SerializeDocumentsAsync to a path, SerializeCollectionToFileAsync) write the file in place as the items arrive: an item that fails, or a cancelled export, leaves the items written before it. Write to a new file and replace the old one yourself when the old content must survive a failure.

Deserialize Incrementally

// Process large files without memory pressure
await foreach (var user in ToonSerializer.DeserializeDocumentsAsync<User>("users_export.toon"))
{
    await ProcessUserAsync(user);  // Only one user in memory at a time
}

Reading the Items of One Large Array

When the document is one large root array (an export of rows, a log written as a table), read its items as they arrive with DeserializeAsyncEnumerable, without holding the document in memory:

await using var stream = File.OpenRead("users.toon");   // [200000]{id,name,...}: followed by the rows

await foreach (var user in ToonSerializer.DeserializeAsyncEnumerable<User>(stream))
{
    Process(user);
}

The root must be an array: a tabular array ([N]{fields}:), an expanded list ([N]: with - items) or an inline array. An array under a key (users[N]{fields}:) is an object with one property, not a root array, and throws a ToonSerializationException ("Expected an array but got Object"); ToonSerializer.Serialize(list) writes a root array, and DeserializeAsyncEnumerableAt (below) reads an array under a key. Rows and list items are read in batches of about 32 KB of text, so memory stays flat however long the array is; on 200,000 rows it is also a little faster than reading the whole document, which keeps the decoded text and the list (47 MB here) in memory.
Errors are those of reading the whole document, with the line in the whole document; items before an error have been returned by then. In strict mode the declared length is checked when the array ends.
DeserializeAsyncEnumerable(stream, MyType.ToonTypeInfo) reads without reflection (trimming, NativeAOT).

An Array Inside a Document

DeserializeAsyncEnumerableAt streams the items of an array at a path, such as the rows of a response envelope:

// status: ok
// data:
//   orders[200000]{id,customer,total}:
//     ...
await foreach (var order in ToonSerializer.DeserializeAsyncEnumerableAt<Order>(response, "$.data.orders"))
{
    Process(order);
}

For a path of member names ($.data.orders, $['the data'].rows), the lines before the array are only followed by their indentation and keys to find it, the items are read in batches as above, and the stream is read no further than the end of the array. Strict mode reports the lines before the array that break indentation or are not members (as DeserializeAt does); the rest of that content, and everything after the array, is not checked.
Other singular paths ($.pages[0].rows) read the whole document first.
A document without the path gives no items; a value at the path that is not an array throws ToonSerializationException. Errors in the array have the line and column of the whole document.
DeserializeAsyncEnumerableAt(stream, path, MyType.ToonTypeInfo) reads without reflection.

For a file of separate documents (one per item), use SerializeDocumentsAsync and DeserializeDocumentsAsync below.

Advanced Configuration

Separator Modes

Choose between two separator modes:

1. BlankLine (Default)

Documents separated by one blank line. This is what the overloads without write options use, and what SerializeCollectionToFileAsync writes.

await ToonSerializer.SerializeDocumentsAsync(
    items,
    "data.toon",
    options: null,
    writeOptions: ToonMultiDocumentWriteOptions.BlankLine
);

Output:

Name: Alice
Age: 25

Name: Bob
Age: 30

Name: Charlie
Age: 35

2. ExplicitSeparator

Documents separated by a --- line - explicit, YAML-like format. The separator text can be changed with DocumentSeparator; the reader recognizes a separator only when the whole line matches it exactly.

await ToonSerializer.SerializeDocumentsAsync(
    items,
    "data.toon",
    options: null,
    writeOptions: ToonMultiDocumentWriteOptions.ExplicitSeparator
);

Output:

Name: Alice
Age: 25
---
Name: Bob
Age: 30
---
Name: Charlie
Age: 35

Separators are always written with \n, regardless of the platform line ending, and the file has no trailing newline.

An item that encodes to an empty document (an object with no properties, or with only null properties and IgnoreNullValues) cannot be told apart from a blank-line separator, so blank-line output throws a ToonSerializationException for it. ExplicitSeparator output keeps such items: the empty document becomes an empty line between separators (and a final newline when it is the last item), and the reader returns it as an object. DocumentSeparator must be a single non-blank line. A document with a line that is the separator (a root string END with the separator END) would read back as two documents, so writing it throws.

Batch Size Configuration

Control how many serialized documents are buffered before each write:

var writeOptions = new ToonMultiDocumentWriteOptions
{
    Mode = ToonMultiDocumentSeparatorMode.BlankLine,
    BatchSize = 100  // Buffer 100 items before writing
};

await ToonSerializer.SerializeDocumentsAsync(
    items: GenerateLargeDatasetAsync(),
    filePath: "export.toon",
    options: serializerOptions,
    writeOptions: writeOptions,
    cancellationToken: cts.Token
);

Batch size guidelines:

The default is 50; values below 1 are treated as 1.
Memory use is roughly batch size × serialized item size, so use smaller batches for large items.
Larger batches mean fewer write calls; measure with your own data to pick a value.

Complete Examples

Database Export

Export an entire database table incrementally:

using Microsoft.EntityFrameworkCore;
using ToonNet.Core.Serialization;

public async Task ExportUsersAsync(CancellationToken cancellationToken)
{
    using var dbContext = new MyDbContext();
    
    await ToonSerializer.SerializeDocumentsAsync(
        items: dbContext.Users
            .AsNoTracking()  // No change tracking for read-only export
            .AsAsyncEnumerable(),
        filePath: "users_backup.toon",
        cancellationToken: cancellationToken
    );
    
    Console.WriteLine("Export completed successfully!");
}

ETL Pipeline

Transform and export data incrementally:

public async Task RunEtlPipelineAsync(CancellationToken cancellationToken)
{
    await ToonSerializer.SerializeDocumentsAsync(
        items: ExtractTransformAsync(),
        filePath: "transformed_data.toon",
        options: null,
        writeOptions: new ToonMultiDocumentWriteOptions { BatchSize = 100 },
        cancellationToken: cancellationToken
    );
}

private async IAsyncEnumerable<TransformedData> ExtractTransformAsync()
{
    await foreach (var raw in ReadRawDataAsync())
    {
        // Transform each record
        var transformed = new TransformedData
        {
            Id = raw.Id,
            ProcessedAt = DateTime.UtcNow,
            // ... transformation logic
        };
        
        yield return transformed;
    }
}

Log Processing

Process multi-GB log files without memory issues:

public async Task AnalyzeLogsAsync(string logFilePath)
{
    int errorCount = 0;
    int warningCount = 0;
    
    await foreach (var logEntry in ToonSerializer.DeserializeDocumentsAsync<LogEntry>(logFilePath))
    {
        switch (logEntry.Level)
        {
            case "ERROR":
                errorCount++;
                await AlertAsync(logEntry);
                break;
            case "WARNING":
                warningCount++;
                break;
        }
    }
    
    Console.WriteLine($"Errors: {errorCount}, Warnings: {warningCount}");
}

Progress Reporting

Report progress during long-running operations:

public async Task ExportWithProgressAsync(IProgress<int> progress, CancellationToken cancellationToken)
{
    await ToonSerializer.SerializeDocumentsAsync(
        items: TrackProgressAsync(progress),
        filePath: "export.toon",
        cancellationToken: cancellationToken
    );
}

private async IAsyncEnumerable<User> TrackProgressAsync(IProgress<int> progress)
{
    var count = 0;

    await foreach (var user in dbContext.Users.AsAsyncEnumerable())
    {
        yield return user;
        
        if (++count % 1000 == 0)
        {
            progress.Report(count);
        }
    }
}

Roundtrip Compatibility

Ensure write and read options match for successful roundtrips:

var writeOptions = ToonMultiDocumentWriteOptions.ExplicitSeparator;
var readOptions = ToonMultiDocumentReadOptions.ExplicitSeparator;

// Write
await ToonSerializer.SerializeDocumentsAsync(items, "data.toon", options: null, writeOptions: writeOptions);

// Read
await foreach (var item in ToonSerializer.DeserializeDocumentsAsync<Item>("data.toon", options: null, multiDocumentOptions: readOptions))
{
    ProcessItem(item);
}

Reading a --- separated file in blank-line mode fails with a ToonParseException, because the --- line ends up inside a document.

Performance Benchmarks

The repository contains a BenchmarkDotNet suite that compares SerializeCollectionToFileAsync with SerializeDocumentsAsync at several batch sizes (benchmark/ToonNet.Benchmarks/StreamingSerializationBenchmarks.cs). No results are published for the current version; see benchmark/ToonNet.Benchmarks/README.md for how to run it on your own hardware.

API Reference

SerializeDocumentsAsync Methods

// File-based, default options
ValueTask SerializeDocumentsAsync<T>(
    IAsyncEnumerable<T> items,
    string filePath,
    ToonSerializerOptions? options = null,
    CancellationToken cancellationToken = default);

// File-based, custom write options
ValueTask SerializeDocumentsAsync<T>(
    IAsyncEnumerable<T> items,
    string filePath,
    ToonSerializerOptions? options,
    ToonMultiDocumentWriteOptions writeOptions,
    CancellationToken cancellationToken = default);

// Stream-based, default options
ValueTask SerializeDocumentsAsync<T>(
    IAsyncEnumerable<T> items,
    Stream stream,
    ToonSerializerOptions? options = null,
    CancellationToken cancellationToken = default);

// Stream-based, custom write options
ValueTask SerializeDocumentsAsync<T>(
    IAsyncEnumerable<T> items,
    Stream stream,
    ToonSerializerOptions? options,
    ToonMultiDocumentWriteOptions writeOptions,
    CancellationToken cancellationToken = default);

// A collection in memory, separated by blank lines
ValueTask SerializeDocumentsAsync<T>(
    IEnumerable<T> values,
    Stream stream,
    ToonSerializerOptions? options = null,
    CancellationToken cancellationToken = default);

DeserializeDocumentsAsync Methods

// File-based, default separator
IAsyncEnumerable<T?> DeserializeDocumentsAsync<T>(
    string filePath,
    ToonSerializerOptions? options = null,
    CancellationToken cancellationToken = default);

// File-based, custom separator
IAsyncEnumerable<T?> DeserializeDocumentsAsync<T>(
    string filePath,
    ToonSerializerOptions? options,
    ToonMultiDocumentReadOptions multiDocumentOptions,
    CancellationToken cancellationToken = default);

// UTF-8 stream (left open), default or custom separator
IAsyncEnumerable<T?> DeserializeDocumentsAsync<T>(
    Stream utf8Stream,
    ToonSerializerOptions? options = null,
    CancellationToken cancellationToken = default);

IAsyncEnumerable<T?> DeserializeDocumentsAsync<T>(
    Stream utf8Stream,
    ToonSerializerOptions? options,
    ToonMultiDocumentReadOptions multiDocumentOptions,
    CancellationToken cancellationToken = default);

// StreamReader, default separator
IAsyncEnumerable<T?> DeserializeDocumentsAsync<T>(
    StreamReader reader,
    ToonSerializerOptions? options = null,
    CancellationToken cancellationToken = default);

// StreamReader, custom separator
IAsyncEnumerable<T?> DeserializeDocumentsAsync<T>(
    StreamReader reader,
    ToonSerializerOptions? options,
    ToonMultiDocumentReadOptions multiDocumentOptions,
    CancellationToken cancellationToken = default);

Best Practices

Always use cancellation tokens for long-running operations
Tune batch size based on item size and available memory
Use AsNoTracking() for Entity Framework read-only queries
Match separator modes for write/read roundtrips
Monitor memory usage in production with metrics
Consider explicit separators when other tools also write the file, so document boundaries are unambiguous
Buffer appropriately - larger batches mean fewer writes but more memory

Troubleshooting

Issue: OutOfMemoryException

Cause: Batch size too large for item size.

Solution: Reduce batch size:

writeOptions: new ToonMultiDocumentWriteOptions { BatchSize = 10 }

Issue: Slow throughput

Cause: Batch size too small, excessive I/O.

Solution: Increase batch size:

writeOptions: new ToonMultiDocumentWriteOptions { BatchSize = 200 }

Issue: Parse errors on deserialization

Cause: Separator mode mismatch.

Solution: Match write and read options:

// Write
await ToonSerializer.SerializeDocumentsAsync(items, "export.toon", null, ToonMultiDocumentWriteOptions.ExplicitSeparator);

// Read
await foreach (var item in ToonSerializer.DeserializeDocumentsAsync<Item>("export.toon", null, ToonMultiDocumentReadOptions.ExplicitSeparator))
{
    Process(item);
}

Overview​

When to Use Streaming​

Basic Streaming Serialization​

Serialize Database Records​

Deserialize Incrementally​

Reading the Items of One Large Array​

An Array Inside a Document​

Advanced Configuration​

Separator Modes​

1. BlankLine (Default)​

2. ExplicitSeparator​

Batch Size Configuration​

Complete Examples​

Database Export​

ETL Pipeline​

Log Processing​

Progress Reporting​

Roundtrip Compatibility​

Performance Benchmarks​

API Reference​

SerializeDocumentsAsync Methods​

DeserializeDocumentsAsync Methods​

Best Practices​

Troubleshooting​

Issue: OutOfMemoryException​

Issue: Slow throughput​

Issue: Parse errors on deserialization​

See Also​