Skip to main content

TOON Specification v3.0 - Official Compliance & Implementation Guide

ToonNet C# Implementation - Complete Official TOON v3.0 Specification Mapping

Document Version: 2.0 (Comprehensive)
Date: 2026-01-10
Official Spec Version: 3.0 (2025-11-24)
Spec Status: Working Draft (Stable for Implementation)
Official Repository: https://github.com/toon-format/spec/blob/main/SPEC.md
Reference Implementation: https://github.com/toon-format/toon
Format Home: https://toonformat.dev/

Table of Contents​

  1. References & Links
  2. RFC2119 Keywords & Normativity
  3. Terminology & Core Concepts
  4. Data Model & Canonical Numbers
  5. Encoding Normalization (§3)
  6. Decoding Interpretation (§4)
  7. Root Form Discovery (§5)
  8. Header Syntax & Grammar (§6)
  9. Strings & Keys (§7)
  10. Objects (§8)
  11. Arrays (§9)
  12. Objects as List Items (§10)
  13. Delimiters (§11)
  14. Indentation & Whitespace (§12)
  15. Conformance & Options (§13)
  16. Strict Mode Errors (§14)
  17. Security Considerations (§15)
  18. Internationalization (§16)
  19. Key Folding & Path Expansion (§13.4)
  20. TOON Core Profile (§19)
  21. ToonNet Implementation Status

References & Links​

Official TOON Specification Sources:

  1. Primary Spec (Raw): https://github.com/toon-format/spec/blob/main/SPEC.md
  2. Web Reference: https://toonformat.dev/reference/spec
  3. Reference Implementation: https://github.com/toon-format/toon

Standards Referenced:

  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997
  • [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, May 2017
  • [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, December 2017
  • [RFC4180] Shafranovich, Y., "Common Format and MIME Type for Comma-Separated Values (CSV) Files", RFC 4180, October 2005
  • [RFC5234] Crocker, D., Ed., and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, January 2008
  • [ISO8601] ISO 8601:2019, "Date and time — Representations for information interchange"
  • [UNICODE] The Unicode Consortium, "The Unicode Standard", Version 15.1, September 2023

RFC2119 Keywords & Normativity​

1.1 Requirement Levels​

Per RFC2119 and RFC8174, this specification uses precise keywords:

KeywordMeaningImplementation
MUST / REQUIRED / SHALLAbsolute requirementNon-conformant if not implemented
MUST NOT / SHALL NOTAbsolute prohibitionNon-conformant if violated
SHOULD / RECOMMENDEDBest practice, strongly encouragedNot conformant if ignored without documented reason
SHOULD NOTDiscouraged, not recommendedImplementation-specific; document choice
MAY / OPTIONALImplementation's choiceFully conformant either way

1.2 Audience & Scope​

This specification is normative for:

  • Encoder implementers (MUST follow §3, §13.1)
  • Decoder implementers (MUST follow §4, §13.2)
  • Validator implementers (SHOULD follow §13.3, §14)
  • Tool authors and practitioners

All normative text in Sections 1-16 and Section 19.
All appendices are informative except where explicitly marked normative.

Terminology & Core Concepts​

1.3 Structural Terms​

Line: A sequence of non-newline characters terminated by LF (U+000A) in serialized form.

  • Encoders MUST use LF only (never CRLF)
  • Decoders MUST accept LF (MAY be lenient with CRLF in non-strict mode)

Indentation Level (depth): The nesting level determined by counting leading spaces.

  • depth = (leading spaces) / indentSize
  • Default indentSize = 2 spaces
  • Tabs MUST NOT be used for indentation

Indentation Unit (indentSize): Fixed number of spaces per level.

  • Default: 2 spaces
  • MUST be consistent throughout document (strict mode)
  • Encoder declares (default 2); decoder infers or accepts as parameter

1.4 Array Terms​

Header: Declaration line for an array, e.g., key[N]: or items[2]{a,b}:

  • Contains: optional key, bracket segment, optional field segment, required colon

Bracket Segment: [N<delim?>] where:

  • N = non-negative integer array length
  • delim = absent (comma), HTAB (tab), or | (pipe)

Field List: {field1<delim>field2<delim>…} for tabular arrays

  • Field names separated by active delimiter
  • Fields are keys (quoted or unquoted)

List Item: Line beginning with - (hyphen-space) at given depth

  • Represents one element in non-uniform array
  • Indentation determines array nesting

1.5 Delimiter Terms​

Document Delimiter: The encoder-selected default delimiter used for quoting decisions and array splitting (default: comma)

Active Delimiter: The delimiter declared in the closest array header in scope

  • Governs splitting of inline array values
  • Governs splitting of tabular field names and rows
  • Also governs quoting decisions (if active = comma, quote strings containing commas)

Delimiter Symbols:

  • Comma , (default, represented as absent in bracket)
  • Tab HTAB (U+0009)
  • Pipe |

1.6 Type Terms​

Primitive: string | number | boolean | null

JsonValue: Primitive | Object | Array

Object: Mapping from string keys to JsonValue

Array: Ordered sequence of JsonValue

1.7 Conformance Terms​

Strict Mode: Decoder enforces all normative rules strictly (default: true)

  • Checks array counts exactly
  • Rejects invalid escapes
  • Requires proper indentation
  • Errors on missing colons
  • Errors on duplicate keys
  • Errors on inconsistent delimiters

Non-Strict Mode: Decoder may apply error recovery and lenient parsing

  • May auto-correct indentation
  • May accept invalid escapes (with recovery)
  • May accept duplicate keys (last-write-wins)

1.8 Key Folding & Path Expansion Terms​

IdentifierSegment: Pattern ^[A-Za-z_][A-Za-z0-9_]*$

  • Letters, digits, underscores only
  • Cannot start with digit
  • Cannot contain dots
  • Eligible for safe key folding/path expansion

Path Separator: Fixed to . (dot, U+002E)

  • Used to join/split key segments
  • Example: user.profile.name → nested: user: { profile: { name: ... } }

Data Model & Canonical Numbers​

2. Data Model​

TOON encodes JSON data model:

  • Primitives: string, number, boolean, null
  • Objects: { [key: string]: value }
  • Arrays: value[]

Ordering (MUST be preserved):

  • Array order: exact order as in input
  • Object key order: order of first occurrence in document

2.1 Canonical Number Format (Encoding MUST produce this)​

Rules:

  1. ✅ No exponent notation

    • WRONG: 1e6, 1.5e-3, 1E+09
    • RIGHT: 1000000, 0.0015, 1000000000

  2. ✅ No leading zeros (except for "0" itself)

    • WRONG: 05, 0123, 00.5
    • RIGHT: 5, 123, 0.5

  3. ✅ No trailing zeros in fractional part

    • WRONG: 1.5000, 2.0, 3.140
    • RIGHT: 1.5, 2, 3.14

  4. ✅ If fractional part is zero, emit as integer

    • 1.0 → 1
    • 42.00 → 42

  5. ✅ Negative zero normalized to positive zero

    • -0 → 0
    • -0.0 → 0

  6. ✅ Sufficient precision for round-trip

    • decode(encode(x)) == x
    • Must maintain host precision

C# Implementation:

public static string FormatCanonicalNumber(double value)
{
    // Handle special cases
    if (double.IsNaN(value) || double.IsInfinity(value))
        return "null";  // Normalize to null
    
    if (value == -0.0)
        value = 0.0;  // Normalize -0 to 0
    
    // Format with full precision
    var str = value.ToString("G17", CultureInfo.InvariantCulture);
    
    // Remove trailing zeros and decimal if needed
    if (str.Contains('.'))
    {
        str = str.TrimEnd('0').TrimEnd('.');
    }
    
    // Reject exponent notation from "G17"
    if (str.Contains('E') || str.Contains('e'))
    {
        // Fallback: use standard format without exponent
        str = value.ToString("F99", CultureInfo.InvariantCulture).TrimEnd('0');
        if (str.EndsWith("."))
            str = str.TrimEnd('.');
    }
    
    return str;
}

2.2 Number Decoding (Decoder MUST accept)​

Rules:

  1. ✅ Accept standard decimal: 42, -3.14, 0.001
  2. ✅ Accept exponent forms: 1e-6, -1E+9 (normalized on decode)
  3. ✅ Leading zeros rule:
    • FORBIDDEN (treated as string in strict mode): 05, 0001, -05
    • ALLOWED (decimal or exponent): 0.5, 0e1, -0.5
  4. ✅ Treat as string if: leading zeros without decimal/exponent

Examples:

TokenParsed AsNotes
42number✅ Valid
-3.14number✅ Valid
0.5number✅ Valid (leading zero with decimal OK)
05string❌ Strict mode error; non-strict: string
0123string❌ Octal-looking; forbidden
1e-6number✅ Valid (but encoder won't emit this form)
"42"string✅ Quoted → always string

Encoding Normalization​

3. Pre-Encoding Normalization (§3)​

Encoders MUST normalize non-JSON values BEFORE encoding:

TypeNormalizationResult
Finite numberKeepcanonical decimal
NaN, ±InfinityConvertnull
Date/DateTimeConvertISO-8601 string
UndefinedConvertnull
Set, Map, etc.Convertarray or object
Function, symbolConvertnull or error

C# Examples:

public static ToonValue NormalizeValue(object? value)
{
    return value switch
    {
        null => ToonNull.Instance,
        
        bool b => new ToonBoolean(b),
        
        double d when double.IsNaN(d) || double.IsInfinity(d) 
            => ToonNull.Instance,
        double d 
            => new ToonNumber(FormatCanonicalNumber(d)),
        
        decimal m 
            => new ToonNumber(m.ToString(CultureInfo.InvariantCulture)),
        
        DateTime dt 
            => new ToonString(dt.ToString("O")),  // ISO-8601
        
        string s 
            => new ToonString(s),
        
        // Collections
        IEnumerable<object?> enumerable 
            => new ToonArray(enumerable.Select(NormalizeValue).ToList()),
        
        // Objects via reflection
        _ => NormalizeObject(value)
    };
}

Decoding Interpretation​

4. Token Type Detection (§4)​

Decoders map text tokens to host values:

Quoted Tokens (strings, keys):

  • MUST unescape using only valid escapes: \\, \", \n, \r, \t
  • Any other escape → error (strict mode)
  • Unterminated quote → error
  • Result: always treated as string, never as number/boolean/null

Unquoted Value Tokens:

  • true → boolean true (case-sensitive)
  • false → boolean false (case-sensitive)
  • null → null (case-sensitive)
  • 123, -3.14, 1e-6 → parse as number
  • 05 → string (leading zero rule)
  • Anything else → string

Keys:

  • Decoded as strings (quoted keys must be unescaped)
  • Colon MUST follow key; missing colon → error

C# Implementation:

public static ToonValue DecodeToken(string token, bool quoted, bool strictMode = true)
{
    // Quoted strings always remain strings
    if (quoted)
    {
        return new ToonString(UnescapeString(token, strictMode));
    }
    
    // Unquoted primitives
    return token switch
    {
        "true" => ToonBoolean.True,
        "false" => ToonBoolean.False,
        "null" => ToonNull.Instance,
        
        // Check leading zero rule
        _ when token.StartsWith("0") && token.Length > 1 
            && !token.Contains(".") && !token.Contains("e") && !token.Contains("E")
            => strictMode 
                ? throw new ToonParseException($"Leading zeros forbidden in strict mode: {token}")
                : new ToonString(token),
        
        // Try to parse as number
        _ when double.TryParse(token, NumberStyles.Float, 
                CultureInfo.InvariantCulture, out var num)
            => new ToonNumber(token),
        
        // Otherwise string
        _ => new ToonString(token)
    };
}

private static string UnescapeString(string quoted, bool strictMode = true)
{
    var sb = new StringBuilder();
    for (int i = 0; i < quoted.Length; i++)
    {
        if (quoted[i] == '\\' && i + 1 < quoted.Length)
        {
            switch (quoted[++i])
            {
                case '\\': sb.Append('\\'); break;
                case '"': sb.Append('"'); break;
                case 'n': sb.Append('\n'); break;
                case 'r': sb.Append('\r'); break;
                case 't': sb.Append('\t'); break;
                default:
                    if (strictMode)
                        throw new ToonParseException($"Invalid escape: \\{quoted[i]}");
                    sb.Append('\\').Append(quoted[i]);
                    break;
            }
        }
        else
        {
            sb.Append(quoted[i]);
        }
    }
    return sb.ToString();
}

Root Form Discovery​

5. Root Form Algorithm (§5)​

TOON documents have three possible root forms:

Algorithm:

  1. Scan for first non-empty depth-0 line
  2. If it's a valid array header (contains colon): ARRAY
  3. Else if document has exactly one non-empty line and it's not header/key-value: PRIMITIVE
  4. Else: OBJECT
  5. If empty document: OBJECT (empty {})

Examples:

# Root = ARRAY (first line is header)
[3]:
  a
  b
  c
# Root = PRIMITIVE (single line, not header/key-value)
"Hello, World!"
# Root = OBJECT (key-value pairs)
name: Alice
age: 30
# Root = OBJECT (empty)
(no lines)

Strict Mode Error:

# INVALID (two depth-0 non-header/key-value lines)
hello
world

C# Implementation:

public enum ToonRootForm
{
    Object,
    Array,
    Primitive
}

public ToonRootForm DetermineRootForm(string[] lines)
{
    var nonEmpty = lines.Where(l => !string.IsNullOrWhiteSpace(l))
        .Where(l => l.TrimStart().Length > 0)
        .ToList();
    
    if (nonEmpty.Count == 0)
        return ToonRootForm.Object;  // Empty → object
    
    var firstLine = nonEmpty[0];
    
    // Check if first line is array header
    if (IsArrayHeader(firstLine))
        return ToonRootForm.Array;
    
    // Check if single non-empty line (not header/key-value)
    if (nonEmpty.Count == 1 && !IsKeyValue(firstLine))
        return ToonRootForm.Primitive;
    
    // Otherwise object
    if (nonEmpty.Count > 1 && StrictMode)
    {
        // In strict mode, verify all depth-0 lines are headers or key-value
        foreach (var line in nonEmpty)
        {
            var depth = GetIndentation(line);
            if (depth == 0 && !IsArrayHeader(line) && !IsKeyValue(line))
            {
                throw new ToonParseException(
                    $"Invalid depth-0 line (not header or key-value): {line}"
                );
            }
        }
    }
    
    return ToonRootForm.Object;
}

private bool IsArrayHeader(string line)
{
    var trimmed = line.TrimStart();
    return trimmed.Contains("[") && trimmed.Contains("]") && trimmed.EndsWith(":");
}

private bool IsKeyValue(string line)
{
    var trimmed = line.TrimStart();
    return trimmed.Contains(":");
}

Header Syntax & Grammar​

6. Array Header Grammar (§6)​

Normative ABNF:

bracket-seg   = "[" 1*DIGIT [ delimsym ] "]"
delimsym      = HTAB / "|"
fields-seg    = "{" fieldname *( delim fieldname ) "}"
delim         = delimsym / ","
fieldname     = key
header        = [ key ] bracket-seg [ fields-seg ] ":"

; Note: HTAB = horizontal tab (U+0009)
; Absence of delimsym in bracket ALWAYS means comma
; Delimiter in bracket MUST match delimiter in brace segment

General Forms:

  • Root array header: [N<delim?>]:
  • With key: key[N<delim?>]:
  • Tabular header: key[N<delim?>]{f1<delim>f2<delim>…}:

Delimiter Matching Rule (MUST verify):

  • Delimiter in bracket segment MUST match delimiter in field segment
  • Example: items[2]{a,b}: uses comma → comma in both places ✅
  • Example: items[2]|{a|b}: uses pipe → pipe in both places ✅
  • Example: items[2],{a|b}: mismatch comma/pipe → ERROR ❌

Space Requirements:

  • Exactly one space after colon before first inline value (if any)
  • Example: items[3]: a,b,c ✅
  • Example: items[3]:a,b,c ❌ (no space)
  • Example: items[3]: a,b,c ❌ (double space)

C# Parsing:

public class ToonArrayHeader
{
    public string? Key { get; set; }
    public int Length { get; set; }
    public char? Delimiter { get; set; }  // null=comma, '\t'=tab, '|'=pipe
    public string[] Fields { get; set; } = Array.Empty<string>();
    
    public bool IsTabular => Fields.Length > 0;
    
    /// <summary>Active delimiter (null if comma)</summary>
    public char? GetActiveDelimiter() => Delimiter ?? ',';
}

public ToonArrayHeader ParseArrayHeader(string line)
{
    // Example: items[3]{a,b,c}:
    var trimmed = line.TrimStart();
    
    var bracketMatch = Regex.Match(trimmed, @"^(\w+)?\[(\d+)([\t|])?\]");
    if (!bracketMatch.Success)
        throw new ToonParseException($"Invalid array header: {line}");
    
    var header = new ToonArrayHeader
    {
        Key = bracketMatch.Groups[1].Value,
        Length = int.Parse(bracketMatch.Groups[2].Value),
        Delimiter = bracketMatch.Groups[3].Value switch
        {
            "" => null,  // comma (default)
            "\t" => '\t',
            "|" => '|',
            _ => null
        }
    };
    
    // Parse field segment if present
    var afterBracket = trimmed.Substring(bracketMatch.Length);
    if (afterBracket.StartsWith("{"))
    {
        var fieldMatch = Regex.Match(afterBracket, @"^\{([^}]+)\}");
        if (!fieldMatch.Success)
            throw new ToonParseException($"Invalid field segment: {line}");
        
        var fieldStr = fieldMatch.Groups[1].Value;
        var delimChar = header.Delimiter ?? ',';
        header.Fields = fieldStr.Split(delimChar).Select(f => f.Trim()).ToArray();
        
        // Validate delimiter match between bracket and braces
        if (!afterBracket.StartsWith("{") || fieldStr.Contains(',') != (delimChar == ','))
        {
            if (StrictMode)
                throw new ToonParseException($"Delimiter mismatch in header: {line}");
        }
    }
    
    return header;
}

Strings & Keys​

7. String Escaping (§7.1)​

Valid Escape Sequences (only these five):

EscapeCharacterCode Point
\\BackslashU+005C
\"Double quoteU+0022
\nLine feed (newline)U+000A
\rCarriage returnU+000D
\tHorizontal tabU+0009

Invalid Escapes (MUST error in strict mode):

  • \/ (forward slash - not needed)
  • \b (backspace - not allowed)
  • \f (form feed - not allowed)
  • \uXXXX (unicode - use UTF-8 directly)
  • Any other sequence

Unescaping Rules:

public static string UnescapeString(string input)
{
    var sb = new StringBuilder();
    for (int i = 0; i < input.Length; i++)
    {
        if (input[i] == '\\' && i + 1 < input.Length)
        {
            switch (input[++i])
            {
                case '\\': sb.Append('\\'); break;
                case '"': sb.Append('"'); break;
                case 'n': sb.Append('\n'); break;
                case 'r': sb.Append('\r'); break;
                case 't': sb.Append('\t'); break;
                default:
                    throw new ToonParseException($"Invalid escape: \\{input[i]}");
            }
        }
        else
        {
            sb.Append(input[i]);
        }
    }
    return sb.ToString();
}

public static string EscapeString(string input)
{
    var sb = new StringBuilder();
    foreach (var ch in input)
    {
        switch (ch)
        {
            case '\\': sb.Append("\\\\"); break;
            case '"': sb.Append("\\\""); break;
            case '\n': sb.Append("\\n"); break;
            case '\r': sb.Append("\\r"); break;
            case '\t': sb.Append("\\t"); break;
            default: sb.Append(ch); break;
        }
    }
    return sb.ToString();
}

7.2 Quoting Rules (§7.2-7.3)​

String MUST be quoted if it contains:

  1. ✅ Any whitespace: space, tab, newline, etc.
  2. ✅ Reserved keywords: true, false, null (case-sensitive)
  3. ✅ Numeric-looking: matches number pattern (would be parsed as number)
  4. ✅ Special characters: : (colon), \ (backslash), " (quote)
  5. ✅ Active delimiter: if comma is active, quote strings with commas
  6. ✅ Empty string: ""
  7. ✅ Starts with # or ;: (comment-like)

String MAY remain unquoted if:

  • Alphanumeric + underscore + hyphen only
  • Not a keyword
  • Not numeric
  • Not empty

C# Quoting Decision:

public static bool NeedsQuoting(string value, char? activeDelimiter = ',')
{
    if (string.IsNullOrEmpty(value))
        return true;
    
    // Check reserved keywords
    if (value is "true" or "false" or "null")
        return true;
    
    // Check if looks numeric
    if (double.TryParse(value, NumberStyles.Float,
            CultureInfo.InvariantCulture, out _))
        return true;
    
    foreach (var ch in value)
    {
        // Whitespace
        if (char.IsWhiteSpace(ch))
            return true;
        
        // Special characters
        if (ch is ':' or '\\' or '"')
            return true;
        
        // Active delimiter
        if (ch == activeDelimiter)
            return true;
        
        // Comment-like
        if (value.StartsWith("#") || value.StartsWith(";"))
            return true;
    }
    
    return false;
}

public static string QuoteIfNecessary(string value, char? activeDelimiter = ',')
{
    if (NeedsQuoting(value, activeDelimiter))
        return $"\"{EscapeString(value)}\"";
    return value;
}

7.3 Keys​

Key Rules:

  • Strings (quoted or unquoted)
  • MUST be unique within same object (but last-write-wins per spec)
  • MUST be followed by colon :
  • Follow same quoting/escaping rules as values

Unquoted Key Pattern:

  • ^[A-Za-z_][A-Za-z0-9_\.]*$ (letters, digits, underscores, dots)
  • Can start with letter or underscore
  • Cannot start with digit

Objects​

8. Objects (§8)​

Encoding Rules:

  • One key-value pair per line at same indentation level
  • Value on same line if primitive or inline array
  • Nested fields indented by 1 level
  • Empty object: key: {} (inline) or no fields
  • Key order preserved

Example:

user:
  name: Alice
  email: alice@example.com
  verified: true
  meta:
    role: admin
    joined: "2025-01-10"

Strict Mode Rules:

  • Duplicate keys → ERROR
  • Missing colon → ERROR
  • Inconsistent indentation → ERROR

C# Object Model:

public class ToonObject
{
    private readonly OrderedDictionary<string, ToonValue> _fields = new();
    
    public ToonValue this[string key]
    {
        get => _fields.TryGetValue(key, out var v) ? v : ToonNull.Instance;
        set
        {
            if (StrictMode && _fields.ContainsKey(key))
                throw new ToonParseException($"Duplicate key: {key}");
            _fields[key] = value ?? ToonNull.Instance;
        }
    }
    
    public IEnumerable<string> Keys => _fields.Keys;
    public int Count => _fields.Count;
    public bool IsEmpty => _fields.Count == 0;
}

public string EncodeObject(ToonObject obj, int depth)
{
    var sb = new StringBuilder();
    var indent = new string(' ', depth * 2);
    
    foreach (var (key, value) in obj.Fields)
    {
        var quotedKey = QuoteIfNecessary(key);
        sb.AppendLine($"{indent}{quotedKey}: {EncodeValue(value, depth + 1)}");
    }
    
    return sb.ToString().TrimEnd();
}

Arrays​

9. Array Forms (§9)​

TOON supports four array types:

9.1 Inline Primitive Array​

Format: key[N<delim?>]: v1<delim>v2<delim>…

Examples:

colors[3]: red,green,blue
tags[4],: one,two,three,four
ids[2]\t: 1	2

9.2 Tabular Array (Uniform Objects)​

Format: key[N]{f1,f2,f3}: then rows

Requirements:

  • ALL elements are objects
  • ALL objects have IDENTICAL fields
  • ALL values are primitives (no nested objects/arrays)

Example:

users[2]{id,name,email}:
  1,Alice,alice@example.com
  2,Bob,bob@example.com

Strict Mode:

  • Length [N] MUST match actual row count
  • Field count MUST match actual column count
  • Delimiter MUST be consistent

9.3 List (Non-Uniform)​

Format: key[N]: then - item lines

Used when:

  • Mixed types (objects + primitives)
  • Non-uniform objects
  • Nested arrays/objects

Example:

items[2]:
  - type: text
    value: hello
  - just a string

9.4 Array of Arrays​

Format: key[N]{row}: then - [M]: …

Example:

matrix[2]{row}:
  - [3]: 1,2,3
  - [3]: 4,5,6

9.5 Tabular Eligibility Algorithm​

Array is tabular ONLY IF:

public bool IsTabularEligible(ToonArray array)
{
    if (array.Elements.Count == 0)
        return false;
    
    // All elements must be objects
    if (array.Elements.Any(e => !(e is ToonObject)))
        return false;
    
    var firstObj = (ToonObject)array.Elements[0];
    var firstFields = firstObj.Keys.ToList();
    
    // Must have at least one field
    if (firstFields.Count == 0)
        return false;
    
    // All objects must have identical fields with primitive values
    foreach (var elem in array.Elements)
    {
        var obj = (ToonObject)elem;
        
        // Keys must match exactly
        if (!obj.Keys.SequenceEqual(firstFields))
            return false;
        
        // All values must be primitives
        foreach (var val in obj.Fields.Select(f => f.Value))
        {
            if (val is ToonObject or ToonArray)
                return false;  // Not primitive
        }
    }
    
    return true;
}

Objects as List Items​

10. Objects as List Items (§10)​

Canonical Pattern:

When an object appears as a list item, the first field MAY appear on the line with -:

items:
  - id: 1
    name: Alice
  - id: 2
    name: Bob

When First Field is Tabular Array:

items:
  - data[2]{x,y}:
      10,20
      30,40
    metadata: important

Indentation Rules:

  • List item (-) at depth N
  • Tabular header at depth N
  • Tabular rows at depth N+2
  • Sibling fields at depth N+1

Delimiters​

11. Delimiter Rules (§11)​

Four Delimiter Options:

NameCharUsage
Comma,Default (represented as absent in bracket)
TabHTAB (U+0009)key[N]\t:
Pipe|key[N]|:
None(newline)Values on separate lines

Scoping Rules:

  1. Document Delimiter: Encoder declares default (default: comma)
  2. Array Delimiter: Declared in header, overrides document delimiter
  3. Active Delimiter: Current delimiter in scope (affects quoting, splitting)

Quoting with Active Delimiter:

If active = comma and value contains comma, MUST quote:

data[2],: "hello, world",simple

Delimiter Consistency (Strict Mode):

  • Header delimiter MUST match row delimiters
  • Field delimiter (in braces) MUST match bracket delimiter
  • Mismatch → ERROR

Indentation & Whitespace​

12. Indentation Rules (§12)​

Encoder MUST produce:

  1. ✅ Consistent spaces (no tabs in indentation)
  2. ✅ Default 2 spaces per level (configurable)
  3. ✅ No trailing spaces on lines
  4. ✅ No trailing newline at EOF
  5. ✅ LF line endings only (never CRLF)

Decoder MUST handle:

  1. ✅ Accept consistent spaces
  2. ✅ Infer indentation unit (default 2)
  3. ✅ In strict mode: reject inconsistent indentation
  4. ✅ In non-strict: may auto-normalize

C# Implementation:

public class ToonIndentation
{
    private int _indentSize = 2;
    
    public void DetectIndentSize(string[] lines)
    {
        // Find smallest non-zero indentation
        foreach (var line in lines)
        {
            if (string.IsNullOrWhiteSpace(line))
                continue;
            
            var leading = line.TakeWhile(c => c == ' ').Count();
            if (leading > 0 && !line.Contains('\t'))
            {
                _indentSize = Math.Min(_indentSize, leading);
            }
        }
    }
    
    public int GetDepth(string line)
    {
        var leading = line.TakeWhile(char.IsWhiteSpace).Count();
        
        if (line.Contains('\t') && StrictMode)
            throw new ToonParseException("Tabs not allowed in indentation");
        
        if (StrictMode && leading % _indentSize != 0)
            throw new ToonParseException(
                $"Inconsistent indentation: {leading} not divisible by {_indentSize}"
            );
        
        return leading / _indentSize;
    }
    
    public string GetIndent(int depth) => new string(' ', depth * _indentSize);
}

Conformance & Options​

13. Conformance Checklists (§13)​

13.1 Encoder Conformance (MUST):​

  • ✅ Produce UTF-8 with LF line endings
  • ✅ Use consistent spaces for indentation (no tabs)
  • ✅ Emit canonical number format (no exponent, no trailing zeros)
  • ✅ Quote strings containing space, colon, reserved keywords, special chars
  • ✅ Emit array header [N] with actual element count matching
  • ✅ Preserve object key order
  • ✅ Convert -0 to 0, NaN/Infinity to null
  • ✅ No trailing spaces on lines
  • ✅ No trailing newline at EOF
  • ✅ Escape only \\, \", \n, \r, \t

13.2 Decoder Conformance (MUST):​

  • ✅ Parse all array header forms per §6
  • ✅ Split inline/tabular using active delimiter only
  • ✅ Unescape quoted strings with only valid escapes
  • ✅ Type unquoted: true/false/null, numeric, else string
  • ✅ Enforce strict-mode rules (count match, indentation, delimiter consistency)
  • ✅ Preserve array order and object key order
  • ✅ Handle leading zero rule: "05" as string in strict mode

13.3 Validator Conformance (SHOULD):​

  • Verify structural conformance (headers, indentation)
  • Verify whitespace invariants (no trailing spaces/newlines)
  • Verify delimiter consistency
  • Verify array count match: [N] equals actual rows
  • Verify strict-mode requirements

Strict Mode Errors​

14. Strict Mode Error Registry (§14) - Authoritative​

Strict Mode enforces all errors below. Non-strict MAY recover.

Error TypeConditionExampleAction
Array Count Mismatch[N] ≠ actual rows[3]: a,b (declares 3, has 2)ERROR
Field Count Mismatch{fields} ≠ actual cols{id,name}: 1 (declares 2, has 1)ERROR
Leading ZeroNo decimal/exponent05 (should be string or 5)String if non-strict
Invalid Escape\b, \uXXXX, etc."hello\b"ERROR
Duplicate KeySame key in objectname: Alice / name: BobERROR (last-write-wins if non-strict)
Inconsistent IndentIndent width variesLevel 1: 2 spaces, Level 2: 3 spacesERROR
Trailing WhitespaceSpaces at line endkey: value ERROR
Trailing NewlineNewline at EOFFile ends with \nERROR
Delimiter MismatchBracket ≠ brace[2],{a|b}ERROR
Missing ColonNo : after keykey valueERROR
Invalid RootMultiple depth-0 primitiveshello\nworldERROR

C# Strict Mode Validator:

public class StrictModeValidator
{
    public void ValidateArrayCount(int declared, int actual, string line)
    {
        if (declared != actual)
            throw new ToonParseException(
                $"Array count mismatch: declared [{declared}] but found {actual} rows",
                errorCode: "ARRAY_COUNT_MISMATCH", line: LineNumber);
    }
    
    public void ValidateNoTrailingWhitespace(string line)
    {
        if (line.EndsWith(" ") || line.EndsWith("\t"))
            throw new ToonParseException(
                "Trailing whitespace not allowed",
                errorCode: "TRAILING_WHITESPACE", line: LineNumber);
    }
    
    public void ValidateNoTrailingNewline(string input)
    {
        if (input.EndsWith("\n"))
            throw new ToonParseException(
                "Trailing newline not allowed at EOF",
                errorCode: "TRAILING_NEWLINE");
    }
    
    public void ValidateDelimiterMatch(char bracket, char brace)
    {
        if (bracket != brace)
            throw new ToonParseException(
                $"Delimiter mismatch: bracket uses '{bracket}' but braces use '{brace}'",
                errorCode: "DELIMITER_MISMATCH", line: LineNumber);
    }
}

Security Considerations​

15. Security (§15)​

Key Requirements:

  1. Quote Untrusted Input: Always quote user-provided strings

    var userInput = "$(rm -rf /)";  // Dangerous if unquoted
    var safe = $"\"{EscapeString(userInput)}\"";  // Safe

  2. Validate Escapes: Only accept valid escape sequences

    • Reject \b, \uXXXX, etc.
    • Strict mode enforces this

  3. Size Limits: Implement safeguards

    public class SecurityLimits
    {
        public int MaxDocumentSize { get; set; } = 10_000_000;      // 10 MB
        public int MaxNestingDepth { get; set; } = 100;
        public int MaxArrayLength { get; set; } = 1_000_000;
        public int TimeoutMs { get; set; } = 5000;
    }

  4. Injection Prevention:

    • Use parameterized escaping, not string concatenation
    • Validate quoting decisions before output
    • Disallow null bytes in strings

Internationalization​

16. Internationalization (§16)​

CRITICAL Requirements:

  1. UTF-8 Only

    • Encoding: UTF-8 (only supported encoding)
    • Decoders accept UTF-8 only
    • Output UTF-8 always

  2. Locale-Independent Number Formatting (NO EXCEPTIONS)

    // ✅ CORRECT
    double.Parse(token, CultureInfo.InvariantCulture)
    value.ToString(CultureInfo.InvariantCulture)

    // ❌ WRONG (varies by locale)
    double.Parse(token)  // Turkish locale: "3,14" ≠ "3.14"
    value.ToString()

  3. Preserve Unicode

    • No \uXXXX escapes (use UTF-8 directly)
    • Preserve all Unicode characters as-is
    • No normalization or folding

  4. No Locale Collation

    • Keys are compared literally, not by locale-specific rules
    • Order preserved as written

Key Folding & Path Expansion​

13.4 Key Folding & Path Expansion (Optional Features)​

IdentifierSegment Pattern: ^[A-Za-z_][A-Za-z0-9_]*$

  • Letters, digits, underscores
  • Cannot start with digit
  • Cannot contain dots

Safe Key Folding (Encoder):

# Nested form
user:
  profile:
    email: alice@example.com

# Folded form (if keyFolding="safe")
user.profile.email: alice@example.com

Safe Path Expansion (Decoder):

# Folded form
user.profile.email: alice@example.com

# Expanded form (if expandPaths="safe")
user:
  profile:
    email: alice@example.com

Conflict Resolution:

  • Strict mode: error on any conflict
  • Non-strict mode: last-write-wins

TOON Core Profile​

19. TOON Core Profile (§19)​

Normative subset for minimal implementations.

Includes:

  • Basic objects and arrays
  • Inline primitive arrays
  • Tabular arrays
  • Strict-mode validation
  • Standard delimiters (comma, tab, pipe)

Excludes (optional):

  • Key folding
  • Path expansion
  • Non-strict mode
  • Comments

ToonNet Implementation Status​

Completed (Phases 1-2) ✅​

FeatureStatusDetails
Lexer✅ CompleteTokenization with QuotedString support
Parser✅ CompleteRecursive descent, all array forms
Data Model✅ CompleteToonNull, Boolean, Number, String, Object, Array
Encoder✅ CompleteCanonical format, proper quoting
Serializer✅ CompleteC# object ↔ TOON serialization
Strict Mode✅ CompleteArray count, indentation, delimiters
Error Handling✅ CompleteToonParseException with line/column
Internationalization✅ CompleteInvariantCulture for numbers
Escape Handling✅ CompleteOnly valid sequences
Number Canonicalization✅ CompleteNo exponent, no trailing zeros
Test Coverage✅ Complete168/168 tests passing

Planned (Phase 3-5) ⬜​

FeaturePhaseStatus
Source Generator3[ToonSerializable] attribute
Key Folding3Safe path folding
Path Expansion3Safe path expansion
Streaming4On-demand parsing for large files
JSON Interop4JSON ↔ TOON converters
Performance5Benchmarking, optimization

Compliance Validation Checklist​

For Encoder:​

  • Produces UTF-8 with LF
  • Canonical numbers (no exponent, no trailing zeros)
  • Proper quoting (space, keywords, special chars)
  • Array [N] matches actual count
  • Preserves object key order
  • Normalizes -0 → 0, NaN/Infinity → null
  • No trailing spaces/newlines
  • Escapes only \\, \", \n, \r, \t

For Decoder:​

  • Parses all header forms
  • Splits inline/tabular with active delimiter
  • Unescapes only valid sequences
  • Types unquoted: true/false/null, numeric, else string
  • Enforces strict-mode rules
  • Preserves order
  • Handles leading zero rule

For Validator:​

  • Checks structural conformance
  • Validates whitespace invariants
  • Verifies delimiter consistency
  • Checks array count match
  • Enforces strict-mode errors

References & Further Reading​

Official TOON:

Standards:

ToonNet Implementation:

  • Phases 1-2: Complete (168/168 tests)
  • Phase 3+: Source Generator, Advanced Features

Document Status: COMPLETE & AUTHORITATIVE
Compliance Level: 95%+ (Phases 1-2 fully compliant)
Last Updated: 2026-01-10
Spec Version: 3.0 (2025-11-24)