View source for Processing TEI critical apparatus data with Lua

<!--
Page title:
Processing TEI critical apparatus data with Lua

Purpose:
Tutorial — Guide 5 of 6 in the collection devoted to building critical
editions from TEI XML with ConTeXt.

Guides 1–4 constructed the TEI documentary and editorial model.
Guide 5 introduces an intermediate processing layer that turns selected
encoded structures into normalized, resolved, checked, and diagnosable
editorial records.

The reason for using Lua is both architectural and editorial:
ConTeXt LMTX runs on LuaMetaTeX, so Lua is already part of the engine
environment; and the accumulated TEI relations have become sufficiently
global, relational, and rule-driven to benefit from a reusable processing
layer.

Lua remains optional for simpler XML workflows. Direct ConTeXt XML setups
remain appropriate when interpretation can stay local and straightforward.

Final typographical composition is reserved for Guide 6.

Visual convention used throughout the TEI guides:

blue   = guiding principle / conceptual point
yellow = distinction / warning
green  = stage reached / where we are
grey   = technical note / further reference

Diátaxis role:
Tutorial. The reader first understands why an intermediate Lua layer is
useful in LMTX, then constructs a witness registry, apparatus records,
reference resolution, project-level editorial checks, diagnostics, and a
stable Lua–ConTeXt contract.

Repetition is retained when it reactivates an earlier TEI distinction,
shows a change in the status of the data, or prepares the next processing
stage.
-->

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.75em 0.95em; margin:1em 0;">
<span style="font-size:1.15em; margin-right:0.35em;">🚧</span>
'''Under construction''' — This page and its subpages are currently being revised. Contributions are welcome: feel free to edit and '''improve''' them.
</div>

<div style="margin:0 0 1.5em 0; padding:0.7em 1em; border:1px solid #a7c7e7; background:#eef6fc;">
'''Guide 5 of 6 — Processing TEI critical apparatus data with Lua'''

'''Previous: Guide 4 —''' [[Encoding complex textual variation in TEI]]
&nbsp;·&nbsp;
[[Building critical editions from TEI XML|Collection overview]]
&nbsp;·&nbsp;
[[Glossary of terms used in the TEI XML critical edition guides|Glossary]]
&nbsp;·&nbsp;
'''Next: Guide 6 —''' [[Typesetting a TEI critical apparatus with ConTeXt]]
</div>

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.9em 1.1em; margin:1.2em 0;">
'''What you will build in this guide'''

Guides 1–4 produced a TEI source containing witness declarations,
apparatus entries, references, responsibility, uncertainty, and other
editorial distinctions.

Guide 5 does not replace that source. It constructs a controlled
intermediate representation:

<pre>
TEI XML
   │
   ▼
Lua processing
├── extract
├── normalize
├── resolve
├── check project rules
└── diagnose
   │
   ▼
editorial records
├── source connection
├── normalized identifiers
├── resolved objects
├── explicit categories
├── validity status
└── diagnostics
   │
   ▼
Guide 6: ConTeXt composition
</pre>

You will build this layer progressively:

# a witness registry;
# normalized apparatus records;
# reference resolution;
# project-level editorial checks;
# errors and warnings;
# accepted and rejected record collections;
# a stable Lua–ConTeXt contract.

The functional examples use a representative subset of Guide 4 so that
the processing pipeline remains readable.
</div>

__TOC__



= Processing TEI critical apparatus data with Lua =

Guides 1–4 progressively constructed the TEI source.

By the end of Guide 4, the document could contain:

<pre>
TEI
├── declarations
│   ├── witnesses
│   └── editorial responsibility
│
└── text
    └── apparatus
        ├── lem / rdg
        ├── @wit
        ├── @resp
        ├── omissions
        ├── documentary alterations
        ├── uncertainty
        └── grouped or complex readings
</pre>

At this point the problem changes.

We no longer ask mainly:

<pre>
How do we encode another textual phenomenon?
</pre>

We now ask:

<pre>
How do we process all these encoded relations consistently and safely?
</pre>

Guide 5 therefore adds a new layer:

<pre>
GUIDES 1–4

TEI documentary source
        │
        ▼
encoded witnesses and apparatus
        │
        ▼

GUIDE 5

Lua processing                         ← NEW
        │
        ▼
checked editorial records
        │
        ▼

GUIDE 6

ConTeXt typesetting
</pre>

The central architecture is:

<pre>
                 DOCUMENTARY SOURCE

                       TEI XML
                          │
                          ▼
              +----------------------+
              |         Lua          |
              |                      |
              | extract              |
              | normalize            |
              | resolve              |
              | validate             |
              | diagnose             |
              +----------------------+
                          │
                          ▼
                 EDITORIAL RECORDS
                          │
                          ▼
                       ConTeXt
                          │
                          ▼
               TYPOGRAPHICAL OUTPUT
</pre>

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''Guiding principle.'''

TEI records the documentary evidence and scholarly relationships.
Lua interprets the encoded structures according to explicit editorial rules.
ConTeXt gives the resulting records their typographical form.

The three layers cooperate, but they do not have the same responsibility.
</div>


== 1. Where this guide fits in the collection ==

The six-guide progression can now be understood as:

<pre>
Guide 1   STRUCTURE
             │
             ▼
Guide 2   IDENTITIES
             │
             ▼
Guide 3   BASIC RELATIONS
             │
             ▼
Guide 4   COMPLEX RELATIONS
             │
             ▼
Guide 5   INTERPRET + VALIDATE          ← YOU ARE HERE
             │
             ▼
Guide 6   COMPOSE
</pre>

'''How to read this diagram.''' Guides 1–4 primarily construct the
scholarly data model. Guide 5 changes the task from encoding to controlled
interpretation and checking. Guide 6 receives the resulting records and
concentrates on composition.

The first four guides primarily built the scholarly data model.

Guide 5 transforms that model into a stable intermediate representation that
Guide 6 can typeset without repeatedly interpreting raw XML.

The processing system developed here will progressively:

# build a registry of textual witnesses;
# convert TEI apparatus entries into Lua records;
# normalize identifiers, witness lists, and textual values;
# resolve witness references against the registry;
# validate the resulting editorial records;
# report errors and warnings;
# separate accepted records from records requiring correction;
# make checked records available to ConTeXt.

<div style="background:#eef7ee; border:1px solid #a8c7a8; border-left:5px solid #5b8a5a; padding:0.8em 1em; margin:1em 0;">
'''Stage reached.'''

The TEI source is already rich enough to describe the edition.

Guide 5 does not replace that source. It constructs a controlled processing
layer between documentary encoding and typographical composition.
</div>


== 2. Why Lua in an LMTX workflow? ==

The question should be asked explicitly:

<pre>
If ConTeXt can already process XML, why introduce Lua?
</pre>

There are two complementary answers.

=== 2.1. Lua is already part of the LMTX environment ===

ConTeXt LMTX runs on LuaMetaTeX. Lua is therefore not an external
programming environment inserted between TEI and ConTeXt.

A useful mental map is:

<pre>
                 ConTeXt LMTX
                      │
              +-------+-------+
              │               │
              ▼               ▼
        ConTeXt MkXL      LuaMetaTeX
                              │
                              └── Lua is available natively
                                     │
                                     ▼
TEI XML ─────────────────────► processing
                                     │
                                     ▼
                                  ConTeXt
                                     │
                                     ▼
                                    page
</pre>

'''How to read this diagram.''' Lua does not introduce a foreign
application into the workflow. It uses the programming language already
available in the engine environment on which ConTeXt LMTX runs. TEI
processing, editorial transformation, and typesetting can therefore remain
inside one integrated toolchain.

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''Why Lua in an LMTX workflow?''' ConTeXt LMTX runs on LuaMetaTeX. Lua is
therefore already available where the XML is being processed and the
document is being composed. When editorial relations require reusable
normalization, cross-reference lookup, project-level checks, or diagnostic
reports, Lua is a natural processing layer within the same environment.
</div>

<div style="background:#f4f4f4; border:1px solid #c8c8c8; border-left:5px solid #8a8a8a; padding:0.8em 1em; margin:1em 0;">
'''Further reference.'''

For the engine architecture, see the
[https://mirror.contextgarden.net/general/manuals/luametatex.pdf LuaMetaTeX reference manual].

For programming Lua from ConTeXt, see [[ConTeXt and Lua programming]].
</div>

=== 2.2. A minimal proof: Lua is already here ===

The following Garden example does not perform editorial processing. Its
only purpose is to demonstrate that Lua code can run directly inside an
LMTX document:

<context source="yes">
\starttext

ConTeXt says: \quotation{TEI will be processed here.}

\startluacode

xml.registerns(
    "tei",
    "http://www.tei-c.org/ns/1.0"
)

context.par()
context("Lua says: the processing layer is already inside LMTX.")
\stopluacode

\stoptext
</context>

The important fact is architectural: there is no separate conversion
program to launch merely in order to use Lua.

=== 2.3. Direct XML processing remains useful ===

Lua is not required simply because the source is XML.

For a small or local transformation, ConTeXt XML setups can remain the
simplest path:

<pre>
TEI XML
   │
   ▼
ConTeXt XML setup
   │
   ▼
typeset result
</pre>

That model was sufficient for the inspection tests in Guides 1–4.

The processing problem changes when each apparatus entry begins to require
operations such as:

<pre>
parse @wit
    ↓
normalize identifiers
    ↓
look up declarations
    ↓
apply project rules
    ↓
record warnings
    ↓
retain invalid objects for diagnosis
    ↓
pass only accepted records to composition
</pre>

At that point, repeating the same interpretation inside every
typographical setup becomes difficult to maintain.

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.8em 1em; margin:1em 0;">
'''Lua is useful, not mandatory.''' Direct ConTeXt XML processing remains
appropriate when the mapping from XML to output is local and simple.
Introduce a Lua intermediate layer when processing becomes relational,
reusable, rule-driven, or diagnostic.
</div>

=== 2.4. The editorial reason for the intermediate layer ===

The accumulated TEI data now contain relationships across the document:

<pre>
witness declaration
        ▲
        │ @wit
        │
apparatus reading

responsibility declaration
        ▲
        │ @resp
        │
editorial intervention
</pre>

The task is no longer only to select one XML element. It is to interpret
relations consistently across many elements.

The architectural choice is therefore:

<pre>
DIRECT XML → TYPESETTING
works well while processing is local and simple

                    BUT

when every apparatus entry may require
normalization + lookup + checks + diagnostics

                    ↓

TEI XML
   │
   ▼
Lua intermediate processing
   │
   ▼
stable editorial records
   │
   ▼
ConTeXt
</pre>

'''How to read this diagram.''' Lua is introduced because the task has
changed, not because XML has become unreadable. The intermediate layer
centralizes operations that would otherwise be repeated in many
typographical macros.

=== 2.5. Division of labour ===

The three layers remain distinct:

<pre>
TEI
├── evidence
├── documentary distinctions
└── scholarly assertions
        │
        ▼
Lua
├── extract
├── normalize
├── resolve
├── check project rules
└── diagnose
        │
        ▼
ConTeXt
├── typography
├── punctuation
├── spacing
├── placement
└── page architecture
</pre>

'''How to read this diagram.''' TEI remains the documentary and scholarly
source. Lua turns selected encoded relations into controlled editorial
objects. ConTeXt determines how those objects appear on the page.

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''Division of labour.''' TEI records the scholarly claims; Lua applies
documented processing rules to those claims; ConTeXt composes the result.
The boundaries are deliberate even though all three layers operate inside
one LMTX workflow.
</div>

<div style="background:#f4f4f4; border:1px solid #c8c8c8; border-left:5px solid #8a8a8a; padding:0.8em 1em; margin:1em 0;">
'''Further reference.'''

For direct XML processing, see
[[Input and compilation/XML source/XML setup commands|XML setup commands]].

For Lua-based XML traversal, see
[[ConTeXt and Lua programming/Processing XML with Lua|Processing XML with Lua]].

For the broader TEI → ConTeXt architecture, see [[TEI xml|TEI XML]].
</div>

<div style="background:#eef7ee; border:1px solid #a8c7a8; border-left:5px solid #5b8a5a; padding:0.8em 1em; margin:1em 0;">
'''Stage reached.''' We now know why Lua belongs in Guide 5: it is already
native to the LMTX environment, and the editorial processing has become
complex enough to benefit from an explicit intermediate layer.
</div>

== Part I — Understand the processing model ==

== 3. From XML syntax to editorial knowledge ==

=== 3.1. Reading is not yet interpreting ===

Consider:

<pre>
<rdg wit="#ms-A #ed-C">λόγος</rdg>
</pre>

An XML parser can expose:

<pre>
element name: rdg
text:         λόγος
@wit:         "#ms-A #ed-C"
</pre>

But the string:

<pre>
"#ms-A #ed-C"
</pre>

has not yet become editorial knowledge.

The processing path is:

<pre>
XML PARSER

wit="#ms-A #ed-C"
     │
     ▼
character string
"#ms-A #ed-C"


LUA PROCESSING

"#ms-A #ed-C"
     │
     ▼
split references
     │
     ▼
 A       C
 │       │
 ▼       ▼
lookup  lookup
 │       │
 ▼       ▼
known?  known?
 │       │
 +---+---+
     │
     ▼
resolved witness relations
</pre>

Lua may need to establish:

* that the value contains two references;
* that <code>#</code> is a reference marker rather than part of the identifier;
* that A and C are declared witnesses;
* that their order is meaningful or should be normalized;
* that neither reference has been duplicated accidentally;
* that the references are admissible in the current editorial model.

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''From syntax to editorial knowledge.'''

XML parsing makes the encoded structure accessible.
Lua processing determines how that structure is to be understood within the
rules of the edition.
</div>


=== 3.2. Normalization: equivalent notation, one internal form ===

Equivalent information may appear as:

<pre>
wit="#ms-A #ms-B"
wit=" #ms-A   #ms-B "
wit="#ms-A #ms-A #ms-B"
</pre>

If the edition treats spacing differences and repeated references as
insignificant, Lua can normalize all three to:

<pre>
{ "ms-A", "ms-B" }
</pre>

The operation can be visualized as:

<pre>
RAW TEI

wit="  #ms-A   #ms-B  #ms-A "
          │
          ▼
      trim spaces
          │
          ▼
    "#ms-A #ms-B #ms-A"
          │
          ▼
        split
          │
      A   B   A
          │
          ▼
      deduplicate
          │
          ▼
      { "ms-A", "ms-B" }
</pre>

Normalization may include:

* removing reference markers;
* trimming insignificant whitespace;
* splitting lists into individual identifiers;
* removing accidental duplicates;
* preserving or standardizing order;
* replacing absent values with explicit internal values;
* retaining editorial categories such as <code>omission</code>.

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.8em 1em; margin:1em 0;">
'''Normalization is an interpretative operation.'''

It defines which differences in the source are editorially significant and
which are merely differences of notation.

Normalization should therefore follow documented project rules rather than
being treated as an invisible programming convenience.
</div>


=== 3.3. Reference resolution changes the status of a value ===

Suppose the TEI header declares:

<pre>
<witness xml:id="ms-A">
  Paris, Bibliothèque nationale de France, grec 1807
</witness>
</pre>

and a reading contains:

<pre>
<rdg wit="#ms-A">...</rdg>
</pre>

Before resolution:

<pre>
#ms-A
│
└── character string
</pre>

After resolution:

<pre>
#ms-A
 │
 ▼
normalize
 │
 ▼
A
 │
 ▼
registry lookup
 │
 ▼
witness record A
 ├── siglum
 ├── description
 ├── kind
 └── order
</pre>

Thus:

<pre>
STRING
  │
  ▼
IDENTIFIER
  │
  ▼
RESOLVED EDITORIAL OBJECT
</pre>

'''How to read this diagram.''' The characters in <code>@wit</code> begin
as encoded text. Normalization turns them into a controlled identifier;
registry lookup turns that identifier into a known editorial object. This
change of status is one of the main reasons for introducing the
intermediate layer.

If no record exists, the same process produces a detectable editorial problem.


=== 3.4. Project-level validation adds another state ===

A document may be well-formed XML and still contain unusable apparatus data.

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.8em 1em; margin:1em 0;">
'''TEI validation and Lua project checks are not the same thing.'''

Schema validation asks whether a document conforms to the selected TEI
model. The Lua checks developed in this guide apply additional rules chosen
for this tutorial or for a particular edition: reference integrity,
controlled categories, required fields, normalization policy, and the
conditions treated as errors or warnings.

A rule enforced here must therefore not be presented as a universal TEI
constraint.
</div>

<div style="background:#f4f4f4; border:1px solid #c8c8c8; border-left:5px solid #8a8a8a; padding:0.8em 1em; margin:1em 0;">
'''Further reference.'''

Guide 1 introduced the distinction between XML well-formedness, TEI
validation, and editorial correctness.

For formal TEI conformance, see the TEI P5 chapter
[https://tei-c.org/release/doc/tei-p5-doc/en/html/USE.html ''Using the TEI''].
The checks implemented below belong to the processing profile of this
tutorial.
</div>

For example:

* a reading may refer to an undeclared witness;
* this tutorial's project profile may require a lemma where the source lacks one;
* an ordinary reading may be empty without being classified as an omission;
* a witness identifier may be declared twice;
* a repeated reference may need normalization;
* an editorial category required by the project may be absent.

The progression is:

<pre>
XML
 │
 ▼
readable structure
 │
 ▼
normalized Lua record
 │
 ▼
resolved references
 │
 ▼
project-level editorial checking
 │
 +---------------+----------------+
 │               │                │
 ▼               ▼                ▼
VALID         WARNING           INVALID
 │               │                │
 │               │                └── retained for diagnosis
 │               │
 └───────┬───────┘
         │
         ▼
usable record
</pre>

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.8em 1em; margin:1em 0;">
'''Well-formed XML is not necessarily editorially usable XML.'''

Syntax may be correct while references, classifications, or apparatus
relationships remain incomplete or contradictory.
</div>


=== 3.5. Separate interpretation from typesetting ===

Without an intermediate layer, one ConTeXt macro might have to:

<pre>
read XML
   ↓
split @wit
   ↓
remove #
   ↓
lookup witnesses
   ↓
check validity
   ↓
decide omission status
   ↓
sort sigla
   ↓
choose punctuation
   ↓
typeset
</pre>

That mixes several responsibilities.

The architecture adopted here is:

<pre>
TEI XML
   │
   ▼
Lua
├── identity
├── normalization
├── resolution
├── consistency
└── editorial status
   │
   ▼
checked records
   │
   ▼
ConTeXt
├── placement
├── visual hierarchy
├── spacing
├── punctuation
└── page construction
</pre>

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''Division of labour.'''

Lua prepares editorially meaningful records.
ConTeXt transforms those records into pages.
</div>


== 4. The processing pipeline ==

The complete conceptual workflow is cumulative:

<pre>
TEI documentary source
        │
        ▼
1. EXTRACTION
        │
        ▼
raw Lua values
        │
        ▼
2. NORMALIZATION
        │
        ▼
consistent fields
        │
        ▼
3. REFERENCE RESOLUTION
        │
        ▼
known editorial objects
        │
        ▼
4. VALIDATION
        │
        ├── errors
        ├── warnings
        └── accepted records
        │
        ▼
5. INTERMEDIATE REPRESENTATION
        │
        ▼
records prepared for ConTeXt
        │
        ▼
6. TYPESETTING                         ← Guide 6
</pre>

'''How to read this diagram.''' Each stage answers a different question.
Extraction selects relevant XML structures; normalization controls their
surface form; resolution connects references to declarations; project
checks test declared editorial rules; the intermediate representation
becomes the stable input contract for Guide 6.

{| class="wikitable"
! Stage
! Input
! Main question
! Result
|-
| Documentary encoding
| TEI elements and attributes
| What evidence and relationships are being recorded?
| Structured XML source
|-
| Extraction
| TEI XML
| Which structures are relevant to this task?
| Raw Lua values
|-
| Normalization
| Raw strings and content
| Which surface differences are insignificant?
| Controlled internal fields
|-
| Reference resolution
| Identifiers and references
| Which declared object does this value denote?
| Resolved editorial objects
|-
| Validation
| Normalized and resolved records
| Are the records usable under project rules?
| Errors, warnings, valid records
|-
| Intermediate representation
| Validated data
| What stable structure should typesetting receive?
| Records prepared for ConTeXt
|-
| Typesetting
| Validated records
| How should the information appear on the page?
| Printed or electronic apparatus
|}

These are conceptual stages. A practical Lua function may perform several of
them together, but the distinctions remain valuable because they show where a
decision belongs and where an error should be reported.


=== 4.1. TEI remains the documentary source ===

The Lua representation is deliberately smaller than the TEI document.

That does not make it a replacement source.

<pre>
TEI
│
├── documentary richness
├── scholarly distinctions
├── source relationships
└── reusable encoding
        │
        │ selective transformation
        ▼
Lua record
│
├── fields needed now
├── normalized values
├── resolved objects
└── source connection
</pre>

Where practical, a record should retain a connection to its source XML node or
identifier so that diagnostics can be traced back to the encoding.

<div style="background:#f4f4f4; border:1px solid #c8c8c8; border-left:5px solid #8a8a8a; padding:0.8em 1em; margin:1em 0;">
'''Further reference.'''

For ConTeXt's Lua-side XML traversal facilities, see
[[ConTeXt and Lua programming/Processing XML with Lua|Processing XML with Lua]].

For the TEI structures being processed, return to Guides 2–4 and the TEI
P5 chapter [https://tei-c.org/release/doc/tei-p5-doc/en/html/TC.html ''Critical Apparatus''].
</div>



=== 4.2. Lua creates task-oriented editorial objects ===

A witness may become:

<pre>
{
    id          = "ms-A",
    siglum      = "A",
    description = "Paris, Bibliothèque nationale de France, grec 1807",
    kind        = "manuscript",
    order       = 1,
}
</pre>

An apparatus entry may become:

<pre>
{
    id       = "app-001",
    lemma    = {
        text      = "λόγος",
        witnesses = { "ms-A" },
    },
    readings = {
        {
            text      = "λέξις",
            witnesses = { "ms-B", "ed-C" },
        },
    },
    valid = true,
}
</pre>

The relationship is:

<pre>
TEI STRUCTURE
     │
     ▼
identify editorial roles
     │
     ▼
LUA RECORD
     │
     ├── predictable fields
     ├── normalized values
     └── source link
</pre>


=== 4.3. The intermediate representation is a contract ===

Once processing is complete, ConTeXt should be able to rely on several
properties:

* identifiers have normalized forms;
* witness references have either resolved or been reported;
* apparatus records have predictable fields;
* exceptional readings have explicit categories;
* invalid records are distinguishable from accepted records;
* diagnostics have already been produced.

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''The intermediate representation is a contract.'''

Lua guarantees the editorial status and internal structure of accepted records.
ConTeXt can therefore concentrate on their presentation instead of
reinterpreting XML syntax at every typographical command.
</div>



<div style="background:#f4f4f4; border:1px solid #c8c8c8; border-left:5px solid #8a8a8a; padding:0.8em 1em; margin:1em 0;">
'''Scope of the processing examples.'''

Guide 4 introduced responsibility, certainty, documentary substitutions,
reading groups, local word-order variation, and unequal textual extent in
addition to simple substitutions and omissions.

The functional MWEs below deliberately process a representative subset so
that the pipeline remains readable. The same intermediate architecture can
be extended with fields for <code>@resp</code>, <code>@cert</code>,
documentary operations, reading groups, and other project-specific
structures.
</div>


== Part II — Build the processing layer ==

== 5. Stage 1 — Build a witness registry ==

Guide 2 declared witnesses in TEI.

Guide 5 now turns those declarations into a processing structure.

=== 5.1. From <code>&lt;listWit&gt;</code> to a Lua registry ===

The TEI structure is:

<pre>
listWit
├── witness A
├── witness B
└── witness C
</pre>

The processing path is:

<pre>
TEI
listWit
   │
   ▼
collect witness declarations
   │
   ▼
normalize identifiers
   │
   ▼
check uniqueness
   │
   ▼
LUA WITNESS REGISTRY
├── by_id
├── in_order
└── errors
</pre>

The two main collections answer different questions:

<pre>
witnesses.by_id["ms-B"]
        │
        └── Which witness is B?

witnesses.in_order[2]
        │
        └── Which witness occupies this declared position?
</pre>

<div style="background:#f4f4f4; border:1px solid #c8c8c8; border-left:5px solid #8a8a8a; padding:0.8em 1em; margin:1em 0;">
'''Technical note.'''

Direct lookup and documentary order are different requirements.
Keep them explicit rather than relying on one Lua table to represent both
implicitly.
</div>


=== 5.2. What a witness record contains ===

A record may contain:

<pre>
witnesses["A"] = {
    id          = "ms-A",
    siglum      = "A",
    kind        = "manuscript",
    description = "Paris, Bibliothèque nationale de France, grec 1807",
    order       = 1,
}
</pre>

Conceptually:

<pre>
TEI witness
   │
   ├── xml:id ─────────► id
   ├── display value ──► siglum
   ├── @type ──────────► kind
   ├── text ───────────► description
   └── position ───────► order
</pre>

The exact fields depend on the needs of the edition.


=== 5.3. Registry lookup ===

Once the registry exists:

<pre>
"B"
 │
 ▼
witnesses.by_id["ms-B"]
 │
 +───────────────+
 │               │
 ▼               ▼
found           absent
 │               │
 ▼               ▼
witness record  unresolved reference
</pre>

Later apparatus processing can therefore check references without repeatedly
searching the original <code>&lt;listWit&gt;</code>.


=== 5.4. Functional MWE: Building a witness registry from TEI XML ===

The following example reads witness declarations from a TEI fragment, builds a
Lua registry, records errors, and prints a diagnostic report.

It does not yet process apparatus entries. Its sole purpose is to establish the
collection of witnesses against which later references will be resolved.

<context source = "yes">
\mainlanguage[en]

\setuppapersize[A5]

\setupbodyfont
  [libertinus,10pt]

\setupwhitespace
  [medium]

\startbuffer[tei-data]
<TEI xmlns="http://www.tei-c.org/ns/1.0">
  <teiHeader>
    <fileDesc>
      <titleStmt>
        <title>Guide 5 witness registry</title>
      </titleStmt>
      <publicationStmt>
        <p>Unpublished teaching example.</p>
      </publicationStmt>
      <sourceDesc>
        <listWit>
          <witness xml:id="ms-A" n="A" type="manuscript">
            Paris, Bibliothèque nationale de France, grec 1807
          </witness>
          <witness xml:id="ms-B" n="B" type="manuscript">
            Vatican City, Biblioteca Apostolica Vaticana, Vat. gr. 1
          </witness>
          <witness xml:id="ed-C" n="C" type="edition">
            Aldine edition, Venice, 1513
          </witness>
        </listWit>
      </sourceDesc>
    </fileDesc>
    <encodingDesc>
      <variantEncoding
        method="parallel-segmentation"
        location="internal"/>
    </encodingDesc>
  </teiHeader>
  <text>
    <body>
      <p xml:id="p1">The mind seeks unity.</p>
    </body>
  </text>
</TEI>
\stopbuffer

\startluacode

xml.registerns(
    "tei",
    "http://www.tei-c.org/ns/1.0"
)


thirddata = thirddata or {}

thirddata.tei_witnesses = {
    by_id    = {},
    in_order = {},
    errors   = {},
}

local registry = thirddata.tei_witnesses

local function normalize_space(value)
    if not value then
        return ""
    end

    value = value:gsub("%s+", " ")
    value = value:gsub("^%s+", "")
    value = value:gsub("%s+$", "")

    return value
end

local function add_error(message)
    registry.errors[#registry.errors + 1] = message
end

local function reset_registry()
    registry.by_id    = {}
    registry.in_order = {}
    registry.errors   = {}
end

local function register_witness(node)
    local attributes = node.at or {}

    local id =
        attributes["xml:id"]
        or attributes.id
        or ""

    local siglum =
        attributes.n
        or ""

    local kind =
        attributes.type
        or ""

    id     = normalize_space(id)
    siglum = normalize_space(siglum)
    kind   = normalize_space(kind)

    if siglum == "" then
        siglum = id
    end

    local description =
        normalize_space(xml.text(node))

    if id == "" then
        add_error(
            "A witness declaration has no xml:id."
        )
        return
    end

    if registry.by_id[id] then
        add_error(
            "Duplicate witness identifier: " .. id
        )
        return
    end

    local record = {
        id          = id,
        siglum      = siglum,
        kind        = kind ~= "" and kind or "unspecified",
        description = description,
        order       = #registry.in_order + 1,
        source      = node,
    }

    registry.by_id[id] = record
    registry.in_order[#registry.in_order + 1] = record
end

function registry.build()
    reset_registry()

    local document = xml.convert(
        buffers.getcontent("tei-data")
    )

    for node in xml.collected(document, "tei:witness") do
        register_witness(node)
    end
end

function registry.lookup(id)
    return registry.by_id[id]
end

function registry.typeset()
    context.subject("Witness registry")

    context.starttabulate {
        "|l|l|p|"
    }

    context.NC()
    context.bold("Siglum")
    context.NC()
    context.bold("Kind")
    context.NC()
    context.bold("Description")
    context.NC()
    context.NR()

    for _, witness in ipairs(registry.in_order) do
        context.NC()
        context(witness.siglum)

        context.NC()
        context(witness.kind)

        context.NC()
        context(witness.description)

        context.NC()
        context.NR()
    end

    context.stoptabulate()

    context.blank()

    context.subject("Lookup test")

    local selected = registry.lookup("ms-B")

    if selected then
        context(
            "Witness B is registered as a %s.",
            selected.kind
        )
    else
        context(
            "Witness B is not registered."
        )
    end

    context.blank()

    context.subject("Registry report")

    context(
        "Registered witnesses: %s",
        #registry.in_order
    )

    context.par()

    if #registry.errors == 0 then
        context(
            "No registry errors were detected."
        )
    else
        context(
            "Errors: %s",
            #registry.errors
        )

        context.startitemize()

        for _, message in ipairs(registry.errors) do
            context.startitem()
            context(message)
            context.stopitem()
        end

        context.stopitemize()
    end
end

\stopluacode

\starttext

\subject{Building a witness registry from TEI XML}

The following report is produced from the witness declarations contained in
the TEI fragment.

\blank

\ctxlua{
    thirddata.tei_witnesses.build()
    thirddata.tei_witnesses.typeset()
}

\stoptext

</context>



<!-- Recompile the revised namespace-aware MWE and update the witness-registry screenshot here. -->


The central result of the example is not the table printed on the page. The
important result is the Lua registry itself:

<pre>
witnesses.by_id
witnesses.in_order
witnesses.errors
</pre>

These three fields represent different kinds of knowledge about the TEI
document:

* <code>by_id</code> records which witnesses are known and makes them
  directly accessible;
* <code>in_order</code> records the sequence in which the witnesses were
  declared;
* <code>errors</code> records conditions that prevent reliable interpretation.

The source XML contains witness declarations. The Lua registry turns those
declarations into a collection of identifiable, ordered, and verifiable
editorial objects.

=== 5.5. Testing missing and duplicate identifiers ===

The validation behaviour can be tested by adding the following declarations to
the TEI buffer:

<pre>
<witness type="manuscript">
  An unidentified manuscript
</witness>

<witness xml:id="ms-A" type="manuscript">
  A second declaration using the identifier A
</witness>
</pre>

The first declaration cannot be referenced because it has no
<code>xml:id</code>. The second makes the identifier <code>A</code>
ambiguous because that identifier is already present in the registry.

The processing layer therefore reports both conditions and excludes the
problematic declarations from the usable registry.

This is an important difference between merely traversing the XML tree and
constructing an editorial representation. Traversal can find both elements.
Registry construction must decide whether they can function as stable objects
within the edition.



=== 5.6. Result of Stage 1 ===

At this point the source declarations have become a controlled registry:

<pre>
TEI witness declarations
        │
        ▼
Lua witness registry
        ├── direct lookup
        ├── preserved order
        └── declaration errors
</pre>

<div style="background:#eef7ee; border:1px solid #a8c7a8; border-left:5px solid #5b8a5a; padding:0.8em 1em; margin:1em 0;">
'''Stage reached.'''

Witnesses are no longer only XML elements. They are now identifiable, ordered,
and verifiable editorial objects against which later references can be checked.
</div>


== 6. Stage 2 — Convert apparatus entries into Lua records ==

We now bring in the structure constructed in Guide 3:

<pre>
app
├── lem: λόγος
│   └── @wit="#ms-A"
└── rdg: λέξις
    └── @wit="#ms-B #ed-C"
</pre>

The next transformation is:

<pre>
TEI APP
   │
   ▼
extract fields
   │
   ▼
normalize witness lists
   │
   ▼
classify editorial roles
   │
   ▼
LUA APPARATUS RECORD
</pre>


=== 6.1. From XML hierarchy to explicit fields ===

For:

<pre>
<app xml:id="app-001">
  <lem wit="#ms-A">λόγος</lem>
  <rdg wit="#ms-B #ed-C">λέξις</rdg>
</app>
</pre>

we want:

<pre>
record
├── id: app-001
├── lemma
│   ├── text: λόγος
│   └── witnesses: { A }
│
└── readings
    └── reading 1
        ├── text: λέξις
        ├── witnesses: { B, C }
        └── kind: reading
</pre>

This is not yet a typeset apparatus entry.
It is an editorial object with predictable fields.


=== 6.2. Normalize witness lists ===

The path is:

<pre>
@wit="#ms-B #ed-C"
      │
      ▼
raw string
      │
      ▼
remove # markers
      │
      ▼
split values
      │
      ▼
remove accidental duplicates
      │
      ▼
{ "ms-B", "ed-C" }
</pre>

An irregular source such as:

<pre>
wit="  #ms-B   #ed-C  #ms-B "
</pre>

can therefore produce the same normalized list while still allowing a warning
to record that B was duplicated in the original value.


=== 6.3. Preserve editorial categories ===

An empty value alone is not enough to explain its meaning.

Compare:

<pre>
{
    kind = "reading",
    text = "λέξις",
}
</pre>

with:

<pre>
{
    kind = "omission",
    text = "",
}
</pre>

The relation is:

<pre>
TEI distinction
       │
       ▼
explicit Lua category
       │
       ▼
later validation + typography
</pre>

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.8em 1em; margin:1em 0;">
'''Do not flatten significant distinctions.'''

An omission, missing data, unreadable text, and a processing failure should not
all become the same empty string merely because their visible text is empty.
</div>


=== 6.4. Functional MWE: Converting TEI apparatus entries into Lua records ===

The following example extends the witness-registry mechanism developed in
section 4.

It reads two apparatus entries, normalizes their witness lists, preserves the
order of their readings, and constructs Lua records.

The example displays the resulting records for inspection. It does not yet
validate whether every witness reference is known.

<context source = "yes">
\mainlanguage[en]

\setuppapersize[A5]

\setupbodyfont
  [libertinus,10pt]

\setupwhitespace
  [medium]

\startbuffer[tei-data]
<TEI xmlns="http://www.tei-c.org/ns/1.0">
  <teiHeader>
    <fileDesc>
      <titleStmt>
        <title>Guide 5 apparatus records</title>
      </titleStmt>
      <publicationStmt>
        <p>Unpublished teaching example.</p>
      </publicationStmt>
      <sourceDesc>
        <listWit>
          <witness xml:id="ms-A" n="A" type="manuscript">
            Paris, Bibliothèque nationale de France, grec 1807
          </witness>
          <witness xml:id="ms-B" n="B" type="manuscript">
            Vatican City, Biblioteca Apostolica Vaticana, Vat. gr. 1
          </witness>
          <witness xml:id="ed-C" n="C" type="edition">
            Aldine edition, Venice, 1513
          </witness>
        </listWit>
      </sourceDesc>
    </fileDesc>
    <encodingDesc>
      <variantEncoding
        method="parallel-segmentation"
        location="internal"/>
    </encodingDesc>
  </teiHeader>
  <text>
    <body>
      <p>
        <app xml:id="app-001">
          <lem wit="#ms-A">λόγος</lem>
          <rdg wit="#ms-B #ed-C">λέξις</rdg>
        </app>
      </p>
      <p>
        <app xml:id="app-002" type="omission">
          <lem wit="#ms-A #ms-B">ἀλήθεια</lem>
          <rdg wit="#ed-C"/>
        </app>
      </p>
    </body>
  </text>
</TEI>
\stopbuffer

\startluacode

xml.registerns(
    "tei",
    "http://www.tei-c.org/ns/1.0"
)


thirddata = thirddata or {}

thirddata.tei_apparatus = {
    records = {},
    errors  = {},
}

local apparatus = thirddata.tei_apparatus

local function normalize_space(value)
    if not value then
        return ""
    end

    value = value:gsub("%s+", " ")
    value = value:gsub("^%s+", "")
    value = value:gsub("%s+$", "")

    return value
end

local function normalize_witnesses(value)
    local result = {}
    local seen   = {}

    value = normalize_space(value)

    for reference in value:gmatch("%S+") do
        local id = reference:gsub("^#", "")

        if id ~= "" and not seen[id] then
            result[#result + 1] = id
            seen[id] = true
        end
    end

    return result
end

local function make_reading_record(node, app_kind)
    local attributes = node.at or {}

    local kind =
        normalize_space(attributes.type)

    if kind == "" then
        if app_kind == "omission" then
            kind = "omission"
        else
            kind = "reading"
        end
    end

    return {
        kind      = kind,
        text      = normalize_space(xml.text(node)),
        witnesses = normalize_witnesses(attributes.wit),
        source    = node,
    }
end

local function make_lemma_record(node)
    if not node then
        return nil
    end

    local attributes = node.at or {}

    return {
        text      = normalize_space(xml.text(node)),
        witnesses = normalize_witnesses(attributes.wit),
        source    = node,
    }
end

local function make_apparatus_record(node, position)
    local attributes = node.at or {}

    local app_kind =
        normalize_space(attributes.type)

    if app_kind == "" then
        app_kind = "variation"
    end

    local lemma_node =
        xml.first(node, "tei:lem")

    local record = {
        id =
            normalize_space(attributes["xml:id"])
            ~= ""
            and normalize_space(attributes["xml:id"])
            or "app-" .. position,

        kind     = app_kind,
        lemma    = make_lemma_record(lemma_node),
        readings = {},
        order    = position,
        source   = node,
        valid    = true,
    }

    for reading_node in xml.collected(node, "tei:rdg") do
        record.readings[#record.readings + 1] =
            make_reading_record(reading_node, app_kind)
    end

    return record
end

function apparatus.build()
    apparatus.records = {}
    apparatus.errors  = {}

    local document = xml.convert(
        buffers.getcontent("tei-data")
    )

    local position = 0

    for node in xml.collected(document, "tei:app") do
        position = position + 1

        apparatus.records[#apparatus.records + 1] =
            make_apparatus_record(node, position)
    end
end

local function typeset_witness_list(list)
    if #list == 0 then
        context("none")
        return
    end

    context(table.concat(list, ", "))
end

function apparatus.typeset()
    context.subject("Apparatus records")

    for _, record in ipairs(apparatus.records) do
        context.subsubject(record.id)

        if record.lemma then
            context.bold("Lemma: ")
            context(record.lemma.text)

            context.par()

            context.bold("Lemma witnesses: ")
            typeset_witness_list(record.lemma.witnesses)
        else
            context.bold("Lemma: ")
            context("missing")
        end

        context.par()
        context.bold("Readings: ")
        context(#record.readings)

        context.startitemize()

        for _, reading in ipairs(record.readings) do
            context.startitem()

            context.bold("Kind: ")
            context(reading.kind)

            context.par()

            context.bold("Text: ")

            if reading.text ~= "" then
                context(reading.text)
            else
                context("empty")
            end

            context.par()

            context.bold("Witnesses: ")
            typeset_witness_list(reading.witnesses)

            context.stopitem()
        end

        context.stopitemize()
    end
end

\stopluacode

\starttext

\subject{Converting TEI apparatus entries into Lua records}

The following report shows the intermediate records constructed from two TEI
apparatus entries.

\blank

\ctxlua{
    thirddata.tei_apparatus.build()
    thirddata.tei_apparatus.typeset()
}

\stoptext

</context>

<!-- Recompile the revised apparatus-record MWE and update its screenshot here. -->

The first record contains:

* the lemma <code>λόγος</code>, associated with identifier
  <code>ms-A</code> (display label <code>A</code>);
* the reading <code>λέξις</code>, associated with identifiers
  <code>ms-B</code> and <code>ed-C</code> (display labels
  <code>B</code> and <code>C</code>).

The second record contains:

* the lemma <code>ἀλήθεια</code>, associated with identifiers
  <code>ms-A</code> and <code>ms-B</code>;
* an explicitly classified omission associated with
  <code>ed-C</code>.

The empty text of the omission is therefore not treated as an ordinary empty
reading. Guide 4 classified the apparatus entry with
<code>type="omission"</code>; the Lua conversion carries that project
classification into the reading record:

<pre>
kind = "omission"
</pre>

This is an example of contextual interpretation: an empty
<code>&lt;rdg/&gt;</code> acquires its editorial meaning from the apparatus
classification rather than from emptiness alone.

=== 6.5. Result of Stage 2 ===

At the end of this stage, each TEI <code>&lt;app&gt;</code> element has become
a structured Lua record with:

* a stable identifier;
* one lemma record;
* an ordered collection of reading records;
* normalized witness identifiers;
* explicit editorial categories;
* a reference to the source XML node;
* a predictable structure for later validation and typesetting.

The transformation has therefore moved beyond XML traversal.

Lua has identified the editorial roles of the elements, separated their
contents into explicit fields, and constructed apparatus objects that can be
examined independently of the original XML syntax.

The witness identifiers contained in these records have been normalized, but
they have not yet been verified against the witness registry.

That verification is the purpose of the next section.



<div style="background:#eef7ee; border:1px solid #a8c7a8; border-left:5px solid #5b8a5a; padding:0.8em 1em; margin:1em 0;">
'''Stage reached.'''

Each <code>&lt;app&gt;</code> now has a predictable Lua representation.
Its witness identifiers are normalized, but they have not yet been proved to
denote declared witnesses.
</div>


== 7. Stage 3 — Resolve references and check project rules ==

The two structures built so far can now be joined.

=== 7.1. Join the witness registry and apparatus records ===

<pre>
WITNESS REGISTRY                    APPARATUS RECORD

by_id["ms-A"] ◄──────────────────────── A
by_id["ms-B"] ◄──────────────────────── B
by_id["ed-C"] ◄──────────────────────── C

by_id["ms-D"] ? ◄────────────────────── D
                unresolved
</pre>

For each normalized identifier:

<pre>
identifier
    │
    ▼
registry lookup
    │
 +--+--+
 │     │
 ▼     ▼
found  absent
 │     │
 ▼     ▼
resolved ERROR
object
</pre>

This is the point where a character string becomes a relation to a known
editorial object.


=== 7.2. Check the project apparatus profile ===

Reference resolution is only one part of validation.

A project may also require additional rules beyond TEI schema
conformance. The small profile used by this tutorial asks, for example:

<pre>
apparatus record
      │
      ├── lemma exists?
      ├── readings exist?
      ├── ordinary readings contain text?
      ├── empty readings classified explicitly?
      ├── witnesses declared?
      └── duplicates reported?
</pre>

A record may therefore be perfectly readable as XML while being unusable as an
editorial record.


=== 7.3. Errors and warnings have different meanings ===

<pre>
IRREGULARITY
     │
     ▼
can processing establish
safe editorial meaning?
     │
  +--+--+
  │     │
 yes    no
  │     │
  ▼     ▼
WARNING ERROR
  │     │
  ▼     ▼
usable invalid
record record
</pre>

Typical errors include:

* an undeclared witness;
* a missing lemma where this tutorial's project profile requires one;
* an ordinary reading whose empty content is unexplained;
* ambiguous duplicate witness declarations.

Typical warnings may include:

* a repeated reference that was normalized;
* a generated identifier;
* a condition that is technically usable but deserves editorial review.

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''Project checks change the status of the data.'''

A reference is no longer merely present: it is resolved or unresolved.
An apparatus entry is no longer merely readable: it is usable, usable with
warnings, or invalid.
</div>


=== 7.4. Functional MWE: Resolving references and checking apparatus records ===

The following example combines a witness registry with several apparatus
records.

It deliberately includes:

* one valid apparatus entry;
* one reading that refers to an undeclared witness;
* one apparatus entry without a lemma;
* one empty reading that is not classified as an omission;
* one repeated witness reference that can be normalized but should be reported.

The output separates errors from warnings and marks each apparatus record as
valid or invalid.

<context source = "yes">
\mainlanguage[en]

\setuppapersize[A5]

\setupbodyfont
  [libertinus,10pt]

\setupwhitespace
  [medium]

\startbuffer[tei-data]
<TEI xmlns="http://www.tei-c.org/ns/1.0">
  <teiHeader>
    <fileDesc>
      <titleStmt>
        <title>Guide 5 project-level checks</title>
      </titleStmt>
      <publicationStmt>
        <p>Unpublished teaching example.</p>
      </publicationStmt>
      <sourceDesc>
        <listWit>
          <witness xml:id="ms-A" n="A" type="manuscript">
            Paris, Bibliothèque nationale de France, grec 1807
          </witness>
          <witness xml:id="ms-B" n="B" type="manuscript">
            Vatican City, Biblioteca Apostolica Vaticana, Vat. gr. 1
          </witness>
          <witness xml:id="ed-C" n="C" type="edition">
            Aldine edition, Venice, 1513
          </witness>
        </listWit>
      </sourceDesc>
    </fileDesc>
    <encodingDesc>
      <variantEncoding
        method="parallel-segmentation"
        location="internal"/>
    </encodingDesc>
  </teiHeader>
  <text>
    <body>
      <p>
        <app xml:id="app-001">
          <lem wit="#ms-A">λόγος</lem>
          <rdg wit="#ms-B #ed-C">λέξις</rdg>
        </app>
      </p>
      <p>
        <app xml:id="app-002">
          <lem wit="#ms-A">ἀλήθεια</lem>
          <rdg wit="#ms-B #ms-D">ἀληθεία</rdg>
        </app>
      </p>
      <p>
        <app xml:id="app-003">
          <rdg wit="#ms-B">φωνή</rdg>
        </app>
      </p>
      <p>
        <app xml:id="app-004">
          <lem wit="#ms-A #ms-B">σοφία</lem>
          <rdg wit="#ed-C"/>
        </app>
      </p>
      <p>
        <app xml:id="app-005">
          <lem wit="#ms-A">ψυχή</lem>
          <rdg wit="#ms-B #ms-B #ed-C">πνοή</rdg>
        </app>
      </p>
    </body>
  </text>
</TEI>
\stopbuffer

\startluacode

xml.registerns(
    "tei",
    "http://www.tei-c.org/ns/1.0"
)


thirddata = thirddata or {}

thirddata.tei_validation = {
    witnesses = {
        by_id    = {},
        in_order = {},
    },
    records  = {},
    errors   = {},
    warnings = {},
}

local validation = thirddata.tei_validation

local function normalize_space(value)
    if not value then
        return ""
    end

    value = value:gsub("%s+", " ")
    value = value:gsub("^%s+", "")
    value = value:gsub("%s+$", "")

    return value
end

local function add_error(message)
    validation.errors[#validation.errors + 1] = message
end

local function add_warning(message)
    validation.warnings[#validation.warnings + 1] = message
end

local function normalize_witnesses(value)
    local result     = {}
    local seen       = {}
    local duplicates = {}

    value = normalize_space(value)

    for reference in value:gmatch("%S+") do
        local id = reference:gsub("^#", "")

        if id ~= "" then
            if seen[id] then
                duplicates[#duplicates + 1] = id
            else
                result[#result + 1] = id
                seen[id] = true
            end
        end
    end

    return result, duplicates
end

local function build_witness_registry(document)
    validation.witnesses.by_id    = {}
    validation.witnesses.in_order = {}

    for node in xml.collected(document, "tei:witness") do
        local attributes = node.at or {}

        local id =
            normalize_space(
                attributes["xml:id"]
                or attributes.id
            )

        if id == "" then
            add_error(
                "A witness declaration has no xml:id."
            )
        elseif validation.witnesses.by_id[id] then
            add_error(
                "Duplicate witness identifier: " .. id
            )
        else
            local siglum =
                normalize_space(attributes.n)

            if siglum == "" then
                siglum = id
            end

            local record = {
                id          = id,
                siglum      = siglum,
                kind        =
                    normalize_space(attributes.type)
                    ~= ""
                    and normalize_space(attributes.type)
                    or "unspecified",
                description =
                    normalize_space(xml.text(node)),
                order       =
                    #validation.witnesses.in_order + 1,
                source      = node,
            }

            validation.witnesses.by_id[id] = record

            validation.witnesses.in_order[
                #validation.witnesses.in_order + 1
            ] = record
        end
    end
end

local function make_text_record(node, default_kind)
    if not node then
        return nil
    end

    local attributes = node.at or {}

    local witnesses, duplicates =
        normalize_witnesses(attributes.wit)

    local kind =
        normalize_space(attributes.type)

    if kind == "" then
        kind = default_kind
    end

    return {
        kind       = kind,
        text       = normalize_space(xml.text(node)),
        witnesses  = witnesses,
        duplicates = duplicates,
        source     = node,
    }
end

local function build_apparatus_records(document)
    validation.records = {}

    local position = 0

    for node in xml.collected(document, "tei:app") do
        position = position + 1

        local attributes = node.at or {}

        local id =
            normalize_space(
                attributes["xml:id"]
                or attributes.id
            )

        if id == "" then
            id = "app-" .. position

            add_warning(
                id .. ": generated identifier."
            )
        end

        local app_kind =
            normalize_space(attributes.type)

        if app_kind == "" then
            app_kind = "variation"
        end

        local record = {
            id       = id,
            kind     = app_kind,
            lemma    =
                make_text_record(
                    xml.first(node, "tei:lem"),
                    "lemma"
                ),
            readings = {},
            order    = position,
            source   = node,
            valid    = true,
        }

        for reading_node in xml.collected(node, "tei:rdg") do
            record.readings[#record.readings + 1] =
                make_text_record(
                    reading_node,
                    app_kind == "omission"
                    and "omission"
                    or "reading"
                )
        end

        validation.records[
            #validation.records + 1
        ] = record
    end
end

local function validate_witness_list(
    record,
    list,
    location
)
    for _, id in ipairs(list) do
        if not validation.witnesses.by_id[id] then
            add_error(
                string.format(
                    "%s: unknown witness %s in %s.",
                    record.id,
                    id,
                    location
                )
            )

            record.valid = false
        end
    end
end

local function report_duplicates(
    record,
    duplicates,
    location
)
    for _, id in ipairs(duplicates) do
        add_warning(
            string.format(
                "%s: duplicate witness %s in %s; "
                .. "the repeated reference was removed.",
                record.id,
                id,
                location
            )
        )
    end
end

local function validate_record(record)
    if not record.lemma then
        add_error(
            record.id .. ": missing lemma."
        )

        record.valid = false
    else
        if record.lemma.text == ""
           and record.kind ~= "addition" then
            add_error(
                record.id
                .. ": the lemma is empty under the current "
                .. "project profile."
            )

            record.valid = false
        end

        validate_witness_list(
            record,
            record.lemma.witnesses,
            "lemma"
        )

        report_duplicates(
            record,
            record.lemma.duplicates,
            "lemma"
        )
    end

    if #record.readings == 0 then
        add_warning(
            record.id .. ": no variant readings."
        )
    end

    for index, reading in ipairs(record.readings) do
        local location =
            "reading " .. index

        validate_witness_list(
            record,
            reading.witnesses,
            location
        )

        report_duplicates(
            record,
            reading.duplicates,
            location
        )

        if reading.kind == "omission" then
            if reading.text ~= "" then
                add_warning(
                    string.format(
                        "%s: %s is classified as an "
                        .. "omission but contains text.",
                        record.id,
                        location
                    )
                )
            end
        elseif reading.text == "" then
            add_error(
                string.format(
                    "%s: %s is empty and is not "
                    .. "classified as an omission.",
                    record.id,
                    location
                )
            )

            record.valid = false
        end
    end
end

function validation.build()
    validation.errors   = {}
    validation.warnings = {}
    validation.records  = {}

    local document = xml.convert(
        buffers.getcontent("tei-data")
    )

    build_witness_registry(document)
    build_apparatus_records(document)

    for _, record in ipairs(validation.records) do
        validate_record(record)
    end
end

local function typeset_status(record)
    if record.valid then
        context("valid")
    else
        context("invalid")
    end
end

function validation.typeset()
    context.subject("Validation summary")

    context.starttabulate {
        "|l|l|l|r|"
    }

    context.NC()
    context.bold("Record")
    context.NC()
    context.bold("Status")
    context.NC()
    context.bold("Lemma")
    context.NC()
    context.bold("Readings")
    context.NC()
    context.NR()

    for _, record in ipairs(validation.records) do
        context.NC()
        context(record.id)

        context.NC()
        typeset_status(record)

        context.NC()

        if record.lemma then
            context(record.lemma.text)
        else
            context("missing")
        end

        context.NC()
        context(#record.readings)

        context.NC()
        context.NR()
    end

    context.stoptabulate()

    context.blank()

    context.subject("Errors")

    if #validation.errors == 0 then
        context("No validation errors were detected.")
    else
        context.startitemize()

        for _, message in ipairs(validation.errors) do
            context.startitem()
            context(message)
            context.stopitem()
        end

        context.stopitemize()
    end

    context.blank()

    context.subject("Warnings")

    if #validation.warnings == 0 then
        context("No validation warnings were produced.")
    else
        context.startitemize()

        for _, message in ipairs(validation.warnings) do
            context.startitem()
            context(message)
            context.stopitem()
        end

        context.stopitemize()
    end
end

\stopluacode

\starttext

\subject{Validating witness references and apparatus records}

The following report compares normalized apparatus records with the witness
registry and applies a small set of project-level editorial checking rules.

\blank

\ctxlua{
    thirddata.tei_validation.build()
    thirddata.tei_validation.typeset()
}

\stoptext
</context>

<!-- Recompile the revised project-check MWE and update its screenshot here. -->

The expected validation summary marks:

* <code>app-001</code> as valid;
* <code>app-002</code> as invalid because witness <code>D</code> has not
  been declared;
* <code>app-003</code> as invalid because it has no lemma;
* <code>app-004</code> as invalid because its empty reading is not classified
  as an omission;
* <code>app-005</code> as valid, but accompanied by a warning because witness
  <code>B</code> was repeated in the original <code>@wit</code> value.

The repeated reference in <code>app-005</code> is removed during
normalization. The warning nevertheless preserves knowledge of the irregularity
found in the source.

=== 7.5. Result of Stage 3 ===

At the end of this stage, the processing layer knows more than the XML parser
alone can establish.

It knows:

* which witness references denote declared editorial objects;
* which apparatus records satisfy the required structural rules;
* which records cannot safely be passed to typesetting;
* which irregularities were normalized without invalidating the record;
* which diagnostic messages must be presented to the editor.

Project-level checking therefore produces a new processing state.

The original TEI document contains encoded claims and relationships. The Lua
layer tests those claims against the declarations and rules of the edition.

A reference is no longer merely syntactically present: it is either resolved or
unresolved.

An apparatus entry is no longer merely well-formed: it is either editorially
usable, usable with warnings, or invalid.

The next section will combine witness registration, apparatus conversion,
reference resolution, and validation in a single complete workflow.



<div style="background:#eef7ee; border:1px solid #a8c7a8; border-left:5px solid #5b8a5a; padding:0.8em 1em; margin:1em 0;">
'''Stage reached.'''

The processing layer can now distinguish accepted editorial objects from
records that require correction, while preserving diagnostic information about
the latter.
</div>


== 8. Assemble the complete workflow ==

The three stages can now be combined while remaining conceptually distinct.

=== 8.1. The cumulative processing map ===

<pre>
                         TEI SOURCE
                             │
             +---------------+---------------+
             │                               │
             ▼                               ▼
     witness declarations              apparatus entries
             │                               │
             ▼                               ▼
       Lua registry                  Lua apparatus records
             │                               │
             │                         normalization
             │                               │
             +-------------+-----------------+
                           │
                           ▼
                 reference resolution
                           │
                           ▼
                      validation
                           │
                  +--------+--------+
                  │                 │
                  ▼                 ▼
              accepted           invalid
               records            records
                  │                 │
                  │                 └── diagnostics
                  ▼
             ConTeXt input
                  │
                  ▼
            TYPOGRAPHY
</pre>

The important collections are:

<pre>
processing.records
    │
    └── every constructed apparatus record
        including invalid records retained for inspection

processing.accepted
    │
    └── records that have passed validation
        and may be passed safely to typesetting
</pre>

This prevents bad data from silently reaching the final apparatus without
throwing away the information needed to diagnose it.


=== 8.2. Preserve normalized identifiers and resolved objects ===

A useful record can retain both levels:

<pre>
reading
├── witness_ids
│   └── { "ms-B", "ed-C" }
│
└── witnesses
    ├── witness record B
    └── witness record C
</pre>

Thus:

<pre>
ENCODED REFERENCE
      │
      ▼
NORMALIZED IDENTIFIER
      │
      ▼
RESOLVED OBJECT
</pre>

The intermediate representation remains traceable to the source while also
being convenient for later processing.


=== 8.3. Diagnostic reports preserve failures ===

The processing system should expose both success and failure:

<pre>
             ALL APPARATUS RECORDS
                     │
                  validate
                     │
          +----------+----------+
          │                     │
          ▼                     ▼
       accepted               invalid
          │                     │
          ▼                     ▼
processing.accepted      retained in records
          │                     │
          ▼                     └── diagnostics
       ConTeXt
</pre>

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''Preserve both success and failure.'''

Accepted records protect the typesetting stage.
Rejected records remain traceable so that the editor can inspect and correct
the source or the processing rules.
</div>


=== 8.4. Complete functional MWE: From TEI apparatus data to checked Lua records ===

The following MWE performs the complete processing sequence developed in this
guide.

<context source = "yes">
\mainlanguage[en]

\setuppapersize[A5]

\setupbodyfont
  [libertinus,10pt]

\setupwhitespace
  [medium]

\startbuffer[tei-data]
<TEI xmlns="http://www.tei-c.org/ns/1.0">
  <teiHeader>
    <fileDesc>
      <titleStmt>
        <title>Guide 5 complete processing workflow</title>
      </titleStmt>
      <publicationStmt>
        <p>Unpublished teaching example.</p>
      </publicationStmt>
      <sourceDesc>
        <listWit>
          <witness xml:id="ms-A" n="A" type="manuscript">
            Paris, Bibliothèque nationale de France, grec 1807
          </witness>
          <witness xml:id="ms-B" n="B" type="manuscript">
            Vatican City, Biblioteca Apostolica Vaticana, Vat. gr. 1
          </witness>
          <witness xml:id="ed-C" n="C" type="edition">
            Aldine edition, Venice, 1513
          </witness>
        </listWit>
      </sourceDesc>
    </fileDesc>
    <encodingDesc>
      <variantEncoding
        method="parallel-segmentation"
        location="internal"/>
    </encodingDesc>
  </teiHeader>
  <text>
    <body>
      <p>
        <app xml:id="app-001">
          <lem wit="#ms-A">λόγος</lem>
          <rdg wit="#ms-B #ed-C">λέξις</rdg>
        </app>
      </p>
      <p>
        <app xml:id="app-002" type="omission">
          <lem wit="#ms-A #ms-B">ἀλήθεια</lem>
          <rdg wit="#ed-C"/>
        </app>
      </p>
      <p>
        <app xml:id="app-003">
          <lem wit="#ms-A">ψυχή</lem>
          <rdg wit="#ms-B #ms-D">πνοή</rdg>
        </app>
      </p>
    </body>
  </text>
</TEI>
\stopbuffer

\startluacode

xml.registerns(
    "tei",
    "http://www.tei-c.org/ns/1.0"
)


thirddata = thirddata or {}

thirddata.tei_complete = {
    witnesses = {
        by_id    = {},
        in_order = {},
    },

    records  = {},
    accepted = {},
    errors   = {},
    warnings = {},
}

local processing = thirddata.tei_complete

local function normalize_space(value)
    if not value then
        return ""
    end

    value = value:gsub("%s+", " ")
    value = value:gsub("^%s+", "")
    value = value:gsub("%s+$", "")

    return value
end

local function add_global_error(message)
    processing.errors[
        #processing.errors + 1
    ] = message
end

local function add_global_warning(message)
    processing.warnings[
        #processing.warnings + 1
    ] = message
end

local function add_record_error(record, message)
    record.errors[
        #record.errors + 1
    ] = message

    record.valid = false

    add_global_error(
        record.id .. ": " .. message
    )
end

local function add_record_warning(record, message)
    record.warnings[
        #record.warnings + 1
    ] = message

    add_global_warning(
        record.id .. ": " .. message
    )
end

local function reset_processing()
    processing.witnesses.by_id    = {}
    processing.witnesses.in_order = {}
    processing.records            = {}
    processing.accepted           = {}
    processing.errors             = {}
    processing.warnings           = {}
end

local function normalize_witnesses(value)
    local identifiers = {}
    local duplicates  = {}
    local seen        = {}

    value = normalize_space(value)

    for reference in value:gmatch("%S+") do
        local id =
            reference:gsub("^#", "")

        if id ~= "" then
            if seen[id] then
                duplicates[
                    #duplicates + 1
                ] = id
            else
                identifiers[
                    #identifiers + 1
                ] = id

                seen[id] = true
            end
        end
    end

    return identifiers, duplicates
end

local function build_witness_registry(document)
    for node in xml.collected(document, "tei:witness") do
        local attributes = node.at or {}

        local id =
            normalize_space(
                attributes["xml:id"]
                or attributes.id
            )

        if id == "" then
            add_global_error(
                "A witness declaration has no xml:id."
            )
        elseif processing.witnesses.by_id[id] then
            add_global_error(
                "Duplicate witness identifier: " .. id
            )
        else
            local kind =
                normalize_space(
                    attributes.type
                )

            local siglum =
                normalize_space(attributes.n)

            if siglum == "" then
                siglum = id
            end

            local record = {
                id          = id,
                siglum      = siglum,
                kind        =
                    kind ~= ""
                    and kind
                    or "unspecified",
                description =
                    normalize_space(
                        xml.text(node)
                    ),
                order       =
                    #processing.witnesses.in_order + 1,
                source      = node,
            }

            processing.witnesses.by_id[id] =
                record

            processing.witnesses.in_order[
                #processing.witnesses.in_order + 1
            ] = record
        end
    end
end

local function make_text_record(
    node,
    default_kind
)
    if not node then
        return nil
    end

    local attributes = node.at or {}

    local witness_ids, duplicates =
        normalize_witnesses(
            attributes.wit
        )

    local kind =
        normalize_space(
            attributes.type
        )

    if kind == "" then
        kind = default_kind
    end

    return {
        kind        = kind,
        text        =
            normalize_space(
                xml.text(node)
            ),
        witness_ids = witness_ids,
        witnesses   = {},
        duplicates  = duplicates,
        valid       = true,
        source      = node,
    }
end

local function build_apparatus_records(document)
    local position = 0

    for node in xml.collected(document, "tei:app") do
        position = position + 1

        local attributes = node.at or {}

        local id =
            normalize_space(
                attributes["xml:id"]
                or attributes.id
            )

        local generated_id = false

        if id == "" then
            id = "app-" .. position
            generated_id = true
        end

        local app_kind =
            normalize_space(attributes.type)

        if app_kind == "" then
            app_kind = "variation"
        end

        local record = {
            id       = id,
            kind     = app_kind,
            lemma    =
                make_text_record(
                    xml.first(node, "tei:lem"),
                    "lemma"
                ),
            readings = {},
            order    = position,
            valid    = true,
            errors   = {},
            warnings = {},
            source   = node,
        }

        if generated_id then
            add_record_warning(
                record,
                "generated apparatus identifier."
            )
        end

        for reading_node in xml.collected(node, "tei:rdg") do
            record.readings[
                #record.readings + 1
            ] = make_text_record(
                reading_node,
                "reading"
            )
        end

        processing.records[
            #processing.records + 1
        ] = record
    end
end

local function resolve_witnesses(
    record,
    text_record,
    location
)
    text_record.witnesses = {}

    for _, id in ipairs(
        text_record.witness_ids
    ) do
        local witness =
            processing.witnesses.by_id[id]

        if witness then
            text_record.witnesses[
                #text_record.witnesses + 1
            ] = witness
        else
            text_record.valid = false

            add_record_error(
                record,
                string.format(
                    "unknown witness %s in %s.",
                    id,
                    location
                )
            )
        end
    end

    for _, id in ipairs(
        text_record.duplicates
    ) do
        add_record_warning(
            record,
            string.format(
                "duplicate witness %s in %s; "
                .. "the repeated reference was removed.",
                id,
                location
            )
        )
    end
end

local function validate_record(record)
    if not record.lemma then
        add_record_error(
            record,
            "missing lemma."
        )
    else
        if record.lemma.text == ""
           and record.kind ~= "addition" then
            add_record_error(
                record,
                "the lemma is empty under the current "
                .. "project profile."
            )
        end

        resolve_witnesses(
            record,
            record.lemma,
            "lemma"
        )
    end

    if #record.readings == 0 then
        add_record_warning(
            record,
            "no variant readings."
        )
    end

    for index, reading in ipairs(
        record.readings
    ) do
        local location =
            "reading " .. index

        resolve_witnesses(
            record,
            reading,
            location
        )

        if reading.kind == "omission" then
            if reading.text ~= "" then
                add_record_warning(
                    record,
                    location
                    .. " is classified as an omission "
                    .. "but contains text."
                )
            end
        elseif reading.text == "" then
            reading.valid = false

            add_record_error(
                record,
                location
                .. " is empty and is not classified "
                .. "as an omission."
            )
        end
    end

    if record.valid then
        processing.accepted[
            #processing.accepted + 1
        ] = record
    end
end

function processing.build()
    reset_processing()

    local document = xml.convert(
        buffers.getcontent("tei-data")
    )

    build_witness_registry(document)
    build_apparatus_records(document)

    for _, record in ipairs(
        processing.records
    ) do
        validate_record(record)
    end
end

local function typeset_identifier_list(list)
    if #list == 0 then
        context("none")
    else
        context(
            table.concat(list, ", ")
        )
    end
end

local function resolved_sigla(list)
    local sigla = {}

    for _, witness in ipairs(list) do
        sigla[#sigla + 1] =
            witness.siglum
    end

    return sigla
end

local function typeset_status(valid)
    if valid then
        context("valid")
    else
        context("invalid")
    end
end

local function typeset_messages(
    title,
    messages,
    empty_message
)
    context.subject(title)

    if #messages == 0 then
        context(empty_message)
        return
    end

    context.startitemize()

    for _, message in ipairs(messages) do
        context.startitem()
        context(message)
        context.stopitem()
    end

    context.stopitemize()
end

function processing.typeset_summary()
    context.subject("Processing summary")

    context.starttabulate {
        "|l|r|"
    }

    context.NC()
    context("Declared witnesses")
    context.NC()
    context(#processing.witnesses.in_order)
    context.NC()
    context.NR()

    context.NC()
    context("Apparatus records")
    context.NC()
    context(#processing.records)
    context.NC()
    context.NR()

    context.NC()
    context("Accepted records")
    context.NC()
    context(#processing.accepted)
    context.NC()
    context.NR()

    context.NC()
    context("Errors")
    context.NC()
    context(#processing.errors)
    context.NC()
    context.NR()

    context.NC()
    context("Warnings")
    context.NC()
    context(#processing.warnings)
    context.NC()
    context.NR()

    context.stoptabulate()

    context.blank()

    context.subject("Record status")

    context.starttabulate {
        "|l|l|l|r|"
    }

    context.NC()
    context.bold("Record")
    context.NC()
    context.bold("Status")
    context.NC()
    context.bold("Lemma")
    context.NC()
    context.bold("Readings")
    context.NC()
    context.NR()

    for _, record in ipairs(
        processing.records
    ) do
        context.NC()
        context(record.id)

        context.NC()
        typeset_status(record.valid)

        context.NC()

        if record.lemma then
            context(record.lemma.text)
        else
            context("missing")
        end

        context.NC()
        context(#record.readings)

        context.NC()
        context.NR()
    end

    context.stoptabulate()

    context.blank()

    typeset_messages(
        "Errors",
        processing.errors,
        "No processing errors were detected."
    )

    context.blank()

    typeset_messages(
        "Warnings",
        processing.warnings,
        "No processing warnings were produced."
    )
end

function processing.typeset_records()
    context.subject(
        "Validated intermediate records"
    )

    for _, record in ipairs(
        processing.records
    ) do
        context.subsubject(record.id)

        context.bold("Status: ")
        typeset_status(record.valid)
        context.par()

        if record.lemma then
            context.bold("Lemma: ")
            context(record.lemma.text)
            context.par()

            context.bold(
                "Normalized lemma references: "
            )

            typeset_identifier_list(
                record.lemma.witness_ids
            )

            context.par()

            context.bold(
                "Resolved lemma witnesses: "
            )

            typeset_identifier_list(
                resolved_sigla(
                    record.lemma.witnesses
                )
            )

            context.par()
        else
            context.bold("Lemma: ")
            context("missing")
            context.par()
        end

        for index, reading in ipairs(
            record.readings
        ) do
            context.bold(
                "Reading %s kind: ",
                index
            )
            context(reading.kind)
            context.par()

            context.bold(
                "Reading %s text: ",
                index
            )

            if reading.text ~= "" then
                context(reading.text)
            elseif reading.kind == "omission" then
                context("omission")
            else
                context("empty")
            end

            context.par()

            context.bold(
                "Normalized reading references: "
            )

            typeset_identifier_list(
                reading.witness_ids
            )

            context.par()

            context.bold(
                "Resolved reading witnesses: "
            )

            typeset_identifier_list(
                resolved_sigla(
                    reading.witnesses
                )
            )

            context.par()
        end
    end
end

\stopluacode

\starttext

\subject{From TEI apparatus data to checked Lua records}

This report shows the complete processing sequence from TEI witness and
apparatus data to normalized, resolved, and checked Lua records.

\blank

\ctxlua{
    thirddata.tei_complete.build()
    thirddata.tei_complete.typeset_summary()
    thirddata.tei_complete.typeset_records()
}

\stoptext
</context>

<!-- Recompile the revised complete MWE and update page 1 of its screenshot here. -->

<!-- Recompile the revised complete MWE and update page 2 of its screenshot here. -->

The expected summary contains:

* three declared witnesses;
* three apparatus records;
* two accepted records;
* one error;
* no warnings.

The records <code>app-001</code> and <code>app-002</code> are accepted.

In <code>app-001</code>, the normalized identifiers
<code>ms-B</code> and <code>ed-C</code> are resolved to witness records
whose reader-facing labels are <code>B</code> and <code>C</code>.

In <code>app-002</code>, the empty reading is accepted because its
<code>kind</code> is explicitly recorded as <code>omission</code>.

The record <code>app-003</code> is rejected because the reference
<code>ms-D</code> cannot be resolved to a declared witness. Its normalized identifiers remain visible,
and the successfully resolved witness <code>B</code> is retained, but the
record is not included in:

<pre>
processing.accepted
</pre>

=== 8.5. Records prepared for ConTeXt ===

The collection:

<pre>
processing.records
</pre>

contains every apparatus record constructed from the TEI source, including
invalid records retained for inspection.

The collection:

<pre>
processing.accepted
</pre>

contains only records that have passed validation and may safely be transmitted
to the typesetting layer.

This distinction prevents invalid data from silently reaching the final
apparatus while preserving enough information for diagnosis and correction.

A record prepared for ConTeXt now possesses:

* a stable apparatus identifier;
* normalized lemma and reading text;
* explicit editorial categories;
* normalized witness identifiers;
* resolved witness objects;
* documentary order;
* source-node references;
* validity information;
* errors and warnings.

ConTeXt will therefore not need to rediscover the meaning of
<code>@wit</code>, search the TEI witness declarations, distinguish omissions
from missing data, or decide whether unresolved references should be printed.

The next guide can begin from records whose editorial status has already been
made explicit.

=== 8.6. Result of the complete workflow ===

The complete workflow does not merely transfer information from one notation
to another.

It successively changes the epistemic status of the data:

{| class="wikitable"
! Stage
! Status of the data
|-
| TEI encoding
| Witnesses and variants are documented through elements, attributes, identifiers, and references.
|-
| Extraction
| The relevant XML structures become accessible to Lua.
|-
| Normalization
| Notational variations are reduced to controlled internal forms.
|-
| Reference resolution
| Character strings become relations to known editorial objects.
|-
| Validation
| Encoded claims are tested against declarations and editorial rules.
|-
| Intermediate representation
| The accepted records form an explicit and reusable body of editorial knowledge.
|}

The final Lua records are therefore not neutral copies of the XML source.

They are the result of documented interpretative operations whose assumptions,
successes, and failures can be inspected.

This intermediate representation is what makes a reliable separation between
scholarly data processing and typographical composition possible.

<div style="background:#eef5fb; border:1px solid #9bbbd4; border-left:6px solid #4f81a8; padding:0.8em 1em; margin:1em 0;">
'''Stage reached.''' the data has not yet been typeset, but its editorial status
has been made explicit. This is the stable point at which Guide 6 can take over.
</div>



== Part III — Define the boundaries ==

== 9. What Lua should not decide ==

The processing layer now performs substantial interpretative work.
That does not mean that every decision belongs in Lua.

The three layers should remain distinguishable. This repetition of the
TEI → Lua → ConTeXt architecture is intentional: here the diagram no longer
explains what each layer ''can'' do, but what each layer should ''not
silently take over'' from the others.

<pre>
TEI
│
├── evidence
├── documentary distinctions
└── scholarly assertions
        │
        ▼
Lua
│
├── extract
├── normalize
├── resolve
├── validate
└── diagnose
        │
        ▼
ConTeXt
│
├── typography
├── punctuation
├── spacing
├── placement
└── page architecture
</pre>


=== 9.1. Lua should not silently repair the TEI source ===

If the source contains:

<pre>
<rdg wit="#ms-D">πνοή</rdg>
</pre>

but no witness D has been declared, Lua may report:

<pre>
app-003: unknown witness D in reading 1.
</pre>

It should not silently invent witness D.

That would turn programmatic invention into apparently documented evidence.

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.8em 1em; margin:1em 0;">
'''Do not repair silently.'''

A controlled default may be legitimate when the project explicitly authorizes
it, but the intervention should remain visible through a warning, generated
value, or another diagnostic field.
</div>


=== 9.2. Lua should not erase significant TEI distinctions ===

These should not all become:

<pre>
text = ""
</pre>

without further qualification:

<pre>
omission
lacuna
unreadable text
absent data
editorial suppression
processing failure
</pre>

A record should retain the relevant category:

<pre>
{
    kind = "omission",
    text = "",
}
</pre>

or, where appropriate:

<pre>
{
    kind = "lacuna",
    text = "",
}
</pre>

The reduction from TEI to Lua should therefore be selective rather than
destructive.


=== 9.3. Lua should not determine typography prematurely ===

A record may state:

<pre>
kind = "omission"
</pre>

It should not normally need to state:

<pre>
print the abbreviation "om."
use an italic font
insert exactly 0.5 em space
</pre>

Those are ConTeXt decisions.

The relation should remain:

<pre>
EDITORIAL CATEGORY
kind = "omission"
       │
       ▼
ConTeXt rendering policy
       │
       ├── om.
       ├── omitted
       ├── ∅
       └── another documented convention
</pre>

Keeping these decisions separate allows typography to change without
reconstructing the editorial records.


=== 9.4. Lua should not replace editorial judgment ===

Lua can test whether:

* a witness has been declared;
* a reference resolves;
* required fields exist;
* a category belongs to a controlled set;
* a record satisfies declared consistency rules.

It cannot determine every scholarly question, for example:

* whether two readings should be grouped together;
* whether a difficult reading is genuinely conjectural;
* whether a witness should be considered primary or secondary;
* whether an apparatus boundary is philologically appropriate.

<div style="background:#fff8d6; border:1px solid #d6b84a; border-left:5px solid #c49a00; padding:0.8em 1em; margin:1em 0;">
'''Project checks make assumptions explicit; they do not replace scholarship.'''

Lua can test the formal consequences of editorial rules. The rules themselves
still depend on scholarly judgment.
</div>


== 10. Prepare the Lua–ConTeXt contract ==

The checked Lua records now form a stable boundary between interpretation and
composition.

=== 10.1. The Lua–ConTeXt contract ===

A record passed to ConTeXt may now contain:

<pre>
record
├── id
├── kind
├── lemma
│   ├── text
│   ├── witness_ids
│   └── resolved witnesses
│
├── readings
│   ├── text
│   ├── kind
│   ├── witness_ids
│   └── resolved witnesses
│
├── responsibility   (when processed)
├── certainty          (when processed)
├── documentary        (when processed)
├── order
├── source
├── valid
└── diagnostics
</pre>

ConTeXt therefore does not need to:

* parse <code>@wit</code>;
* remove reference markers;
* search <code>&lt;listWit&gt;</code>;
* decide whether C exists;
* distinguish an omission from unexplained missing text;
* determine whether an unresolved record should be printed.

It receives records whose editorial status is already explicit.


=== 10.2. What ConTeXt will decide ===

Guide 6 can now concentrate on questions such as:

* how lemmas and readings are formatted;
* how witness sigla are separated;
* how omissions are displayed;
* how apparatus entries are ordered and grouped;
* how several apparatus layers are distinguished;
* where notes and apparatus blocks are placed;
* how page and line breaking are controlled;
* how the same editorial data adapts to different layouts.

The transition is:

<pre>
GUIDES 1–4
BUILD TEI
      │
      ▼
GUIDE 5
INTERPRET + VALIDATE
      │
      ▼
checked Lua records
      │
      ▼
GUIDE 6
COMPOSE
</pre>


=== 10.3. Preparing complex scholarly pages ===

The same intermediate model can support richer passage records containing, for
example:

<pre>
{
    passage_id = "passage-001",

    texts = {
        greek       = { ... },
        latin       = { ... },
        translation = { ... },
    },

    apparatus = {
        textual     = { ... },
        sources     = { ... },
        translation = { ... },
    },

    bibliography = { ... },
    alignment    = { ... },

    validation = {
        valid    = true,
        errors   = {},
        warnings = {},
    },
}
</pre>

The point is not to reproduce the final page as a giant typesetting macro.
The processing layer prepares structured editorial objects from which ConTeXt
can construct several coordinated typographical regions.


== 11. What you have built ==

Guide 5 has introduced a conceptual shift.

The first four guides primarily built encoded scholarly structures.
This guide has shown how those structures become controlled editorial objects.

=== 11.1. The cumulative transformation ===

<pre>
TEI ENCODING
witnesses + apparatus
        │
        ▼
EXTRACTION
accessible XML structures
        │
        ▼
NORMALIZATION
controlled identifiers and values
        │
        ▼
REFERENCE RESOLUTION
strings become known objects
        │
        ▼
VALIDATION
claims tested against rules
        │
        ▼
DIAGNOSTICS
successes and failures remain visible
        │
        ▼
INTERMEDIATE REPRESENTATION
validated editorial knowledge
        │
        ▼
ConTeXt
</pre>


=== 11.2. The status of the information changes at every stage ===

A witness value can move through:

<pre>
encoded string
"#ms-A #ms-B"
    │
    ▼
normalized identifiers
{ "ms-A", "ms-B" }
    │
    ▼
resolved objects
{ witness_A, witness_B }
    │
    ▼
validated relation
    │
    ▼
editorial knowledge
ready for typesetting
</pre>

Normalization and validation are therefore not merely technical cleanup.
They are documented interpretative operations.


=== 11.3. Diagnostics preserve the limits of interpretation ===

An invalid record should not disappear without explanation.

The processing layer can preserve:

<pre>
invalid record
├── normalized fields
├── successfully resolved references
├── unresolved references
├── source connection
└── diagnostic messages
</pre>

Thus failure remains inspectable and correctable.


=== 11.4. The complete six-guide architecture is now visible ===

<pre>
GUIDE 1
TEI structure
    │
    ▼
GUIDE 2
witness identities
    │
    ▼
GUIDE 3
basic apparatus relations
    │
    ▼
GUIDE 4
complex textual relations
    │
    ▼
GUIDE 5
normalized + resolved + checked records
    │
    ▼
GUIDE 6
page composition
</pre>

<div style="background:#eef7ee; border:1px solid #a8c7a8; border-left:5px solid #5b8a5a; padding:0.8em 1em; margin:1em 0;">
'''Guide 5 complete.'''

The data has not yet received its final typography, but its editorial status is
explicit. This is the stable point at which Guide 6 can take over.
</div>


=== 11.5. Continue with Guide 6 ===

The central question is no longer:

<pre>
What does this TEI data mean?
</pre>

Guide 5 has established a controlled answer in the intermediate records.

The next question is:

<pre>
How should this validated editorial knowledge
be composed on the page?
</pre>

Guide 6 shows how ConTeXt can turn those records into a scholarly page,
including:

* formatted lemmas, readings, and witness sigla;
* representations of omissions and other editorial categories;
* one or more apparatus layers;
* coordinated source text and translation;
* bibliographical notes;
* complex page architectures whose documentary richness remains legible.

: '''Next: Guide 6 —''' [[Typesetting a TEI critical apparatus with ConTeXt]]


----


<div style="margin:1.5em 0 1em 0; padding:0.7em 1em; border:1px solid #a7c7e7; background:#eef6fc;">
'''Guide 5 of 6 — Processing TEI critical apparatus data with Lua'''

'''Previous: Guide 4 —''' [[Encoding complex textual variation in TEI]]
&nbsp;·&nbsp;
[[Building critical editions from TEI XML|Collection overview]]
&nbsp;·&nbsp;
[[Glossary of terms used in the TEI XML critical edition guides|Glossary]]
&nbsp;·&nbsp;
'''Next: Guide 6 —''' [[Typesetting a TEI critical apparatus with ConTeXt]]
</div>


<!--
Glossary follow-up after Guide 5:
verify or add:
LuaMetaTeX; LMTX; Lua table; Lua record; witness registry; normalization;
reference resolution; resolved object; TEI validation; project-level
editorial checking / project validation; diagnostic; error; warning;
accepted record; invalid record; intermediate representation; source
connection / source traceability; processing pipeline; Lua–ConTeXt contract.

Also retain the Guide 4 vocabulary already marked for audit.
-->

== Related pages ==

* [[Building critical editions from TEI XML]]
* [[Understanding TEI documents for critical editions]]
* [[Declaring witnesses in TEI critical editions]]
* [[Encoding a basic critical apparatus in TEI]]
* [[Encoding complex textual variation in TEI]]
* [[Typesetting a TEI critical apparatus with ConTeXt]]
* [[Glossary of terms used in the TEI XML critical edition guides]]
* [[TEI xml|TEI XML]]
* [[Input and compilation/XML source|XML]]
* [[ConTeXt and Lua programming]]
* [[ConTeXt and Lua programming/Processing XML with Lua|Processing XML with Lua]]


{{DEFAULTSORT:Processing TEI critical apparatus data with Lua}}

[[Category:ConTeXt]]
[[Category:XML]]
[[Category:Lua]]
[[Category:Scholarly publishing]]
[[Category:Critical editions]]