How to Use the HTML Tokenizer in Python

The tokenizer in Aspose.HTML FOSS for Python converts markup into a stream of standards-defined tokens — useful for linters, sanitizer pre-passes, and any tool that inspects markup below the tree level. This guide covers the tokenizing entry points, token vocabulary, and states, using pip for installation.

Step-by-Step Guide

Step 1: Install the Package

pip install aspose-html-foss

Step 2: Import Required Classes

from aspose_html.tokenizer import Tokenizer, TokenizerState

Step 3: Tokenize Markup

Tokenizer.tokenize() processes complete input; Tokenizer.tokenize_fragment() applies fragment rules. The stream contains typed tokens: StartTagToken and EndTagToken (with attributes), CharacterToken for text, CommentToken, DoctypeToken, and a final EofToken.

Step 4: Control the Initial State

TokenizerState mirrors the specification’s state machine. For context-sensitive input — content that would appear inside <script> or <style> — set the matching initial state with Tokenizer.set_state() before tokenizing.

Step 5: Resolve Character References

Named and numeric character references resolve through resolve_named_char_ref(), resolve_numeric_char_ref(), and find_named_char_ref_match() — backed by the full named-reference table.

Common Issues and Fixes

Script or style content tokenizes as markup. The initial state was wrong for the context; set it with set_state().

Entities appear unresolved in output. Raw text consumers must run the character-reference resolvers; resolution is explicit at this level.

Fragment output differs from a full parse. Fragment tokenization rules intentionally differ — use tokenize() for full documents.

Frequently Asked Questions

Can I consume tokens without building a tree?

Yes — the token stream is a public output, independent of tree construction.

Which token types exist?

StartTagToken, EndTagToken, CharacterToken, CommentToken, DoctypeToken, and EofToken.

Are the tokenizer states the standard ones?

Yes — TokenizerState models the specification’s state machine.