Parse Source

The parseSource utility parses source code into HAST (Hypertext Abstract Syntax Tree) nodes with syntax highlighting using Starry Night. It converts code into highlighted HTML structures for display in documentation and demos.

import { createParseSource } from '@mui/internal-docs-infra/pipeline/parseSource';

// Initialize the parser (do this once, typically at app startup)
const parseSource = await createParseSource();

// Parse and highlight JavaScript code
const highlighted = parseSource('const x = 42;', 'example.js');

// Use the HAST tree for rendering (e.g., with hastToReact)

The parser automatically:

• Detects the language from the file extension
• Applies syntax highlighting
• Adds line gutter elements
• Returns a HAST tree ready for rendering

import { createParseSource } from '@mui/internal-docs-infra/pipeline/parseSource';

const parseSource = await createParseSource();

// JavaScript
const jsCode = parseSource('const x = 42;', 'example.js');

// TypeScript
const tsCode = parseSource('interface User { name: string; }', 'types.ts');

// CSS
const cssCode = parseSource('.button { color: blue; }', 'styles.css');

// HTML
const htmlCode = parseSource('<div>Hello</div>', 'index.html');

import { createParseSource } from '@mui/internal-docs-infra/pipeline/parseSource';

const parseSource = await createParseSource();

const files = [
  { name: 'App.tsx', content: 'export default function App() {}' },
  { name: 'styles.css', content: '.app { margin: 0; }' },
  { name: 'utils.ts', content: 'export const helper = () => {};' },
];

const highlighted = files.map((file) => ({
  name: file.name,
  hast: parseSource(file.content, file.name),
}));

For unsupported file extensions, the parser returns plain text:

const parseSource = await createParseSource();

// Unsupported extension - returns plain text node
const result = parseSource('Some content', 'file.xyz');
// {
//   type: 'root',
//   children: [{ type: 'text', value: 'Some content' }]
// }

// File without extension - also returns plain text
const readme = parseSource('# README', 'README');

After initialization, parseSource can be used directly from anywhere:

import { createParseSource, parseSource } from '@mui/internal-docs-infra/pipeline/parseSource';

// Initialize once (e.g., in app startup)
await createParseSource();

// Use the global instance anywhere in your code
function highlightCode(code: string, fileName: string) {
  return parseSource(code, fileName); // Works without re-initialization
}

import { createParseSource } from '@mui/internal-docs-infra/pipeline/parseSource';
import { hastToReact } from '@mui/internal-docs-infra/hastUtils';

const parseSource = await createParseSource();

function HighlightedCode({ code, fileName }: { code: string; fileName: string }) {
  const hast = parseSource(code, fileName);
  const jsx = hastToReact(hast);

  return <pre>{jsx}</pre>;
}

1. Initialization: createParseSource() creates a Starry Night instance with grammar definitions
2. Language Detection: File extension is extracted and mapped to a grammar
3. Syntax Highlighting: Starry Night processes the source code and generates HAST nodes with pl-* classes
4. Syntax Extensions: Adds fine-grained di-* classes (numbers, booleans, nullish, attributes) without removing existing classes
5. Line Gutters: starryNightGutter() adds line number elements to the tree
6. Fallback: Unsupported file types return a simple text node

The Starry Night instance is cached globally using __docs_infra_starry_night_instance__ to avoid re-initialization.

Starry Night maps TextMate scopes to a small set of pl-* CSS classes (the GitHub Prettylights theme). Many token types share the same class — pl-c1 covers numbers, booleans, null, component names, and more.

During parsing, parseSource extends the highlighted tree with additive di-* classes that distinguish these cases, giving CSS full control over each token type:

di-this applies to JS/TS family grammars (.js, .ts, .tsx, .jsx, .mdx). di-bt applies to TS family grammars only (.ts, .tsx, .jsx, .mdx) — plain JS is excluded because string, number, etc. are valid variable names. di-jsx and di-jv only apply to JSX grammars (.tsx, .jsx, .mdx); di-jv is added to identifier spans (pl-smi, pl-v) appearing inside JSX expression braces ({...}), including spread arguments, arrow-function parameters, and pl-c1 member-access property names (e.g. name in {row.name}). Object property keys inside JSX expressions also receive di-jv in addition to di-op. di-ak, di-ae, di-av only apply to HTML/JSX/MDX grammars. di-pu applies to all grammars and tags pl-k operator tokens whose text is purely symbolic (it is not added to word keywords like const or if). di-op applies to JS-family grammars and tags both bare-identifier object keys (e.g. height in { height: 400 }) and string keys (e.g. 'aria-label'); string keys additionally receive di-ps. di-da, di-cp, and di-cv only apply to CSS. di-ht and di-jt are added by enhanceCodeInline (not parseSource) as wrapper spans around tag brackets and the inner pl-ent or pl-c1 span.

di-te and di-td apply to JS-family grammars and handle template-literal interpolations (`...${expr}...`). Starry Night tokenizes the whole backtick string as a single pl-s span, so without these classes the ${/} delimiters and any bare punctuation inside the expression (e.g. . in ${a.b}) inherit the string color. parseSource wraps each ${ ... } slice in a di-te span so the interpolated expression resets from the string color, and wraps the ${ and } glyphs themselves in di-td spans. Inner pl-* tokens (and additive di-* classes such as di-num on ${42}) still apply on top of the reset. Only backtick template literals are processed — a ${...} sequence inside a regular "/' string is inert text and is left untouched. Because a multi-line template literal is tokenized as one pl-s span per line, an interpolation that spans several lines is wrapped with a separate di-te slice on each line (mirroring how the string itself continues across lines).

Existing pl-* classes are never removed — <span class="pl-c1">42</span> becomes <span class="pl-c1 di-num">42</span>.

The parser supports languages based on file extensions:

See grammars.ts for the complete list of supported languages.

import { parseSource } from '@mui/internal-docs-infra/pipeline/parseSource';

// ❌ This will throw an error
parseSource('code', 'file.js');
// Error: Starry Night not initialized. Use createParseSource to create an initialized parseSource function.

// ✓ Initialize first
import { createParseSource } from '@mui/internal-docs-infra/pipeline/parseSource';
await createParseSource();
parseSource('code', 'file.js'); // Now works

const parseSource = await createParseSource();

// Handles empty content gracefully
const result = parseSource('', 'empty.js');
// Returns valid HAST tree with empty content

• Syntax highlighting - When you need to highlight code for display
• Code documentation - Preparing code examples for docs
• Build-time processing - Pre-highlighting code during build
• Custom code viewers - Building code display components

When NOT to use:

• Simple text display - Use plain <pre> tags if highlighting isn't needed

string[] | undefined

Promise<ParseSource>

string

string

string | undefined

HastRoot

string

string | undefined

string[]

Promise<void>

type extensionMap = Record<string, string>

type languageToGrammarMap = Record<string, string>

• hastUtils - For converting HAST to React JSX
• CodeHighlighter - Uses parseSource for syntax highlighting
• loadPrecomputedCodeHighlighter - Build-time optimization using parseSource