[WIP] rustdoc: Add tree-sitter syntax highlighting for non-Rust code blocks #149944
+540
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Integrate arborium (tree-sitter based highlighting) to provide syntax highlighting for non-Rust code blocks in documentation. Previously, code blocks like ```python or ```javascript were rendered as plain text. Supported languages: bash, c, cpp, css, go, html, java, javascript, json, python, ruby, sql, toml, typescript, yaml. The highlighting uses custom HTML elements (a-k for keywords, a-s for strings, etc.) which are styled via CSS to match rustdoc's existing color scheme across all themes (light, dark, ayu).
Gate the arborium-based syntax highlighting behind an unstable flag (-Z unstable-options --highlight-foreign-code) so it can be tested before becoming the default behavior. The flag is threaded through: - RenderOptions in config.rs - SharedContext in context.rs - Markdown/MarkdownWithToc structs in markdown.rs - CodeBlocks iterator for actual highlighting
Test cases: - Highlighting enabled: Python, JavaScript, JSON produce arborium tags - Language aliases work (py -> python, js -> javascript) - Highlighting disabled: no arborium tags produced - Unsupported languages fall back to plain escaped text
rustbot
added
S-waiting-on-author
This comment has been minimized.
This comment has been minimized.
Cached crates, build artifacts, cloned repositories etc. quickly sum up to tens of gigabytes of used storage capacity for a developer. From that viewpoint, the question of including either a 171 MiB vs. 22 MiB (quoted from the blog post) binary is that it does not really matter that much. Of course, other reasons may be much more important.
Going only by the viewpoint above, all would be convenient. If it's not feasible, at least assembly would be nice to have. |
Member
|
@bors try @rust-timer queue |
Collaborator
|
Awaiting bors try build completion. @rustbot label: +S-waiting-on-perf |
rust-bors bot
added a commit
that referenced
this pull request
Dec 14, 2025
[WIP] rustdoc: Add tree-sitter syntax highlighting for non-Rust code blocks
This comment has been minimized.
This comment has been minimized.
rustbot
added
the
S-waiting-on-perf
fmease
added
the
needs-fcp
Member
|
I've personally never encountered code blocks other than Rust in docs.rs I'm sure there are some crates that do include this, but I'm uncertain how valuable this addition would be, particularly if it negatively impacts binary size or other aspects I'm curious: has there been any user research or data gathering on this specific need? |
Contributor
Author
Surely that's not true - TOML or JSON at least. The downsides of this approach are documented in excruciating detail here: https://fasterthanli.me/articles/my-gift-to-the-rust-docs-team — there are two other approaches mentioned.
None as far as I'm aware! There's no better time than the present, though. I wanted to make sure we were covered implementation-wise — again, as described in the article. It would be fairly easy to run a Rust user survey. Collecting the quantity of non-Rust code blocks on docs.rs is also a fairly simple affair. Syntax highlighting could ship as a separate (optional) component. There are many options when one is willing to put in the work. (I don't necessarily think we should increase the size of the rustdoc binary from 22M to 171M — I wonder what that looks like compressed — the WASM version of grammars compress very well). If there's one thing I can ask, it's to do your own research before commenting. Thanks! |
Member
To clarify, I meant other programming languages |
Member
|
Just went through |
Collaborator
|
The job Click to see the possible cause of the failure (guessed by this bot) |
|
💔 Test for d9d0c0a failed: CI. Failed jobs:
|
Given how well Rust works in the Web Dev world, I can see JavaScript, TypeScript, JSON, TOML, SQL, CSS, and HTML doc blocks being used frequently. Maybe this would go a step forward and encourage better documentation across the board when integrating with those other languages. I mean no shade, I just really love good documentation. Anything that the rust team can do to facilitate that would be extremely helpful and welcome. |
Contributor
Author
Just want to mention that this is not a system library, it's innocent portable C11 code. I would swear on my life that it's not gonna cause any problem down the line but I understand this isn't going to be enough. Can I get a glimpse of the future? Say suddenly arborium uses a Rust core (instead of tree-sitter-core), a Rust compiler that generates Rust parsers, and that all grammars in the 96 that have an external scanner.c or scanner.cc are ported to pure Rust — what would you look at next? What would the vetting process look like? @GuillaumeGomez |
Member
|
That would remove one blocker. Next would be to look into reducing the size of code base overall (if possible). This feature would be very nice to have, however, it's not "core" so if it adds tens thousands of lines for that, might be a bit too much (when I talk about lines of code, I don't include each language "tree", these are mostly fine I guess). |
Contributor
Author
I'm interested in where the line is (and maybe we should take this discussion elsewhere). A minimal Rust re-implementation of tree-sitter for docs.rs wouldn't need all the incremental parsing stuff which constitutes the bulk of the complexity of TS (a I understand it). We could also look into using bytecode instead of DFA for size gains at the expend of speed. When you looked at the current state of arborium, which dependencies specifically looked worrying? Does it matter as much if it's shipped as a separate, optional rustup component that rustdoc knows how to call to over... stdio or something? Help me understand the constraints here. |
Member
|
Trying to describe bigger picture (not an official rustdoc position, just my opinion here): First thing to precise: there isn't a clear line on most things. We check and if majority of the team agrees, we add it. There are a few exceptions when something stands out too much, like having a C dependency (immediate blocker) or having more than 5 dependencies (which are not optional and not already in the rustc tree). These would require a debate to be accepted. Stating how I see dependencies: more dependency means potentially more bugs, more security issues, increases compile-time (strongly depends on each dep of course), and eventually means to increase our maintenance burden because the rustdoc team needs to keep an eye on our deps (we contribute heavily to all of our direct dependencies to fix bugs, for perf reasons, etc, so that's part of the maintenance burden). So before adding a dependency, we need to check what it does etc. So if the crate has 100+ optional dependencies, if in the end we only use 3, it's fine. And then comes the last two points which both influence the other: the impact on performance and how widely used it would be. If widely used and small impact, then we have the golden path, other cases are more tricky. Based on all this, do you have some more specific questions? |
Contributor
|
One possible alternative is to have simple extension points in rustdoc (e.g. HTML postprocessors or Markdown preprocessors), so crates can choose which “plugins” they want to use. Running a configured command on every HTML file is much less code to add for rustdoc and it will allow integrating with arborium, LaTeX renderers or any other kind of doc niceness one might want. |
Contributor
Author
I do! So you're not worried at all about the redistributable size? (22MB => 171MB uncompressed?) But you are worried about having to patch "tree-sitter but in Rust" for performance / security / etc. ? I don't think you've commented on the "separate component" option, either redistributed as a rustup component or simply internal to docs.rs — like the arborium-rustdoc proof of concept here. |
Member
You mean the generated docs? If so yes we are but we're getting close to the best we can do for pure HTML output. And it's not only for "tree-sitter but in Rust" we're worried but about all new dependencies.
I don't think the complexity of a rustup component is worth the added value from highlighting other languages, especially when an alternative solution based on JS and highlight.js works perfectly. As for docs.rs, the issues I listed for rustdoc are basically the same but even worse: we're only two working on docs.rs. |
Contributor
the 22M -> 171M is for |
Member
|
Oof, yeah that's also a blocker imo. |
Contributor
Author
No! I mean the rustdoc binary that is distributed via rustup. That one: The generated HTML inflates very little — have you had a chance to check out the Angle 3 section of my article? The video shows My worry is that, when this discussion resumes after having "some good enough rust version of tree-sitter", the size of the rustdoc binary will be the next blocker (I thought it would be the current blocker!). Then the only reasonable move would be to "run it in the docs.rs backend" — at which point maybe "every C dependency is a blocker" no longer applies. So. Just trying to save some work here. Maybe I should go a little more in detail about the technical aspects of the tree-sitter solution. Grammars are defined as Sometimes, defining a grammar in a The However, these are not sufficient to perform parsing, querying and highlighting. You can think of them as mostly "definitions" or "state machines" (with some imperative code) that are loaded by the tree-sitter runtime itself, which is also written in portable C11. The authors saw it as an advantage! It let it have zero dependencies and compile anywhere, on the web, on any native platform with a C compiler, etc. Now, under a "no C policy", the plan would be to:
Part number 3 has already been attempted via C2Rust — I don't believe this is necessarily the right approach. The result would be probably slightly slower, it might be slightly smaller too if we chose a different representation of "compiled grammars", and there wouldn't be any C in the build process of rustdoc or "some tool internal to docs.rs". This is a significant engineering effort that I'm happy to shoulder, I just want to make sure you understand what the end result would look like! I would love it if you could ask me questions to ensure there is no misunderstanding here as to what's being proposed. @GuillaumeGomez |
Member
I did, hence why I was surprised. Although I find the test to be a bit "off". It directly depends on the number of code examples, we have the same issue with rust highlighting: almost no changes between highlighted and not highlighted, but as you add more Rust code examples, the difference will increase. But that's another debate and quite secondary, if we reach this discussion, then it's pretty much that all other problems have been solved.
I confirm that going from 22MB to 177MB is also a blocker. You did well asking confirmation. ^^'
Thanks for the technical details! Just to confirm: it is done when updating
I can already say that this approach is not viable last I check. Or more like "not maintainable", ie it generates working code but good luck trying to maintain it. Overall I'm surprised that a Rust equivalent of |
Contributor
Author
Just want to highlight that's A) those sizes are uncompressed and unstripped B) 177MB is including all 96 grammars which I think is probably overkill.
Correct. By the time you're compiling arborium or something that depends on it, you are building a lot of C code (all of it vendored with zero system dependencies outside of malloc + a handful of isupper/islower functions), but not generating anything.
highlight.js is a pile of regexp. My personal position is that this is no way to highlight a programming language, unless you have literally exhausted all other options. Here, docs.rs has an easy, literally drop-in solution (insert arborium-rustdoc into the build pipeline) to get high-quality syntax highlighting with zero JavaScript dependencies to maintain or care about. I must admit I'm surprised at all the shields being raised re: that option specifically. |
Member
|
Just to sum things up: you suggest to add a new dependency for a minor feature (which would still be nice to have) which:
So in short: being worried about greatly increasing maintenance burden of rustdoc for a minor feature doesn't seem that far of a fetch to me. I understand your frustration, you spent a lot of time and this and would love for it to be handled by rustdoc by default directly, but based on all the reasons I listed, I think my position shouldn't come as a surprise. |
Contributor
Author
Nope! You're talking about Angle 2, which I always predicted would reveal untenable. I'm asking why is Angle 3 not being discussed. Angle 3 only involves post-processing rustdoc output. It's a horse of a completely different color. edit: also the VERY FIRST THING this PR says is "This is not intended to merge as-is, but as a basis for discussion." |
Member
|
You mean doing it in docs.rs instead? Quoting myself:
|
Contributor
Author
Do you mean that building a binary for
I have a lot of empathy there, but I was hoping to have a technical discussion. |
Member
|
I think it's natural that people are discussing Angle 2 here; after all, your PR implements Angle 2. It can be quite confusing to understand what exactly do you mean by the alternatives, and it's IMO not ideal to force reviewers to read an article (as much as I think your articles are amazing, and I think pretty much everyone would agree with that!) to understand that. If you want to discuss Angle 3, I would suggest to create an issue specifically about Angle 3 (maybe on the docs.rs repo?), and explain in that issue what exactly that would entail. |
Member
The binary size is secondary for docs.rs, however all other points remain valid with some new concerns:
I'll put that on account of your frustration but not very nice to read when I'm trying to explain why, even though I'd love to have this feature, the proposed solutions come with their issues which we can't overlook. @Kobzol I read the blog post, that's how I discovered this PR. 😉 |
Member
I tried to do this today, as the author asked me to research the topic independently (which I'm not happy with, but I was really curious about actual numbers). Unfortunately, I’m not a professional in this area, so all I could do (given the limited free time I had) was collect information from the first 20k crates in alphabetical order. This limitation exists because only the first 200 pages of crates are accessible (as page 201 explains because of performance reasons). I also don’t know how to obtain a repository link by crate name, which would have made the task much easier (if there a way please tell me so I could do research by all crates) FWIW, I don't think that JSON, shell and TOML are really needs any kind of highlighting, they perfectly readable even without it There is a screenshot of most popular results in this selection, I will try spend a bit more time to this to make full crates analysis, but there is what I have for now (important to mention that this is only README analysis of crates, not sure if there any other way to analyse this) 
|
I guess the reason syntect wasn't used is because the Rust grammar is out of date? Or is the bundled oniguruma library the issue since it's C? |
Contributor
|
|
|
True, I had issues with many grammars with it, but maybe just for Rust syntax it's fine. |
Contributor
|
there's still definitely issues with a few, but https://github.com/CosmicHorrorDev/two-face/blob/main/README.md#syntaxes |
Member
|
Even if the rust syntax isn't working, we don't care since all Rust code highlighting is handled internally by rustdoc. :) |
Even these numbers, just being in the readme file is pretty fascinating. I would have thought that crates like rustqlite would have had more SQL syntax highlighting. Their example code shows rust code with the SQL in the 'static str. So the real question is, can this highlight SQL in rust strings? Such as when a rust string starts with SELECT, or UPDATE, or DELETE? If so, this becomes a massive benefit. |
Member
I've just run my new version of the program. It's still running and will take a couple more hours, despite already running for an hour. Overall, the results are very different from the simple analysis of README files, which are now excluded. The scan now only checks .rs files across all crates For example, the term "shell" appeared only 420 times in the last 76k crates, which is significantly lower than what we saw in the first 20k README files. Also, the new version is producing many false positives due to some odd C code (screenshot below, for example "void" appeared 7600 times at this moment) I'm thinking I'll either share the final JSON file as-is, or manualy filter for specific languages and try to clean up the JSON before sharing. I might even do both 
|
Member
|
I'm back with the results! Here is the cleaned version of the JSON. As I mentioned before, there were a lot of false positives - like the one above with "void" https://gist.github.com/Kivooeo/2abd126715e4d84e7568da06771fd23c And for those who are truly curious about everything found after the code fences, here's a very large list - about 10k different entries https://gist.github.com/Kivooeo/25294c105536a236d8df30d1cf66448b Personally and if I understood all pros and cons correctly, I don't think this is worth to add (though given the concerns raised, if we had an easy way to do it, then sure), what do others think? |
Member
|
Thanks a lot for the data @Kivooeo! So yeah, kinda confirm my suspicions. My take on this would be "it's a feature we want, but not with this approach". |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds syntax highlighting for non-Rust code blocks in rustdoc using arborium, a tree-sitter based highlighting library.
Currently, code blocks like
```pythonor```javascriptare rendered as plain text. This PR enables proper syntax highlighting for 15 languages: Python, JavaScript, TypeScript, Bash, C, C++, Go, Java, JSON, TOML, YAML, SQL, Ruby, CSS, and HTML.Status
This is not intended to merge as-is, but as a basis for discussion.
There are other approaches being explored, such as post-processing the rustdoc HTML output rather than integrating directly into the rendering pipeline. See bearcove/arborium#36 for context on the different approaches.
Usage
Behind an unstable flag:
Implementation
arboriumas a dependency with 15 language grammarsCodeBlocksiterator inhtml/markdown.rs<a-k>for keywords,<a-s>for strings, etc.)Open questions