Functionality for truncating string to width

[danielrainer]

For use cases printing to an area with a maximum size, the width of a string is relevant for deciding if and where to truncate it. Currently, this crate can be used for such computations, but not well. Either one computes width per character, but that can't give correct results in all cases, if I understand the logic correctly. Alternatively, one could repeatedly pass prefixes of the string to the functions for computing string width until finding the maximal prefix which fits into the available area. While this would work, it's inefficient, both syntactically and performance-wise. Ideally, there would be functions like

pub fn longest_prefix_below_width(s: &str, width_limit: usize) -> (&str, usize);

which return the longest prefix which results in a width below the width_limit.
Additional useful information to return is the width of the prefix, which might be shorter than the width_limit - 1 if the input string is not wide enough to fill to the limit, or of a width-2 character is at the width boundary, resulting in the prefix having width width_limit - 2.
It is also useful to know whether truncation occurred. For &str, that's easy to check based on their length. If an interface for iterators is implemented, which would be nice, it might be considered to add truncation information in some way.

I think implementing this might require reworking the width computation logic, as it currently seems to work in reverse. Doing so would also allow adding a regular width computation interface for non-double-ended iterators, but I don't know how much effort it would be to reverse the logic, or why the computation is currently done starting from the end of the string.

[Manishearth]

Be very careful with assumptions like "truncating a string makes it visually shorter", that was the source of the 2015 widespread iOS crash.

AIUI the width computation basically must happen in reverse based on how the spec works. I'm not sure.

So I'm overall reluctant to add this.

[danielrainer]

This is a good point I didn't fully consider. Does this mean that there is no correct way of getting the maximal-length prefix of a string which fits into a fixed width other than testing all prefixes?

[danielrainer]

I guess if a forward implementation is possible it could go through the entire string, storing latest prefix whose width fits within the limit, and return that prefix once the entire string was checked.

[Manishearth]

Does this mean that there is no correct way of getting the maximal-length prefix of a string which fits into a fixed width other than testing all prefixes?

No, this means that you need to really understand what you're trying to do in a cross-script way first.

I don't yet know what you're trying to do. The concept of "width" of arbitrary text is already marginal. As noted in the readme, this crate follows a particular spec that deals with the concept of East Asian Width, and then has more tweaks on top of that. It is not going to be accurate for terminal width, since monospace is a Latin-centric concept that does not easily generalize cross-script.

This library is a useful approximation. Trying to do complicated things with a useful approximation will have diminishing returns.

[danielrainer]

We have several use cases involving width computation in the fish shell, including:

• Cursor placement (see Cursor position is wrong in history-pager when contains chinese fish-shell/fish-shell#12444 for an example)
• Truncating strings such that they fit into the current line, e.g. truncating the description of a command.
• Using width semantics for printf-style printing, e.g. printf '%.3s' $variable should limit the output to a width of 3 columns, rather than printing at most 3 bytes or codepoints or grapheme clusters.

The first use case does not necessarily require truncation, but the others do. We currently use a per-codepoint implementation (a Rust version of wcwidth), which works for most cases, but clearly can't work as well as an implementation which considers the entire string. The text we are working with can be arbitrary user input, as well as our own localized messages. For now, the available localization options mostly use Latin scripts, with the exception of Chinese. New language options could be added whenever someone decides to contribute translations.

So the currently available options for truncation I see for fish are (without writing our own width computation logic):

• Rework the UI such that truncation is no longer needed. I don't think this would work well for fish.
• Stick with using per-codepoint width calculations.
• Build quadratic-time truncation logic on top of unicode-width.

None of these options is great, and I think having truncation capabilities available directly in the width computation logic would clearly be an improvement over the second and third option. But if width computation cannot be done (or at least not as well) in the forward direction, this might not be a realistic option.

In any case, I wonder how other projects handle truncation, both with the limitations of terminals and monospace fonts, and in GUIs free from these limitations.

[Manishearth]

You could use grapheme segmentation to "guess" at a truncated string and then see if that works.

But also a fundamental problem is that this is entirely up to the font. Whatever you do will be a guess here, unless you use the font as an input (and then you still have to do something quadratic".

Using width semantics for printf-style printing, e.g. printf '%.3s' $variable should limit the output to a width of 3 columns, rather than printing at most 3 bytes or codepoints or grapheme clusters.

To me it feels like it should print at most 3 GCs and then have that behave as 3 columns. Note that you have to do this kind of thing anyway: "汉字" has a width of 4 and if truncated would give you a width of 2. However you truncate it, you would still need to artificially treat the output as 3.

---

I think an API that would be pretty well suited for your needs would be one that collected up the width_in_str output into a vector and then searched it from the front for the right character.

Exposing the core iterator as an API would be in general useful: Something which takes a backwards iterator of characters and produces an iterator of their corresponding widths in the string. I'd support that.

[danielrainer]

Exposing the core iterator as an API would be in general useful: Something which takes a backwards iterator of characters and produces an iterator of their corresponding widths in the string.

To confirm we are discussing the same thing, would your proposed API work as follows?

• Original string s: g̈测
• The input to the API would be s.chars().rev(), i.e. an iterator yielding 测, \u{308}, g
• The output would be an iterator yielding 2, 0, 1

The caller would then have to reverse the returned iterator (which I think requires allocation, so doing it in unicode-width is probably not desirable, except maybe by introducing an optional feature which enables allocation in the crate). Then, a prefix sum of the correctly oriented width iterator can be calculated and used for truncation.

You could use grapheme segmentation to "guess" at a truncated string and then see if that works.

Maybe this could also be the basis of a decent solution; first split the string on GC boundaries, then compute the width of each GC separately and add them up until the width limit is reached. This wouldn't require allocation, or even changes to unicode-width, but assumes that width calculation would still work in this case, which I'm not sure is true. If GC widths are always positive (or at least non-negative), it also wouldn't be necessary to consider the entire string.

To me it feels like it should print at most 3 GCs and then have that behave as 3 columns. Note that you have to do this kind of thing anyway: "汉字" has a width of 4 and if truncated would give you a width of 2. However you truncate it, you would still need to artificially treat the output as 3.

I think the main reason to use printf that way, or for doing most truncation in a terminal, is to get predictable maximum widths in terminal cells. So I think I would like the string to be truncated to width 2, and then maybe add padding to the exact limit, depending on the use case. Even if width computation is not perfect, I think this should lead to decent results in most cases. If that's insufficient, one might also consider writing to specific terminal cells, but I think that should be avoided if possible. The semantics of interpreting the format string as "at most 3 GCs" do not seem particularly useful to me, since different GCs can have different widths. The primary reason I see for using this kind of format specifier is to ensure that the result fits into the available space, and not using an available column is much better than overflowing in such cases. I can't think of a good reason to limit a string to a maximum number of GCs, so I don't see much point in supporting this use case.

[Manishearth]

To confirm we are discussing the same thing, would your proposed API work as follows?

Yes, and we just carefully document that all the iterators are backwards.

[danielrainer]

Seems reasonable. Do you think adding support for reversing the output as an optional feature (only enabled when allocation is available) makes sense?

Taking a reversed iterator could also be useful as an alternative to char_iter_width, to avoid having to provide a DoubleEndedIterator.

[Manishearth]

Not really; there's nothing there that the user can't do themselves with just a little bit of code.

Yes, taking a reversed iterator is fine: this is a power user API, we can be weird about it.