`convert` and `preview` commands crash on certain Markdown pages

[MaxDesiatov]

Description

When rendering Swift compiler documentation the diagnostics formatting code crashes on certain Markdown pages.

Checklist

• If possible, I've reproduced the issue using the main branch of this package.
• This issue hasn't been addressed in an existing GitHub issue.

Expected Behavior

No crashes.

Actual behavior

Swift/StringCharacterView.swift:158: Fatal error: String index is out of bounds

Steps To Reproduce

1. Clone https://github.com/apple/swift.git
2. Install a recent development snapshot of Swift, I'm using swift-DEVELOPMENT-SNAPSHOT-2023-07-23-a.
3. Run DocC from this snapshot this way (assuming swift is the clone from step 1):

docc preview --allow-arbitrary-catalog-directories swift/docs

Swift-DocC Version Information

swift-DEVELOPMENT-SNAPSHOT-2023-07-23-a

Swift Compiler Version Information

Apple Swift version 5.9-dev (LLVM 9b562f55c38e378, Swift b4ee68bd37c4f7d)
Target: arm64-apple-macosx14.0

[franklinsch]

The code that crashes is in the new diagnostics formatter (#535). cc @arthurcro if you have any ideas 🙂

[arthurcro]

The code that crashes is in the new diagnostics formatter (#535). cc @arthurcro if you have any ideas 🙂

Thank you for the heads up, I’ll take a look asap! 🙂

[arthurcro]

Hi! 🙂I have identified the issue and I have found a proper fix.
The crashes were caused by poor UTF-8 encoding handling while highlighting a source line using the diagnostic source range. I will open a pull request with the fix and several tests covering the issue shortly!

Thanks again for reporting this! @MaxDesiatov cc @franklinsch

[MaxDesiatov]

@arthurcro this is still reproducible on main with f70b3ea, I'm reopening the issue

[arthurcro]

@arthurcro this is still reproducible on main with f70b3ea, I'm reopening the issue

Thank you for reopening the issue and apologies for the inconvenience.
The crash is surprising, I did not face any crashes when I tested it after implementing the fix. Did you change anything this time compared to the steps mentioned in the Steps to reproduce ?

[MaxDesiatov]

I didn't, potentially more docs were published in the docs directory in question. But this crash looked similar if it wasn't the same as the original issue, so I felt that reopening this one would work better than creating a new one.

[arthurcro]

Hi! 😄 I'm unfortunately unable to reproduce the issue on main with the latest docs from https://github.com/apple/swift.
@franklinsch @d-ronnqvist could you check if you can reporduce if you have some time? 😊

[MaxDesiatov]

Reproduced again with main DocC, checkout this commit with the docs: swiftlang/swift@1de261b

[arthurcro]

Great! I can reproduce with this commit, thank you! 🙇🏻

[arthurcro]

Hi! 👋 After some investigation, I've identified what seems to be the underlying cause of the problem.

Interestingly, the issue doesn't seem to be directly caused by the improved default diagnostic formatter, even though the formatter does highlight it.

While I might not have captured all the details accurately, here's my understanding so far:

When using:

`` my content ``

DocC will see this as a symbol link.

For this specific crash, when using:

  for the target platform and verify diagnostics, like ``swift -frontend -typecheck -verify
%s``

DocC will try to resolve this symbol link as a TopicReference but it will eventually fail.

https://github.com/apple/swift-docc/blob/f70b3ea281773121a468d9c0966ab8a34bba58d2/Sources/SwiftDocC/Semantics/MarkupReferenceResolver.swift#L161

To resolve the symbol link and eventually emit a diagnostic for this resolution error, DocC uses the symbol link SourceRange provided by MarkupData.swift from swift-markdown.

This source range is used as the diagnostic's source range in case of resolution error. It's then used by the default diagnostic formatter to highlight the unresolved reference. If the source range does not match the actual content of the source file, it crashes.

If we look at your crash, we get a SourceRange 253:56-253:97 while if we look at the source file content, we should be getting 253:56-254:5. It seems we are getting an incorrect source range for some symbol links.

I hope that made sense.
@d-ronnqvist I'm not sure how to proceed to fix this issue. Could you advice possible next steps? 🙇🏻