mirror of
https://github.com/neovim/neovim.git
synced 2026-02-18 02:11:06 +10:00
fa24df3264
Problem: bw_rest was used as an extra buffer to save incomplete byte sequences between calls to buf_write_bytes. Besides being unnecessarily complicated, this introduced a number of issues: 1) The bytes stored in bw_rest could still be there at the end of writing the file, never having been written, thus losing some of the file content on write. 2) bw_rest was not cleared out after the "checking_conversion" phase, leaving them to affect the written file content during the writing phase, corrupting the file. 3) bw_rest could contain extra bytes that need to be written to the output buffer during a buf_write_convert call, potentially before any bytes are consumed. But some conversions are in-place, without a separate output buffer. Writing bytes from bw_rest to the "output" buffer actually overwrote bytes from the input buffer before they were read, corrupting the data to be written. 4) The extra bytes in bw_rest that need to be written to the conversion output buffer were not originally accounted for in the size calculation for the output buffer, causing a buffer overflow (previously fixed in Vim patch 9.1.2028). Solution: Rather than maintaining a separate buffer, the unconverted bytes at the end of the buffer can just be shifted to the beginning of the buffer, and the buffer size updated. This requires a bit of refactoring, and buf_write_convert and buf_write_convert_with_iconv need to report the number of bytes they consumed so that buf_write_bytes can handle the remaining bytes. Following conversion, bw_buf can be checked for any remaining bytes. Leftover bytes in this case result in a conversion error, which is better than silently dropping them. A short section of dead code was removed from buf_write_convert, for converting a non-UTF-8 buffer to UTF-8. Neovim buffers are always UTF-8. A few additional tests for iconv conversions have been added. Vim's iconv tests are disabled in Neovim because they use unsupported values for 'encoding'.