diff --git a/runtime/doc/debug.txt b/runtime/doc/debug.txt deleted file mode 100644 index 5d9ed9c984..0000000000 --- a/runtime/doc/debug.txt +++ /dev/null @@ -1,163 +0,0 @@ -*debug.txt* Nvim - - - VIM REFERENCE MANUAL by Bram Moolenaar - - -Debugging Vim *debug-vim* - -This is for debugging Vim itself, when it doesn't work properly. -For debugging Vim scripts, functions, etc. see |debug-scripts| - - Type |gO| to see the table of contents. - -============================================================================== - -1. Location of a crash, using gcc and gdb *debug-gcc* *gdb* - -When Vim crashes in one of the test files, and you are using gcc for -compilation, here is what you can do to find out exactly where Vim crashes. -This also applies when using the MingW tools. - -1. Compile Vim with the "-g" option (there is a line in the src/Makefile for - this, which you can uncomment). Also make sure "strip" is disabled (do not - install it, or use the line "STRIP = /bin/true"). - -2. Execute these commands (replace "11" with the test that fails): > - cd testdir - gdb ../vim - run -u unix.vim -U NONE -s dotest.in test11.in - -3. Check where Vim crashes, gdb should give a message for this. - -4. Get a stack trace from gdb with this command: > - where -< You can check out different places in the stack trace with: > - frame 3 -< Replace "3" with one of the numbers in the stack trace. - -============================================================================== - -2. Locating memory leaks *debug-leaks* *valgrind* - -If you suspect Vim is leaking memory and you are using Linux, the valgrind -tool is very useful to pinpoint memory leaks. - -First of all, build Vim with EXITFREE defined. Search for this in MAKEFILE -and uncomment the line. - -Use this command to start Vim: -> - valgrind --log-file=valgrind.log --leak-check=full ./vim - -Note: Vim will run much slower. If your vimrc is big or you have several -plugins you need to be patient for startup, or run with the "-u NONE" -argument. - -There are often a few leaks from libraries, such as getpwuid() and -XtVaAppCreateShell(). Those are unavoidable. The number of bytes should be -very small a Kbyte or less. - -============================================================================== - -3. Windows Bug Reporting *debug-win32* - -If the Windows version of Vim crashes in a reproducible manner, you can take -some steps to provide a useful bug report. - - -3.1 GENERIC ~ - -You must obtain the debugger symbols (PDB) file for your executable: gvim.pdb -for gvim.exe, or vim.pdb for vim.exe. The PDB should be available from the -same place that you obtained the executable. Be sure to use the PDB that -matches the EXE (same date). - -If you built the executable yourself with the Microsoft Visual C++ compiler, -then the PDB was built with the EXE. - -If you have Visual Studio, use that instead of the VC Toolkit and WinDbg. - -For other compilers, you should always use the corresponding debugger: gdb -(see above |debug-gcc|) for the Cygwin and MinGW compilers. - - - *debug-vs2005* -3.2 Debugging Vim crashes with Visual Studio 2005/Visual C++ 2005 Express ~ - -First launch vim.exe or gvim.exe and then launch Visual Studio. (If you don't -have Visual Studio, follow the instructions at |get-ms-debuggers| to obtain a -free copy of Visual C++ 2005 Express Edition.) - -On the Tools menu, click Attach to Process. Choose the Vim process. - -In Vim, reproduce the crash. A dialog will appear in Visual Studio, telling -you about the unhandled exception in the Vim process. Click Break to break -into the process. - -Visual Studio will pop up another dialog, telling you that no symbols are -loaded and that the source code cannot be displayed. Click OK. - -Several windows will open. Right-click in the Call Stack window. Choose Load -Symbols. The Find Symbols dialog will open, looking for (g)vim.pdb. Navigate -to the directory where you have the PDB file and click Open. - -At this point, you should have a full call stack with vim function names and -line numbers. Double-click one of the lines and the Find Source dialog will -appear. Navigate to the directory where the Vim source is (if you have it.) - -If you don't know how to debug this any further, follow the instructions -at ":help bug-report". Paste the call stack into the bug report. - -If you have a non-free version of Visual Studio, you can save a minidump via -the Debug menu and send it with the bug report. A minidump is a small file -(<100KB), which contains information about the state of your process. -Visual C++ 2005 Express Edition cannot save minidumps and it cannot be -installed as a just-in-time debugger. Use WinDbg, |debug-windbg|, if you -need to save minidumps or you want a just-in-time (postmortem) debugger. - - *debug-windbg* -3.3 Debugging Vim crashes with WinDbg ~ - -See |get-ms-debuggers| to obtain a copy of WinDbg. - -As with the Visual Studio IDE, you can attach WinDbg to a running Vim process. -You can also have your system automatically invoke WinDbg as a postmortem -debugger. To set WinDbg as your postmortem debugger, run "windbg -I". - -To attach WinDbg to a running Vim process, launch WinDbg. On the File menu, -choose Attach to a Process. Select the Vim process and click OK. - -At this point, choose Symbol File Path on the File menu, and add the folder -containing your Vim PDB to the sympath. If you have Vim source available, -use Source File Path on the File menu. You can now open source files in -WinDbg and set breakpoints, if you like. Reproduce your crash. WinDbg should -open the source file at the point of the crash. Using the View menu, you can -examine the call stack, local variables, watch windows, and so on. - -If WinDbg is your postmortem debugger, you do not need to attach WinDbg to -your Vim process. Simply reproduce the crash and WinDbg will launch -automatically. As above, set the Symbol File Path and the Source File Path. - -To save a minidump, type the following at the WinDbg command line: > - .dump vim.dmp -< - *debug-minidump* -3.4 Opening a Minidump ~ - -If you have a minidump file, you can open it in Visual Studio or in WinDbg. - -In Visual Studio 2005: on the File menu, choose Open, then Project/Solution. -Navigate to the .dmp file and open it. Now press F5 to invoke the debugger. -Follow the instructions in |debug-vs2005| to set the Symbol File Path. - -In WinDbg: choose Open Crash Dump on the File menu. Follow the instructions -in |debug-windbg| to set the Symbol File Path. - - *get-ms-debuggers* -3.5 Obtaining Microsoft Debugging Tools ~ - -Visual Studio 2017 Community Edition can be downloaded for free from: - https://visualstudio.microsoft.com/downloads/ - - vim:tw=78:ts=8:noet:ft=help:norl: diff --git a/runtime/doc/dev_arch.txt b/runtime/doc/dev_arch.txt index 1e4aa4eb19..3fe50ddb7e 100644 --- a/runtime/doc/dev_arch.txt +++ b/runtime/doc/dev_arch.txt @@ -6,34 +6,67 @@ How to develop Nvim, explanation of modules and subsystems *dev-arch* -The top of each major module has (or should have) an overview in a comment at -the top of its file. The purpose of this document is to give: +Module-specific details are documented at the top of each module +(`terminal.c`, `undo.c`, …). The top of each major module has (or should have) +an overview in a comment at the top of its file. + +The purpose of this document is to give: 1. an overview of how it all fits together 2. how-to guides for common tasks such as: - - deprecating public functions - - adding a new public (API) function - - adding a new public (UI) event -3. TODO: move src/nvim/README.md into this doc. + - (TODO) deprecating public functions + - (TODO) adding a new public (API) function or (UI) event Type |gO| to see the table of contents. +============================================================================== +Filename conventions + +The source filenames use extensions to hint about their purpose. + +- `*.c`, `*.generated.c` - full C files, with all includes, etc. +- `*.c.h` - parametrized C files, contain all necessary includes, but require + defining macros before actually using. Example: `typval_encode.c.h` +- `*.h` - full headers, with all includes. Does *not* apply to `*.generated.h`. +- `*.h.generated.h` - exported functions’ declarations. +- `*.c.generated.h` - static functions’ declarations. + ============================================================================== Data structures +- StringBuilder +- kvec or garray.c for dynamic lists / vectors (use StringBuilder for strings) + Use `kvec.h` for most lists. When you absolutely need a linked list, use `lib/queue_defs.h` which defines an "intrusive" linked list. -============================================================================== -Events +Buffer text is stored as a tree of line segments, defined in `src/nvim/memline.c`. +The central idea is found in `ml_find_line`. -All new events must be implemented using `aucmd_defer()` (and where possible, -old events should be migrated to this), so that they are processed in -a predictable manner, which avoids crashes and race conditions. See +Many of the editor concepts are defined as Lua data files: + +- Events (autocmds): src/nvim/auevents.lua +- Ex (cmdline) commands: src/nvim/ex_cmds.lua +- Options: src/nvim/options.lua +- Vimscript functions: src/nvim/eval.lua +- v: variables: src/nvim/vvars.lua + +============================================================================== +Events *dev-events* + +The events historically called "autocmds", referred to here as "editor events" +or simply "events", are high-level events for use by plugins, user config, and +the Nvim editor. (There is an unrelated, low-level concept defined by the +`event/defs.h#Event` struct, which is just a bag of data passed along the +internal |event-loop|.) + +All new editor events must be implemented using `aucmd_defer()` (and where +possible, old events should be migrated to this), so that they are processed +in a predictable manner, which avoids crashes and race conditions. See `do_markset_autocmd` for an example. ============================================================================== -UI events +UI events *dev-ui-events* The long-term vision is that UI events are just another type of "editor event" (formerly known as "autocmds"). There is no real reason that we have separate @@ -91,6 +124,269 @@ Common examples of non-fast code: regexp matching, wildcard expansion, expression evaluation. +============================================================================== +The event-loop *event-loop* + +The internal, low-level, libuv event-loop (|luv-event-loop|) is used to +schedule arbitrary work in a predictable way. One such obvious use-case for +scheduling is deferred editor-events (autocmds). Another example is +|job-control|. + +ASYNC EVENT SUPPORT + +One of the features Nvim added is the support for handling arbitrary +asynchronous events, which can include: + +- RPC requests +- job control callbacks +- timers + +Nvim implements this functionality by entering another event loop while +waiting for characters, so instead of: >py + + def state_enter(state_callback, data): + do + key = readkey() # read a key from the user + while state_callback(data, key) # invoke the callback for the current state +< + +The Nvim program loop is more like: >py + + def state_enter(state_callback, data): + do + event = read_next_event() # read an event from the operating system + while state_callback(data, event) # invoke the callback for the current state +< + +where `event` is something the operating system delivers to us, including (but +not limited to) user input. The `read_next_event()` part is internally +implemented by libuv, the platform layer used by Nvim. + +Since Nvim inherited its code from Vim, the states are not prepared to receive +"arbitrary events", so we use a special key to represent those (When a state +receives an "arbitrary event", it normally doesn't do anything other than +update the screen). + +MAIN LOOP + +The `Loop` structure (which describes `main_loop`) abstracts multiple queues +into one loop: > + + uv_loop_t uv; + MultiQueue *events; + MultiQueue *thread_events; + MultiQueue *fast_events; + +`loop_poll_events` checks `Loop.uv` and `Loop.fast_events` whenever Nvim is +idle, and also at `os_breakcheck` intervals. + +MultiQueue is cool because you can attach throw-away "child queues" trivially. +For example `do_os_system()` does this (for every spawned process!) to +automatically route events onto the `main_loop`: > + + Process *proc = &uvproc.process; + MultiQueue *events = multiqueue_new_child(main_loop.events); + proc->events = events; + + +NVIM LIFECYCLE + +How Nvim processes input. + +Consider a typical Vim-like editing session: + +01. Vim displays the welcome screen +02. User types: `:` +03. Vim enters command-line mode +04. User types: `edit README.txt` +05. Vim opens the file and returns to normal mode +06. User types: `G` +07. Vim navigates to the end of the file +09. User types: `5` +10. Vim enters count-pending mode +11. User types: `d` +12. Vim enters operator-pending mode +13. User types: `w` +14. Vim deletes 5 words +15. User types: `g` +16. Vim enters the "g command mode" +17. User types: `g` +18. Vim goes to the beginning of the file +19. User types: `i` +20. Vim enters insert mode +21. User types: `word` +22. Vim inserts "word" at the beginning and returns to normal mode + +Note that we split user actions into sequences of inputs that change the state +of the editor. While there's no documentation about a "g command mode" (step +16), internally it is implemented similarly to "operator-pending mode". + +From this we can see that Vim has the behavior of an input-driven state machine +(more specifically, a pushdown automaton since it requires a stack for +transitioning back from states). Assuming each state has a callback responsible +for handling keys, this pseudocode represents the main program loop: >py + + def state_enter(state_callback, data): + do + key = readkey() # read a key from the user + while state_callback(data, key) # invoke the callback for the current state +< + +That is, each state is entered by calling `state_enter` and passing a +state-specific callback and data. Here is a high-level pseudocode for a program +that implements something like the workflow described above: >py + + def main() + state_enter(normal_state, {}): + + def normal_state(data, key): + if key == ':': + state_enter(command_line_state, {}) + elif key == 'i': + state_enter(insert_state, {}) + elif key == 'd': + state_enter(delete_operator_state, {}) + elif key == 'g': + state_enter(g_command_state, {}) + elif is_number(key): + state_enter(get_operator_count_state, {'count': key}) + elif key == 'G' + jump_to_eof() + return true + + def command_line_state(data, key): + if key == '': + if data['input']: + execute_ex_command(data['input']) + return false + elif key == '' + return false + + if not data['input']: + data['input'] = '' + + data['input'] += key + return true + + def delete_operator_state(data, key): + count = data['count'] or 1 + if key == 'w': + delete_word(count) + elif key == '$': + delete_to_eol(count) + return false # return to normal mode + + def g_command_state(data, key): + if key == 'g': + go_top() + elif key == 'v': + reselect() + return false # return to normal mode + + def get_operator_count_state(data, key): + if is_number(key): + data['count'] += key + return true + unshift_key(key) # return key to the input buffer + state_enter(delete_operator_state, data) + return false + + def insert_state(data, key): + if key == '': + return false # exit insert mode + self_insert(key) + return true +< + +The above gives an idea of how Nvim is organized internally. Some states like +the `g_command_state` or `get_operator_count_state` do not have a dedicated +`state_enter` callback, but are implicitly embedded into other states (this +will change later as we continue the refactoring effort). To start reading the +actual code, here's the recommended order: + +1. `state_enter()` function (state.c). This is the actual program loop, + note that a `VimState` structure is used, which contains function pointers + for the callback and state data. +2. `main()` function (main.c). After all startup, `normal_enter` is called + at the end of function to enter normal mode. +3. `normal_enter()` function (normal.c) is a small wrapper for setting + up the NormalState structure and calling `state_enter`. +4. `normal_check()` function (normal.c) is called before each iteration of + normal mode. +5. `normal_execute()` function (normal.c) is called when a key is read in normal + mode. + +The basic structure described for normal mode in 3, 4 and 5 is used for other +modes managed by the `state_enter` loop: + +- command-line mode: `command_line_{enter,check,execute}()`(`ex_getln.c`) +- insert mode: `insert_{enter,check,execute}()`(`edit.c`) +- terminal mode: `terminal_{enter,execute}()`(`terminal.c`) + +IMPORTANT VARIABLES + +The current mode is stored in `State`. The values it can have are `MODE_NORMAL`, +`MODE_INSERT`, `MODE_CMDLINE`, and a few others. + +The current window is `curwin`. The current buffer is `curbuf`. These point +to structures with the cursor position in the window, option values, the file +name, etc. + +All the global variables are declared in `globals.h`. + +THE MAIN EVENT-LOOP + +The main loop is implemented in state_enter. The basic idea is that Vim waits +for the user to type a character and processes it until another character is +needed. Thus there are several places where Vim waits for a character to be +typed. The `vgetc()` function is used for this. It also handles mapping. + +What we consider the "Nvim event loop" is actually a wrapper around `uv_run` to +handle both the `fast_events` queue and possibly (a suitable subset of) deferred +events. Therefore "raw" `vim.uv.run()` is often not enough to "yield" from Lua +plugins; instead they can call `vim.wait(0)`. + +Updating the screen is mostly postponed until a command or a sequence of +commands has finished. The work is done by `update_screen()`, which calls +`win_update()` for every window, which calls `win_line()` for every line. +See the start of [drawscreen.c](drawscreen.c) for more explanations. + +COMMAND-LINE MODE + +When typing a `:`, `normal_cmd()` will call `getcmdline()` to obtain a line with +an Ex command. `getcmdline()` calls a loop that will handle each typed +character. It returns when hitting `` or `` or some other character that +ends the command line mode. + +EX COMMANDS + +Ex commands are handled by the function `do_cmdline()`. It does the generic +parsing of the `:` command line and calls `do_one_cmd()` for each separate +command. It also takes care of while loops. + +`do_one_cmd()` parses the range and generic arguments and puts them in the +exarg_t and passes it to the function that handles the command. + +The `:` commands are listed in [ex_cmds.lua](ex_cmds.lua). + +NORMAL MODE COMMANDS + +The Normal mode commands are handled by the `normal_cmd()` function. It also +handles the optional count and an extra character for some commands. These +are passed in a `cmdarg_T` to the function that handles the command. + +There is a table `nv_cmds` in [normal.c](normal.c) which +lists the first character of every +command. The second entry of each item is the name of the function that +handles the command. + +INSERT MODE COMMANDS + +When doing an `i` or `a` command, `normal_cmd()` will call the `edit()` function. +It contains a loop that waits for the next character and handles it. It +returns when leaving Insert mode. + + ============================================================================== vim:tw=78:ts=8:sw=4:et:ft=help:norl: diff --git a/runtime/doc/dev_tools.txt b/runtime/doc/dev_tools.txt index 13f45eceec..4ca7af2c3a 100644 --- a/runtime/doc/dev_tools.txt +++ b/runtime/doc/dev_tools.txt @@ -6,15 +6,221 @@ Tools and techniques for developing Nvim *dev-tools* -The following advice is helpful when working on or debugging issues with Nvim -itself. - -TODO: merge |debug.txt| into here. +This is for developing or debugging Nvim itself. Type |gO| to see the table of contents. ============================================================================== -Backtraces *dev-tools-backtrace* +Logs *dev-tools-logs* + +Low-level log messages sink to `$NVIM_LOG_FILE`. + +UI events are logged at DEBUG level. > + + rm -rf build/ + make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG" + +Use `LOG_CALLSTACK()` (Linux only) to log the current stacktrace. To log to an +alternate file (e.g. stderr) use `LOG_CALLSTACK_TO_FILE(FILE*)`. Requires +`-no-pie` ([ref](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860394#15)): > + + rm -rf build/ + make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG -DCMAKE_C_FLAGS=-no-pie" + +Many log messages have a shared prefix, such as "UI" or "RPC". Use the shell to +filter the log, e.g. at DEBUG level you might want to exclude UI messages: > + + tail -F ~/.local/state/nvim/log | cat -v | stdbuf -o0 grep -v UI | stdbuf -o0 tee -a log + + +============================================================================== +Reproducible build + +To make a reproducible build of Nvim, set cmake variable `LUA_GEN_PRG` to +a LuaJIT binary built with `LUAJIT_SECURITY_PRN=0`. See commit +cb757f2663e6950e655c6306d713338dfa66b18d. + + +============================================================================== +Debug TUI *dev-tools-tui* + +TUI INSPECT + +Use the Ghostty https://ghostty.org/ inspector tool to observe and query the +output and events from any terminal application such as Nvim. + +TERMINFO LOGGING + +At 'verbose' level 3, Nvim logs its internal terminfo state, so you can see +exactly what terminfo values it is using on the current system. > + + nvim -V3log + +TUI DEBUGGING WITH GDB LLDB + +Launching the Nvim TUI involves two processes, one for main editor state and one +for rendering the TUI. Both of these processes use the nvim binary, so somewhat +confusingly setting a breakpoint in either will generally succeed but may not be +hit depending on which process the breakpoints were set in. + +To debug the main process, you can debug the nvim binary with the `--headless` +flag which does not launch the TUI and will allow you to set breakpoints in code +not related to TUI rendering like so: > + + lldb -- ./build/bin/nvim --headless --listen ~/.cache/nvim/debug-server.pipe + +While in lldb, enter `run`. You can then attach to the headless process in a +new terminal window to interact with the editor like so: > + + ./build/bin/nvim --remote-ui --server ~/.cache/nvim/debug-server.pipe + +Conversely for debugging TUI rendering, you can start a headless process and +debug the remote-ui process multiple times without losing editor state. + +For details on using nvim-dap and automatically debugging the child (main) +process, see [here](https://zignar.net/2023/02/17/debugging-neovim-with-neovim-and-nvim-dap/) + +TUI REDRAW + +For debugging Nvim TUI redraw behavior it is sometimes useful to slow down its +redraws. Set the 'writedelay' and 'redrawdebug' options to see where and when +the UI is painted. > + + :set writedelay=50 rdb=compositor + +Note: Nvim uses an internal screenbuffer to only send minimal updates even if a large +region is repainted internally. To also highlight excess internal redraws, use > + + :set writedelay=50 rdb=compositor,nodelta + +TUI TRACE + +In the rare case that you want to collect a trace of terminal output, the +ancient `script` command is still the "state of the art". The libvterm +`vterm-dump` utility formats the result for human-readability. + +Record a Nvim terminal session and format it with `vterm-dump`: >sh + + script foo + ./build/bin/nvim -u NONE + # Exit the script session with CTRL-d + + # Use `vterm-dump` utility to format the result. + ./.deps/usr/bin/vterm-dump foo > bar + +Then you can compare `bar` with another session, to debug TUI behavior. + +TERMINAL REFERENCE + +- `man terminfo` +- https://github.com/vim/vim/blob/0124320c97b0fbbb44613f42fc1c34fee6181fc8/src/libvterm/doc/seqs.txt +- https://invisible-island.net/xterm/ctlseqs/ctlseqs.html + +============================================================================== +Debug Performance *dev-tools-perf* + +PROFILING (EASY) + +For debugging performance bottlenecks in any code, there is a simple (and very +effective) approach: + +1. Run the slow code in a loop. +2. Break execution ~5 times and save the stacktrace. +3. The cause of the bottleneck will (almost always) appear in most of the stacktraces. + + +PROFILING (FANCY) + +For more advanced profiling, consider `perf` + `flamegraph`. + + +USDT PROFILING (POWERFUL) + +Or you can use USDT probes via `NVIM_PROBE` ([#12036](https://github.com/neovim/neovim/pull/12036)). + +> USDT is basically a way to define stable probe points in userland binaries. +> The benefit of bcc is the ability to define logic to go along with the probe +> points. + +Tools: +- bpftrace provides an awk-like language to the kernel bytecode, BPF. +- BCC provides a subset of C. Provides more complex logic than bpftrace, but takes a bit more effort. + +Example using bpftrace to track slow vim functions, and print out any files +that were opened during the trace. At the end, it prints a histogram of +function timing: > + + #!/usr/bin/env bpftrace + + BEGIN { + @depth = -1; + } + + tracepoint:sched:sched_process_fork /@pidmap[args->parent_pid]/ { + @pidmap[args->child_pid] = 1; + } + + tracepoint:sched:sched_process_exit /@pidmap[args->pid]/ { + delete(@pidmap[args->pid]); + } + + usdt:build/bin/nvim:neovim:eval__call_func__entry { + @pidmap[pid] = 1; + @depth++; + @funcentry[@depth] = nsecs; + } + + usdt:build/bin/nvim:neovim:eval__call_func__return { + $func = str(arg0); + $msecs = (nsecs - @funcentry[@depth]) / 1000000; + + @time_histo = hist($msecs); + + if ($msecs >= 1000) { + printf("%u ms for %s\n", $msecs, $func); + print(@files); + } + + clear(@files); + delete(@funcentry[@depth]); + @depth--; + } + + tracepoint:syscalls:sys_enter_open, + tracepoint:syscalls:sys_enter_openat { + if (@pidmap[pid] == 1 && @depth >= 0) { + @files[str(args->filename)] = count(); + } + } + + END { + clear(@depth); + } + + $ sudo bpftrace funcslower.bt + 1527 ms for Slower + @files[/usr/lib/libstdc++.so.6]: 2 + @files[/etc/fish/config.fish]: 2 + + + ^C + @time_histo: + [0] 71430 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| + [1] 346 | | + [2, 4) 208 | | + [4, 8) 91 | | + [8, 16) 22 | | + [16, 32) 85 | | + [32, 64) 7 | | + [64, 128) 0 | | + [128, 256) 0 | | + [256, 512) 6 | | + [512, 1K) 1 | | + [1K, 2K) 5 | | +< + +============================================================================== +Backtraces *dev-tools-backtrace* LINUX @@ -83,6 +289,53 @@ https://developer.apple.com/library/archive/technotes/tn2124/_index.html#//apple but note that some of the things on this page are out of date (such as enabling core dumps with `/etc/launchd.conf`). + +WINDOWS + +If the Windows version of Nvim crashes in a reproducible manner, you can take +some steps to provide a useful bug report. + +You must obtain the debugger symbols (PDB) file for the Nvim executable: nvim.pdb. +The PDB should be available from the same place that you obtained the +executable (TODO: not currently provided by Nvim CI releases). Be sure to use +the PDB that matches the EXE (same build). + +If you built the executable yourself with the Microsoft Visual C++ compiler, +then the PDB was built with the EXE. + +If you have Visual Studio, use that instead of the VC Toolkit and WinDbg. + +For other compilers, always use the corresponding debugger: gdb or lldb. + +Debugging Nvim crashes with Visual Studio 2005 ~ + +First launch nvim.exe and then launch Visual Studio. (If you don't have +Visual Studio, get it from https://visualstudio.microsoft.com/downloads/). + +On the Tools menu, click Attach to Process. Choose the Nvim process. + +In Nvim, reproduce the crash. A dialog will appear in Visual Studio, telling +you about the unhandled exception in the Nvim process. Click Break to break +into the process. + +Visual Studio will pop up another dialog, telling you that no symbols are +loaded and that the source code cannot be displayed. Click OK. + +Several windows will open. Right-click in the Call Stack window. Choose Load +Symbols. The Find Symbols dialog will open, looking for (g)vim.pdb. Navigate +to the directory where you have the PDB file and click Open. + +At this point, you should have a full call stack with vim function names and +line numbers. Double-click one of the lines and the Find Source dialog will +navigate to the directory where the Nvim source is (if you have it.) + +If you don't know how to debug this any further, follow the instructions +at ":help bug-report". Paste the call stack into the bug report. + +From Visual Studio you can also try saving a minidump via the Debug menu and +send it with the bug report. A minidump is a small file (<100KB), which +contains information about the state of your process. + ============================================================================== Gdb *dev-tools-gdb* @@ -194,5 +447,42 @@ example: (gdb) target remote localhost:6666 (gdb) br main < +============================================================================== +Debugging crashes or memory leaks *dev-tools-asan* + +BUILD WITH ASAN + +Building Nvim with Clang sanitizers (Address Sanitizer: ASan, Undefined +Behavior Sanitizer: UBSan, Memory Sanitizer: MSan, Thread Sanitizer: TSan) is +a good way to catch undefined behavior, leaks and other errors as soon as they +happen. It's significantly faster than Valgrind. + +Requires clang 3.4 or later, and `llvm-symbolizer` must be in `$PATH`: > + + clang --version + +Build Nvim with sanitizer instrumentation (choose one): > + + CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_ASAN_UBSAN=ON" + CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_MSAN=ON" + CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_TSAN=ON" + +Create a directory to store logs: > + + mkdir -p "$HOME/logs" + +Configure the sanitizer(s) via these environment variables: > + + # Change to detect_leaks=1 to detect memory leaks (slower, noisier). + export ASAN_OPTIONS="detect_leaks=0:log_path=$HOME/logs/asan" + # Show backtraces in the logs. + export MSAN_OPTIONS="log_path=${HOME}/logs/msan" + export TSAN_OPTIONS="log_path=${HOME}/logs/tsan" + +Logs will be written to `${HOME}/logs/*san.PID` then. + +For more information: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags + + vim:tw=78:ts=8:et:ft=help:norl: diff --git a/src/gen/gen_help_html.lua b/src/gen/gen_help_html.lua index 48dbcf9a5b..ebd004f448 100644 --- a/src/gen/gen_help_html.lua +++ b/src/gen/gen_help_html.lua @@ -91,6 +91,7 @@ local new_layout = { local redirects = { ['api-ui-events'] = 'ui', ['credits'] = 'backers', + ['dev_tools'] = 'debug', ['plugins'] = 'editorconfig', ['terminal'] = 'nvim_terminal_emulator', ['tui'] = 'term', diff --git a/src/nvim/README.md b/src/nvim/README.md index 6abc56ad1f..06e4dc0b30 100644 --- a/src/nvim/README.md +++ b/src/nvim/README.md @@ -1,526 +1,5 @@ -Nvim core -========= +## Moved to: -Module-specific details are documented at the top of each module (`terminal.c`, `undo.c`, …). - -See `:help dev` for guidelines. - -Filename conventions --------------------- - -The source files use extensions to hint about their purpose. - -- `*.c`, `*.generated.c` - full C files, with all includes, etc. -- `*.c.h` - parametrized C files, contain all necessary includes, but require - defining macros before actually using. Example: `typval_encode.c.h` -- `*.h` - full headers, with all includes. Does *not* apply to `*.generated.h`. -- `*.h.generated.h` - exported functions’ declarations. -- `*.c.generated.h` - static functions’ declarations. - -Common structures ------------------ - -- StringBuilder -- kvec or garray.c for dynamic lists / vectors (use StringBuilder for strings) - -Logs ----- - -Low-level log messages sink to `$NVIM_LOG_FILE`. - -UI events are logged at DEBUG level. - - rm -rf build/ - make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG" - -Use `LOG_CALLSTACK()` (Linux only) to log the current stacktrace. To log to an -alternate file (e.g. stderr) use `LOG_CALLSTACK_TO_FILE(FILE*)`. Requires -`-no-pie` ([ref](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860394#15)): - - rm -rf build/ - make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG -DCMAKE_C_FLAGS=-no-pie" - -Many log messages have a shared prefix, such as "UI" or "RPC". Use the shell to -filter the log, e.g. at DEBUG level you might want to exclude UI messages: - - tail -F ~/.local/state/nvim/log | cat -v | stdbuf -o0 grep -v UI | stdbuf -o0 tee -a log - -Build with ASAN ---------------- - -Building Nvim with Clang sanitizers (Address Sanitizer: ASan, Undefined -Behavior Sanitizer: UBSan, Memory Sanitizer: MSan, Thread Sanitizer: TSan) is -a good way to catch undefined behavior, leaks and other errors as soon as they -happen. It's significantly faster than Valgrind. - -Requires clang 3.4 or later, and `llvm-symbolizer` must be in `$PATH`: - - clang --version - -Build Nvim with sanitizer instrumentation (choose one): - - CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_ASAN_UBSAN=ON" - CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_MSAN=ON" - CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_TSAN=ON" - -Create a directory to store logs: - - mkdir -p "$HOME/logs" - -Configure the sanitizer(s) via these environment variables: - - # Change to detect_leaks=1 to detect memory leaks (slower, noisier). - export ASAN_OPTIONS="detect_leaks=0:log_path=$HOME/logs/asan" - # Show backtraces in the logs. - export MSAN_OPTIONS="log_path=${HOME}/logs/msan" - export TSAN_OPTIONS="log_path=${HOME}/logs/tsan" - -Logs will be written to `${HOME}/logs/*san.PID` then. - -For more information: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags - -Reproducible build ------------------- - -To make a reproducible build of Nvim, set cmake variable `LUA_GEN_PRG` to -a LuaJIT binary built with `LUAJIT_SECURITY_PRN=0`. See commit -cb757f2663e6950e655c6306d713338dfa66b18d. - -Debug: Performance ------------------- - -### Profiling (easy) - -For debugging performance bottlenecks in any code, there is a simple (and very -effective) approach: - -1. Run the slow code in a loop. -2. Break execution ~5 times and save the stacktrace. -3. The cause of the bottleneck will (almost always) appear in most of the stacktraces. - -### Profiling (fancy) - -For more advanced profiling, consider `perf` + `flamegraph`. - -### USDT profiling (powerful) - -Or you can use USDT probes via `NVIM_PROBE` ([#12036](https://github.com/neovim/neovim/pull/12036)). - -> USDT is basically a way to define stable probe points in userland binaries. -> The benefit of bcc is the ability to define logic to go along with the probe -> points. - -Tools: -- bpftrace provides an awk-like language to the kernel bytecode, BPF. -- BCC provides a subset of C. Provides more complex logic than bpftrace, but takes a bit more effort. - -Example using bpftrace to track slow vim functions, and print out any files -that were opened during the trace. At the end, it prints a histogram of -function timing: - - #!/usr/bin/env bpftrace - - BEGIN { - @depth = -1; - } - - tracepoint:sched:sched_process_fork /@pidmap[args->parent_pid]/ { - @pidmap[args->child_pid] = 1; - } - - tracepoint:sched:sched_process_exit /@pidmap[args->pid]/ { - delete(@pidmap[args->pid]); - } - - usdt:build/bin/nvim:neovim:eval__call_func__entry { - @pidmap[pid] = 1; - @depth++; - @funcentry[@depth] = nsecs; - } - - usdt:build/bin/nvim:neovim:eval__call_func__return { - $func = str(arg0); - $msecs = (nsecs - @funcentry[@depth]) / 1000000; - - @time_histo = hist($msecs); - - if ($msecs >= 1000) { - printf("%u ms for %s\n", $msecs, $func); - print(@files); - } - - clear(@files); - delete(@funcentry[@depth]); - @depth--; - } - - tracepoint:syscalls:sys_enter_open, - tracepoint:syscalls:sys_enter_openat { - if (@pidmap[pid] == 1 && @depth >= 0) { - @files[str(args->filename)] = count(); - } - } - - END { - clear(@depth); - } - - $ sudo bpftrace funcslower.bt - 1527 ms for Slower - @files[/usr/lib/libstdc++.so.6]: 2 - @files[/etc/fish/config.fish]: 2 - - - ^C - @time_histo: - [0] 71430 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| - [1] 346 | | - [2, 4) 208 | | - [4, 8) 91 | | - [8, 16) 22 | | - [16, 32) 85 | | - [32, 64) 7 | | - [64, 128) 0 | | - [128, 256) 0 | | - [256, 512) 6 | | - [512, 1K) 1 | | - [1K, 2K) 5 | | - -Debug: TUI ----------- - -### TUI troubleshoot - -Nvim logs its internal terminfo state at 'verbose' level 3. This makes it -possible to see exactly what terminfo values Nvim is using on any system. - - nvim -V3log - -### TUI Debugging with gdb/lldb - -Launching the nvim TUI involves two processes, one for main editor state and one -for rendering the TUI. Both of these processes use the nvim binary, so somewhat -confusingly setting a breakpoint in either will generally succeed but may not be -hit depending on which process the breakpoints were set in. - -To debug the main process, you can debug the nvim binary with the `--headless` -flag which does not launch the TUI and will allow you to set breakpoints in code -not related to TUI rendering like so: - - lldb -- ./build/bin/nvim --headless --listen ~/.cache/nvim/debug-server.pipe - -While in lldb, enter `run`. You can then attach to the headless process in a -new terminal window to interact with the editor like so: - - ./build/bin/nvim --remote-ui --server ~/.cache/nvim/debug-server.pipe - -Conversely for debugging TUI rendering, you can start a headless process and -debug the remote-ui process multiple times without losing editor state. - -For details on using nvim-dap and automatically debugging the child (main) -process, see -[here](https://zignar.net/2023/02/17/debugging-neovim-with-neovim-and-nvim-dap/) - -### TUI trace - -The ancient `script` command is still the "state of the art" for tracing -terminal behavior. The libvterm `vterm-dump` utility formats the result for -human-readability. - -Record a Nvim terminal session and format it with `vterm-dump`: - - script foo - ./build/bin/nvim -u NONE - # Exit the script session with CTRL-d - - # Use `vterm-dump` utility to format the result. - ./.deps/usr/bin/vterm-dump foo > bar - -Then you can compare `bar` with another session, to debug TUI behavior. - -### TUI redraw - -Set the 'writedelay' and 'redrawdebug' options to see where and when the UI is painted. - - :set writedelay=50 rdb=compositor - -Note: neovim uses an internal screenbuffer to only send minimal updates even if a large -region is repainted internally. To also highlight excess internal redraws, use - - :set writedelay=50 rdb=compositor,nodelta - -### Terminal reference - -- `man terminfo` -- http://bazaar.launchpad.net/~libvterm/libvterm/trunk/view/head:/doc/seqs.txt -- http://invisible-island.net/xterm/ctlseqs/ctlseqs.html - -Data structures ---------------- - -Buffer text is stored as a tree of line segments, defined in [memline.c](https://github.com/neovim/neovim/blob/v0.9.5/src/nvim/memline.c#L8-L35). -The central idea is found in [ml_find_line](https://github.com/neovim/neovim/blob/v0.9.5/src/nvim/memline.c#L2800). - -Nvim lifecycle --------------- - -Following describes how Nvim processes input. - -Consider a typical Vim-like editing session: - -01. Vim displays the welcome screen -02. User types: `:` -03. Vim enters command-line mode -04. User types: `edit README.txt` -05. Vim opens the file and returns to normal mode -06. User types: `G` -07. Vim navigates to the end of the file -09. User types: `5` -10. Vim enters count-pending mode -11. User types: `d` -12. Vim enters operator-pending mode -13. User types: `w` -14. Vim deletes 5 words -15. User types: `g` -16. Vim enters the "g command mode" -17. User types: `g` -18. Vim goes to the beginning of the file -19. User types: `i` -20. Vim enters insert mode -21. User types: `word` -22. Vim inserts "word" at the beginning and returns to normal mode - -Note that we split user actions into sequences of inputs that change the state -of the editor. While there's no documentation about a "g command mode" (step -16), internally it is implemented similarly to "operator-pending mode". - -From this we can see that Vim has the behavior of an input-driven state machine -(more specifically, a pushdown automaton since it requires a stack for -transitioning back from states). Assuming each state has a callback responsible -for handling keys, this pseudocode represents the main program loop: - -```py -def state_enter(state_callback, data): - do - key = readkey() # read a key from the user - while state_callback(data, key) # invoke the callback for the current state -``` - -That is, each state is entered by calling `state_enter` and passing a -state-specific callback and data. Here is a high-level pseudocode for a program -that implements something like the workflow described above: - -```py -def main() - state_enter(normal_state, {}): - -def normal_state(data, key): - if key == ':': - state_enter(command_line_state, {}) - elif key == 'i': - state_enter(insert_state, {}) - elif key == 'd': - state_enter(delete_operator_state, {}) - elif key == 'g': - state_enter(g_command_state, {}) - elif is_number(key): - state_enter(get_operator_count_state, {'count': key}) - elif key == 'G' - jump_to_eof() - return true - -def command_line_state(data, key): - if key == '': - if data['input']: - execute_ex_command(data['input']) - return false - elif key == '' - return false - - if not data['input']: - data['input'] = '' - - data['input'] += key - return true - -def delete_operator_state(data, key): - count = data['count'] or 1 - if key == 'w': - delete_word(count) - elif key == '$': - delete_to_eol(count) - return false # return to normal mode - -def g_command_state(data, key): - if key == 'g': - go_top() - elif key == 'v': - reselect() - return false # return to normal mode - -def get_operator_count_state(data, key): - if is_number(key): - data['count'] += key - return true - unshift_key(key) # return key to the input buffer - state_enter(delete_operator_state, data) - return false - -def insert_state(data, key): - if key == '': - return false # exit insert mode - self_insert(key) - return true -``` - -The above gives an idea of how Nvim is organized internally. Some states like -the `g_command_state` or `get_operator_count_state` do not have a dedicated -`state_enter` callback, but are implicitly embedded into other states (this -will change later as we continue the refactoring effort). To start reading the -actual code, here's the recommended order: - -1. `state_enter()` function (state.c). This is the actual program loop, - note that a `VimState` structure is used, which contains function pointers - for the callback and state data. -2. `main()` function (main.c). After all startup, `normal_enter` is called - at the end of function to enter normal mode. -3. `normal_enter()` function (normal.c) is a small wrapper for setting - up the NormalState structure and calling `state_enter`. -4. `normal_check()` function (normal.c) is called before each iteration of - normal mode. -5. `normal_execute()` function (normal.c) is called when a key is read in normal - mode. - -The basic structure described for normal mode in 3, 4 and 5 is used for other -modes managed by the `state_enter` loop: - -- command-line mode: `command_line_{enter,check,execute}()`(`ex_getln.c`) -- insert mode: `insert_{enter,check,execute}()`(`edit.c`) -- terminal mode: `terminal_{enter,execute}()`(`terminal.c`) - -## Important variables - -The current mode is stored in `State`. The values it can have are `MODE_NORMAL`, -`MODE_INSERT`, `MODE_CMDLINE`, and a few others. - -The current window is `curwin`. The current buffer is `curbuf`. These point -to structures with the cursor position in the window, option values, the file -name, etc. - -All the global variables are declared in `globals.h`. - -### The main event-loop - -The main loop is implemented in state_enter. The basic idea is that Vim waits -for the user to type a character and processes it until another character is -needed. Thus there are several places where Vim waits for a character to be -typed. The `vgetc()` function is used for this. It also handles mapping. - -What we consider the "Nvim event loop" is actually a wrapper around `uv_run` to -handle both the `fast_events` queue and possibly (a suitable subset of) deferred -events. Therefore "raw" `vim.uv.run()` is often not enough to "yield" from Lua -plugins; instead they can call `vim.wait(0)`. - -Updating the screen is mostly postponed until a command or a sequence of -commands has finished. The work is done by `update_screen()`, which calls -`win_update()` for every window, which calls `win_line()` for every line. -See the start of [drawscreen.c](drawscreen.c) for more explanations. - -### Command-line mode - -When typing a `:`, `normal_cmd()` will call `getcmdline()` to obtain a line with -an Ex command. `getcmdline()` calls a loop that will handle each typed -character. It returns when hitting `` or `` or some other character that -ends the command line mode. - -### Ex commands - -Ex commands are handled by the function `do_cmdline()`. It does the generic -parsing of the `:` command line and calls `do_one_cmd()` for each separate -command. It also takes care of while loops. - -`do_one_cmd()` parses the range and generic arguments and puts them in the -exarg_t and passes it to the function that handles the command. - -The `:` commands are listed in [ex_cmds.lua](ex_cmds.lua). - -### Normal mode commands - -The Normal mode commands are handled by the `normal_cmd()` function. It also -handles the optional count and an extra character for some commands. These -are passed in a `cmdarg_T` to the function that handles the command. - -There is a table `nv_cmds` in [normal.c](normal.c) which -lists the first character of every -command. The second entry of each item is the name of the function that -handles the command. - -### Insert mode commands - -When doing an `i` or `a` command, `normal_cmd()` will call the `edit()` function. -It contains a loop that waits for the next character and handles it. It -returns when leaving Insert mode. - -### Options - -There is a list with all option names in [options.lua](options.lua). - -Async event support -------------------- - -One of the features Nvim added is the support for handling arbitrary -asynchronous events, which can include: - -- RPC requests -- job control callbacks -- timers - -Nvim implements this functionality by entering another event loop while -waiting for characters, so instead of: - -```py -def state_enter(state_callback, data): - do - key = readkey() # read a key from the user - while state_callback(data, key) # invoke the callback for the current state -``` - -Nvim program loop is more like: - -```py -def state_enter(state_callback, data): - do - event = read_next_event() # read an event from the operating system - while state_callback(data, event) # invoke the callback for the current state -``` - -where `event` is something the operating system delivers to us, including (but -not limited to) user input. The `read_next_event()` part is internally -implemented by libuv, the platform layer used by Nvim. - -Since Nvim inherited its code from Vim, the states are not prepared to receive -"arbitrary events", so we use a special key to represent those (When a state -receives an "arbitrary event", it normally doesn't do anything other than -update the screen). - -Main loop ---------- - -The `Loop` structure (which describes `main_loop`) abstracts multiple queues -into one loop: - - uv_loop_t uv; - MultiQueue *events; - MultiQueue *thread_events; - MultiQueue *fast_events; - -`loop_poll_events` checks `Loop.uv` and `Loop.fast_events` whenever Nvim is -idle, and also at `os_breakcheck` intervals. - -MultiQueue is cool because you can attach throw-away "child queues" trivially. -For example `do_os_system()` does this (for every spawned process!) to -automatically route events onto the `main_loop`: - - Process *proc = &uvproc.process; - MultiQueue *events = multiqueue_new_child(main_loop.events); - proc->events = events; +- [dev_arch.txt](../../runtime/doc/dev_arch.txt) +- [dev_tools.txt](../../runtime/doc/dev_tools.txt) +- [develop.txt](../../runtime/doc/develop.txt)