This is a better way to prevent parallel tests from interfering with
each other, as there are many ways files can be created and deleted in
tests, so enforcing different file names is hard.
Using $TMPDIR can also work in most cases, but 'backipskip' etc. have
special defaults for $TMPDIR.
Symlink runtime/, src/, test/ and README.md to Xtest_xdg dir to make
tests more convenient (and symlinking test/ is required for busted).
Also, use README.md instead of test/README.md in the Ex mode inccommand
test, as test/README.md no longer contains 'N' char.
- Rename ".nvimlog" to "nvim.log"
- doesn't need to be "hidden"/dotfile
- ".log" extension helps with filetype detection
- Also rename "nvim/log" => "nvim/nvim.log"
Define a CMake target for every subdirectory of test/functional that
contains functional tests, and a functionaltest_parallel target that
depends on all those targets, allowing multiple test runners to run in
parallel.
On CI, use at most 2 parallel test runners, as using more may increase
system load and make tests unstable.
Problem:
We want to encourage implementing core features in Lua instead of C, but
it's clumsy because:
- Core Lua code (built into `nvim` so it is available even if VIMRUNTIME
is missing/invalid) requires manually updating CMakeLists.txt, or
stuffing it into `_editor.lua`.
- Core Lua modules are not organized similar to C modules, `_editor.lua`
is getting too big.
Solution:
- Introduce `_core/` where core Lua code can live. All Lua modules added
there will automatically be included as bytecode in the `nvim` binary.
- Move these core modules into `_core/*`:
```
_defaults.lua
_editor.lua
_options.lua
_system.lua
shared.lua
```
TODO:
- Move `_extui/ => _core/ui2/`
Problem:
Every CI log has a lot of noise at the end, which makes it harder to
find relevant test failures:
Running tests from test/functional/terminal/tui_spec.lua
...
T5831 TUI bg color queries the terminal for background color:
T5832 TUI bg color triggers OptionSet from automatic background processing:
T5833 TUI bg color sends theme update notifications when background changes #31652:
...
Running tests from test/functional/ui/output_spec.lua
...
WRN 2025-12-02T03:36:47.304 ui/c/T5831.28003.0 tui_handle_term_mode:223: TUI: terminal mode 2026 unavailable, state 0
WRN 2025-12-02T03:36:47.359 ui/c/T5832.28006.0 tui_handle_term_mode:223: TUI: terminal mode 2048 unavailable, state 0
WRN 2025-12-02T03:36:47.414 ui/c/T5833.28009.0 tui_handle_term_mode:223: TUI: terminal mode 2048 unavailable, state 0
Solution:
- Skip logging in test-mode.
- This can be reverted later, when these logs are changed to "INFO"
level, per this TODO comment:
```
// TODO(bfredl): This is really ILOG but we want it in all builds.
// add to show_verbose_terminfo() without being too racy ????
WLOG("TUI: terminal mode %d unavailable, state %d", mode, state);
```
This prevents Nvim TUI running in tests from thinking it's inside tmux.
Another solution is make :terminal unset $TMUX instead, but I'm not sure
if that'll break some other use cases.
This is a diabolical anti-pattern and is hiding errors which exist
currently in unittests. Also we want to see _where_ in the process
stderr was emitted, stashing it away at the end erases important
context.
NEW BUILD SYSTEM!
This is a MVP implementation which supports building the "nvim" binary,
including cross-compilation for some targets.
As an example, you can build a aarch64-macos binary from
an x86-64-linux-gnu host, or vice versa
Add CI target for build.zig currently for functionaltests on linux
x86_64 only
Follow up items:
- praxis for version and dependency bumping
- windows 💀
- full integration of libintl and gettext (or a desicion not to)
- update help and API metadata files
- installation into a $PREFIX
- more tests and linters
- Move all generator Lua scripts to the `src/gen/`
- Add a `.luarc.json` to `src/gen/`
- Add a `preload.lua` to `src/gen/`
- Add `src` to `package.path` so it aligns with `.luarc.json'
- Fix all `require` statements in `src/gen/` so they are consistent:
- `require('scripts.foo')` -> `require('gen.foo')`
- `require('src.nvim.options')` -> `require('nvim.options')`
- `require('api.dispatch_deprecated')` -> `require('nvim.api.dispatch_deprecated')`
- add EXTERNALPROJECT_OPTIONS variable to main build
- use `REQUIRED` keyword for IWYU.
- remove check_c_compiler_flag checks when `ENABLE_COMPILER_SUGGESTIONS`
is `ON`. If we explicitly enable it then we probably want it to give
an error if it doesn't exist, rather than silently skip it.
- Move dependency interface libraries to their find module and use them
as a pseudo-imported target.
- Remove BUSTED_OUTPUT_TYPE. It's not used and we can reintroduce it
again if something similar is needed.
- Use LINK_OPTIONS intead of LINK_FLAGS when generating the `--version`
output.
Problem:
Some tests fail with $SHELL=fish #6172
Related: https://github.com/neovim/neovim/pull/6176
Solution:
Replace "echo -n" with "printf", because "echo" in sh may be provided
as a shell builtin, which does not accept an "-n" flag to avoid a
trailing newline (e.g. on macos). "printf" is more portable (defined by
POSIX) and it does not output a trailing newline by itself.
Fixes#6172
TODO:
Other test failures may be related to "session leader" issue: https://github.com/neovim/neovim/issues/2354
Checked by running `:terminal ./build/bin/tty-test` from Nvim with
`shell=/bin/fish` (inherited from `$SHELL`) and it indeed complains
about "process does not own the terminal". With `shell=sh` it doesn't complain. And
unsetting `$SHELL` seems to make `nvim` to fall back to `shell=sh`.
FAILED test/functional/terminal/tui_spec.lua @ 1017: TUI paste: terminal mode
test/functional/terminal/tui_spec.lua:1024: Row 1 did not match.
Expected:
|*tty ready |
|*{1: } |
|* |
| |
|{5:^^^^^^^ }|
|{3:-- TERMINAL --} |
|{3:-- TERMINAL --} |
Actual:
|*process does not own the terminal |
|* |
|*[Process exited 2]{1: } |
| |
|{5:^^^^^^^ }|
|{3:-- TERMINAL --} |
|{3:-- TERMINAL --} |
To print the expect() call that would assert the current screen state, use
screen:snapshot_util(). In case of non-deterministic failures, use
screen:redraw_debug() to show all intermediate screen states.
stack traceback:
test/functional/ui/screen.lua:622: in function '_wait'
test/functional/ui/screen.lua:352: in function 'expect'
test/functional/terminal/tui_spec.lua:1024: in function <test/functional/terminal/tui_spec.lua:1017>
FAILED test/functional/terminal/tui_spec.lua @ 1551: TUI forwards :term palette colors with termguicolors
test/functional/terminal/tui_spec.lua:1567: Row 1 did not match.
Expected:
|*{1:t}ty ready |
| |
|* |
| |
|{2:^^^^^^^ }|
| |
|{3:-- TERMINAL --} |
Actual:
|*{1:p}rocess does not own the terminal |
| |
|*[Process exited 2] |
| |
|{2:^^^^^^^ }|
| |
|{3:-- TERMINAL --} |
To print the expect() call that would assert the current screen state, use
screen:snapshot_util(). In case of non-deterministic failures, use
screen:redraw_debug() to show all intermediate screen states.
stack traceback:
test/functional/ui/screen.lua:622: in function '_wait'
test/functional/ui/screen.lua:352: in function 'expect'
test/functional/terminal/tui_spec.lua:1567: in function <test/functional/terminal/tui_spec.lua:1551>
Problem:
`nvim -v` and `:version` prints system vimrc, fallback files, and compilation
info by default, which most people don't care about and just clutters up the
output.
Solution:
Omit extra info unless 'verbose' is set.
- Simplify error checking when using execute_process.
- Set BUILD_SHARED_LIBS to OFF when building dependencies.
This is normally not needed, but msgpack interprets an unset
BUILD_SHARED_LIBS to build a shared library, which is the opposite of
the cmake behavior.
- Move function check_lua_module to Util.cmake.
- Remove unnecessary code.
- Make variable naming more consistent
Eliminates lua-client and non-static libluv as test time dependencies
Note: the API for a public lua-client is not yet finished.
The interface needs to be adjusted to work in the embedded loop
of a nvim instance (to use it to talk between instances)
This partially reverts commit 42aeb5c5b1.
Setting cmake policies is normally not required as
cmake_minimum_required automatically sets these. One exception is cmake
script mode (-P) since it automatically resets all policy changes.
Closes: https://github.com/neovim/neovim/issues/20286
Problem:
RunTests.cmake adds $TEST_PATH to $TMPDIR with the implication that it
gives more isolation. But this is misleading because $TEST_PATH is only
defined once. Full test runs use the same $TMPDIR for all tests.
This was likely added with the intention of invoking RunTests.cmake
once-per-testfile from a wrapper than does the isolation/orchestration.
But even so, Nvim's vim_maketempdir() / vim_mktempdir() _already_
creates a unique tempdir per session.
Solution:
Don't append $TEST_PATH to $TMPDIR. Avoids confusion and makes the path
shorter.
Problem:
Since 22b52dd462#11501, log_path_init is called in log_init, so it is
now called at a deterministic time. So the "just in time" complexity of
log_path_init is no longer needed.
Solution:
Remove logic intended to try to "heal" partial initialization.
Problem:
Some tests were not passing on my machine, specifically in
`test/functional/api/vim_spec.lua` the two tests under
`describe('nvim_get_runtime_file...`
Solution:
Unset `XDG_DATA_DIRS` in the test runner.
Note: Window CI failed if we set it to the same value as `XDG_DATA_HOME`.
Set a maximum test run-time of 20 minutes to:
- fail faster
- avoid wasting CI resources
- set a bound on the test time (if tests take longer than 20 minutes we
need to invest in parallelizing them...)
Timeout looks like:
-- Tests exited non-zero: Process terminated due to timeout
-- No output to stderr.
CMake Error at /…/neovim/cmake/RunTests.cmake:86
(message):
functional tests failed with error: Process terminated due to timeout
* tests: move "busted" dir to "test"
It is used for outputHandlers only, and clearly belongs to the tests.
Use the full module name with the `-o` option to `busted` then for
clarity.
* luacheck
* test/busted/outputHandlers/TAP.lua: use/extend upstream
Just set it from `${CMAKE_HOST_SYSTEM_NAME}` directly, instead of passing it
from the main CMake file (CMAKE_SYSTEM_NAME defaults to it, but is empty
in script mode).
Initially added in 9ce81f7b2, but then even used with unrelated commands
(that do not use RunTests.cmake, e.g. 221f6fffa).
The feature is very rudimentary [1], and causes problems when not using
bundled deps, where it might then fail to find `busted.runner` due to
this.
E.g. with "-DUSE_BUNDLED=OFF -DUSE_BUNDLED_LUAROCKS=ON":
`.deps/usr/bin/busted` contains `exec '/usr/bin/luajit' -e …`, i.e. it
run luajit with adjusted lpath etc.
But using `--lua /usr/bin/luajit` then causes this `busted` wrapper to
be replaced with just `/usr/bin/luajit`, i.e. it is missing the adjusted
lpath then.
1: 9eb6c9cf93
- allow for passing in BUSTED_ARGS via env
- quote values of TEST_TAG/TEST_FILTER
Previously TEST_FILTER="'foo bar'" was required.
This allows for:
make functionaltest TEST_FILE=test/functional/terminal/tui_spec.lua \
BUSTED_ARGS="--no-keep-going --shuffle" \
TEST_FILTER="TUI background color handles"
* RunTests.cmake: BUILD_DIR for Xtest files, isolated TMPDIR
Assume relative path for given TEST_PATH.
ref #9280
Introduce the `vim.compat` module, to help environments with system Lua
5.2+ run the build/tests. Include the module implicitly in all tests.
ref #8677
legacy `vim` module:
beep
buffer
command
dict
eval
firstline
lastline
line
list
open
type
window
Tests that check localized error messages need a stable locale, else
errors like this occur:
[ FAILED ] 2 tests, listed below:
[ FAILED ] ...npack/file/vim/neovim/test/functional/eval/null_spec.lua @ 29: NULL list is accepted as an empty list by writefile()
...npack/file/vim/neovim/test/functional/eval/null_spec.lua:30: Expected objects to be the same.
Passed in:
(string) '
E484: Cannot open file Xtest-functional-viml-null'
Expected:
(string) '
E484: Can't open file Xtest-functional-viml-null'
stack traceback:
...npack/file/vim/neovim/test/functional/eval/null_spec.lua:30: in function <...npack/file/vim/neovim/test/functional/eval/null_spec.lua:29>
[ FAILED ] ...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua @ 81: :write errors out correctly
...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:97: Expected objects to be the same.
Passed in:
(string) 'Vim(write):E510: Cannot make backup file (add ! to override)'
Expected:
(string) 'Vim(write):E510: Can't make backup file (add ! to override)'
stack traceback:
...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:97: in function <...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:81>
10 SKIPPED TESTS
2 FAILED TESTS
-- Output to stderr:
2017/07/29 00:41:32 ERROR 31133/open_log_file:170: Logging to stderr, failed to open $NVIM_LOG_FILE: Xtest-startup-xdg-logpath/nvim/log
2017/07/29 00:41:32 WARN 31133/call_set_error:815: RPC: ch 1 was closed by the client
CMake Error at /home/shlomif/Download/unpack/file/vim/neovim/cmake/RunTests.cmake:50 (message):
Running functional tests failed with error: 1.
FAILED: CMakeFiles/functionaltest
cd /home/shlomif/Download/unpack/file/vim/neovim/build && /usr/bin/cmake -DBUSTED_PRG=/home/shlomif/Download/unpack/file/vim/neovim/.deps/usr/bin/busted -DLUA_PRG=/home/shlomif/Download/unpack/file/vim/neovim/.deps/usr/bin/luajit -DNVIM_PRG=/home/shlomif/Download/unpack/file/vim/neovim/build/bin/nvim -DWORKING_DIR=/home/shlomif/Download/unpack/file/vim/neovim -DBUSTED_OUTPUT_TYPE=nvim -DTEST_DIR=/home/shlomif/Download/unpack/file/vim/neovim/test -DBUILD_DIR=/home/shlomif/Download/unpack/file/vim/neovim/build -DTEST_TYPE=functional -DSYSTEM_NAME=Linux -P /home/shlomif/Download/unpack/file/vim/neovim/cmake/RunTests.cmake
ninja: build stopped: subcommand failed.
Makefile:102: recipe for target 'functionaltest' failed
make: *** [functionaltest] Error 1
- Do not delete it: may need to inspect it after tests finished.
- Avoids writing to stderr in cases where the test-local $XDG_DATA_HOME
was not created yet.
