Closes#1234
multiqueue:
- Implement multiqueue_size()
- Rename MultiQueueItem.parent to MultiQueueItem.parent_item, to avoid confusion
with MultiQueue.parent.
Closes#3529Closes#5241
In Vim,
:echo system('cat - &', 'foo')
works because for both system() and :! Vim writes input to a temp file and uses
shell syntax to redirect the file to the backgrounded `cat` (get_cmd_output()
.. make_filter_cmd()).
In Nvim,
:echo system('cat - &', 'foo')
fails because we write the input directly via pipes (shell.c:do_os_system()),
but (per POSIX[1]) backgrounded process input stream is redirected from
/dev/null (unless overridden by shell redirection; supported only by some shells
[2]), so our writes are ignored, the process exits quickly, and if we are
writing data larger than the buffer size we'll see EPIPE.
This still works:
:%w !tee > foo1358.txt &
but this does not:
:%w !tee foo1358.txt &
though it *should* (why doesn't it?) because we still do the temp file dance
in do_bang() .. do_filter().
[1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_03_02
[2] http://unix.stackexchange.com/a/71218
`lib/queue.h` implements a basic queue. `event/queue.c` implements
a specialized data structure on top of lib/queue.h; it is not a "normal"
queue.
Rename the specialized multi-level queue implemented in event/queue.c to
"multiqueue", to avoid confusion when reading the code.
Before this change one can eventually notice that "macros (uppercase
symbols) are for the normal queue, lowercase operations are for the
multi-level queue", but that is unnecessary friction for new developers
(or existing developers just visiting this part of the codebase).
Because terminfo_stop() already ran, there is not much reason to wait
for the loop to teardown.
Helped-by: Björn Linse <bjorn.linse@gmail.com>
Helped-by: oni-link <knil.ino@gmail.com>
Closes#4778
References #3541
---
Bug report:
> After pressing `ZZ` I can find two threads freezing, occupying 100% CPU:
|-systemd-+
|-nvim,11567 /home/lz/code/1.rs +set title
| `-{nvim},11574
> 11567 has two threads:
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f7622907780 (LWP 11567) "nvim" 0x00007f76222e66bd in pthread_join () from /usr/lib/libpthread.so.0
2 Thread 0x7f761f5ff700 (LWP 11574) "nvim" 0x00007ffcec9e9c59 in clock_gettime ()
(gdb) thread apply all bt
Thread 2 (Thread 0x7f761f5ff700 (LWP 11574)):
#0 0x00007ffcec9e9c59 in clock_gettime ()
#1 0x00007f76210b9356 in clock_gettime () from /usr/lib/libc.so.6
#2 0x00007f7622513e3c in ?? () from /usr/lib/libuv.so.1
#3 0x00007f7622505e94 in uv_run () from /usr/lib/libuv.so.1
#4 0x00000000004822ed in loop_close (loop=loop@entry=0x7f761f5fe870) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/loop.c:87
#5 0x00000000005a7ec0 in tui_main (bridge=0x7f761f6ac000, ui=0x7f761f69ace0) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/tui/tui.c:234
#6 0x00000000005a9b47 in ui_thread_run (data=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/ui_bridge.c:87
#7 0x00007f7622510d07 in ?? () from /usr/lib/libuv.so.1
#8 0x00007f76222e5474 in start_thread () from /usr/lib/libpthread.so.0
#9 0x00007f76210ac69d in clone () from /usr/lib/libc.so.6
Thread 1 (Thread 0x7f7622907780 (LWP 11567)):
#0 0x00007f76222e66bd in pthread_join () from /usr/lib/libpthread.so.0
#1 0x00007f7622510dae in uv_thread_join () from /usr/lib/libuv.so.1
#2 0x00000000005aac9e in ui_bridge_stop (b=0x7f761f6ac000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/ui_bridge.c:104
#3 0x00000000005a90e4 in ui_builtin_stop () at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/ui.c:91
#4 0x000000000052be09 in mch_exit (r=r@entry=1) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os_unix.c:143
#5 0x00000000004db1cc in getout (exitval=exitval@entry=1) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/main.c:623
#6 0x00000000004fa43c in preserve_exit () at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/misc1.c:2652
#7 0x000000000052b77a in deadly_signal (signum=1) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/signal.c:120
#8 0x000000000052b7cf in on_signal (handle=<optimized out>, signum=<optimized out>, data=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/signal.c:145
#9 0x0000000000484178 in signal_event (argv=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/signal.c:44
#10 0x0000000000483b7f in queue_process_events (queue=0x7f7620417360) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/queue.c:142
#11 0x0000000000482208 in loop_poll_events (loop=0x84dec0 <loop>, ms=ms@entry=4000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/loop.c:56
#12 0x000000000052a364 in input_poll (ms=ms@entry=4000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/input.c:325
#13 0x000000000052a3e2 in inbuf_poll (ms=4000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/input.c:347
#14 0x000000000052a839 in os_inchar (buf=buf@entry=0x0, maxlen=maxlen@entry=0, ms=ms@entry=-1, tb_change_cnt=tb_change_cnt@entry=0) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/input.c:107
#15 0x0000000000592eeb in state_enter (s=s@entry=0x7ffcec9d3560) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/state.c:46
#16 0x0000000000508533 in normal_enter (cmdwin=cmdwin@entry=false, noexmode=noexmode@entry=false) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/normal.c:464
#17 0x00000000004dc17e in main (argc=<optimized out>, argv=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/main.c:538
Inherited signal mask may block SIGCHLD, which causes libuv to hang at
epoll_wait.
Closes#5230
Helped-by: Nicolas Hillegeer <nicolas@hillegeer.com>
Helped-by: John Szakmeister <john@szakmeister.net>
Note: the #pragma gymnastics are a workaround for broken system headers on
macOS.
signal.h:
int sigaddset(sigset_t *, int);
#define sigaddset(set, signo) (*(set) |= __sigbits(signo), 0)
sys/_types/_sigset.h:
typedef __darwin_sigset_t sigset_t;
sys/_types.h:
typedef __uint32_t __darwin_sigset_t; /* [???] signal set */
sigset_t is defined as unsigned int, but the sigaddset() ORs it with an int,
mixing the types. So GCC generates a sign-conversion warning:
sig.c:9:13: warning: implicit conversion changes signedness: 'int' to 'unsigned int' [-Wsign-conversion]
(*(&s) |= __sigbits((sigset_t) 20), 0);
~~ ^~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
System headers are normally ignored when the compiler generates warnings:
https://gcc.gnu.org/onlinedocs/cpp/System-Headers.html
> GCC gives code found in system headers special treatment. All warnings,
> other than those generated by ‘#warning’ (see Diagnostics), are suppressed
> while GCC is processing a system header. Macros defined in a system header
> are immune to a few warnings wherever they are expanded. This immunity is
> granted on an ad-hoc basis, when we find that a warning generates lots of
> false positives because of code in macros defined in system headers.
Instead of the #pragma workaround, we could cast the sigset_t pointer:
# if defined(__APPLE__)
sigaddset((int *)&mask, SIGCHLD);
# else
sigaddset(&mask, SIGCHLD);
# endif
but that could break if the headers are later fixed.
Processes in vim are always started in the current directory, which
causes issues when the process is a daemon and the current directory is
a mountpoint. Fix this by adding an option to set the cwd of the new
process with jobstart(). In addition, fix termopen() so that it actually
uses the cwd option from the dict (it couldn't previously set the cwd
value due to dead code).
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Compiling with macro -DEXITFREE opens a code path on which the event
loop is used after it was teared down, because not all close events
were processed yet.
nvim blocking can be tested with "nvim +te +'!xclip' +qa"
By closing all handles for a pty process, we unblock the event loop if
the process has not terminated yet.
The only data loss should be, if a process forked a child that keeps
sending data after the parent terminated.
While not in teardown mode we could keep reading child data, but then
`:!cmd` would block after `cmd` exited. In teardown mode we want to exit
nvim so we cannot keep reading child data.
In windows libuv does not return -errno, instead it uses negative
error codes e.g. UV_ENOENT. This commit changes the comments in os_*
functions to reflect this.
When calling uv_spawn to launch a process set the libuv process flag
UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS to disable escaping for the
arguments otherwise libuv will wrap arguments in quotes.
- Implement `loop_schedule` method for queueing events from other threads
- Make `loop_poll_events` `recursive` static variable a field of the Loop
structure
Processing a stream's output can be queued. If stream_close() is called
before the queue is processed, the RBuffer containing the stream's data
is freed and the next read event would try to access freed memory.
To fix this behavior, use the stream's pending requests counter.
The new event processing architecture changed `jobwait()` semantics: Only one
job is processed at time since process_wait only focuses on one queue.
This fixes the problem with a few changes:
- Allow the event queue polled by `process_wait` to be overriden by a new
argument.
- Allow the parent queue to be overriden with `queue_replace_parent`
- Create a temporary queue that serves as the parent for all jobs passed to
`jobwait()`
Since pty events are queued, it is possible that the reads will be reordered.
Example scenario:
In the terminal you have output combined from stdout and stderr. A program
generates output, first you have some output on stdout, then output on stderr,
output on stdout, output on stderr,... The whole output should be interleaved
from both streams.
Each output generates a read_event and they are placed in the same queue. If the
queue is processed, the first read_event will send the whole stdout output to
the terminal (on_job_output() consumes the whole buffer). The next read_event is
similar for stderr. The remaining read events do nothing because now both
RBuffer are already empty. So the terminal would show first the stdout output
and after that the stderr output.
This commit fixes the problem by disabling stderr stream in pty processes.
That's ok because they all represent the same stream(duplicate file
descriptors), plus one stream is simpler to deal with.
- Improve the implementation of deferred/immediate events.
- Use the new queue module to change how/when events are queued/processed by
giving a private queue to each emitter.
- Immediate events(which only exist to break uv_run recursion) are now
represented in the `loop->fast_events` queue.
- Events pushed to child queues are propagated to the event loop main queue and
processed as K_EVENT keys.
- New libuv/pty process abstraction with simplified API and no globals.
- Remove nvim/os/job*. Jobs are now a concept that apply only to programs
spawned by vimscript job* functions.
- Refactor shell.c/channel.c to use the new module, which brings a number of
advantages:
- Simplified API, less code
- No slots in the user job table are used
- Not possible to acidentally receive data from vimscript
- Implement job table in eval.c, which is now a hash table with unilimited job
slots and unique job ids.
- Simplify RStream/WStream API and make it more consistent with libuv.
- Move into the event loop layer(event subdirectory)
- Remove uv_helpers module.
- Simplify job/process internal modules/API.
- Unify RStream and WStream into a single structure. This is necessary because
libuv streams can be readable and writable at the same time(and because the
uv_helpers.c hack to associate multiple streams with libuv handle was removed)
- Make struct definition public, allowing more flexible/simple memory
management by users of the module.
- Adapt channel/job modules to cope with the changes.
- Add event loop abstraction module under src/nvim/event. The
src/nvim/event/loop module replaces src/nvim/os/event
- Remove direct dependency on libuv signal/timer API and use the new abstraction
instead.
- Replace all references to uv_default_loop() by &loop.uv, a new global variable
that wraps libuv main event loop but allows the event loop functions to be
reused in other contexts.
