`<div dir="ltr">Be sure you have the following fix; without it I recall seeing spins from the ZPL similar to that stack trace. With only 1 cpu, if a kernel thread spins, it can be very hard to get other threads to run.<div>
<br></div><div>
<p class="">commit e722410c49fe67cbf0f639cbcc288bd6cbcf7dd1</p>
<p class="">Author: Matthew <span class="">Ahrens</span> <<a href="mailto:mahrens@delphix.com">mahrens@delphix.com</a>></p>
<p class="">Date: Tue Nov 26 13:47:33 2013 -0500</p>
<p class=""><br></p>
<p class=""> 4347 ZPL can use dmu_tx_assign(TXG_WAIT)</p>
<p class=""> Reviewed by: George Wilson <<a href="mailto:george.wilson@delphix.com">george.wilson@delphix.com</a>></p>
<p class=""> Reviewed by: Adam Leventhal <<a href="mailto:ahl@delphix.com">ahl@delphix.com</a>></p>
<p class=""> Reviewed by: Dan McDonald <<a href="mailto:danmcd@nexenta.com">danmcd@nexenta.com</a>></p>
<p class=""> Reviewed by: Boris Protopopov <<a href="mailto:boris.protopopov@nexenta.com">boris.protopopov@nexenta.com</a>></p>
<p class=""> Approved by: Dan McDonald <<a href="mailto:danmcd@nexenta.com">danmcd@nexenta.com</a>></p></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 5, 2013 at 8:14 PM, Saso Kiselkov <span dir="ltr"><<a href="mailto:skiselkov.ml@gmail.com" target="_blank">skiselkov.ml@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm investigating a bizarre hang situation which I noticed by accident<br>
on the latest stable omnios release. When I'm running in VMware Fusion<br>
on a 1-CPU VM and doing any significant write IO to the pool (e.g. just<br>
dd'ing something around is enough to trigger this), the VM will, with<br>
100% certainty, hang. Console input works, but all userspace programs<br>
are stopped and nothing responds (e.g. attempting to telnet to sshd over<br>
the network establishes the socket, but then sshd doesn't print the<br>
version string).<br>
<br>
Using some dtrace foo and kmdb I was able to trace it (roughly, the<br>
exact stack trace changes between hangs, which is mighty weird in itself):<br>
<br>
atomic_dec_32_nv+8()<br>
dbuf_read+0x179(ffffff00d2393600, ffffff00c72f98f0, a)<br>
dmu_tx_check_ioerr+0x76(ffffff00c72f98f0, ffffff00d2279cf0, 0, 1e0)<br>
dmu_tx_count_write+0x395(ffffff00ce0536e0, 3c04000, 4000)<br>
dmu_tx_hold_write+0x5a(ffffff00d1a55300, 4009, 3c04000, 4000)<br>
zfs_write+0x3e3(ffffff00d09ef540, ffffff00028e7e60, 0,<br>
ffffff00cd511748, 0)<br>
fop_write+0x5b(ffffff00d09ef540, ffffff00028e7e60, 0,<br>
ffffff00cd511748, 0)<br>
write+0x250(1, 440660, 4000)<br>
sys_syscall+0x17a()<br>
<br>
(usually the trace is identical up to dmu_tx_hold_write)<br>
<br>
I can definitely confirm that this doesn't happen on omnios r151006 and<br>
it doesn't happen on my vanilla kernels either. My suspicion is that<br>
something got botched in the "OMNIOS#72 Integrate Joyent updated zone<br>
write throttle" commit, but I can't put my finger on it.<br>
<br>
Can somebody please confirm this?<br>
<br>
Cheers,<br>
<span class="HOEnZb"><font color="#888888">--<br>
Saso<br>
<br>
<br>
-------------------------------------------<br>
illumos-zfs<br>
Archives: <a href="https://www.listbox.com/member/archive/182191/=now" target="_blank">https://www.listbox.com/member/archive/182191/=now</a><br>
RSS Feed: <a href="https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460" target="_blank">https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460</a><br>
Modify Your Subscription: <a href="https://www.listbox.com/member/?member_id=21635000&id_secret=21635000-73dc201a" target="_blank">https://www.listbox.com/member/?member_id=21635000&id_secret=21635000-73dc201a</a><br>
Powered by Listbox: <a href="http://www.listbox.com" target="_blank">http://www.listbox.com</a><br>
</font></span></blockquote></div><br></div>
`