`<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Dan,<br>
<br>
just wanted to know, if you would be interested in the core dump of
this crash:<br>
<br>
Jan 11 07:28:52 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk@g600144f090d09613000056b8a83b0007 (sd27):<br>
Jan 11 07:28:52 zfsha02gh79 incomplete write- retrying<br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79
/scsi_vhci/disk@g600144f0564d504f4f4c3035534c3133 (sd47): Command
Timeout on path
<a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,3">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,3</a><br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79
/scsi_vhci/disk@g600144f0564d504f4f4c3035534c3039 (sd44): Command
Timeout on path
<a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,0">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,0</a><br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79
/scsi_vhci/disk@g600144f0564d504f4f4c3035534c3131 (sd46): Command
Timeout on path
<a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,2">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,2</a><br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79
/scsi_vhci/disk@g600144f0564d504f4f4c3035534c3130 (sd45): Command
Timeout on path
<a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,1">iscsi0/disk@0000iqn.2016-02.de.jvm:nfsvmpool05ssd030002,1</a><br>
Jan 12 17:30:22 zfsha02gh79 iscsi: [ID 431120 kern.warning] WARNING:
iscsi connection(26/3f) closing connection - target requested
reason:0x7<br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79
/scsi_vhci/disk@g600144f090d09613000056b8a7f10003 (sd19): Command
Timeout on path
<a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,2">iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,2</a><br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79
/scsi_vhci/disk@g600144f090d09613000056b8a7fc0004 (sd21): Command
Timeout on path
<a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,3">iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,3</a><br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79
/scsi_vhci/disk@g600144f090d09613000056b8a84a0008 (sd29): Command
Timeout on path
<a class="moz-txt-link-abbreviated" href="mailto:iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,7">iscsi0/disk@0000iqn.2015-03.de.jvm:nfsvmpool05ssd010002,7</a><br>
Jan 12 17:30:22 zfsha02gh79 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 836849 kern.notice]<br>
Jan 12 17:30:22 zfsha02gh79 ^Mpanic[cpu1]/thread=ffffff00f6539c40:<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 335743 kern.notice] BAD
TRAP: type=e (#pf Page fault) rp=ffffff00f6539610 addr=10 occurred
in module "scsi_vhci" due to a NULL pointer dereference<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice]<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 839527 kern.notice] sched:<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 753105 kern.notice] #pf Page
fault<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 532287 kern.notice] Bad kernel
fault at addr=0x10<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 243837 kern.notice] pid=0,
pc=0xfffffffff7948e15, sp=0xffffff00f6539700, eflags=0x10246<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 211416 kern.notice] cr0:
8005003b<pg,wp,ne,et,ts,mp,pe> cr4:
1426f8<smep,osxsav,vmxe,xmme,fxsr,pge,mce,pae,pse,de><br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 624947 kern.notice] cr2: 10<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 625075 kern.notice] cr3:
c000000<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 625715 kern.notice] cr8: 0<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice]<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice] rdi:
ffffff226adb90d8 rsi: 1 rdx: ffffff227063d400<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
rcx: 2 r8: 0 r9: fffffffff794bd10<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
rax: 0 rbx: 1 rbp: ffffff00f6539780<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
r10: 0 r11: ffffff00f65397b0 r12: 2<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
r13: 1 r14: 0 r15: 4<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
fsb: 0 gsb: ffffff21f0e81040 ds: 4b<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
es: 4b fs: 0 gs: 1c3<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
trp: e err: 0 rip: fffffffff7948e15<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 592667 kern.notice]
cs: 30 rfl: 10246 rsp: ffffff00f6539700<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 266532 kern.notice]
ss: 38<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice]<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f65394f0 unix:die+df ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539600 unix:trap+dd8 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539610 unix:_cmntrap+e6 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539780 scsi_vhci:vhci_scsi_reset_target+75 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f65397d0 scsi_vhci:vhci_recovery_reset+7d ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539820 scsi_vhci:vhci_pathinfo_offline+e5 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f65398c0 scsi_vhci:vhci_pathinfo_state_change+d5 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539950 genunix:i_mdi_pi_state_change+16a ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539990 genunix:mdi_pi_offline+39 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539a20 iscsi:iscsi_lun_offline+b3 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539a60 iscsi:iscsi_sess_offline_luns+4d ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539ab0 iscsi:iscsi_sess_state_logged_in+11e ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539b00 iscsi:iscsi_sess_state_machine+13e ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539b60 iscsi:iscsi_client_notify_task+17e ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539c20 genunix:taskq_thread+2d0 ()<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 655072 kern.notice]
ffffff00f6539c30 unix:thread_start+8 ()<br>
Jan 12 17:30:22 zfsha02gh79 unix: [ID 100000 kern.notice]<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 672855 kern.notice] syncing
file systems...<br>
Jan 12 17:30:24 zfsha02gh79 genunix: [ID 904073 kern.notice] done<br>
Jan 12 17:30:22 zfsha02gh79 genunix: [ID 111219 kern.notice] dumping
to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel<br>
Jan 12 17:30:22 zfsha02gh79 ahci: [ID 405573 kern.info] NOTICE:
ahci0: ahci_tran_reset_dport port 0 reset port<br>
Jan 12 17:48:01 zfsha02gh79 genunix: [ID 100000 kern.notice]<br>
Jan 12 17:48:02 zfsha02gh79 genunix: [ID 665016 kern.notice] ^M100%
done: 4721646 pages dumped,<br>
<br>
This happend on a rather higher load situation, when I was copying a
200G file from a snapshot back to it's original place on its zvol,
when this happened. Luckily these are RSF-1 nodes and the other one
took over very quickliy, such as that my VM cluster didn't even seem
to notice this issue. However, at that time I was conencted to the
crashing host via ssh and my heart skipped a beat. ;)<br>
<br>
As I have (unvoluntarily) freed this node of it's duties, I could
jump to r020 on it, but I wonder if there has been any changes to
the scsi_vhci layer at all in recent times…<br>
<br>
Cheers,<br>
Stephan<br>
<pre class="moz-signature" cols="72">--
Krebs's 3 Basic Rules for Online Safety
1st - "If you didn't go looking for it, don't install it!"
2nd - "If you installed it, update it."
3rd - "If you no longer need it, remove it."
<a class="moz-txt-link-freetext" href="http://krebsonsecurity.com/2011/05/krebss-3-basic-rules-for-online-safety">http://krebsonsecurity.com/2011/05/krebss-3-basic-rules-for-online-safety</a>
Stephan Budach
Head of IT
Jung von Matt AG
Glashüttenstraße 79
20357 Hamburg
Tel: +49 40-4321-1353
Fax: +49 40-4321-1114
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:stephan.budach@jvm.de">stephan.budach@jvm.de</a>
Internet: <a class="moz-txt-link-freetext" href="http://www.jvm.com">http://www.jvm.com</a>
CiscoJabber Video: <a class="moz-txt-link-freetext" href="https://exp-e2.jvm.de/call/stephan.budach">https://exp-e2.jvm.de/call/stephan.budach</a>
Vorstand: Dr. Peter Figge, Jean-Remy von Matt, Larissa Pohl, Thomas Strerath, Götz Ulmer
Vorsitzender des Aufsichtsrates: Hans Hermann Münchmeyer
AG HH HRB 72893
</pre>
</body>
</html>
`