`<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>This may be a given, but, since you didn't mention this in your network topology.. Make sure the 1g LAN link is on a different subnet than the 20g iscsi link. Otherwise iscsi traffic might be flowing through the 1g link. Also jumbo frames can help with iscsi.</div><div><br></div><div>Additionally, dd speed tests from /dev/zero to a zfs disk are highly misleading if you have any compression enabled on the zfs disk (since only 512 bytes of disk is actually written for nearly any amount of consecutive zeros)<br><br>Michael<br>Sent from my iPhone</div><div><br>On Aug 30, 2015, at 7:17 AM, Steffen Wagner <<a href="mailto:mail@steffenwagner.com">mail@steffenwagner.com</a>> wrote:<br><br></div><blockquote type="cite"><div><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><meta name="Generator" content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--><div class="WordSection1"><p class="MsoNormal"><span lang="EN-US">Hi everyone!<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">I just setup a small network with 2 nodes:<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US">* 1 proxmox host on Debian Wheezy hosting KVM VMs<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US">* 1 napp-it host on OmniOS stable<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">The systems are currently connected through a 1 GBit link for general WAN and LAN communitcation and a 20 GBit link (two 10 GBit links aggregated) for the iSCSI communication.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US">Both connection's bandwidth was confirmed using iperf.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">The napp-it system currently has one pool (tank) consisting of 2 mirror vdevs. The 4 disks are SAS3 disks connected to a SAS2 backplane and directly attached (no expander) to the LSI SAS3008 (9300-8i) HBA.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US">Comstar is running on that Machine with 1 target (vm-storage) in 1 target group (vm-storage-group).<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">Proxmox has this iSCSI target configured as a "ZFS over iSCSI" storage using a block size of 8k and the "Write cache" option enabled.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US">This is where the problem starts:<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">dd if=/dev/zero of=/tank/test bs=1G count=20 conv=fdatasync<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">This dd test yields around 300 MB/s directly on the napp-it system.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">dd if=/dev/zero of=/home/test bs=1G count=20 conv=fdatasync<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">This dd test yields around 100 MB/s on a VM with it's disk on the napp-it system connected via iSCSI.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">The problem here is not the absolute numbers as these tests do not provide accurate numbers, the problem is the difference between the two values. I expected at least something around 80% of the local bandwidth, but this is usually around 30% or less.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">What I noticed during the tests: When running the test locally on the napp-it system, all disks will be fully utilized (read using iostat -x 1). When running the test inside a VM, the disk utilization barely reaches 30% (which seems to reflect the results of the bandwidth displayed by dd).<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">These 30% are only reached, if the locical unit of the VM disk has the writeback cache enabled. Disabling it results in 20-30 MB/s with the dd test mentioned above. Enabling it also increases the disk utilization.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">These values are also seen during the disk migration. Migrating one disk results in slow speed and low disk utilization. Migrating several disks in parallel will evetually cause 100% disk utilization.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">I also tested a NFS share as VM storage in proxmox. Running the same test inside a VM on the NFS share yields results around 200-220 MB/s. This is better (and shows that the traffic is going over the fast link between the servers), but not really yet as I still lose a third.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal"><span lang="EN-US">I am fairly new to the Solaris and ZFS world, so any help is greatly appreciated.<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p><p class="MsoNormal">Thanks in advance!<o:p></o:p></p><p class="MsoNormal"><o:p> </o:p></p><p class="MsoNormal">Steffen<o:p></o:p></p></div></div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>OmniOS-discuss mailing list</span><br><span><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a></span><br><span><a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a></span><br></div></blockquote></body></html>`