`<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 23, 2015, at 12:58 AM, Dan Vatca <<a href="mailto:dan@syneto.eu" class="">dan@syneto.eu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">If you need latency, you will most likely need a latency distribution histogram, and not an average latency.<div class="">With averages you will lose latency outliers that are very important. Here's a good read with lots of references on this topic: <a href="https://www.vividcortex.com/blog/why-percentiles-dont-work-the-way-you-think" class="">https://www.vividcortex.com/blog/why-percentiles-dont-work-the-way-you-think</a></div><div class="">To currently do this on OmniOS, you need to use dtrace to aggregate (quantize) time differences between nfsv3:::op-read-start and nfsv3:::op-read-done (same for write).</div></div></div></blockquote><div><br class=""></div><div>Indeed, distributions are much more enlightening than averages.</div><div><br class=""></div><div>Unfortunately, the new kstats added for NFS server operations on a per-mountpoint basis</div><div>are implemented using the Riemann sums (KSTAT_TYPE_IO) and it is not possible to obtain</div><div>per-operation information needed for min/max or distribution. These are the same type of </div><div>kstat used for the iostat command.</div><div><br class=""></div><div>Shameless plug, nfssvrtop has proven to be useful in watching NFS traffic and uses the </div><div>op-read-start/op-read-done method.</div><div><a href="https://github.com/richardelling/tools" class="">https://github.com/richardelling/tools</a></div><div><br class=""></div><div> — richard</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div></div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div style="font-family: Monaco; font-size: 12px;" class=""><div style="color: rgb(34, 34, 34); margin: 0px; font-size: 10px; font-family: Verdana;" class="">Dan Vâtca</div><div style="color: rgb(34, 34, 34); margin: 0px; font-size: 10px; font-family: Verdana;" class="">CTO at Syneto</div><div style="color: rgb(34, 34, 34); margin: 0px; font-size: 10px; font-family: Verdana;" class="">Tel: +40723604357, Skype: dan_vatca</div><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px" class=""><a href="http://www.syneto.net/" target="_blank" class=""><img src="https://syneto.eu/wp-content/uploads/2014/07/storage_logo_icons8.png" class=""></a></p></div></div></div></div></div></div>
<br class=""><div class="gmail_quote">On Wed, Dec 23, 2015 at 2:44 AM, 張峻宇 <span dir="ltr" class=""><<a href="mailto:jerry1209@cht.com.tw" target="_blank" class="">jerry1209@cht.com.tw</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="ZH-TW" link="#0563C1" vlink="#954F72" class="">
<div class=""><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class="">Hi all,
<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class=""> According to the release note of OmniOS r151016</span><span lang="EN-US" class="">, we could get “</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class="">IOPS,
bandwidth, and latency kstats for NFS server”<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class=""> there is lots of information showing when I use enter command #kstat,<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class=""> I want to get the “nfs read & write latency for NFS server”<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class="">
<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class=""> Q1 : Is the ‘nfs:0:rfsprocio_v4_write:wtime’ & ‘nfs:0:rfsprocio_v4_read:wtime’ meant write & read latency ?<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040" class=""> Q2 : I mounted the nfs share directory, and write lots file to it, the number of ‘nfs:0:rfsprocio_v4_write:wtime’ & ‘nfs:0:rfsprocio_v4_read:wtime’
still zero. Why ? <u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040;background:white" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040;background:white" class=""> #kstat –p –m nfs –n rfsprocio_v4_write<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" class=""> nfs:0:rfsprocio_v4_write:class rfsprocio_v4<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:crtime 50.833043074<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:nread 3932160<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:nwritten 5374607360<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:rcnt 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:reads 163840<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:rlastupdate 12048225488385<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:rlentime 33429565743<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:rtime 23992279289<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:snaptime 269635.483575440<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:wcnt 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:wlastupdate 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:wlentime 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:writes 163840 / number of writes /<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_write:wtime 0 / wait queue - time spent waiting /<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" class=""> </span><span lang="EN-US" style="font-size:11.0pt;font-family:"Lucida Sans Unicode",sans-serif;color:#404040;background:white" class="">#kstat –p –m nfs –n rfsprocio_v4_read</span><span lang="EN-US" class=""><u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" class=""> nfs:0:rfsprocio_v4_read:class rfsprocio_v4<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:crtime 50.833003263<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:nread 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:nwritten 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:rcnt 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:reads 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:rlastupdate 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:rlentime 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:rtime 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:snaptime 269635.483080962<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:wcnt 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:wlastupdate 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:wlentime 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:writes 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-left:24.0pt"><span lang="EN-US" class="">nfs:0:rfsprocio_v4_read:wtime 0<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" class=""> <u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"\005fae\008edf\006b63\009ed1\009ad4",sans-serif;color:#1f497d" class="">Best regards,<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"\0065b0\007d30\00660e\009ad4",serif;color:#1f497d" class="">---------------------------------------------<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-family:"\0065b0\007d30\00660e\009ad4",serif;color:#1f497d" class="">張峻宇<span lang="EN-US" class=""><u class=""></u><u class=""></u></span></span></p><p class="MsoNormal"><span style="font-family:"\0065b0\007d30\00660e\009ad4",serif;color:#1f497d" class="">中華電信研究院雲端運算研究所<span lang="EN-US" class=""><u class=""></u><u class=""></u></span></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"\0065b0\007d30\00660e\009ad4",serif;color:#1f497d" class="">TEL: 03-4245663</span><span lang="EN-US" style="color:#1f497d" class=""><u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span lang="EN-US" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<b class=""><br class=""><br class=""><font size="-1" class="">本信件可能包含中華電信股份有限公司機密資訊,非指定之收件者,請勿蒐集、處理或利用本信件內容,並請銷毀此信件.
如為指定收件者,應確實保護郵件中本公司之營業機密及個人資料,不得任意傳佈或揭露,並應自行確認本郵件之附檔與超連結之安全性,以共同善盡資訊安全與個資保護責任.
<br class="">Please be advised that this email message (including any attachments) contains confidential information and may be legally privileged. If you are not the intended recipient, please destroy this message and all attachments from your system and do not further collect, process, or use them. Chunghwa Telecom and all its subsidiaries and associated companies shall not be liable for the improper or incomplete transmission of the information contained in this email nor for any delay in its receipt or damage to your system. If you are the intended recipient, please protect the confidential and/or personal information contained in this email with due care. Any unauthorized use, disclosure or distribution of this message in whole or in part is strictly prohibited. Also, please self-inspect attachments and hyperlinks contained in this email to ensure the information security and to protect personal information.</font></b>
</div>
<br class="">_______________________________________________<br class="">
OmniOS-discuss mailing list<br class="">
<a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a><br class="">
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" rel="noreferrer" target="_blank" class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br class="">
<br class=""></blockquote></div><br class=""></div>
_______________________________________________<br class="">OmniOS-discuss mailing list<br class=""><a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a><br class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss<br class=""></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; line-height: normal; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class="">--</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><br class=""></div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><a href="mailto:Richard.Elling@richardelling.com" class="">Richard.Elling@RichardElling.com</a><br class="">+1-760-896-4422<br class=""><br class=""><br class=""></div></span></div></span></div></span></div></span></div>
</div>
<br class=""></body></html>`