`<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>I want to start 12 kvm machines in my global zone and I end up in an memlock issue after the 6th machine:</p>
<p><br>
</p>
<p>qemu_mlock: have only locked 0 of 7516192768 bytes; still trying…</p>
<p><br>
</p>
<p>The details:</p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;"># echo "::memstat" | mdb -k</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page Summary Pages MB %Tot</span></p>
<p><span style="font-family: 'Courier New', monospace;">------------ ---------------- ---------------- ----</span></p>
<p><span style="font-family: 'Courier New', monospace;">Kernel 1462114 5711 5%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Boot pages 166 0 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">ZFS File Data 143529 560 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Anon 27477 107 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Exec and libs 1244 4 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page cache 4911 19 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (cachelist) 3597 14 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (freelist) 29809914 116444 95%</span></p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;">Total 31452952 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">Physical 31452951 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p>My pagesize is:</p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;"># pagesize</span></p>
<p><span style="font-family: 'Courier New', monospace;">4096</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p><br>
</p>
<p>so 31452952 pages = (122863*1024*1024) bytes / 4096 (bytes)</p>
<p>thus: 31452952 / 256 = 122863MB</p>
<p><br>
</p>
<p>That's correct. So pages to MB we can divide by 256. MB to pages, multiply with 256.</p>
<p><br>
</p>
<p>I want 8GB per vm and I want 12vm's. This is 96GB (98304MB) </p>
<p>-> needs 98304 * 256 = 25165824 pages. </p>
<p><br>
</p>
<p><br>
</p>
<p>31452952 - 25165824 = 6287128 left (or 24559 MB ~ 23.9GB )</p>
<p><br>
</p>
<p>I reserved (in /etc/system ) 15 GB for zfs arc </p>
<p>15GB = 15360 MB = 3932160 pages</p>
<p><br>
</p>
<p>So this gives 6287128 - 3932160 = 2354968 pages left (or 9199MB ~ 8.98 GB)</p>
<p><br>
</p>
<p>-> this should be sufficient for omnios to run. Unless I'm not aware of other things, which is very well possible. </p>
<p><br>
</p>
<p>So remember </p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;">Page Summary Pages MB %Tot</span></p>
<p><span style="font-family: 'Courier New', monospace;">------------ ---------------- ---------------- ----</span></p>
<p><span style="font-family: 'Courier New', monospace;">...</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (cachelist) 3597 14 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (freelist) 29809914 116444 95%</span></p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;">Total 31452952 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">Physical 31452951 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p>Let's boot one kvm with 8GB ram for the vm. We expect to see (8GB *1024) * 256 = 2097152 pages be eaten up. So we should end up with 31452952 - 2097152 = 29355800 pages.</p>
<p>OR if we are eating from the "Free" part, we should end up with 29809914 - 2097152 = 27712762 pages (or 108252MB ~105.71 GB) </p>
<p><br>
</p>
<p>Let's confirm:</p>
<p><br>
</p>
<p>Before boot:</p>
<p><span style="font-family: 'Courier New', monospace;"># mdb -ke 'availrmem/D ; pages_pp_maximum/D'</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem:</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem: 29845807</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum:</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum: 1216998</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p>Boot the machine.</p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;"># echo "::memstat" | mdb -k</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page Summary Pages MB %Tot</span></p>
<p><span style="font-family: 'Courier New', monospace;">------------ ---------------- ---------------- ----</span></p>
<p><span style="font-family: 'Courier New', monospace;">Kernel 1472257 5751 5%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Boot pages 166 0 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">ZFS File Data 149321 583 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Anon 2139596 8357 7%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Exec and libs 1549 6 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page cache 4955 19 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (cachelist) 3445 13 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (freelist) 27681663 108131 88%</span><span class="Apple-tab-span" style="white-space: pre; font-family: 'Courier New', monospace;">
</span><span style="font-family: 'Courier New', monospace;"><<<----- 27712762 pages </span></p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;">Total 31452952 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">Physical 31452951 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;"># mdb -ke 'availrmem/D ; pages_pp_maximum/D'</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem:</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem: 26621718</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum:</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum: 1216998</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p><br>
</p>
<p>look there. We expected to end up with 29355800 pages or 27712762 free pages. </p>
<p><br>
</p>
<p>If we check in the available memory 29845807 - 26621718 = 3224089 pages were consumed (12594.09MB ~ 12.29GB ) </p>
<p>Double check: 29809914 - 27681663 = 2128251 pages consumed ( 8313.48MB ~ 8.11GB )</p>
<p>And this for a machine of 8GB. </p>
<p><br>
</p>
<p>Let's do a test with a vm with 4GB allocated ( 1048576 pages )</p>
<p><br>
</p>
<p><br>
</p>
<p>Before boot:</p>
<p><span style="font-family: 'Courier New', monospace;"># echo "::memstat" | mdb -k</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page Summary Pages MB %Tot</span></p>
<p><span style="font-family: 'Courier New', monospace;">------------ ---------------- ---------------- ----</span></p>
<p><span style="font-family: 'Courier New', monospace;">Kernel 1476857 5768 5%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Boot pages 166 0 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">ZFS File Data 176243 688 1%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Anon 2139416 8357 7%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Exec and libs 1549 6 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page cache 4961 19 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (cachelist) 3440 13 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (freelist) 27650320 108009 88%</span></p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;">Total 31452952 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">Physical 31452951 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;"># mdb -ke 'availrmem/D ; pages_pp_maximum/D'</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem:</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem: 26596813</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum:</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum: 1216998</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p>So we expect </p>
<p>27650320 - 1048576 = 26601744 free pages (103913.06 MB ~ 101.47 GB free)</p>
<p>or</p>
<p>26596813 - 1048576 = 25548237 free pages (99797.80 MB ~ 97.45 GB free) </p>
<p><br>
</p>
<p>Booting the vm. </p>
<p>And the aftermath:</p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;"># ./start_kvm_pogo.sh</span></p>
<p><span style="font-family: 'Courier New', monospace;">qemu-system-x86_64: -net vnic,vlan=40,name=net0,ifname=vnic_kvm_pogo0,macaddr=2:8:20:87:2f:69: vnic dhcp disabled</span></p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;">Started VM: pogo</span></p>
<p><span style="font-family: 'Courier New', monospace;">VNC available at: host IP 127.0.0.1</span></p>
<p><span style="font-family: 'Courier New', monospace;">10.0.1.171</span></p>
<p><span style="font-family: 'Courier New', monospace;">::1/128</span></p>
<p><span style="font-family: 'Courier New', monospace;">::/0 port 5900</span></p>
<p><span style="font-family: 'Courier New', monospace;">QEMU Monitor, do: # telnet localhost . Note: use Control ] to exit monitor before quit!</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p>Check the values</p>
<p><span style="font-family: 'Courier New', monospace;"># echo "::memstat" | mdb -k</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page Summary Pages MB %Tot</span></p>
<p><span style="font-family: 'Courier New', monospace;">------------ ---------------- ---------------- ----</span></p>
<p><span style="font-family: 'Courier New', monospace;">Kernel 1482301 5790 5%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Boot pages 166 0 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">ZFS File Data 180958 706 1%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Anon 3198541 12494 10%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Exec and libs 1549 6 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Page cache 4961 19 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (cachelist) 3440 13 0%</span></p>
<p><span style="font-family: 'Courier New', monospace;">Free (freelist) 26581036 103832 85%</span></p>
<p><br>
</p>
<p><span style="font-family: 'Courier New', monospace;">Total 31452952 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">Physical 31452951 122863</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><span style="font-family: 'Courier New', monospace;"># mdb -ke 'availrmem/D ; pages_pp_maximum/D'</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem:</span></p>
<p><span style="font-family: 'Courier New', monospace;">availrmem: 24472324</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum:</span></p>
<p><span style="font-family: 'Courier New', monospace;">pages_pp_maximum: 1216998</span></p>
<p><span style="font-family: 'Courier New', monospace;">#</span></p>
<p><br>
</p>
<p>let's verify. </p>
<p>Test1:</p>
<p>27650320 - 26581036 = 1069284 pages consumed ( 4176.89 MB ~ 4GB ) </p>
<p>Test 2:</p>
<p>26596813 - 24472324 = 2124489 pages consumed ( 8298.78 MB ~ 8.10GB )</p>
<p><br>
</p>
<p><br>
</p>
<p>3 questions remain.</p>
<p>- Why does this happen. If we check using the memstat it's fairly ok, but appearently this is not correct as I can't start the 12 machines. </p>
<p>- How to avoid this. If I say to kvm "use 8GB" I want it to use 8GB not 12GB. Or am I missing something crucial? </p>
<p>- If it's a (kernel) parameter ... which one and how to set it? </p>
<p><br>
</p>
<p>Thanks and best regards,</p>
<p>Pieter</p>
<p><br>
</p>
<p><br>
</p>
<div style="color: rgb(0, 0, 0);">
<div></div>
</div>
</div>
</body>
</html>
`