`<div dir="ltr"><div>Here's the screen shot:</div><img src="https://lh3.googleusercontent.com/-xVxdRkbs3g0/WQ5dVdg53nI/AAAAAAAAAAo/RBtlwJCdF70EvVN1erOVcdpkxQaJp3HKACK8B/s512/2017-05-06.jpg" class="uqvYjb KgFPz" alt=""><div><br></div><div>Given the rarity of this, I wouldn't be surprised it never happens again to me. The major difficulty was locating the offending system. All we were finding was very poor TCP connections everywhere on or network. Even VLANs that were not active on the server, but trunked on its switch ports.</div><div><br></div><div>That mac address corresponds to another OmniOS server, that is not part of the same HA cluster as this one. It has not been shut down since the incident. </div><div><br></div><div>-Chip</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 9, 2017 at 3:22 PM, Dan McDonald <span dir="ltr"><<a href="mailto:danmcd@omniti.com" target="_blank">danmcd@omniti.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
> On May 9, 2017, at 3:32 PM, Schweiss, Chip <<a href="mailto:chip@innovates.com">chip@innovates.com</a>> wrote:<br>
><br>
> This was a first for me and extremely painful to locate.<br>
><br>
> In the middle of the night between last Friday and Saturday, I started getting down alerts from most of my network. It took 4 engineers including myself 9 hours to pinpoint the source of the problem.<br>
><br>
> The problem turned out to be one of my OmniOS boxes sending out pure garbage constantly on layer 2 out the 10G network ports. This disrupted ARP caches on every machine on every VLAN that was trunked on these ports, not just the VLANs that were configured on the server. The switches reported every port healthy and without error. The traffic on the bad port was not high either, just severely disruptive.<br>
<br>
</span>Whoa! On L2 (like non-TCP/IP ethernet frames)?<br>
<span class=""><br>
> The affected OmniOS box appear to be healthy, as it was still serving the VM data stores for over 350 virtual machines. However, it like every other service on the network appeared to be up and down repeatedly, but NFS kept on recovering gracefully.<br>
><br>
> The only thing that finally identified this server was when one of us plug a monitor to the console and saw "WARNING: proxy ARP problem?" happening so fast that it took taking a cellphone picture of it a high frame rate to read it. Powering off this server, cleared the problem for the entire network, and its pools were taken over by its HA sister.<br>
<br>
</span>If it's easy to do so, unplug or "ifconfig down" the interface next time this happens.<br>
<span class=""><br>
> Googling for that warning brings up nothing useful.<br>
><br>
> Has anyone ever seen a problem like this? How did you locate it?<br>
<br>
</span>Should search <a href="http://src.illumos.org" rel="noreferrer" target="_blank">src.illumos.org</a>, you'll find this:<br>
<br>
<a href="http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/inet/ip/ip_arp.c#1449" rel="noreferrer" target="_blank">http://src.illumos.org/source/<wbr>xref/illumos-gate/usr/src/uts/<wbr>common/inet/ip/ip_arp.c#1449</a><br>
<br>
We appear to be freaking out over another node having our IP. The only caller with AR_CN_BOGON is after ip_nce_resolve_all() returns AR_BOGON.<br>
<br>
I wonder if some other entity had the same IP, and they fed-back-upon-each-other negatively?<br>
<br>
The message you cite should show an IP address with it:<br>
<br>
"proxy ARP problem? Node '%s' is using %s on %s",<br>
<br>
where the %s-es are MAC-address, IP-address, and interface-name respectively. You didn't get examples with your digital camera, did you?<br>
<span class="HOEnZb"><font color="#888888"><br>
Dan<br>
<br>
</font></span></blockquote></div><br></div>
`