`<div dir="ltr">I've used BeeGFS on Ubuntu 16.04 for about a year now. I like your idea of putting it lxzones on OmniOS for scratch space. <div><br></div><div>I have found it to scale with millions of files very well. It's running on a 4 node cluster. Each node is client, metadata and data nodes. These are very big GPU boxes with 9 Tesla GPUs, 40 CPU cores and 256GB ram. <div><br></div><div>The metadata is mirrored on 2 Samsung Pro SSDs on each node. It sustains about 33k metadata ops with never more than one queued.</div><div><br></div><div>This is my third iteration of setting it up. Metadata performance was our bottleneck each time previously. What I have found is that latency and horizontal scaling is king with BeeGFS metadata. It doesn't take a lot of CPU, but keep it close as possible on the network to the clients and keep latency low with fast network and SSDs. </div><div><br></div><div>My complaints about BeeGFS is lack of snapshots, so backup is limited to rsync of a live file system. For this reason it's only used for this very high read demand cluster. I still use ZFS on OmniOS for our PBs of data where snapshots and replication are priceless.</div><div><br></div><div><br></div><div>-Chip</div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jun 16, 2018 at 12:45 PM, Michael Talbott <span dir="ltr"><<a href="mailto:mtalbott@lji.org" target="_blank">mtalbott@lji.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We've been using OmniOS happily for years now for our storage server needs. But we're rapidly increasing our data footprint and growing so much (multiple PBs per year) that ideally I'd like to move to a cluster based object store based system ontop of OmniOS. I successfully use BeeGFS inside lxzones in OmniOS which seems to work nicely for our HPC scratch volume, but, it doesn't sound like it scales to hundreds of millions of files very well.<br>
<br>
I am hoping that someone has some ideas for me. Ideally I'd like something that's cluster capable and has erasure coding like Ceph and have cluster aware snapshots (not local zfs snaps) and an s3 compatibility/access layer.<br>
<br>
Any thoughts on the topic are greatly appreciated.<br>
<br>
Thanks,<br>
<br>
Michael<br>
Sent from my iPhone<br>
______________________________<wbr>_________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.<wbr>com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" rel="noreferrer" target="_blank">http://lists.omniti.com/<wbr>mailman/listinfo/omnios-<wbr>discuss</a><br>
</blockquote></div><br></div>
`