Print this page
NEX-19394 backport 9337 zfs get all is slow due to uncached metadata
Reviewed by: Joyce McIntosh <joyce.mcintosh@nexenta.com>
Reviewed by: Roman Strashkin <roman.strashkin@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Thomas Caputi <tcaputi@datto.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
 Conflicts:
  usr/src/uts/common/fs/zfs/dbuf.c
  usr/src/uts/common/fs/zfs/dmu.c
  usr/src/uts/common/fs/zfs/sys/dmu_objset.h
NEX-9752 backport illumos 6950 ARC should cache compressed data
Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
6950 ARC should cache compressed data
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Don Brady <don.brady@intel.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
6267 dn_bonus evicted too early
Reviewed by: Richard Yao <ryao@gentoo.org>
Reviewed by: Xin LI <delphij@freebsd.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
NEX-4582 update wrc test cases for allow to use write back cache per tree of datasets
Reviewed by: Steve Peng <steve.peng@nexenta.com>
Reviewed by: Alex Aizman <alex.aizman@nexenta.com>
5960 zfs recv should prefetch indirect blocks
5925 zfs receive -o origin=
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
5911 ZFS "hangs" while deleting file
Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
Reviewed by: Alek Pinchuk <alek@nexenta.com>
Reviewed by: Simon Klinkert <simon.klinkert@gmail.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
NEX-1823 Slow performance doing of a large dataset
5911 ZFS "hangs" while deleting file
Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com>
Reviewed by: Bayard Bell <bayard.bell@nexenta.com>
NEX-3165 segregate ddt in arc
SUP-507 Delete or truncate of large files delayed on datasets with small recordsize
Reviewed by: Albert Lee <trisk@nexenta.com>
Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
Reviewed by: Ilya Usvyatsky <ilya.usvyatsky@nexenta.com>
Reviewed by: Tony Nguyen <tony.nguyen@nexenta.com>
4370 avoid transmitting holes during zfs send
4371 DMU code clean up
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
Approved by: Garrett D'Amore <garrett@damore.org>
OS-80 support for vdev and CoS properties for the new I/O scheduler
OS-95 lint warning introduced by OS-61
Make special vdev subtree topology the same as regular vdev subtree to simplify testcase setup
Fixup merge issues
Issue #7: add cacheability to the properties
          Contributors: Boris Protopopov
DDT is placed either into special or to L2ARC but not in both
Support for secondarycache=data option
Align mutex tables in arc.c and dbuf.c to 64 bytes (cache line), place each kmutex_t on cache line by itself to avoid false sharing
re #12585 rb4049 ZFS++ work port - refactoring to improve separation of open/closed code, bug fixes, performance improvements - open code
Bug 11205: add missing libzfs_closed_stubs.c to fix opensource-only build.
ZFS plus work: special vdevs, cos, cos/vdev properties


   6  * You may not use this file except in compliance with the License.
   7  *
   8  * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
   9  * or http://www.opensolaris.org/os/licensing.
  10  * See the License for the specific language governing permissions
  11  * and limitations under the License.
  12  *
  13  * When distributing Covered Code, include this CDDL HEADER in each
  14  * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
  15  * If applicable, add the following below this CDDL HEADER, with the
  16  * fields enclosed by brackets "[]" replaced with your own identifying
  17  * information: Portions Copyright [yyyy] [name of copyright owner]
  18  *
  19  * CDDL HEADER END
  20  */
  21 /*
  22  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
  23  * Copyright (c) 2012, 2015 by Delphix. All rights reserved.
  24  * Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  25  * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.

  26  */
  27 
  28 #ifndef _SYS_DBUF_H
  29 #define _SYS_DBUF_H
  30 
  31 #include <sys/dmu.h>
  32 #include <sys/spa.h>
  33 #include <sys/txg.h>
  34 #include <sys/zio.h>
  35 #include <sys/arc.h>
  36 #include <sys/zfs_context.h>
  37 #include <sys/refcount.h>
  38 #include <sys/zrlock.h>
  39 #include <sys/multilist.h>
  40 
  41 #ifdef  __cplusplus
  42 extern "C" {
  43 #endif
  44 
  45 #define IN_DMU_SYNC 2
  46 
  47 /*
  48  * define flags for dbuf_read
  49  */
  50 
  51 #define DB_RF_MUST_SUCCEED      (1 << 0)
  52 #define DB_RF_CANFAIL           (1 << 1)
  53 #define DB_RF_HAVESTRUCT        (1 << 2)
  54 #define DB_RF_NOPREFETCH        (1 << 3)
  55 #define DB_RF_NEVERWAIT         (1 << 4)
  56 #define DB_RF_CACHED            (1 << 5)
  57 


  58 /*
  59  * The simplified state transition diagram for dbufs looks like:
  60  *
  61  *              +----> READ ----+
  62  *              |               |
  63  *              |               V
  64  *  (alloc)-->UNCACHED            CACHED-->EVICTING-->(free)
  65  *              |               ^        ^
  66  *              |               |        |
  67  *              +----> FILL ----+     |
  68  *              |                        |
  69  *              |                        |
  70  *              +--------> NOFILL -------+
  71  *
  72  * DB_SEARCH is an invalid state for a dbuf. It is used by dbuf_free_range
  73  * to find all dbufs in a range of a dnode and must be less than any other
  74  * dbuf_states_t (see comment on dn_dbufs in dnode.h).
  75  */
  76 typedef enum dbuf_states {
  77         DB_SEARCH = -1,
  78         DB_UNCACHED,
  79         DB_FILL,
  80         DB_NOFILL,
  81         DB_READ,
  82         DB_CACHED,
  83         DB_EVICTING
  84 } dbuf_states_t;
  85 







  86 struct dnode;
  87 struct dmu_tx;
  88 
  89 /*
  90  * level = 0 means the user data
  91  * level = 1 means the single indirect block
  92  * etc.
  93  */
  94 
  95 struct dmu_buf_impl;
  96 
  97 typedef enum override_states {
  98         DR_NOT_OVERRIDDEN,
  99         DR_IN_DMU_SYNC,
 100         DR_OVERRIDDEN
 101 } override_states_t;
 102 
 103 typedef struct dbuf_dirty_record {
 104         /* link on our parents dirty list */
 105         list_node_t dr_dirty_node;


 108         uint64_t dr_txg;
 109 
 110         /* zio of outstanding write IO */
 111         zio_t *dr_zio;
 112 
 113         /* pointer back to our dbuf */
 114         struct dmu_buf_impl *dr_dbuf;
 115 
 116         /* pointer to next dirty record */
 117         struct dbuf_dirty_record *dr_next;
 118 
 119         /* pointer to parent dirty record */
 120         struct dbuf_dirty_record *dr_parent;
 121 
 122         /* How much space was changed to dsl_pool_dirty_space() for this? */
 123         unsigned int dr_accounted;
 124 
 125         /* A copy of the bp that points to us */
 126         blkptr_t dr_bp_copy;
 127 



 128         union dirty_types {
 129                 struct dirty_indirect {
 130 
 131                         /* protect access to list */
 132                         kmutex_t dr_mtx;
 133 
 134                         /* Our list of dirty children */
 135                         list_t dr_children;
 136                 } di;
 137                 struct dirty_leaf {
 138 
 139                         /*
 140                          * dr_data is set when we dirty the buffer
 141                          * so that we can retain the pointer even if it
 142                          * gets COW'd in a subsequent transaction group.
 143                          */
 144                         arc_buf_t *dr_data;
 145                         blkptr_t dr_overridden_by;
 146                         override_states_t dr_override_state;
 147                         uint8_t dr_copies;


 212          * If nonzero, the buffer can't be destroyed.
 213          * Protected by db_mtx.
 214          */
 215         refcount_t db_holds;
 216 
 217         /* buffer holding our data */
 218         arc_buf_t *db_buf;
 219 
 220         kcondvar_t db_changed;
 221         dbuf_dirty_record_t *db_data_pending;
 222 
 223         /* pointer to most recent dirty record for this buffer */
 224         dbuf_dirty_record_t *db_last_dirty;
 225 
 226         /*
 227          * Our link on the owner dnodes's dn_dbufs list.
 228          * Protected by its dn_dbufs_mtx.
 229          */
 230         avl_node_t db_link;
 231 
 232         /*
 233          * Link in dbuf_cache.
 234          */
 235         multilist_node_t db_cache_link;
 236 



 237         /* Data which is unique to data (leaf) blocks: */
 238 
 239         /* User callback information. */
 240         dmu_buf_user_t *db_user;
 241 
 242         /*
 243          * Evict user data as soon as the dirty and reference
 244          * counts are equal.
 245          */
 246         uint8_t db_user_immediate_evict;
 247 
 248         /*
 249          * This block was freed while a read or write was
 250          * active.
 251          */
 252         uint8_t db_freed_in_flight;
 253 
 254         /*
 255          * dnode_evict_dbufs() or dnode_evict_bonus() tried to
 256          * evict this dbuf, but couldn't due to outstanding
 257          * references.  Evict once the refcount drops to 0.
 258          */
 259         uint8_t db_pending_evict;
 260 
 261         uint8_t db_dirtycnt;
 262 } dmu_buf_impl_t;
 263 
 264 /* Note: the dbuf hash table is exposed only for the mdb module */
 265 #define DBUF_MUTEXES 256
 266 #define DBUF_HASH_MUTEX(h, idx) (&(h)->hash_mutexes[(idx) & (DBUF_MUTEXES-1)])








 267 typedef struct dbuf_hash_table {
 268         uint64_t hash_table_mask;
 269         dmu_buf_impl_t **hash_table;
 270         kmutex_t hash_mutexes[DBUF_MUTEXES];
 271 } dbuf_hash_table_t;
 272 
 273 uint64_t dbuf_whichblock(struct dnode *di, int64_t level, uint64_t offset);
 274 
 275 dmu_buf_impl_t *dbuf_create_tlib(struct dnode *dn, char *data);
 276 void dbuf_create_bonus(struct dnode *dn);
 277 int dbuf_spill_set_blksz(dmu_buf_t *db, uint64_t blksz, dmu_tx_t *tx);
 278 void dbuf_spill_hold(struct dnode *dn, dmu_buf_impl_t **dbp, void *tag);
 279 
 280 void dbuf_rm_spill(struct dnode *dn, dmu_tx_t *tx);
 281 
 282 dmu_buf_impl_t *dbuf_hold(struct dnode *dn, uint64_t blkid, void *tag);
 283 dmu_buf_impl_t *dbuf_hold_level(struct dnode *dn, int level, uint64_t blkid,
 284     void *tag);
 285 int dbuf_hold_impl(struct dnode *dn, uint8_t level, uint64_t blkid,
 286     boolean_t fail_sparse, boolean_t fail_uncached,
 287     void *tag, dmu_buf_impl_t **dbp);
 288 
 289 void dbuf_prefetch(struct dnode *dn, int64_t level, uint64_t blkid,
 290     zio_priority_t prio, arc_flags_t aflags);
 291 
 292 void dbuf_add_ref(dmu_buf_impl_t *db, void *tag);
 293 boolean_t dbuf_try_add_ref(dmu_buf_t *db, objset_t *os, uint64_t obj,
 294     uint64_t blkid, void *tag);
 295 uint64_t dbuf_refcount(dmu_buf_impl_t *db);
 296 
 297 void dbuf_rele(dmu_buf_impl_t *db, void *tag);
 298 void dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag);
 299 
 300 dmu_buf_impl_t *dbuf_find(struct objset *os, uint64_t object, uint8_t level,
 301     uint64_t blkid);
 302 
 303 int dbuf_read(dmu_buf_impl_t *db, zio_t *zio, uint32_t flags);
 304 void dmu_buf_will_not_fill(dmu_buf_t *db, dmu_tx_t *tx);
 305 void dmu_buf_will_fill(dmu_buf_t *db, dmu_tx_t *tx);
 306 void dmu_buf_fill_done(dmu_buf_t *db, dmu_tx_t *tx);
 307 void dbuf_assign_arcbuf(dmu_buf_impl_t *db, arc_buf_t *buf, dmu_tx_t *tx);
 308 dbuf_dirty_record_t *dbuf_dirty(dmu_buf_impl_t *db, dmu_tx_t *tx);


 309 arc_buf_t *dbuf_loan_arcbuf(dmu_buf_impl_t *db);
 310 void dmu_buf_write_embedded(dmu_buf_t *dbuf, void *data,
 311     bp_embedded_type_t etype, enum zio_compress comp,
 312     int uncompressed_size, int compressed_size, int byteorder, dmu_tx_t *tx);
 313 
 314 void dbuf_destroy(dmu_buf_impl_t *db);
 315 
 316 void dbuf_setdirty(dmu_buf_impl_t *db, dmu_tx_t *tx);
 317 void dbuf_unoverride(dbuf_dirty_record_t *dr);
 318 void dbuf_sync_list(list_t *list, int level, dmu_tx_t *tx);
 319 void dbuf_release_bp(dmu_buf_impl_t *db);
 320 
 321 boolean_t dbuf_can_remap(const dmu_buf_impl_t *buf);
 322 
 323 void dbuf_free_range(struct dnode *dn, uint64_t start, uint64_t end,
 324     struct dmu_tx *);
 325 
 326 void dbuf_new_size(dmu_buf_impl_t *db, int size, dmu_tx_t *tx);
 327 
 328 #define DB_DNODE(_db)           ((_db)->db_dnode_handle->dnh_dnode)
 329 #define DB_DNODE_LOCK(_db)      ((_db)->db_dnode_handle->dnh_zrlock)
 330 #define DB_DNODE_ENTER(_db)     (zrl_add(&DB_DNODE_LOCK(_db)))
 331 #define DB_DNODE_EXIT(_db)      (zrl_remove(&DB_DNODE_LOCK(_db)))
 332 #define DB_DNODE_HELD(_db)      (!zrl_is_zero(&DB_DNODE_LOCK(_db)))
 333 
 334 void dbuf_init(void);
 335 void dbuf_fini(void);
 336 
 337 boolean_t dbuf_is_metadata(dmu_buf_impl_t *db);



 338 
 339 #define DBUF_GET_BUFC_TYPE(_db) \
 340         (dbuf_is_metadata(_db) ? ARC_BUFC_METADATA : ARC_BUFC_DATA)

 341 
 342 #define DBUF_IS_CACHEABLE(_db)                                          \
 343         ((_db)->db_objset->os_primary_cache == ZFS_CACHE_ALL ||           \
 344         (dbuf_is_metadata(_db) &&                                       \
 345         ((_db)->db_objset->os_primary_cache == ZFS_CACHE_METADATA)))
 346 








 347 #define DBUF_IS_L2CACHEABLE(_db)                                        \
 348         ((_db)->db_objset->os_secondary_cache == ZFS_CACHE_ALL || \
 349         (dbuf_is_metadata(_db) &&                                       \
 350         ((_db)->db_objset->os_secondary_cache == ZFS_CACHE_METADATA)))




 351 
 352 #define DNODE_LEVEL_IS_L2CACHEABLE(_dn, _level)                         \
 353         ((_dn)->dn_objset->os_secondary_cache == ZFS_CACHE_ALL || \
 354         (((_level) > 0 ||                                            \
 355         DMU_OT_IS_METADATA((_dn)->dn_handle->dnh_dnode->dn_type)) &&   \
 356         ((_dn)->dn_objset->os_secondary_cache == ZFS_CACHE_METADATA)))
 357 
 358 #ifdef ZFS_DEBUG
 359 
 360 /*
 361  * There should be a ## between the string literal and fmt, to make it
 362  * clear that we're joining two strings together, but gcc does not
 363  * support that preprocessor token.
 364  */
 365 #define dprintf_dbuf(dbuf, fmt, ...) do { \
 366         if (zfs_flags & ZFS_DEBUG_DPRINTF) { \
 367         char __db_buf[32]; \
 368         uint64_t __db_obj = (dbuf)->db.db_object; \
 369         if (__db_obj == DMU_META_DNODE_OBJECT) \
 370                 (void) strcpy(__db_buf, "mdn"); \




   6  * You may not use this file except in compliance with the License.
   7  *
   8  * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
   9  * or http://www.opensolaris.org/os/licensing.
  10  * See the License for the specific language governing permissions
  11  * and limitations under the License.
  12  *
  13  * When distributing Covered Code, include this CDDL HEADER in each
  14  * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
  15  * If applicable, add the following below this CDDL HEADER, with the
  16  * fields enclosed by brackets "[]" replaced with your own identifying
  17  * information: Portions Copyright [yyyy] [name of copyright owner]
  18  *
  19  * CDDL HEADER END
  20  */
  21 /*
  22  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
  23  * Copyright (c) 2012, 2015 by Delphix. All rights reserved.
  24  * Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  25  * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
  26  * Copyright 2015 Nexenta Systems, Inc. All rights reserved.
  27  */
  28 
  29 #ifndef _SYS_DBUF_H
  30 #define _SYS_DBUF_H
  31 
  32 #include <sys/dmu.h>
  33 #include <sys/spa.h>
  34 #include <sys/txg.h>
  35 #include <sys/zio.h>
  36 #include <sys/arc.h>
  37 #include <sys/zfs_context.h>
  38 #include <sys/refcount.h>
  39 #include <sys/zrlock.h>
  40 #include <sys/multilist.h>
  41 
  42 #ifdef  __cplusplus
  43 extern "C" {
  44 #endif
  45 
  46 #define IN_DMU_SYNC 2
  47 
  48 /*
  49  * define flags for dbuf_read
  50  */
  51 
  52 #define DB_RF_MUST_SUCCEED      (1 << 0)
  53 #define DB_RF_CANFAIL           (1 << 1)
  54 #define DB_RF_HAVESTRUCT        (1 << 2)
  55 #define DB_RF_NOPREFETCH        (1 << 3)
  56 #define DB_RF_NEVERWAIT         (1 << 4)
  57 #define DB_RF_CACHED            (1 << 5)
  58 
  59 #define DBUF_EVICT_ALL          -1
  60 
  61 /*
  62  * The simplified state transition diagram for dbufs looks like:
  63  *
  64  *              +----> READ ----+
  65  *              |               |
  66  *              |               V
  67  *  (alloc)-->UNCACHED            CACHED-->EVICTING-->(free)
  68  *              |               ^        ^
  69  *              |               |        |
  70  *              +----> FILL ----+     |
  71  *              |                        |
  72  *              |                        |
  73  *              +--------> NOFILL -------+
  74  *
  75  * DB_SEARCH is an invalid state for a dbuf. It is used by dbuf_free_range
  76  * to find all dbufs in a range of a dnode and must be less than any other
  77  * dbuf_states_t (see comment on dn_dbufs in dnode.h).
  78  */
  79 typedef enum dbuf_states {
  80         DB_SEARCH = -1,
  81         DB_UNCACHED,
  82         DB_FILL,
  83         DB_NOFILL,
  84         DB_READ,
  85         DB_CACHED,
  86         DB_EVICTING
  87 } dbuf_states_t;
  88 
  89 typedef enum dbuf_cached_state {
  90         DB_NO_CACHE = -1,
  91         DB_DBUF_CACHE,
  92         DB_DBUF_METADATA_CACHE,
  93         DB_CACHE_MAX
  94 } dbuf_cached_state_t;
  95 
  96 struct dnode;
  97 struct dmu_tx;
  98 
  99 /*
 100  * level = 0 means the user data
 101  * level = 1 means the single indirect block
 102  * etc.
 103  */
 104 
 105 struct dmu_buf_impl;
 106 
 107 typedef enum override_states {
 108         DR_NOT_OVERRIDDEN,
 109         DR_IN_DMU_SYNC,
 110         DR_OVERRIDDEN
 111 } override_states_t;
 112 
 113 typedef struct dbuf_dirty_record {
 114         /* link on our parents dirty list */
 115         list_node_t dr_dirty_node;


 118         uint64_t dr_txg;
 119 
 120         /* zio of outstanding write IO */
 121         zio_t *dr_zio;
 122 
 123         /* pointer back to our dbuf */
 124         struct dmu_buf_impl *dr_dbuf;
 125 
 126         /* pointer to next dirty record */
 127         struct dbuf_dirty_record *dr_next;
 128 
 129         /* pointer to parent dirty record */
 130         struct dbuf_dirty_record *dr_parent;
 131 
 132         /* How much space was changed to dsl_pool_dirty_space() for this? */
 133         unsigned int dr_accounted;
 134 
 135         /* A copy of the bp that points to us */
 136         blkptr_t dr_bp_copy;
 137 
 138         /* use special class of dirty entry */
 139         boolean_t dr_usesc;
 140 
 141         union dirty_types {
 142                 struct dirty_indirect {
 143 
 144                         /* protect access to list */
 145                         kmutex_t dr_mtx;
 146 
 147                         /* Our list of dirty children */
 148                         list_t dr_children;
 149                 } di;
 150                 struct dirty_leaf {
 151 
 152                         /*
 153                          * dr_data is set when we dirty the buffer
 154                          * so that we can retain the pointer even if it
 155                          * gets COW'd in a subsequent transaction group.
 156                          */
 157                         arc_buf_t *dr_data;
 158                         blkptr_t dr_overridden_by;
 159                         override_states_t dr_override_state;
 160                         uint8_t dr_copies;


 225          * If nonzero, the buffer can't be destroyed.
 226          * Protected by db_mtx.
 227          */
 228         refcount_t db_holds;
 229 
 230         /* buffer holding our data */
 231         arc_buf_t *db_buf;
 232 
 233         kcondvar_t db_changed;
 234         dbuf_dirty_record_t *db_data_pending;
 235 
 236         /* pointer to most recent dirty record for this buffer */
 237         dbuf_dirty_record_t *db_last_dirty;
 238 
 239         /*
 240          * Our link on the owner dnodes's dn_dbufs list.
 241          * Protected by its dn_dbufs_mtx.
 242          */
 243         avl_node_t db_link;
 244 
 245         /* Link in dbuf_cache or dbuf_metadata_cache */


 246         multilist_node_t db_cache_link;
 247 
 248         /* Tells us which dbuf cache this dbuf is in, if any */
 249         dbuf_cached_state_t db_caching_status;
 250 
 251         /* Data which is unique to data (leaf) blocks: */
 252 
 253         /* User callback information. */
 254         dmu_buf_user_t *db_user;
 255 
 256         /*
 257          * Evict user data as soon as the dirty and reference
 258          * counts are equal.
 259          */
 260         uint8_t db_user_immediate_evict;
 261 
 262         /*
 263          * This block was freed while a read or write was
 264          * active.
 265          */
 266         uint8_t db_freed_in_flight;
 267 
 268         /*
 269          * dnode_evict_dbufs() or dnode_evict_bonus() tried to
 270          * evict this dbuf, but couldn't due to outstanding
 271          * references.  Evict once the refcount drops to 0.
 272          */
 273         uint8_t db_pending_evict;
 274 
 275         uint8_t db_dirtycnt;
 276 } dmu_buf_impl_t;
 277 
 278 /* Note: the dbuf hash table is exposed only for the mdb module */
 279 #define DBUF_MUTEXES    256
 280 #define DBUF_LOCK_PAD   64
 281 typedef struct {
 282         kmutex_t mtx;
 283 #ifdef _KERNEL
 284         unsigned char pad[(DBUF_LOCK_PAD - sizeof (kmutex_t))];
 285 #endif
 286 } dbuf_mutex_t;
 287 #define DBUF_HASH_MUTEX(h, idx) \
 288         (&((h)->hash_mutexes[(idx) & (DBUF_MUTEXES-1)].mtx))
 289 typedef struct dbuf_hash_table {
 290         uint64_t hash_table_mask;
 291         dmu_buf_impl_t **hash_table;
 292         dbuf_mutex_t hash_mutexes[DBUF_MUTEXES];
 293 } dbuf_hash_table_t;
 294 
 295 uint64_t dbuf_whichblock(struct dnode *di, int64_t level, uint64_t offset);
 296 
 297 dmu_buf_impl_t *dbuf_create_tlib(struct dnode *dn, char *data);
 298 void dbuf_create_bonus(struct dnode *dn);
 299 int dbuf_spill_set_blksz(dmu_buf_t *db, uint64_t blksz, dmu_tx_t *tx);
 300 void dbuf_spill_hold(struct dnode *dn, dmu_buf_impl_t **dbp, void *tag);
 301 
 302 void dbuf_rm_spill(struct dnode *dn, dmu_tx_t *tx);
 303 
 304 dmu_buf_impl_t *dbuf_hold(struct dnode *dn, uint64_t blkid, void *tag);
 305 dmu_buf_impl_t *dbuf_hold_level(struct dnode *dn, int level, uint64_t blkid,
 306     void *tag);
 307 int dbuf_hold_impl(struct dnode *dn, uint8_t level, uint64_t blkid,
 308     boolean_t fail_sparse, boolean_t fail_uncached,
 309     void *tag, dmu_buf_impl_t **dbp);
 310 
 311 void dbuf_prefetch(struct dnode *dn, int64_t level, uint64_t blkid,
 312     zio_priority_t prio, arc_flags_t aflags);
 313 
 314 void dbuf_add_ref(dmu_buf_impl_t *db, void *tag);
 315 boolean_t dbuf_try_add_ref(dmu_buf_t *db, objset_t *os, uint64_t obj,
 316     uint64_t blkid, void *tag);
 317 uint64_t dbuf_refcount(dmu_buf_impl_t *db);
 318 
 319 void dbuf_rele(dmu_buf_impl_t *db, void *tag);
 320 void dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag);
 321 
 322 dmu_buf_impl_t *dbuf_find(struct objset *os, uint64_t object, uint8_t level,
 323     uint64_t blkid);
 324 
 325 int dbuf_read(dmu_buf_impl_t *db, zio_t *zio, uint32_t flags);
 326 void dmu_buf_will_not_fill(dmu_buf_t *db, dmu_tx_t *tx);
 327 void dmu_buf_will_fill(dmu_buf_t *db, dmu_tx_t *tx);
 328 void dmu_buf_fill_done(dmu_buf_t *db, dmu_tx_t *tx);
 329 void dbuf_assign_arcbuf(dmu_buf_impl_t *db, arc_buf_t *buf, dmu_tx_t *tx);
 330 dbuf_dirty_record_t *dbuf_dirty(dmu_buf_impl_t *db, dmu_tx_t *tx);
 331 dbuf_dirty_record_t *dbuf_dirty_sc(dmu_buf_impl_t *db, dmu_tx_t *tx,
 332     boolean_t usesc);
 333 arc_buf_t *dbuf_loan_arcbuf(dmu_buf_impl_t *db);
 334 void dmu_buf_write_embedded(dmu_buf_t *dbuf, void *data,
 335     bp_embedded_type_t etype, enum zio_compress comp,
 336     int uncompressed_size, int compressed_size, int byteorder, dmu_tx_t *tx);
 337 
 338 void dbuf_destroy(dmu_buf_impl_t *db);
 339 
 340 void dbuf_setdirty(dmu_buf_impl_t *db, dmu_tx_t *tx);
 341 void dbuf_unoverride(dbuf_dirty_record_t *dr);
 342 void dbuf_sync_list(list_t *list, int level, dmu_tx_t *tx);
 343 void dbuf_release_bp(dmu_buf_impl_t *db);
 344 


 345 void dbuf_free_range(struct dnode *dn, uint64_t start, uint64_t end,
 346     struct dmu_tx *);
 347 
 348 void dbuf_new_size(dmu_buf_impl_t *db, int size, dmu_tx_t *tx);
 349 
 350 #define DB_DNODE(_db)           ((_db)->db_dnode_handle->dnh_dnode)
 351 #define DB_DNODE_LOCK(_db)      ((_db)->db_dnode_handle->dnh_zrlock)
 352 #define DB_DNODE_ENTER(_db)     (zrl_add(&DB_DNODE_LOCK(_db)))
 353 #define DB_DNODE_EXIT(_db)      (zrl_remove(&DB_DNODE_LOCK(_db)))
 354 #define DB_DNODE_HELD(_db)      (!zrl_is_zero(&DB_DNODE_LOCK(_db)))
 355 
 356 void dbuf_init(void);
 357 void dbuf_fini(void);
 358 
 359 boolean_t dbuf_is_metadata(dmu_buf_impl_t *db);
 360 boolean_t dbuf_is_ddt(dmu_buf_impl_t *db);
 361 boolean_t dbuf_ddt_is_l2cacheable(dmu_buf_impl_t *db);
 362 boolean_t dbuf_meta_is_l2cacheable(dmu_buf_impl_t *db);
 363 
 364 #define DBUF_GET_BUFC_TYPE(_db) \
 365         (dbuf_is_ddt(_db) ? ARC_BUFC_DDT :\
 366         (dbuf_is_metadata(_db) ? ARC_BUFC_METADATA : ARC_BUFC_DATA))
 367 
 368 #define DBUF_IS_CACHEABLE(_db)                                          \
 369         ((_db)->db_objset->os_primary_cache == ZFS_CACHE_ALL ||           \
 370         (dbuf_is_metadata(_db) &&                                       \
 371         ((_db)->db_objset->os_primary_cache == ZFS_CACHE_METADATA)))
 372 
 373 /*
 374  * Checks whether we need to cache dbuf in l2arc.
 375  * Metadata is l2cacheable if it is not placed on special device
 376  * or it is placed on special device in "dual" mode. We need to check
 377  * for ddt in ZFS_CACHE_ALL and ZFS_CACHE_METADATA because it is in MOS.
 378  * ZFS_CACHE_DATA mode actually means to cache both data and cacheable
 379  * metadata.
 380  */
 381 #define DBUF_IS_L2CACHEABLE(_db)                                        \
 382         (((_db)->db_objset->os_secondary_cache == ZFS_CACHE_ALL &&        \
 383         (dbuf_ddt_is_l2cacheable(_db) == B_TRUE)) ||                    \
 384         ((_db)->db_objset->os_secondary_cache == ZFS_CACHE_METADATA &&  \
 385         (dbuf_is_metadata(_db)) &&                                      \
 386         (dbuf_ddt_is_l2cacheable(_db) == B_TRUE)) ||                    \
 387         ((dbuf_meta_is_l2cacheable(_db) == B_TRUE) &&                   \
 388         ((_db)->db_objset->os_secondary_cache == ZFS_CACHE_DATA)))
 389 
 390 #define DNODE_LEVEL_IS_L2CACHEABLE(_dn, _level)                         \
 391         ((_dn)->dn_objset->os_secondary_cache == ZFS_CACHE_ALL || \
 392         (((_level) > 0 ||                                            \
 393         DMU_OT_IS_METADATA((_dn)->dn_handle->dnh_dnode->dn_type)) &&   \
 394         ((_dn)->dn_objset->os_secondary_cache == ZFS_CACHE_METADATA)))
 395 
 396 #ifdef ZFS_DEBUG
 397 
 398 /*
 399  * There should be a ## between the string literal and fmt, to make it
 400  * clear that we're joining two strings together, but gcc does not
 401  * support that preprocessor token.
 402  */
 403 #define dprintf_dbuf(dbuf, fmt, ...) do { \
 404         if (zfs_flags & ZFS_DEBUG_DPRINTF) { \
 405         char __db_buf[32]; \
 406         uint64_t __db_obj = (dbuf)->db.db_object; \
 407         if (__db_obj == DMU_META_DNODE_OBJECT) \
 408                 (void) strcpy(__db_buf, "mdn"); \