How to reset kstat counters

Sam Zaydel of RackTop Systems says:

I have been thinking about this one for a while, because every now and again, mostly for testing purposes I find that I really wish I could reset a kstat. I have been able to figure out how to reset certain kstat(s) with MDB, but wondering if perhaps you have one method that makes it relatively easy to find and reset any kstat.

In the kernel, all kstats are on 2 different AVL trees, one by name and the other by a kstat id. You can use mdb to set the value of a kstat to 0, but note that this might not always work (see more below). The other way to set the value to 0 is to write some code. First, let's go through a simple example.

Let's say you want to reset the values of the dnlc (directory name lookup cache) kstats to 0. Here are the values:

# kstat -m unix -n dnlcstatsmodule: unix                            instance: 0name:   dnlcstats                       class:    misc        crtime                          23.002235736        dir_add_abort                   0        dir_add_max                     0        dir_add_no_memory               0        dir_cached_current              4        dir_cached_total                4        dir_entries_cached_current      4708        dir_fini_purge                  0        dir_hits                        4634        dir_misses                      16094        dir_reclaim_any                 0        dir_reclaim_last                0        dir_remove_entry_fail           0        dir_remove_space_fail           0        dir_start_no_memory             0        dir_update_fail                 0        double_enters                   1        enters                          23789        hits                            5123316        misses                          26670        negative_cache_hits             152228        pick_free                       0        pick_heuristic                  3793        pick_last                       431        purge_all                       0        purge_fs1                       0        purge_total_entries             112        purge_vfs                       6        purge_vp                        63        snaptime                        96916.809565119#

We'll use mdb to dump all of the kstats. Note this may give lots of output.

# mdb -kwLoading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci ufsip hook neti sockfs arp usba stmf_sbd stmf zfs lofs idm mpt crypto random sdcpc logindmux ptm sppp nfs ]> kstat_avl_byname::walk avl !wc  1126    1126   19142>

So, there are 1126 ekstat_t structures currently in the kernel. Each ekstat_t may refer to multiple kstat values. Let's take a closer look.

> ::log kstats.out  <-- log output to file "kstats.out"mdb: logging to "kstats.out"> kstat_avl_byname::walk avl | ::print -t ekstat_tekstat_t {    kstat_t e_ks = {        hrtime_t ks_crtime = 0        struct kstat *ks_next = 0        kid_t ks_kid = 0        char [31] ks_module = [ "unix" ]        uchar_t ks_resv = 0        int ks_instance = 0        char [31] ks_name = [ "kstat_headers" ]        uchar_t ks_type = 0        char [31] ks_class = [ "kstat" ]        uchar_t ks_flags = 0x3        void *ks_data = 0        uint_t ks_ndata = 0x3f0        size_t ks_data_size = 0x2d480        hrtime_t ks_snaptime = 0x4785c1c2c60f        int (*)() ks_update = header_kstat_update        void *ks_private = 0        int (*)() ks_snapshot = header_kstat_snapshot        void *ks_lock = kstat_chain_lock    }    size_t e_size = 0x110    kthread_t *e_owner = 0    kcondvar_t e_cv = {        ushort_t _opaque = 0    }    avl_node_t e_avl_bykid = {        struct avl_node *[2] avl_child = [ 0, 0 ]        uintptr_t avl_pcb = 0xfffffffffbc15ab1    }    avl_node_t e_avl_byname = {        struct avl_node *[2] avl_child = [ kstat_initial+0x32b8,kstat_initial+0x9258 ]        uintptr_t avl_pcb = 0xfffffffffbc24618    }    kstat_zone_t e_zone = {        zoneid_t zoneid = 0xffffffff        struct kstat_zone *next = 0    }}...ekstat_t {    kstat_t e_ks = {        hrtime_t ks_crtime = 0x55b0a4358        struct kstat *ks_next = 0        kid_t ks_kid = 0x176        char [31] ks_module = [ "unix" ]        uchar_t ks_resv = 0        int ks_instance = 0        char [31] ks_name = [ "dnlcstats" ]        uchar_t ks_type = 0x1        char [31] ks_class = [ "misc" ]        uchar_t ks_flags = 0x1        void *ks_data = ncs        uint_t ks_ndata = 0x1c        size_t ks_data_size = 0x540        hrtime_t ks_snaptime = 0x15d0d20f9c9f        int (*)() ks_update = default_kstat_update        void *ks_private = 0        int (*)() ks_snapshot = default_kstat_snapshot        void *ks_lock = 0    }    size_t e_size = 0x110    kthread_t *e_owner = 0    kcondvar_t e_cv = {        ushort_t _opaque = 0    }    avl_node_t e_avl_bykid = {        struct avl_node *[2] avl_child = [ 0, 0 ]        uintptr_t avl_pcb = 0xffffff01482adc85    }    avl_node_t e_avl_byname = {        struct avl_node *[2] avl_child = [ kstat_initial+0xeb98,0xffffff014829b1f8 ]        uintptr_t avl_pcb = 0xfffffffffbc23e4e    }    kstat_zone_t e_zone = {        zoneid_t zoneid = 0xffffffff        struct kstat_zone *next = 0    }}...

The field that we want to examine is ks_data. For dnlcstats, this is a global variable: ncs.

Here's the ncs structure.

> ncs::print -t -afffffffffbc97780 struct nc_stats {    fffffffffbc97780 kstat_named_t ncs_hits = {        fffffffffbc97780 char [31] name = [ "hits" ]        fffffffffbc9779f uchar_t data_type = 0x4        fffffffffbc977a0 union  value = {            fffffffffbc977a0 char [16] c = [ 'e', '\245', 'N', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ]            fffffffffbc977a0 int32_t i32 = 0x4ea565            fffffffffbc977a0 uint32_t ui32 = 0x4ea565            fffffffffbc977a0 struct  str = {                fffffffffbc977a0 union  addr = {                    fffffffffbc977a0 char *ptr = 0x4ea565                    fffffffffbc977a0 caddr32_t ptr32 = 0x4ea565                    fffffffffbc977a0 char [8] __pad = [ 'e', '\245', 'N', '\0','\0', '\0', '\0', '\0' ]                }                fffffffffbc977a8 uint32_t len = 0            }            fffffffffbc977a0 int64_t i64 = 0x4ea565            fffffffffbc977a0 uint64_t ui64 = 0x4ea565            fffffffffbc977a0 long l = 0x4ea565            fffffffffbc977a0 ulong_t ul = 0x4ea565            fffffffffbc977a0 longlong_t ll = 0x4ea565            fffffffffbc977a0 u_longlong_t ull = 0x4ea565            fffffffffbc977a0 float f = +7.2225011e-39            fffffffffbc977a0 double d = +2.5464880e-317        }    }    fffffffffbc977b0 kstat_named_t ncs_misses = {        fffffffffbc977b0 char [31] name = [ "misses" ]...

The values are stored in a 128-bit union (note that a given kstat might not use all 128 bits).To change the value for "hits" to 0, simply:

> fffffffffbc977a0,2/Z 0ncs+0x20:       0x4ebf18                =       0x0ncs+0x28:       0                       =       0x0>

If you wanted to change all of the values to 0, you would have to go by hand through the nc_stats structure and change the value to 0 for each kstat_named_t in the structure. Or you could possibly write an mdb "dcmd"...

And now let's look at the value.

# kstat -m unix -n dnlcstatsmodule: unix                            instance: 0name:   dnlcstats                       class:    misc...   hits                            1675...#

So, the value is not 0, but it is much less than 5123316 that we saw when we ran the kstat before. I ran the kstat command a few minutes after setting the value to 0. There could have been 1675 hits since resetting the value, and you could use DTrace to trace them.

# dtrace -n 'dnlc_lookup:return{printf("hits = %d, misses = %d\n", genunix`ncs.ncs_hits.value.ui64, genunix`ncs.ncs_misses.value.ui64);}'dtrace: description 'dnlc_lookup:return' matched 1 probeCPU     ID              FUNCTION:NAME0       17229          dnlc_lookup:return hits = 28486, misses = 272760       17229          dnlc_lookup:return hits = 28487, misses = 272760       17229          dnlc_lookup:return hits = 28488, misses = 272760       17229          dnlc_lookup:return hits = 28489, misses = 272760       17229          dnlc_lookup:return hits = 28490, misses = 27276...

Resetting other kstat values requires the same method. In other words, find the address of the value variable within the kstat you want to modify, and setting it to 0 via mdb.

I mentioned at the beginning that this might not always work. If you are on a multiprocessor system, it is possible that the variable is being updated at the same time you are resetting it to 0.

Depending on atomicity of the operation of increment and zero-ing, it might not work. I also mentioned at the beginning that you could write code. The kstat driver has an ioctl command, KSTAT_IOC_WRITE that could be used for the purpose of setting the value to 0. See for instance the stats_zero_stats() function in src.illumos.org/source/xref/illumos-gate/usr/src/cmd/fs.d/cachefs/common/stats_stats.c for details.

We offer comprehensive training for Triton Developers, Operators and End Users.



Post written by Mr. Max Bruning