Bruning Questions: How to reset kstat counters

January 11, 2013 - by Mr. Max Bruning

ask-mr-bruning-logo

Sam Zaydel of RackTop Systems says:

I have been thinking about this one for a while, because every now and again, mostly for testing purposes I find that I really wish I could reset a kstat. I have been able to figure out how to reset certain kstat(s) with MDB, but wondering if perhaps you have one method that makes it relatively easy to find and reset any kstat.

In the kernel, all kstats are on 2 different AVL trees, one by name and the other by a kstat id. You can use mdb to set the value of a kstat to 0, but note that this might not always work (see more below). The other way to set the value to 0 is to write some code. First, let's go through a simple example.

Let's say you want to reset the values of the dnlc (directory name lookup cache) kstats to 0. Here are the values:

# kstat -m unix -n dnlcstats
module: unix                            instance: 0     
name:   dnlcstats                       class:    misc
        crtime                          23.002235736
        dir_add_abort                   0
        dir_add_max                     0
        dir_add_no_memory               0
        dir_cached_current              4
        dir_cached_total                4
        dir_entries_cached_current      4708
        dir_fini_purge                  0
        dir_hits                        4634
        dir_misses                      16094
        dir_reclaim_any                 0
        dir_reclaim_last                0
        dir_remove_entry_fail           0
        dir_remove_space_fail           0
        dir_start_no_memory             0
        dir_update_fail                 0
        double_enters                   1
        enters                          23789
        hits                            5123316
        misses                          26670
        negative_cache_hits             152228
        pick_free                       0
        pick_heuristic                  3793
        pick_last                       431
        purge_all                       0
        purge_fs1                       0
        purge_total_entries             112
        purge_vfs                       6
        purge_vp                        63
        snaptime                        96916.809565119
#

We'll use mdb to dump all of the kstats. Note this may give lots of output.

# mdb -kw
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci ufs
ip hook neti sockfs arp usba stmf_sbd stmf zfs lofs idm mpt crypto random sd
cpc logindmux ptm sppp nfs ]
> kstat_avl_byname::walk avl !wc
  1126    1126   19142
>

So, there are 1126 ekstat_t structures currently in the kernel. Each ekstat_t may refer to multiple kstat values. Let's take a closer look.

> ::log kstats.out  <-- log output to file "kstats.out"
mdb: logging to "kstats.out"
> kstat_avl_byname::walk avl | ::print -t ekstat_t
ekstat_t {
    kstat_t e_ks = {
        hrtime_t ks_crtime = 0
        struct kstat *ks_next = 0
        kid_t ks_kid = 0
        char [31] ks_module = [ "unix" ]
        uchar_t ks_resv = 0
        int ks_instance = 0
        char [31] ks_name = [ "kstat_headers" ]
        uchar_t ks_type = 0
        char [31] ks_class = [ "kstat" ]
        uchar_t ks_flags = 0x3
        void *ks_data = 0
        uint_t ks_ndata = 0x3f0
        size_t ks_data_size = 0x2d480
        hrtime_t ks_snaptime = 0x4785c1c2c60f
        int (*)() ks_update = header_kstat_update
        void *ks_private = 0
        int (*)() ks_snapshot = header_kstat_snapshot
        void *ks_lock = kstat_chain_lock
    }
    size_t e_size = 0x110
    kthread_t *e_owner = 0
    kcondvar_t e_cv = {
        ushort_t _opaque = 0
    }
    avl_node_t e_avl_bykid = {
        struct avl_node *[2] avl_child = [ 0, 0 ]
        uintptr_t avl_pcb = 0xfffffffffbc15ab1
    }
    avl_node_t e_avl_byname = {
        struct avl_node *[2] avl_child = [ kstat_initial+0x32b8, 
kstat_initial+0x9258 ]
        uintptr_t avl_pcb = 0xfffffffffbc24618
    }
    kstat_zone_t e_zone = {
        zoneid_t zoneid = 0xffffffff
        struct kstat_zone *next = 0
    }
}
...
ekstat_t {
    kstat_t e_ks = {
        hrtime_t ks_crtime = 0x55b0a4358
        struct kstat *ks_next = 0
        kid_t ks_kid = 0x176
        char [31] ks_module = [ "unix" ]
        uchar_t ks_resv = 0
        int ks_instance = 0
        char [31] ks_name = [ "dnlcstats" ]
        uchar_t ks_type = 0x1
        char [31] ks_class = [ "misc" ]
        uchar_t ks_flags = 0x1
        void *ks_data = ncs
        uint_t ks_ndata = 0x1c
        size_t ks_data_size = 0x540
        hrtime_t ks_snaptime = 0x15d0d20f9c9f
        int (*)() ks_update = default_kstat_update
        void *ks_private = 0
        int (*)() ks_snapshot = default_kstat_snapshot
        void *ks_lock = 0
    }
    size_t e_size = 0x110
    kthread_t *e_owner = 0
    kcondvar_t e_cv = {
        ushort_t _opaque = 0
    }
    avl_node_t e_avl_bykid = {
        struct avl_node *[2] avl_child = [ 0, 0 ]
        uintptr_t avl_pcb = 0xffffff01482adc85
    }
    avl_node_t e_avl_byname = {
        struct avl_node *[2] avl_child = [ kstat_initial+0xeb98, 
0xffffff014829b1f8 ]
        uintptr_t avl_pcb = 0xfffffffffbc23e4e
    }
    kstat_zone_t e_zone = {
        zoneid_t zoneid = 0xffffffff
        struct kstat_zone *next = 0
    }
}
...

The field that we want to examine is ks_data. For dnlcstats, this is a global variable: ncs.

Here's the ncs structure.

> ncs::print -t -a
fffffffffbc97780 struct nc_stats {
    fffffffffbc97780 kstat_named_t ncs_hits = {
        fffffffffbc97780 char [31] name = [ "hits" ]
        fffffffffbc9779f uchar_t data_type = 0x4
        fffffffffbc977a0 union  value = {
            fffffffffbc977a0 char [16] c = [ 'e', '\245', 'N', '\0', '\0', '\0',
 '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ]
            fffffffffbc977a0 int32_t i32 = 0x4ea565
            fffffffffbc977a0 uint32_t ui32 = 0x4ea565
            fffffffffbc977a0 struct  str = {
                fffffffffbc977a0 union  addr = {
                    fffffffffbc977a0 char *ptr = 0x4ea565
                    fffffffffbc977a0 caddr32_t ptr32 = 0x4ea565
                    fffffffffbc977a0 char [8] __pad = [ 'e', '\245', 'N', '\0', 
'\0', '\0', '\0', '\0' ]
                }
                fffffffffbc977a8 uint32_t len = 0
            }
            fffffffffbc977a0 int64_t i64 = 0x4ea565
            fffffffffbc977a0 uint64_t ui64 = 0x4ea565
            fffffffffbc977a0 long l = 0x4ea565
            fffffffffbc977a0 ulong_t ul = 0x4ea565
            fffffffffbc977a0 longlong_t ll = 0x4ea565
            fffffffffbc977a0 u_longlong_t ull = 0x4ea565
            fffffffffbc977a0 float f = +7.2225011e-39
            fffffffffbc977a0 double d = +2.5464880e-317
        }
    }
    fffffffffbc977b0 kstat_named_t ncs_misses = {
        fffffffffbc977b0 char [31] name = [ "misses" ]
...

The values are stored in a 128-bit union (note that a given kstat might not use all 128 bits). To change the value for "hits" to 0, simply:

> fffffffffbc977a0,2/Z 0
ncs+0x20:       0x4ebf18                =       0x0
ncs+0x28:       0                       =       0x0
>

If you wanted to change all of the values to 0, you would have to go by hand through the nc_stats structure and change the value to 0 for each kstat_named_t in the structure. Or you could possibly write an mdb "dcmd"...

And now let's look at the value.

# kstat -m unix -n dnlcstats
module: unix                            instance: 0     
name:   dnlcstats                       class:    misc
...
   hits                            1675
...
#

So, the value is not 0, but it is much less than 5123316 that we saw when we ran the kstat before. I ran the kstat command a few minutes after setting the value to 0. There could have been 1675 hits since resetting the value, and you could use DTrace to trace them.

# dtrace -n 'dnlc_lookup:return{printf("hits = %d, misses = %d\n", genunix`ncs.ncs_hits.value.ui64, genunix`ncs.ncs_misses.value.ui64);}'
dtrace: description 'dnlc_lookup:return' matched 1 probe
CPU     ID              FUNCTION:NAME
0       17229          dnlc_lookup:return hits = 28486, misses = 27276
0       17229          dnlc_lookup:return hits = 28487, misses = 27276
0       17229          dnlc_lookup:return hits = 28488, misses = 27276
0       17229          dnlc_lookup:return hits = 28489, misses = 27276
0       17229          dnlc_lookup:return hits = 28490, misses = 27276
...

Resetting other kstat values requires the same method. In other words, find the address of the value variable within the kstat you want to modify, and setting it to 0 via mdb.

I mentioned at the beginning that this might not always work. If you are on a multiprocessor system, it is possible that the variable is being updated at the same time you are resetting it to 0.

Depending on atomicity of the operation of increment and zero-ing, it might not work. I also mentioned at the beginning that you could write code. The kstat driver has an ioctl command, KSTAT_IOC_WRITE that could be used for the purpose of setting the value to 0. See for instance the stats_zero_stats() function in http://src.illumos.org/source/xref/illumos-gate/usr/src/cmd/fs.d/cachefs/common/stats_stats.c for details.

Submit your DTrace, MDB, or SmartOS questions to be answered in next week's column by emailing MrBruning@joyent.com, ask at #BruningQuestions, or attend one of my upcoming courses.

:

Sign up Now for Instant Cloud Access

Get Started

View PricingSee Benchmarks