How to reset kstat counters
Sam Zaydel of RackTop Systems says:
I have been thinking about this one for a while, because every now and again, mostly for testing purposes I find that I really wish I could reset a kstat. I have been able to figure out how to reset certain kstat(s) with MDB, but wondering if perhaps you have one method that makes it relatively easy to find and reset any kstat.
In the kernel, all kstats are on 2 different AVL trees, one by name and the other by a kstat id. You can use mdb
to set the value of a kstat to 0, but note that this might not always work (see more below). The other way to set the value to 0 is to write some code. First, let's go through a simple example.
Let's say you want to reset the values of the dnlc (directory name lookup cache) kstats to 0. Here are the values:
# kstat -m unix -n dnlcstatsmodule: unix instance: 0name: dnlcstats class: misc crtime 23.002235736 dir_add_abort 0 dir_add_max 0 dir_add_no_memory 0 dir_cached_current 4 dir_cached_total 4 dir_entries_cached_current 4708 dir_fini_purge 0 dir_hits 4634 dir_misses 16094 dir_reclaim_any 0 dir_reclaim_last 0 dir_remove_entry_fail 0 dir_remove_space_fail 0 dir_start_no_memory 0 dir_update_fail 0 double_enters 1 enters 23789 hits 5123316 misses 26670 negative_cache_hits 152228 pick_free 0 pick_heuristic 3793 pick_last 431 purge_all 0 purge_fs1 0 purge_total_entries 112 purge_vfs 6 purge_vp 63 snaptime 96916.809565119#
We'll use mdb
to dump all of the kstats. Note this may give lots of output.
# mdb -kwLoading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci ufsip hook neti sockfs arp usba stmf_sbd stmf zfs lofs idm mpt crypto random sdcpc logindmux ptm sppp nfs ]> kstat_avl_byname::walk avl !wc 1126 1126 19142>
So, there are 1126 ekstat_t
structures currently in the kernel. Each ekstat_t
may refer to multiple kstat values. Let's take a closer look.
> ::log kstats.out <-- log output to file "kstats.out"mdb: logging to "kstats.out"> kstat_avl_byname::walk avl | ::print -t ekstat_tekstat_t { kstat_t e_ks = { hrtime_t ks_crtime = 0 struct kstat *ks_next = 0 kid_t ks_kid = 0 char [31] ks_module = [ "unix" ] uchar_t ks_resv = 0 int ks_instance = 0 char [31] ks_name = [ "kstat_headers" ] uchar_t ks_type = 0 char [31] ks_class = [ "kstat" ] uchar_t ks_flags = 0x3 void *ks_data = 0 uint_t ks_ndata = 0x3f0 size_t ks_data_size = 0x2d480 hrtime_t ks_snaptime = 0x4785c1c2c60f int (*)() ks_update = header_kstat_update void *ks_private = 0 int (*)() ks_snapshot = header_kstat_snapshot void *ks_lock = kstat_chain_lock } size_t e_size = 0x110 kthread_t *e_owner = 0 kcondvar_t e_cv = { ushort_t _opaque = 0 } avl_node_t e_avl_bykid = { struct avl_node *[2] avl_child = [ 0, 0 ] uintptr_t avl_pcb = 0xfffffffffbc15ab1 } avl_node_t e_avl_byname = { struct avl_node *[2] avl_child = [ kstat_initial+0x32b8,kstat_initial+0x9258 ] uintptr_t avl_pcb = 0xfffffffffbc24618 } kstat_zone_t e_zone = { zoneid_t zoneid = 0xffffffff struct kstat_zone *next = 0 }}...ekstat_t { kstat_t e_ks = { hrtime_t ks_crtime = 0x55b0a4358 struct kstat *ks_next = 0 kid_t ks_kid = 0x176 char [31] ks_module = [ "unix" ] uchar_t ks_resv = 0 int ks_instance = 0 char [31] ks_name = [ "dnlcstats" ] uchar_t ks_type = 0x1 char [31] ks_class = [ "misc" ] uchar_t ks_flags = 0x1 void *ks_data = ncs uint_t ks_ndata = 0x1c size_t ks_data_size = 0x540 hrtime_t ks_snaptime = 0x15d0d20f9c9f int (*)() ks_update = default_kstat_update void *ks_private = 0 int (*)() ks_snapshot = default_kstat_snapshot void *ks_lock = 0 } size_t e_size = 0x110 kthread_t *e_owner = 0 kcondvar_t e_cv = { ushort_t _opaque = 0 } avl_node_t e_avl_bykid = { struct avl_node *[2] avl_child = [ 0, 0 ] uintptr_t avl_pcb = 0xffffff01482adc85 } avl_node_t e_avl_byname = { struct avl_node *[2] avl_child = [ kstat_initial+0xeb98,0xffffff014829b1f8 ] uintptr_t avl_pcb = 0xfffffffffbc23e4e } kstat_zone_t e_zone = { zoneid_t zoneid = 0xffffffff struct kstat_zone *next = 0 }}...
The field that we want to examine is ks_data
. For dnlcstats, this is a global variable: ncs
.
Here's the ncs
structure.
> ncs::print -t -afffffffffbc97780 struct nc_stats { fffffffffbc97780 kstat_named_t ncs_hits = { fffffffffbc97780 char [31] name = [ "hits" ] fffffffffbc9779f uchar_t data_type = 0x4 fffffffffbc977a0 union value = { fffffffffbc977a0 char [16] c = [ 'e', '\245', 'N', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ] fffffffffbc977a0 int32_t i32 = 0x4ea565 fffffffffbc977a0 uint32_t ui32 = 0x4ea565 fffffffffbc977a0 struct str = { fffffffffbc977a0 union addr = { fffffffffbc977a0 char *ptr = 0x4ea565 fffffffffbc977a0 caddr32_t ptr32 = 0x4ea565 fffffffffbc977a0 char [8] __pad = [ 'e', '\245', 'N', '\0','\0', '\0', '\0', '\0' ] } fffffffffbc977a8 uint32_t len = 0 } fffffffffbc977a0 int64_t i64 = 0x4ea565 fffffffffbc977a0 uint64_t ui64 = 0x4ea565 fffffffffbc977a0 long l = 0x4ea565 fffffffffbc977a0 ulong_t ul = 0x4ea565 fffffffffbc977a0 longlong_t ll = 0x4ea565 fffffffffbc977a0 u_longlong_t ull = 0x4ea565 fffffffffbc977a0 float f = +7.2225011e-39 fffffffffbc977a0 double d = +2.5464880e-317 } } fffffffffbc977b0 kstat_named_t ncs_misses = { fffffffffbc977b0 char [31] name = [ "misses" ]...
The values are stored in a 128-bit union (note that a given kstat might not use all 128 bits).To change the value for "hits" to 0, simply:
> fffffffffbc977a0,2/Z 0ncs+0x20: 0x4ebf18 = 0x0ncs+0x28: 0 = 0x0>
If you wanted to change all of the values to 0, you would have to go by hand through the nc_stats
structure and change the value to 0 for each kstat_named_t
in the structure. Or you could possibly write an mdb "dcmd"
...
And now let's look at the value.
# kstat -m unix -n dnlcstatsmodule: unix instance: 0name: dnlcstats class: misc... hits 1675...#
So, the value is not 0, but it is much less than 5123316 that we saw when we ran the kstat before. I ran the kstat
command a few minutes after setting the value to 0. There could have been 1675 hits since resetting the value, and you could use DTrace to trace them.
# dtrace -n 'dnlc_lookup:return{printf("hits = %d, misses = %d\n", genunix`ncs.ncs_hits.value.ui64, genunix`ncs.ncs_misses.value.ui64);}'dtrace: description 'dnlc_lookup:return' matched 1 probeCPU ID FUNCTION:NAME0 17229 dnlc_lookup:return hits = 28486, misses = 272760 17229 dnlc_lookup:return hits = 28487, misses = 272760 17229 dnlc_lookup:return hits = 28488, misses = 272760 17229 dnlc_lookup:return hits = 28489, misses = 272760 17229 dnlc_lookup:return hits = 28490, misses = 27276...
Resetting other kstat values requires the same method. In other words, find the address of the value variable within the kstat you want to modify, and setting it to 0 via mdb.
I mentioned at the beginning that this might not always work. If you are on a multiprocessor system, it is possible that the variable is being updated at the same time you are resetting it to 0.
Depending on atomicity of the operation of increment and zero-ing, it might not work. I also mentioned at the beginning that you could write code. The kstat driver has an ioctl
command, KSTAT_IOC_WRITE
that could be used for the purpose of setting the value to 0. See for instance the stats_zero_stats()
function in src.illumos.org/source/xref/illumos-gate/usr/src/cmd/fs.d/cachefs/common/stats_stats.c for details.
We offer comprehensive training for Triton Developers, Operators and End Users.
Post written by Mr. Max Bruning