The following are demonstrations of the hotkernel DTrace program.


Here hotkernel is run for a couple of seconds then Ctrl-C is hit,

   # ./hotkernel
   Sampling... Hit Ctrl-C to end.
   ^C
   FUNCTION                                                COUNT   PCNT
   unix`swtch                                                  1   0.1%
   pcplusmp`apic_redistribute_compute                          1   0.1%
   genunix`strrput                                             1   0.1%
   unix`sys_call                                               1   0.1%
   genunix`fsflush_do_pages                                    1   0.1%
   TS`ts_wakeup                                                1   0.1%
   genunix`callout_schedule_1                                  1   0.1%
   unix`page_create_putback                                    1   0.1%
   unix`mutex_enter                                            4   0.3%
   unix`cpu_halt                                            1575  99.2%

The output summarises which kernel-level function was sampled on the
CPU the most. This report shows that unix`cpu_halt was sampled 1575 
times, which was 99.2% of the kernel-level samples.

As it turns out, unix`cpu_halt is called on this x86 server as part of the
kernel idle thread - explaining why it is so often found on the CPU,

   # dtrace -n 'fbt::cpu_halt:entry { @[stack()] = count(); }'
   dtrace: description 'fbt::cpu_halt:entry ' matched 1 probe
   ^C
   
                 unix`idle+0x3b
                 unix`thread_start+0x3
                 956

This kernel stack trace indicates that cpu_halt() is called by idle().

The following is a SPARC example,

   # ./hotkernel 
   Sampling... Hit Ctrl-C to end.
   ^C
   FUNCTION                                                COUNT   PCNT
   genunix`fop_ioctl                                           1   0.1%
   genunix`allocb_cred                                         1   0.1%
   genunix`poll_common                                         1   0.1%
   genunix`cv_block                                            1   0.1%
   genunix`strioctl                                            1   0.1%
   genunix`disp_lock_exit                                      1   0.1%
   genunix`crfree                                              1   0.1%
   ufs`ufs_getpage                                             1   0.1%
   SUNW,UltraSPARC-IIi`copyin                                  1   0.1%
   genunix`strmakedata                                         1   0.1%
   genunix`cv_waituntil_sig                                    1   0.1%
   SUNW,UltraSPARC-IIi`prefetch_page_r                         1   0.1%
   unix`set_freemem                                            1   0.1%
   unix`page_trylock                                           1   0.1%
   genunix`anon_get_ptr                                        1   0.1%
   unix`page_hashin                                            1   0.1%
   genunix`bt_getlowbit                                        1   0.1%
   unix`pp_load_tlb                                            1   0.1%
   unix`_resume_from_idle                                      1   0.1%
   unix`hat_pageunload                                         1   0.1%
   genunix`strrput                                             1   0.1%
   genunix`strpoll                                             1   0.1%
   unix`page_do_hashin                                         1   0.1%
   unix`cpu_vm_stats_ks_update                                 1   0.1%
   genunix`sleepq_wakeone_chan                                 1   0.1%
   unix`lock_set_spl                                           1   0.1%
   tl`tl_wput                                                  1   0.1%
   genunix`kstrgetmsg                                          1   0.1%
   genunix`qbackenable                                         1   0.1%
   genunix`releasef                                            1   0.1%
   genunix`callout_execute                                     1   0.1%
   uata`ata_hba_start                                          1   0.1%
   genunix`pcacheset_cmp                                       1   0.1%
   genunix`sleepq_insert                                       1   0.1%
   genunix`syscall_mstate                                      1   0.1%
   sockfs`sotpi_recvmsg                                        1   0.1%
   genunix`strput                                              1   0.1%
   genunix`timespectohz                                        1   0.1%
   unix`lock_clear_splx                                        1   0.1%
   genunix`read                                                1   0.1%
   genunix`as_segcompar                                        1   0.1%
   unix`atomic_cas_64                                          1   0.1%
   unix`mutex_exit                                             1   0.1%
   genunix`cv_unsleep                                          1   0.1%
   unix`putnext                                                1   0.1%
   unix`intr_thread                                            1   0.1%
   genunix`hrt2tv                                              1   0.1%
   sockfs`socktpi_poll                                         1   0.1%
   unix`sfmmu_mlspl_enter                                      1   0.1%
   SUNW,UltraSPARC-IIi`get_ecache_tag                          1   0.1%
   SUNW,UltraSPARC-IIi`gethrestime                             1   0.1%
   genunix`cv_timedwait_sig                                    1   0.1%
   genunix`getq_noenab                                         1   0.1%
   SUNW,UltraSPARC-IIi`flushecacheline                         1   0.1%
   unix`utl0                                                   1   0.1%
   genunix`anon_alloc                                          1   0.1%
   unix`page_downgrade                                         1   0.1%
   unix`setfrontdq                                             1   0.1%
   genunix`timeout_common                                      1   0.1%
   unix`bzero                                                  1   0.1%
   unix`ktl0                                                   2   0.1%
   genunix`canputnext                                          2   0.1%
   genunix`clear_active_fd                                     2   0.1%
   unix`sfmmu_tlb_demap                                        2   0.1%
   unix`page_vpadd                                             2   0.1%
   SUNW,UltraSPARC-IIi`check_ecache_line                       2   0.1%
   genunix`cyclic_softint                                      2   0.1%
   genunix`restore_mstate                                      2   0.1%
   genunix`anon_map_getpages                                   2   0.1%
   genunix`putq                                                2   0.1%
   unix`page_lookup_create                                     2   0.1%
   dtrace`dtrace_dynvar_clean                                  2   0.1%
   unix`sfmmu_pageunload                                       2   0.1%
   genunix`cpu_decay                                           2   0.1%
   genunix`kmem_cache_alloc                                    3   0.2%
   unix`rw_exit                                                3   0.2%
   tl`tl_wput_data_ser                                         3   0.2%
   unix`page_get_replacement_page                              3   0.2%
   unix`page_sub                                               3   0.2%
   genunix`clock                                               3   0.2%
   SUNW,UltraSPARC-IIi`copyout                                 3   0.2%
   unix`mutex_enter                                            4   0.2%
   genunix`pcache_poll                                         5   0.3%
   SUNW,UltraSPARC-IIi`scrub_ecache_line                       5   0.3%
   SUNW,UltraSPARC-IIi`hwblkpagecopy                          22   1.2%
   SUNW,UltraSPARC-IIi`hwblkclr                               39   2.1%
   unix`generic_idle_cpu                                     506  26.8%
   unix`idle                                                1199  63.5%

Which shows the most common function is unix`idle.




Now the hotkernel tool is demonstrated with the -m option, to only print
out samples by module,

   # ./hotkernel -m
   Sampling... Hit Ctrl-C to end.
   ^C
   MODULE                                                  COUNT   PCNT
   usbms                                                       1   0.0%
   specfs                                                      1   0.0%
   uhci                                                        1   0.0%
   sockfs                                                      2   0.0%
   genunix                                                    28   0.6%
   unix                                                     4539  99.3%

Here, genunix and unix (the two core parts of the kernel) were the most
common module to be executing on-CPU.
