Some more changes, 0.6.0 seems to be a tough one.

ChangeLog

@@ -31,6 +31,9 @@
 - Rewrote cprint/cprintlist logic
 - Get rid of python-curses requirement for SLES9 (although it helps)
 - Fixed dstat_disk24old for newer 2.4 kernels without CONFIG_BLK_STATS support (Susan G. Kleinmann)
+- Improved dstat_disk24 for newer 2.4 kernels with CONFIG_BLK_STATS support (Susan G. Kleinmann)
+- Allow for specifying compatible stats on command line (eg. -M disk24,page24)
+- Make time stat more detailed when --debug is used.
 
 * 0.5.10 - released 08/04/2005
 - Small fix to restore terminal for all exit paths (Dbt 303526, Modesto Alexandre)

README.examples

@@ -10,7 +10,8 @@ The following examples currently exist:
 Please send other examples or tools that make use of dstat classes
 or changes to extend the current infrastructure.
 
-I'm not particularly happy with how to interface with dstat from
-the outside, so any hints on how to improve it are welcome.
+I'm not particularly happy with the current interface to dstat,
+so any hints on how to improve it are welcome. Also look at the
+TODO for future changes.
 
 Please send feedback to: <dag@wieers.com>

README.performance

@@ -2,16 +2,32 @@ Dstat performance
 """""""""""""""""
 Since dstat is written in python, it's not optimized for performance.
 
-When doing performance analysis, it's important to verify that the
-monitoring tool is not messing with the performance numbers.
+When doing performance analysis, it's always important to verify that
+the monitoring tool is not messing with the performance numbers.
+(eg. writing to disk, using cpu/memory/network, increasing load)
 
 Depending on the stats being used and the load on the server itself
-the impact Dstat has on the system you're monitoring might be
-considerable.
+the impact Dstat has on the system you are monitoring might be
+considerable. A lot of plugins are pretty fast (less than 0.1ms on
+an modest 1.2Ghz laptop, but some plugins may use up to 3ms using
+up to 2% of your CPU).
 
 Before performing any tests please verify for yourself what impact
 Dstat has on your test results and keep that in mind when analysing
 the results afterwards.
 
 In case the impact is higher than expected, reduce the number of stats
-and remove expensive stats.
+and remove expensive stats or even look at the plugin you're using and
+send me optimisations.
+
+Newer python versions are also faster than older ones, and hardware is
+only becoming faster at a pace that these considerations may not hold
+anylonger.
+
+If you need feedback about plugin performance, use the --debug option
+to profile different plugins. If you use -t together with --debug, you
+can see the time deviation on your system in relation to load/plugins.
+
+Remember that invisible plugins (that run out of your terminal window)
+do take up cycles because the information is still being collected and
+possibly written to CSV output.

TODO

@@ -36,9 +36,9 @@
 ### Stat bugs
 + Implement better (?) protection against counter rollovers
 + tcp stat is very slow and generates lots of softirqs (on busy systems), to be confirmed
-+ proc stats (run, blk and new) does not work on 2.4.30+ (possibly 2.4.x), to be confirmed
++ proc stats (run and blk) does not work on 2.4.24+
 + proc stats seem to be off-by-one compared to vmstat (some of the time, this is what I get from /proc/stat)
-+ /proc/partitions can have negative numbers, seen on systems with long uptime. dstat handles this except for calculating the very first stat, no work-around possible
++ /proc/partitions can have negative numbers, seen on systems with long uptime. dstat handles this except for calculating the very first stat, no work-around possible?
 
 ### Redesign (v2.0)
 + Create modules that can contain samples of different units
@@ -52,4 +52,4 @@
 	blk_read/sec, blk_wrtn/sec (kB/sec)
 
 + Design proper object model and namespace for _all_ possible stats
-+ Create a seperate tool, much like nmon
++ Create a seperate screen-based tool, much like nmon (dstat is line-based)

dstat

@@ -597,7 +597,7 @@ class dstat_disk24(dstat):
 		if self.fd and not self.discover():
 			raise Exception, 'kernel is not compiled with CONFIG_BLK_STATS'
 #			info(1, 'dstat_disk24: kernel is lacking disk stat support, please compile CONFIG_BLK_STATS into your kernel')
-		self.name = ['disk/'+name for name in self.vars]
+		self.name = ['disk/'+sysfs_dev(name) for name in self.vars]
 		self.init(self.vars + ['total',], 2)
 
 	def discover(self, *list):
@@ -1162,7 +1162,8 @@ class dstat_time(dstat):
 	def __init__(self):
 		self.name = 'time'
 		self.format = ('t', 10, 0)
-#		self.format = ('t', 15, 0)
+		if op.debug:
+			self.format = ('t', 14, 0)
 		self.nick = ('epoch',)
 		self.vars = self.nick
 		self.init(self.vars, 1)
@@ -1314,12 +1315,12 @@ def dchg(var, max, base):
 def fchg(var, max, base):
 	c = 0
 	while True:
-#		ret = repr(round(var))
-#		ret = repr(long(round(var,max)))
-		ret = str(long(round(var,max)))
 		if var == 0:
 			ret = str('0')
 			break
+#		ret = repr(round(var))
+#		ret = repr(long(round(var,max)))
+		ret = str(long(round(var,max)))
 		if len(ret) <= max:
 			i = max - len(ret)
 			while i > 0:
@@ -1395,7 +1396,7 @@ def cprint(var, format = ('f', 4, 1000)):
 	ret = ansi[color] + ret.rjust(max)
 
 	if unit:
-		if c != -1 and var != 0:
+		if c != -1 and round(var) != 0:
 			ret = ret + units[c]
 		else:
 			ret = ret + ' '
@@ -1431,6 +1432,7 @@ def die(ret, str):
 	exit(ret)
 
 def getwinsize():
+	"Return the terminal geometry using several methods"
 	if op.nolimit:
 		return 1024, 1024
 	try:
@@ -1449,25 +1451,47 @@ def getwinsize():
 				return 25, 80
 
 def getcpunr():
+	"Return the number of CPUs in the system"
 	cpunr = -1
 	for line in dopen('/proc/stat').readlines():
 		if line[0:3] == 'cpu':
 			cpunr = cpunr + 1
+	if cpunr < 0:
+		raise "Problem finding number of CPUs in system."
 	return cpunr
 
-def scsidev(nr):
+### FIXME: Add scsi support too and improve
+def sysfs_dev(device):
+	"Convert sysfs device names into device names"
+	ide_re = re.compile('ide/host([0-9])/bus([0-9])/target([0-9])/lun([0-9])/disc')
+	scsi_re = re.compile('placeholder')
+	m = ide_re.match(device)
+	if m:
+		l = m.groups()
+		# ide/host0/bus0/target0/lun0/disc -> 0 -> hda
+		# ide/host0/bus1/target0/lun0/disc -> 2 -> hdc
+		nr = int(l[1]) * 2 + int(l[3])
+		return 'hd' + chr(ord('a') + nr)
+	m = scsi_re.match(device)
+	if m:
+		return 'scsi'
+	return device
+
+def scsi_dev(nr):
+	"Convert sequential number into scsi device names"
 	if nr < 26:
 		return 'sd' + chr(ord('a') + nr)
 	else:
 		return 'sd' + chr(ord('a') - 1 + nr / 26) + chr(ord('a') + nr % 26)
 
 def dev(maj, min):
+	"Convert major/minor pairs into device names"
 	scsi = [8, 65, 66, 67, 68, 69, 70, 71, 128, 129, 130, 131, 132, 133, 134, 135]
 	ide = [3, 22, 33, 34, 56, 57, 88, 89, 90, 91]
 	if maj in scsi:
 		nr = scsi.index(maj) * 16 + min
 #		nr = scsi.index(maj) * 16 + min / 16
-		return scsidev(nr)
+		return scsi_dev(nr)
 	elif maj in ide:
 		nr = ide.index(maj) * 2 + min / 64
 		return 'hd' + chr(ord('a') + nr)
@@ -1550,8 +1574,7 @@ def main():
 		elif module == 'disk': mods = ( 'disk', 'disk24', 'disk24old' )
 		elif module == 'int':  mods = ( 'int', 'int24' )
 		elif module == 'page': mods = ( 'page', 'page24' )
-		else:
-			mods = ( module, )
+		else: mods = ( module, )
 
 		for mod in mods:
 			if 'dstat_'+mod not in globals():
@@ -1570,16 +1593,17 @@ def main():
 					linewidth = linewidth + o.statwidth() + 1
 					totlist.append(o)
 					break
+
 			except Exception, e:
-				info(1, 'Module "dstat_%s" has problems. (%s)' % (mod, e))
+				info(1, 'Module dstat_%s has problems. (%s)' % (mod, e))
 				continue
 
 	if not totlist:
 		die(8, 'None of the stats you selected are available.')
 
 	if op.debug:
 		for o in totlist:
-			print str(o.__class__).split('.')[1],
+			print 'Module', str(o.__class__).split('.')[1],
 			if hasattr(o, 'file'): print 'requires', o.file,
 			print
 

examples/dstat.py

@@ -0,0 +1 @@
+../dstat

proc/diskstats-2.6.11

@@ -0,0 +1,92 @@
+   1    0 ram0 0 0 0 0 0 0 0 0 0 0 0
+   1    1 ram1 0 0 0 0 0 0 0 0 0 0 0
+   1    2 ram2 0 0 0 0 0 0 0 0 0 0 0
+   1    3 ram3 0 0 0 0 0 0 0 0 0 0 0
+   1    4 ram4 0 0 0 0 0 0 0 0 0 0 0
+   1    5 ram5 0 0 0 0 0 0 0 0 0 0 0
+   1    6 ram6 0 0 0 0 0 0 0 0 0 0 0
+   1    7 ram7 0 0 0 0 0 0 0 0 0 0 0
+   1    8 ram8 0 0 0 0 0 0 0 0 0 0 0
+   1    9 ram9 0 0 0 0 0 0 0 0 0 0 0
+   1   10 ram10 0 0 0 0 0 0 0 0 0 0 0
+   1   11 ram11 0 0 0 0 0 0 0 0 0 0 0
+   1   12 ram12 0 0 0 0 0 0 0 0 0 0 0
+   1   13 ram13 0 0 0 0 0 0 0 0 0 0 0
+   1   14 ram14 0 0 0 0 0 0 0 0 0 0 0
+   1   15 ram15 0 0 0 0 0 0 0 0 0 0 0
+   3    0 hda 701516 942900 12172907 6225444 284041 770510 6545342 14685351 0 3033115 20954254
+   3    1 hda1 51 73 0 0
+   3    2 hda2 970571 970620 265871 265871
+   3    3 hda3 3 6 0 0
+   3    5 hda5 633013 10305474 647709 5181672
+   3    6 hda6 22653 181209 51758 414064
+   3    7 hda7 4505 538299 1 8
+   3    8 hda8 1075 32107 1 8
+   3    9 hda9 4685 80135 90117 683719
+  22    0 hdc 3699 183366 748408 154422 0 0 0 0 0 153383 154422
+   9    0 md0 0 0 0 0 0 0 0 0 0 0 0
+ 253    0 dm-0 0 0 0 0 0 0 0 0 0 0 0
+   7    0 loop0 0 0 0 0 0 0 0 0 0 0 0
+   7    1 loop1 0 0 0 0 0 0 0 0 0 0 0
+   7    2 loop2 0 0 0 0 0 0 0 0 0 0 0
+   7    3 loop3 0 0 0 0 0 0 0 0 0 0 0
+   7    4 loop4 0 0 0 0 0 0 0 0 0 0 0
+   7    5 loop5 0 0 0 0 0 0 0 0 0 0 0
+   7    6 loop6 0 0 0 0 0 0 0 0 0 0 0
+   7    7 loop7 0 0 0 0 0 0 0 0 0 0 0
+   7    8 loop8 0 0 0 0 0 0 0 0 0 0 0
+   7    9 loop9 0 0 0 0 0 0 0 0 0 0 0
+   7   10 loop10 0 0 0 0 0 0 0 0 0 0 0
+   7   11 loop11 0 0 0 0 0 0 0 0 0 0 0
+   7   12 loop12 0 0 0 0 0 0 0 0 0 0 0
+   7   13 loop13 0 0 0 0 0 0 0 0 0 0 0
+   7   14 loop14 0 0 0 0 0 0 0 0 0 0 0
+   7   15 loop15 0 0 0 0 0 0 0 0 0 0 0
+   7   16 loop16 0 0 0 0 0 0 0 0 0 0 0
+   7   17 loop17 0 0 0 0 0 0 0 0 0 0 0
+   7   18 loop18 0 0 0 0 0 0 0 0 0 0 0
+   7   19 loop19 0 0 0 0 0 0 0 0 0 0 0
+   7   20 loop20 0 0 0 0 0 0 0 0 0 0 0
+   7   21 loop21 0 0 0 0 0 0 0 0 0 0 0
+   7   22 loop22 0 0 0 0 0 0 0 0 0 0 0
+   7   23 loop23 0 0 0 0 0 0 0 0 0 0 0
+   7   24 loop24 0 0 0 0 0 0 0 0 0 0 0
+   7   25 loop25 0 0 0 0 0 0 0 0 0 0 0
+   7   26 loop26 0 0 0 0 0 0 0 0 0 0 0
+   7   27 loop27 0 0 0 0 0 0 0 0 0 0 0
+   7   28 loop28 0 0 0 0 0 0 0 0 0 0 0
+   7   29 loop29 0 0 0 0 0 0 0 0 0 0 0
+   7   30 loop30 0 0 0 0 0 0 0 0 0 0 0
+   7   31 loop31 0 0 0 0 0 0 0 0 0 0 0
+   7   32 loop32 0 0 0 0 0 0 0 0 0 0 0
+   7   33 loop33 0 0 0 0 0 0 0 0 0 0 0
+   7   34 loop34 0 0 0 0 0 0 0 0 0 0 0
+   7   35 loop35 0 0 0 0 0 0 0 0 0 0 0
+   7   36 loop36 0 0 0 0 0 0 0 0 0 0 0
+   7   37 loop37 0 0 0 0 0 0 0 0 0 0 0
+   7   38 loop38 0 0 0 0 0 0 0 0 0 0 0
+   7   39 loop39 0 0 0 0 0 0 0 0 0 0 0
+   7   40 loop40 0 0 0 0 0 0 0 0 0 0 0
+   7   41 loop41 0 0 0 0 0 0 0 0 0 0 0
+   7   42 loop42 0 0 0 0 0 0 0 0 0 0 0
+   7   43 loop43 0 0 0 0 0 0 0 0 0 0 0
+   7   44 loop44 0 0 0 0 0 0 0 0 0 0 0
+   7   45 loop45 0 0 0 0 0 0 0 0 0 0 0
+   7   46 loop46 0 0 0 0 0 0 0 0 0 0 0
+   7   47 loop47 0 0 0 0 0 0 0 0 0 0 0
+   7   48 loop48 0 0 0 0 0 0 0 0 0 0 0
+   7   49 loop49 0 0 0 0 0 0 0 0 0 0 0
+   7   50 loop50 0 0 0 0 0 0 0 0 0 0 0
+   7   51 loop51 0 0 0 0 0 0 0 0 0 0 0
+   7   52 loop52 0 0 0 0 0 0 0 0 0 0 0
+   7   53 loop53 0 0 0 0 0 0 0 0 0 0 0
+   7   54 loop54 0 0 0 0 0 0 0 0 0 0 0
+   7   55 loop55 0 0 0 0 0 0 0 0 0 0 0
+   7   56 loop56 0 0 0 0 0 0 0 0 0 0 0
+   7   57 loop57 0 0 0 0 0 0 0 0 0 0 0
+   7   58 loop58 0 0 0 0 0 0 0 0 0 0 0
+   7   59 loop59 0 0 0 0 0 0 0 0 0 0 0
+   7   60 loop60 0 0 0 0 0 0 0 0 0 0 0
+   7   61 loop61 0 0 0 0 0 0 0 0 0 0 0
+   7   62 loop62 0 0 0 0 0 0 0 0 0 0 0
+   7   63 loop63 0 0 0 0 0 0 0 0 0 0 0

proc/partitions-2.4.21

@@ -0,0 +1,21 @@
+major minor  #blocks  name     rio rmerge rsect ruse wio wmerge wsect wuse running use aveq
+
+  22     0   20094480 hdc 18502 68378 614922 575620 63 16 632 660 -9 28571324 42702177
+  22     1      48163 hdc1 46 70 232 620 0 0 0 0 0 310 620
+  22     2    5116702 hdc2 15272 58372 588450 543550 63 16 632 660 0 31290 544240
+  22     3    5116702 hdc3 46 70 232 620 0 0 0 0 0 310 620
+  22     4          1 hdc4 0 0 0 0 0 0 0 0 0 0 0
+  22     5    5116671 hdc5 46 70 232 540 0 0 0 0 0 280 540
+  22     6    2562336 hdc6 46 70 232 610 0 0 0 0 0 310 610
+  22     7     514048 hdc7 46 70 232 580 0 0 0 0 0 290 580
+  22     8     265041 hdc8 58 98 312 360 0 0 0 0 0 180 360
+  22     9    1349428 hdc9 46 70 232 320 0 0 0 0 0 160 320
+   3     0   80043264 hda 165456390 118310028 -2026323812 11081595 319064897 471719179 2045961426 3973385 -6386 23654554 26608111
+   3     1    1020096 hda1 1022946 359030 11055850 13886300 18524433 42441169 487840520 19795761 0 15952587 33844701
+   3     2   33792727 hda2 16867355 41487758 466839562 3680943 49279950 136588121 1489574368 3787 0 7851978 6830330
+   3     3   20482875 hda3 21222823 25575094 374388250 40543161 47823459 36322836 675746654 7388934 0 15848238 6467323
+   3     4          1 hda4 0 0 0 0 0 0 0 0 0 0 0
+   3     5   19968763 hda5 12553421 16972432 236204218 36866208 132089301 133660978 2128837078 24946643 0 4783292 19031628
+   3     6    3068383 hda6 2219407 1764003 31864346 28167600 15941594 29654857 365639372 10253628 0 18729247 38579828
+   3     7    1020096 hda7 39200 370913 3278858 960450 939530 2371952 26950130 41566540 0 10284610 42535080
+   3     8     522081 hda8 111525236 31538692 1144516184 15779080 54466614 90679266 1166340568 28867099 0 7075683 3835907

proc/partitions-2.4.24

@@ -0,0 +1,12 @@
+major minor  #blocks  name     rio rmerge rsect ruse wio wmerge wsect wuse running use aveq
+
+  22     0    4224150 ide/host0/bus1/target0/lun0/disc 24958 39400 473150 343900 53587 23309 500312 5447730 0 380490 5795870
+  22     1    1023876 ide/host0/bus1/target0/lun0/part1 3592 4076 61498 43970 27109 5490 261392 546010 0 110060 590040
+  22     2     130882 ide/host0/bus1/target0/lun0/part2 1 0 8 10 0 0 0 0 0 10 10
+  22     3    2047815 ide/host0/bus1/target0/lun0/part3 16859 32873 397730 253060 21988 2699 199512 3131240 0 277340 3388480
+  22     4    1021545 ide/host0/bus1/target0/lun0/part4 4505 2448 13906 46850 4490 15120 39408 1770480 0 73390 1817330
+   3     0    8257032 ide/host0/bus0/target0/lun0/disc 1 3 8 10 0 0 0 0 0 10 10
+   3     1    2048256 ide/host0/bus0/target0/lun0/part1 0 0 0 0 0 0 0 0 0 0 0
+   3     2    3068415 ide/host0/bus0/target0/lun0/part2 0 0 0 0 0 0 0 0 0 0 0
+   3     3    1534207 ide/host0/bus0/target0/lun0/part3 0 0 0 0 0 0 0 0 0 0 0
+   3     4    1598467 ide/host0/bus0/target0/lun0/part4 0 0 0 0 0 0 0 0 0 0 0

proc/partitions-2.6.11

@@ -0,0 +1,56 @@
+major minor  #blocks  name
+
+   3     0   78150744 hda
+   3     1    4611568 hda1
+   3     2    3591000 hda2
+   3     3          1 hda3
+   3     5   11801128 hda5
+   3     6     536728 hda6
+   3     7    6168928 hda7
+   3     8    6191608 hda8
+   3     9   45249592 hda9
+ 253     0    4587520 dm-0
+   7     0     652862 loop0
+   7     1     653886 loop1
+   7     2     506174 loop2
+   7     3     631824 loop3
+   7     4     652852 loop4
+   7     5     651854 loop5
+   7     6     395278 loop6
+   7     7     652478 loop7
+   7     8     653444 loop8
+   7     9     652756 loop9
+   7    10     619972 loop10
+   7    11     137586 loop11
+   7    12     666538 loop12
+   7    13     659278 loop13
+   7    14     152052 loop14
+   7    15     157376 loop15
+   7    16     636162 loop16
+   7    17     649870 loop17
+   7    18     201976 loop18
+   7    19     102298 loop19
+   7    20     596550 loop20
+   7    21     250010 loop21
+   7    22       4332 loop22
+   7    23      47104 loop23
+   7    24     145808 loop24
+   7    25     640022 loop25
+   7    26     653232 loop26
+   7    27     208950 loop27
+   7    28     164716 loop28
+   7    29     641798 loop29
+   7    30     645894 loop30
+   7    31     614270 loop31
+   7    32     177882 loop32
+   7    33     594366 loop33
+   7    34     649578 loop34
+   7    35     641730 loop35
+   7    36     242246 loop36
+   7    37     205478 loop37
+   7    38     631000 loop38
+   7    39     653880 loop39
+   7    40     653894 loop40
+   7    41     183950 loop41
+   7    42      63614 loop42
+   7    43     157376 loop43