support mechanism to verify power command completed

[chu11]

In some protocols, such as IPMI and Redfish, power commands like on and off can be sent and responded to, but the server will perform the actual power on/off at a later time. It could result in some unexpected behavior, such as the following hypothetical:

> pm --off node1
Command Completed Successfully
> pm -q
On: node1
Off:
<wait a little bit>
> pm -q
On: 
Off: node1

In order to solve this in the ipmipower power solution, wait-until-on and wait-until-off options were implemented. Internally in ipmipower, after an on or off request is sent, the power status of the node is checked and the power on/off will not return to the user until it has been verified that it was completed by the server.

It would be nice if this is something that could be scripted via powerman device files. Hypothetically:

	script on_verify {
		send "on %s\n"
		expect "ipmipower> "
	}
	script verify_on {
		send "stat %s\n"
		expect "([^\n:]+): ([^\n]+\n)"
		setplugstate $1 $2 on="^on\n" off="^off\n"
		expect "ipmipower> "
	}

and powerman would know not to return to the user until the on has been verified to have been on.

[garlick]

Another way do it might be to add an option in the device specification to enable that behavior and configure timing.

Edit: maybe something like:

specification "test" {
    timeout 60

    verify-status           # get status after power cmd, fail if not desired state(s)
    verify-status-retry 4   # retry verify up to N times
    verify-status-wait 2    # wait N secs before each verify
    ...
}