The other day I setup Emergency Management Services on one of my machines. The idea is that you connect your machine to another machine via their COM ports, using a null modem cable. Then, from the other machine you can type in some limited commands when the system becomes unresponsive via normal routes (i.e. Remote Desktop/Terminal Services).
The OS setup involves using bootcfg.exe to edit the boot.ini file to add a few switches. Once that is setup, when you boot the OS, the following will come over the serial port:
Computer is booting, SAC started and initialized.
Use the "ch -?" command for information about using channels.
Use the "?" command for general help.
SAC>
EVENT: The CMD command is now available.
SAC>
Enter ? and press return for help:
SAC>?
ch Channel management commands. Use ch -? for more help.
cmd Create a Command Prompt channel.
d Dump the current kernel log.
f Toggle detailed or abbreviated tlist info.
? or help Display this list.
i List all IP network numbers and their IP addresses.
i <#> <ip> <subnet> <gateway> Set IP addr., subnet and gateway.
id Display the computer identification information.
k <pid> Kill the given process.
l <pid> Lower the priority of a process to the lowest possible.
lock Lock access to Command Prompt channels.
m <pid> <MB-allow> Limit the memory usage of a process to <MB-allow>.
p Toggle paging the display.
r <pid> Raise the priority of a process by one.
s Display the current time and date (24 hour clock used).
s mm/dd/yyyy hh:mm Set the current time and date (24 hour clock used).
t Tlist.
restart Restart the system immediately.
shutdown Shutdown the system immediately.
crashdump Crash the system. You must have crash dump enabled.SAC>
At this point during my investigation my machine was inaccessible from the network, so I entered 'i' for IP address info:
Hmm, that was really suspicious because a few minutes ago I was successfully using Remote Desktop with the machine. So then I figured I'd try to get a Command Prompt on the machine to do further diagnosis:SAC>i
Could not retrieve IP Address(es).
SAC>cmd
The Command Prompt session was successfully launched.
SAC>
EVENT: A new channel has been created. Use "ch -?" for channel help.
Channel: Cmd0001
SAC>
So far so good. To access the "new channel", I pressed <esc><tab>, which showed the following:
Name: Cmd0001
Description: Command Prompt
Type: VT-UTF8
Channel GUID: ead5a758-408c-11db-998c-0030485adfcb
Application Type GUID: 63d02271-8aa4-11d5-bccf-00b0d014a2d0
Press <esc><tab> for next channel.
Press <esc><tab>0 to return to the SAC channel.
Use any other key to view this channel.
Then I pressed some key on my keyboard so that I could "view this channel". It allowed me to enter credentials:
Please enter login credentials.
Username: Administrator
Domain :
Password: ********
Attempting to authenticate...
But then it displayed:
The Command Console session is exiting.
??? Very strange. At this point, I thought I was stuck, so I entered "restart" to reboot the system:
SAC>restart
SAC>SAC failed to restart the system.
Failed with status 0xC000009A.
Yikes, what is going on? It sounds like 0xC000009A means out of paged pool. So I ran the 't' command to get more info:
SAC>t
memory: 2095456 kb uptime: 0 0:20:20.609
PageFile: \??\C:\pagefile.sys
Current Size: 2095104 kb Total Used: 3528 kb Peak Used 36040 kb
Memory:2095456K Avail:1527800K TotalWs: 323948K InRam Kernel: 1720K P: 9372K
Commit: 417392K/ 253420K Limit:4039728K Peak: 486732K Pool N:259904K P: 9792K
User Time Kernel Time Ws Faults Commit Pri Hnd Thd Pid Name
32864 44048 File Cache
0:00:00.000 0:13:18.484 28 0 0 0 0 2 0 Idle Process
0:00:00.000 0:00:24.968 236 4626 28 8 372 68 4 System
0:00:00.000 0:00:00.062 452 181 124 11 20 2 536 smss.exe
0:00:00.078 0:00:00.484 3200 1438 1600 13 361 12 624 csrss.exe
0:00:00.109 0:00:00.390 8580 2587 6032 13 476 20 664 winlogon.exe
0:00:00.093 0:00:00.484 3328 966 1496 9 285 16 708 services.exe
0:00:00.093 0:00:00.203 6764 1904 6900 9 393 25 720 lsass.exe
0:00:00.015 0:00:00.000 2524 696 840 8 79 5 900 svchost.exe
0:00:00.359 0:00:00.453 3172 859 1224 8 207 10 984 svchost.exe
0:00:00.000 0:00:00.031 4328 1354 3732 8 133 7 1084 svchost.exe
0:00:00.031 0:00:00.015 4792 1222 2912 8 156 13 1168 svchost.exe
0:00:07.609 0:01:33.750205392 58331 200384 8 909 69 1184 svchost.exe
0:00:00.000 0:00:00.015 4384 1250 3428 8 127 14 1712 spoolsv.exe
0:00:00.015 0:00:00.031 3892 1123 1604 8 149 13 1736 msdtc.exe
0:00:00.000 0:00:00.000 2036 505 524 8 56 2 1876 svchost.exe
0:00:00.000 0:00:00.000 1616 397 368 8 96 3 1912 IAANTmon.exe
0:00:00.000 0:00:00.000 1284 358 300 8 39 2 1956 svchost.exe
0:00:00.000 0:00:02.546 3972 1044 1348 8 64 3 2040 NTService.exe
0:00:00.015 0:00:00.140 4520 1470 2556 8 165 24 488 svchost.exe
0:00:00.000 0:00:00.000 2696 706 740 8 75 6 212 alg.exe
0:00:00.015 0:00:00.046 4384 1572 2464 8 119 4 432 wmiprvse.exe
0:00:00.109 0:00:00.031 4948 1510 1724 8 162 4 1232 wmiprvse.exe
0:00:00.453 0:00:00.359 12980 7658 12724 8 275 13 3716 HelpSvc.exe
0:00:00.000 0:00:00.015 1576 388 368 4 16 1 3380 logon.scr
Note the huge non-paged pool size of 259904K. It's no wonder networking doesn't work and I can't even create a new cmd.exe process on the machine.
At this point, I should have typed "crashdump" to take a full memory dump of the system to analyze later, but I forgot about that and instead I powercycled the system. :-)
But at least now I know what to look for and what to do next time.