Procedures for Virgo Daemons Restarting/Starting/Checking

Restarting

This is the case in which the Daemon is  misbehaving or denying user correct actions but is still receiving Cm messages.

Here the restart-commands to be run (note: they can all be issued on any control room workstation by any user):

CmNameServer:  issue "cm restart_server"

Database Server: issue "db restart_server"

Error Logger Server: issue "el restart_server"

Virgo Main Master Server: issue "vmm restart_server"
 

Starting

This is the case in which the server has crashed or do not respond anymore to any Cm message (like the restart).

The important point to be taken in account is that the daemons (*.exe) are started by a start script (*.startcascina) that automatically restarts them in case they crash.

Login into the machine olserver as virgod (note: for CascinaVTF or CascinaVCS you can also log with your normal user account at ctrlvtf/olvtf, respectively):

ssh olserver -l virgod

To check if the start scripts run:

ps -ef | grep start

And this is the output if all are running:
0 S virgod   19803     1  0  69   0    -  1066 wait4  Jul12 ?        00:00:00 /bin/sh /virgoApp/Cm/v7r16/mgr/NameServer.startcascina Cascina
0 S virgod   20137     1  0  69   0    -  1177 rt_sig Jul12 ?        00:00:00 /bin/csh -f /virgo/VCS-2.0/VIRGOSW//El/v4r7/mgr/ELServer.startcascina Cascina
0 S virgod   20143     1  0  69   0    -  1177 rt_sig Jul12 ?        00:00:00 /bin/csh -f /virgo/VCS-2.0/VIRGOSW//El/v4r7/mgr/ESServer.startcascina Cascina
0 S virgod   20511     1  0  69   0    -  1177 rt_sig Jul12 ?        00:00:00 /bin/csh -f /virgo/VCS-2.0/VIRGOSW//VMM/v1r6/mgr/VMMServer.startcascina Cascina
0 S virgod    6130     1  0  69   0    -  1187 rt_sig Jul28 ?        00:00:00 /bin/csh -f /virgo/VCS-2.0/VIRGOSW//Db/v4r8/mgr/DbServer.startcascina

To check if the daemons are running:

ps -ef | grep exe 

And this is the output if all are running:
0 S virgod   19840 19803  0  70   0    -   876 do_sel Jul12 ?        01:07:27 NameServer.exe Cascina v7r16
0 S virgod   20141 20137  0  69   0    -  1044 do_sel Jul12 ?        00:01:20 ErrorLogger.exe Cascina Cascina
0 S virgod   20147 20143  0  69   0    -  2328 do_sel Jul12 ?        00:01:08 ErrorScanner.exe Cascina Cascina
0 S virgod   20517 20511  0  69   0    -  1718 do_sel Jul12 ?        00:00:30 VMMServer.exe Cascina
0 S virgod    6155  6130  0  69   0    -  1354 do_sel Jul28 ?        00:02:02 DbServer.exe Cascina DbServerv4

There can be then different cases:

1) Both start scripts and daemons (exe) are not running

Start the start scripts and the daemons via the following commands (note: If you wish to start either the CascinaVTF or CascinaVCS domains (both are located on different machines to the Cascina domain) use the same syntax as listed below, but replace "Cascina" with "CascinaVTF" or "CascinaVCS").

$CMROOT/mgr/NameServer.startcascina Cascina&
$DBROOT/mgr/DbServer.startcascina Cascina&
$ELROOT/mgr/ELServer.startcascina Cascina&
$ELROOT/mgr/ESServer.startcascina Cascina&
$VMMROOT/mgr/VMMServer.startcascina Cascina&

2) Deamons are running but start scripts are not

Kill the daemons (exe) manually, then proceed as indicated in point 1).

3) Daemons are not running but start scripts are

Kill the start scripts, the proceed as indicated in 1).

4) Both daemons (exe) and start scripts are running

Kill the daemons (exe) and let the start scripts restart the daemon. If this is not working  kill the start script and proceed as indicated in 1).
 

Checking

Those are the commands to be run in order to check the correct Restarting/Starting. They can all be issued on any Control Room workstation.

CmNameServer:

cm names 

To check if the server is up and running.

Database Server:

db load_object Virgo/Partition/OSF 

To check if the database is accessible.

Error Logger Server:
Run "elui" and verify that the GUI reports on its lower right corner the text "ELErrorLogger <version>"

Virgo Main Master Server:
Run "vmmui" and verify that the GUI reports the text "VMMServer <version>"
 
Remember that all daemons are monitored via BigBrother. You may want to check the trend status under the Virgo Daemon subpage.
 

dampingBackup.sh

On olserver is currently also running the process bacukpping all  the damping cards at any download on DSP.