File:  [LON-CAPA] / loncom / loncnew
Revision 1.19: download - view: text, annotated - select for diffs
Tue Aug 19 09:31:46 2003 UTC (20 years, 8 months ago) by foxr
Branches: MAIN
CVS tags: HEAD
Get socket directory from configuration rather than the old hard coded test
way that I forgot to un-hard code.

    1: #!/usr/bin/perl
    2: # The LearningOnline Network with CAPA
    3: # lonc maintains the connections to remote computers
    4: #
    5: # $Id: loncnew,v 1.19 2003/08/19 09:31:46 foxr Exp $
    6: #
    7: # Copyright Michigan State University Board of Trustees
    8: #
    9: # This file is part of the LearningOnline Network with CAPA (LON-CAPA).
   10: ## LON-CAPA is free software; you can redistribute it and/or modify
   11: # it under the terms of the GNU General Public License as published by
   12: # the Free Software Foundation; either version 2 of the License, or
   13: # (at your option) any later version.
   14: #
   15: # LON-CAPA is distributed in the hope that it will be useful,
   16: # but WITHOUT ANY WARRANTY; without even the implied warranty of
   17: # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   18: # GNU General Public License for more details.
   19: #
   20: # You should have received a copy of the GNU General Public License
   21: # along with LON-CAPA; if not, write to the Free Software
   22: # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
   23: #
   24: # /home/httpd/html/adm/gpl.txt
   25: #
   26: # http://www.lon-capa.org/
   27: #
   28: #
   29: # new lonc handles n request out bver m connections to londs.
   30: # This module is based on the Event class.
   31: #   Development iterations:
   32: #    - Setup basic event loop.   (done)
   33: #    - Add timer dispatch.       (done)
   34: #    - Add ability to accept lonc UNIX domain sockets.  (done)
   35: #    - Add ability to create/negotiate lond connections (done).
   36: #    - Add general logic for dispatching requests and timeouts. (done).
   37: #    - Add support for the lonc/lond requests.          (done).
   38: #    - Add logging/status monitoring.
   39: #    - Add Signal handling - HUP restarts. USR1 status report.
   40: #    - Add Configuration file I/O                       (done).
   41: #    - Add management/status request interface.
   42: #    - Add deferred request capability.                  (done)
   43: #    - Detect transmission timeouts.
   44: #
   45: 
   46: # Change log:
   47: #    $Log: loncnew,v $
   48: #    Revision 1.19  2003/08/19 09:31:46  foxr
   49: #    Get socket directory from configuration rather than the old hard coded test
   50: #    way that I forgot to un-hard code.
   51: #
   52: #    Revision 1.18  2003/08/06 09:52:29  foxr
   53: #    Also needed to remember to fail in-flight transactions if their sends fail.
   54: #
   55: #    Revision 1.17  2003/08/03 00:44:31  foxr
   56: #    1. Correct handling of connection failure: Assume it means the host is
   57: #       unreachable and fail all of the queued transactions.  Note that the
   58: #       inflight transactions should fail on their own time due either to timeout
   59: #       or send/receive failures.
   60: #    2. Correct handling of logs for forced death signals.  Pull the signal
   61: #       from the event watcher.
   62: #
   63: #    Revision 1.16  2003/07/29 02:33:05  foxr
   64: #    Add SIGINT processing to child processes to toggle annoying trace mode
   65: #    on/off.. will try to use this to isolate the compute boud process issue.
   66: #
   67: #    Revision 1.15  2003/07/15 02:07:05  foxr
   68: #    Added code for lonc/lond transaction timeouts.  Who knows if it works right.
   69: #    The intent is for a timeout to fail any transaction in progress and kill
   70: #    off the sockt that timed out.
   71: #
   72: #    Revision 1.14  2003/07/03 02:10:18  foxr
   73: #    Get all of the signals to work correctly.
   74: #
   75: #    Revision 1.13  2003/07/02 01:31:55  foxr
   76: #    Added kill -HUP logic (restart).
   77: #
   78: #    Revision 1.11  2003/06/25 01:54:44  foxr
   79: #    Fix more problems with transaction failure.
   80: #
   81: #    Revision 1.10  2003/06/24 02:46:04  foxr
   82: #    Put a limit on  the number of times we'll retry a connection.
   83: #    Start getting the signal stuff put in as well...note that need to get signals
   84: #    going or else 6the client will permanently give up on dead servers.
   85: #
   86: #    Revision 1.9  2003/06/13 02:38:43  foxr
   87: #    Add logging in 'expected format'
   88: #
   89: #    Revision 1.8  2003/06/11 02:04:35  foxr
   90: #    Support delayed transactions... this is done uniformly by encapsulating
   91: #    transactions in an object ... a LondTransaction that is implemented by
   92: #    LondTransaction.pm
   93: #
   94: #    Revision 1.7  2003/06/03 01:59:39  foxr
   95: #    complete coding to support deferred transactions.
   96: #
   97: #
   98: 
   99: use lib "/home/httpd/lib/perl/";
  100: use lib "/home/foxr/newloncapa/types";
  101: use Event qw(:DEFAULT );
  102: use POSIX qw(:signal_h);
  103: use POSIX;
  104: use IO::Socket;
  105: use IO::Socket::INET;
  106: use IO::Socket::UNIX;
  107: use IO::File;
  108: use IO::Handle;
  109: use Socket;
  110: use Crypt::IDEA;
  111: use LONCAPA::Queue;
  112: use LONCAPA::Stack;
  113: use LONCAPA::LondConnection;
  114: use LONCAPA::LondTransaction;
  115: use LONCAPA::Configuration;
  116: use LONCAPA::HashIterator;
  117: 
  118: 
  119: #
  120: #   Disable all signals we might receive from outside for now.
  121: #
  122: #$SIG{QUIT}  = IGNORE;
  123: #$SIG{HUP}   = IGNORE;
  124: #$SIG{USR1}  = IGNORE;
  125: #$SIG{INT}   = IGNORE;
  126: #$SIG{CHLD}  = IGNORE;
  127: #$SIG{__DIE__}  = IGNORE;
  128: 
  129: 
  130: # Read the httpd configuration file to get perl variables
  131: # normally set in apache modules:
  132: 
  133: my $perlvarref = LONCAPA::Configuration::read_conf('loncapa.conf');
  134: my %perlvar    = %{$perlvarref};
  135: 
  136: #
  137: #  parent and shared variables.
  138: 
  139: my %ChildHash;			# by pid -> host.
  140: 
  141: 
  142: my $MaxConnectionCount = 10;	# Will get from config later.
  143: my $ClientConnection = 0;	# Uniquifier for client events.
  144: 
  145: my $DebugLevel = 0;
  146: my $NextDebugLevel= 10;		# So Sigint can toggle this.
  147: my $IdleTimeout= 3600;		# Wait an hour before pruning connections.
  148: 
  149: #
  150: #  The variables below are only used by the child processes.
  151: #
  152: my $RemoteHost;			# Name of host child is talking to.
  153: my $UnixSocketDir= $perlvar{'lonSockDir'}
  154: my $IdleConnections = Stack->new(); # Set of idle connections
  155: my %ActiveConnections;		# Connections to the remote lond.
  156: my %ActiveTransactions;		# LondTransactions in flight.
  157: my %ActiveClients;		# Serial numbers of active clients by socket.
  158: my $WorkQueue       = Queue->new(); # Queue of pending transactions.
  159: my $ConnectionCount = 0;
  160: my $IdleSeconds     = 0;	# Number of seconds idle.
  161: my $Status          = "";	# Current status string.
  162: my $RecentLogEntry  = "";
  163: my $ConnectionRetries=5;	# Number of connection retries allowed.
  164: my $ConnectionRetriesLeft=5;	# Number of connection retries remaining.
  165: 
  166: #
  167: #   The hash below gives the HTML format for log messages
  168: #   given a severity.
  169: #    
  170: my %LogFormats;
  171: 
  172: $LogFormats{"CRITICAL"} = "<font color=red>CRITICAL: %s</font>";
  173: $LogFormats{"SUCCESS"}  = "<font color=green>SUCCESS: %s</font>";
  174: $LogFormats{"INFO"}     = "<font color=yellow>INFO: %s</font>";
  175: $LogFormats{"WARNING"}  = "<font color=blue>WARNING: %s</font>";
  176: $LogFormats{"DEFAULT"}  = " %s ";
  177: 
  178: 
  179: 
  180: =pod
  181: 
  182: =head2 LogPerm
  183: 
  184: Makes an entry into the permanent log file.
  185: 
  186: =cut
  187: sub LogPerm {
  188:     my $message=shift;
  189:     my $execdir=$perlvar{'lonDaemons'};
  190:     my $now=time;
  191:     my $local=localtime($now);
  192:     my $fh=IO::File->new(">>$execdir/logs/lonnet.perm.log");
  193:     print $fh "$now:$message:$local\n";
  194: }
  195: 
  196: =pod
  197: 
  198: =head2 Log
  199: 
  200: Logs a message to the log file.
  201: Parameters:
  202: 
  203: =item severity
  204: 
  205: One of CRITICAL, WARNING, INFO, SUCCESS used to select the
  206: format string used to format the message.  if the severity is
  207: not a defined severity the Default format string is used.
  208: 
  209: =item message
  210: 
  211: The base message.  In addtion to the format string, the message
  212: will be appended to a string containing the name of our remote
  213: host and the time will be formatted into the message.
  214: 
  215: =cut
  216: 
  217: sub Log {
  218:     my $severity = shift;
  219:     my $message  = shift;
  220:    
  221:     if(!$LogFormats{$severity}) {
  222: 	$severity = "DEFAULT";
  223:     }
  224: 
  225:     my $format = $LogFormats{$severity};
  226:     
  227:     #  Put the window dressing in in front of the message format:
  228: 
  229:     my $now   = time;
  230:     my $local = localtime($now);
  231:     my $finalformat = "$local ($$) [$RemoteHost] [$Status] ";
  232:     my $finalformat = $finalformat.$format."\n";
  233: 
  234:     # open the file and put the result.
  235: 
  236:     my $execdir = $perlvar{'lonDaemons'};
  237:     my $fh      = IO::File->new(">>$execdir/logs/lonc.log");
  238:     my $msg = sprintf($finalformat, $message);
  239:     $RecentLogEntry = $msg;
  240:     print $fh $msg;
  241:     
  242:     
  243: }
  244: 
  245: 
  246: =pod
  247: 
  248: =head2 GetPeerName
  249: 
  250: Returns the name of the host that a socket object is connected to.
  251: 
  252: =cut
  253: 
  254: sub GetPeername {
  255:     my $connection = shift;
  256:     my $AdrFamily  = shift;
  257:     my $peer       = $connection->peername();
  258:     my $peerport;
  259:     my $peerip;
  260:     if($AdrFamily == AF_INET) {
  261: 	($peerport, $peerip) = sockaddr_in($peer);
  262: 	my $peername    = gethostbyaddr($iaddr, $AdrFamily);
  263: 	return $peername;
  264:     } elsif ($AdrFamily == AF_UNIX) {
  265: 	my $peerfile;
  266: 	($peerfile) = sockaddr_un($peer);
  267: 	return $peerfile;
  268:     }
  269: }
  270: #----------------------------- Timer management ------------------------
  271: =pod
  272: 
  273: =head2 Debug
  274: 
  275: Invoked to issue a debug message.
  276: 
  277: =cut
  278: 
  279: sub Debug {
  280:     my $level   = shift;
  281:     my $message = shift;
  282:     if ($level <= $DebugLevel) {
  283: 	Log("INFO", "-Debug- $message host = $RemotHost");
  284:     }
  285: }
  286: 
  287: sub SocketDump {
  288:     my $level = shift;
  289:     my $socket= shift;
  290:     if($level <= $DebugLevel) {
  291: 	$socket->Dump();
  292:     }
  293: }
  294: 
  295: =pod
  296: 
  297: =head2 ShowStatus
  298: 
  299:  Place some text as our pid status.
  300:  and as what we return in a SIGUSR1
  301: 
  302: =cut
  303: sub ShowStatus {
  304:     my $state = shift;
  305:     my $now = time;
  306:     my $local = localtime($now);
  307:     $Status   = $local.": ".$state;
  308:     $0='lonc: '.$state.' '.$local;
  309: }
  310: 
  311: =pod
  312: 
  313: =head 2 SocketTimeout
  314: 
  315:     Called when an action on the socket times out.  The socket is 
  316:    destroyed and any active transaction is failed.
  317: 
  318: 
  319: =cut
  320: sub SocketTimeout {
  321:     my $Socket = shift;
  322:     
  323:     KillSocket($Socket);
  324: }
  325: 
  326: =pod
  327: 
  328: =head2 Tick
  329: 
  330: Invoked  each timer tick.
  331: 
  332: =cut
  333: 
  334: 
  335: sub Tick {
  336:     my $client;
  337:     ShowStatus(GetServerHost()." Connection count: ".$ConnectionCount);
  338: 
  339:     # Is it time to prune connection count:
  340: 
  341: 
  342:     if($IdleConnections->Count()  && 
  343:        ($WorkQueue->Count() == 0)) { # Idle connections and nothing to do?
  344: 	$IdleSeconds++;
  345: 	if($IdleSeconds > $IdleTimeout) { # Prune a connection...
  346: 	    $Socket = $IdleConnections->pop();
  347: 	    KillSocket($Socket);
  348: 	}
  349:     } else {
  350: 	$IdleSeconds = 0;	# Reset idle count if not idle.
  351:     }
  352:     #
  353:     #  For each inflight transaction, tick down its timeout counter.
  354:     #
  355:     foreach $item (keys %ActiveTransactions) {
  356: 	my $Socket = $ActiveTransactions{$item}->getServer();
  357: 	$Socket->Tick();
  358:     }
  359:     # Do we have work in the queue, but no connections to service them?
  360:     # If so, try to make some new connections to get things going again.
  361:     #
  362:     
  363:     my $Requests = $WorkQueue->Count();
  364:     if (($ConnectionCount == 0)  && ($Requests > 0)) { 
  365: 	if ($ConnectionRetriesLeft > 0) {
  366: 	    my $Connections = ($Requests <= $MaxConnectionCount) ?
  367: 		$Requests : $MaxConnectionCount;
  368: 	    Debug(1,"Work but no connections, start ".$Connections." of them");
  369: 	    for ($i =0; $i < $Connections; $i++) {
  370: 		MakeLondConnection();
  371: 	    }
  372: 	} else {
  373: 	    Debug(1,"Work in queue, but gave up on connections..flushing\n");
  374: 	    EmptyQueue();	# Connections can't be established.
  375: 	}
  376:        
  377:     }
  378: }
  379: 
  380: =pod
  381: 
  382: =head2 SetupTimer
  383: 
  384: Sets up a 1 per sec recurring timer event.  The event handler is used to:
  385: 
  386: =item
  387: 
  388: Trigger timeouts on communications along active sockets.
  389: 
  390: =item
  391: 
  392: Trigger disconnections of idle sockets.
  393: 
  394: =cut
  395: 
  396: sub SetupTimer {
  397:     Debug(6, "SetupTimer");
  398:     Event->timer(interval => 1, debug => 1, cb => \&Tick );
  399: }
  400: 
  401: =pod
  402: 
  403: =head2 ServerToIdle
  404: 
  405: This function is called when a connection to the server is
  406: ready for more work.
  407: 
  408: If there is work in the Work queue the top element is dequeued
  409: and the connection will start to work on it.  If the work queue is
  410: empty, the connection is pushed on the idle connection stack where
  411: it will either get another work unit, or alternatively, if it sits there
  412: long enough, it will be shut down and released.
  413: 
  414: =cut
  415: 
  416: sub ServerToIdle {
  417:     my $Socket   = shift;	# Get the socket.
  418:     delete($ActiveTransactions{$Socket}); # Server has no transaction
  419: 
  420:     &Debug(6, "Server to idle");
  421: 
  422:     #  If there's work to do, start the transaction:
  423: 
  424:     $reqdata = $WorkQueue->dequeue(); # This is a LondTransaction
  425:     unless($reqdata eq undef)  {
  426: 	Debug(9, "Queue gave request data: ".$reqdata->getRequest());
  427: 	&StartRequest($Socket,  $reqdata);
  428: 
  429:     } else {
  430: 	
  431:     #  There's no work waiting, so push the server to idle list.
  432: 	&Debug(8, "No new work requests, server connection going idle");
  433: 	$IdleConnections->push($Socket);
  434:     }
  435: }
  436: 
  437: =pod
  438: 
  439: =head2 ClientWritable
  440: 
  441: Event callback for when a client socket is writable.
  442: 
  443: This callback is established when a transaction reponse is
  444: avaiable from lond.  The response is forwarded to the unix socket
  445: as it becomes writable in this sub.
  446: 
  447: Parameters:
  448: 
  449: =item Event
  450: 
  451: The event that has been triggered. Event->w->data is
  452: the data and Event->w->fd is the socket to write.
  453: 
  454: =cut
  455: 
  456: sub ClientWritable {
  457:     my $Event    = shift;
  458:     my $Watcher  = $Event->w;
  459:     my $Data     = $Watcher->data;
  460:     my $Socket   = $Watcher->fd;
  461: 
  462:     # Try to send the data:
  463: 
  464:     &Debug(6, "ClientWritable writing".$Data);
  465:     &Debug(9, "Socket is: ".$Socket);
  466: 
  467:     if($Socket->connected) {
  468: 	my $result = $Socket->send($Data, 0);
  469: 	
  470: 	# $result undefined: the write failed.
  471: 	# otherwise $result is the number of bytes written.
  472: 	# Remove that preceding string from the data.
  473: 	# If the resulting data is empty, destroy the watcher
  474: 	# and set up a read event handler to accept the next
  475: 	# request.
  476: 	
  477: 	&Debug(9,"Send result is ".$result." Defined: ".defined($result));
  478: 	if(defined($result)) {
  479: 	    &Debug(9, "send result was defined");
  480: 	    if($result == length($Data)) { # Entire string sent.
  481: 		&Debug(9, "ClientWritable data all written");
  482: 		$Watcher->cancel();
  483: 		#
  484: 		#  Set up to read next request from socket:
  485: 		
  486: 		my $descr     = sprintf("Connection to lonc client %d",
  487: 					$ActiveClients{$Socket});
  488: 		Event->io(cb    => \&ClientRequest,
  489: 			  poll  => 'r',
  490: 			  desc  => $descr,
  491: 			  data  => "",
  492: 			  fd    => $Socket);
  493: 		
  494: 	    } else {		# Partial string sent.
  495: 		$Watcher->data(substr($Data, $result));
  496: 		if($result == 0) {    # client hung up on us!!
  497: 		    Log("INFO", "lonc pipe client hung up on us!");
  498: 		    $Watcher->cancel;
  499: 		    $Socket->shutdown(2);
  500: 		    $Socket->close();
  501: 		}
  502: 	    }
  503: 	    
  504: 	} else {			# Error of some sort...
  505: 	    
  506: 	    # Some errnos are possible:
  507: 	    my $errno = $!;
  508: 	    if($errno == POSIX::EWOULDBLOCK   ||
  509: 	       $errno == POSIX::EAGAIN        ||
  510: 	       $errno == POSIX::EINTR) {
  511: 		# No action taken?
  512: 	    } else {		# Unanticipated errno.
  513: 		&Debug(5,"ClientWritable error or peer shutdown".$RemoteHost);
  514: 		$Watcher->cancel;	# Stop the watcher.
  515: 		$Socket->shutdown(2); # Kill connection
  516: 		$Socket->close();	# Close the socket.
  517: 	    }
  518: 	    
  519: 	}
  520:     } else {
  521: 	$Watcher->cancel();	# A delayed request...just cancel.
  522:     }
  523: }
  524: 
  525: =pod
  526: 
  527: =head2 CompleteTransaction
  528: 
  529: Called when the reply data has been received for a lond 
  530: transaction.   The reply data must now be sent to the
  531: ultimate client on the other end of the Unix socket.  This is
  532: done by setting up a writable event for the socket with the
  533: data the reply data.
  534: 
  535: Parameters:
  536: 
  537: =item Socket
  538: 
  539: Socket on which the lond transaction occured.  This is a
  540: LondConnection. The data received is in the TransactionReply member.
  541: 
  542: =item Transaction
  543: 
  544: The transaction that is being completed.
  545: 
  546: =cut
  547: 
  548: sub CompleteTransaction {
  549:     &Debug(6,"Complete transaction");
  550:     my $Socket = shift;
  551:     my $Transaction = shift;
  552: 
  553:     if (!$Transaction->isDeferred()) { # Normal transaction
  554: 	my $data   = $Socket->GetReply(); # Data to send.
  555: 	StartClientReply($Transaction, $data);
  556:     } else {			# Delete deferred transaction file.
  557: 	Log("SUCCESS", "A delayed transaction was completed");
  558: 	LogPerm("S:$Client:".$Transaction->getRequest());
  559: 	unlink $Transaction->getFile();
  560:     }
  561: }
  562: =pod
  563: =head1 StartClientReply
  564: 
  565:    Initiates a reply to a client where the reply data is a parameter.
  566: 
  567: =head2  parameters:
  568: 
  569: =item Transaction
  570: 
  571:     The transaction for which we are responding to the client.
  572: 
  573: =item data
  574: 
  575:     The data to send to apached client.
  576: 
  577: =cut
  578: sub StartClientReply {
  579:     my $Transaction   = shift;
  580:     my $data     = shift;
  581: 
  582: 
  583:     my $Client   = $Transaction->getClient();
  584: 
  585:     &Debug(8," Reply was: ".$data);
  586:     my $Serial         = $ActiveClients{$Client};
  587:     my $desc           = sprintf("Connection to lonc client %d",
  588: 
  589: 				 $Serial);
  590:     Event->io(fd       => $Client,
  591: 	      poll     => "w",
  592: 	      desc     => $desc,
  593: 	      cb       => \&ClientWritable,
  594: 	      data     => $data);
  595: }
  596: =pod
  597: =head2 FailTransaction
  598: 
  599:   Finishes a transaction with failure because the associated lond socket
  600:   disconnected.  There are two possibilities:
  601:   - The transaction is deferred: in which case we just quietly
  602:     delete the transaction since there is no client connection.
  603:   - The transaction is 'live' in which case we initiate the sending
  604:     of "con_lost" to the client.
  605: 
  606: Deleting the transaction means killing it from the 
  607: %ActiveTransactions hash.
  608: 
  609: Parameters:
  610: 
  611: =item client  
  612:  
  613:    The LondTransaction we are failing.
  614:  
  615: =cut
  616: 
  617: sub FailTransaction {
  618:     my $transaction = shift;
  619:     Log("WARNING", "Failing transaction ".$transaction->getRequest());
  620:     Debug(1, "Failing transaction: ".$transaction->getRequest());
  621:     if (!$transaction->isDeferred()) { # If the transaction is deferred we'll get to it.
  622: 	my $client  = $transaction->getClient();
  623: 	Debug(1," Replying con_lost to ".$transaction->getRequest());
  624: 	StartClientReply($transaction, "con_lost\n");
  625:     }
  626: 
  627: }
  628: 
  629: =pod
  630: =head1  EmptyQueue
  631: 
  632:   Fails all items in the work queue with con_lost.
  633:   Note that each item in the work queue is a transaction.
  634: 
  635: =cut
  636: sub EmptyQueue {
  637:     while($WorkQueue->Count()) {
  638: 	my $request = $WorkQueue->dequeue(); # This is a transaction
  639: 	FailTransaction($request);
  640:     }
  641: }
  642: 
  643: =pod
  644: 
  645: =head2 CloseAllLondConnections
  646: 
  647: Close all connections open on lond prior to exit e.g.
  648: 
  649: =cut
  650: sub CloseAllLondConnections {
  651:     foreach $Socket (keys %ActiveConnections) {
  652: 	KillSocket($Socket);
  653:     }
  654: }
  655: =cut
  656: 
  657: =pod
  658: 
  659: =head2 KillSocket
  660:  
  661: Destroys a socket.  This function can be called either when a socket
  662: has died of 'natural' causes or because a socket needs to be pruned due to
  663: idleness.  If the socket has died naturally, if there are no longer any 
  664: live connections a new connection is created (in case there are transactions
  665: in the queue).  If the socket has been pruned, it is never re-created.
  666: 
  667: Parameters:
  668: 
  669: =item Socket
  670:  
  671:   The socket to kill off.
  672: 
  673: =item Restart
  674: 
  675: nonzero if we are allowed to create a new connection.
  676: 
  677: 
  678: =cut
  679: sub KillSocket {
  680:     my $Socket = shift;
  681: 
  682:     Log("WARNING", "Shutting down a socket");
  683:     $Socket->Shutdown();
  684: 
  685:     #  If the socket came from the active connection set,
  686:     #  delete its transaction... note that FailTransaction should
  687:     #  already have been called!!!
  688:     #  otherwise it came from the idle set.
  689:     #  
  690:     
  691:     if(exists($ActiveTransactions{$Socket})) {
  692: 	delete ($ActiveTransactions{$Socket});
  693:     }
  694:     if(exists($ActiveConnections{$Socket})) {
  695: 	delete($ActiveConnections{$Socket});
  696:     }
  697:     $ConnectionCount--;
  698: 
  699:     #  If the connection count has gone to zero and there is work in the
  700:     #  work queue, the work all gets failed with con_lost.
  701:     #
  702:     if($ConnectionCount == 0) {
  703: 	EmptyQueue;
  704:     }
  705: }
  706: 
  707: =pod
  708: 
  709: =head2 LondReadable
  710: 
  711: This function is called whenever a lond connection
  712: is readable.  The action is state dependent:
  713: 
  714: =head3 State=Initialized
  715: 
  716: We''re waiting for the challenge, this is a no-op until the
  717: state changes.
  718: 
  719: =head3 State=Challenged 
  720: 
  721: The challenge has arrived we need to transition to Writable.
  722: The connection must echo the challenge back.
  723: 
  724: =head3 State=ChallengeReplied
  725: 
  726: The challenge has been replied to.  The we are receiveing the 
  727: 'ok' from the partner.
  728: 
  729: =head3 State=RequestingKey
  730: 
  731: The ok has been received and we need to send the request for
  732: an encryption key.  Transition to writable for that.
  733: 
  734: =head3 State=ReceivingKey
  735: 
  736: The the key has been requested, now we are reading the new key.
  737: 
  738: =head3 State=Idle 
  739: 
  740: The encryption key has been negotiated or we have finished 
  741: reading data from the a transaction.   If the callback data has
  742: a client as well as the socket iformation, then we are 
  743: doing a transaction and the data received is relayed to the client
  744: before the socket is put on the idle list.
  745: 
  746: =head3 State=SendingRequest
  747: 
  748: I do not think this state can be received here, but if it is,
  749: the appropriate thing to do is to transition to writable, and send
  750: the request.
  751: 
  752: =head3 State=ReceivingReply
  753: 
  754: We finished sending the request to the server and now transition
  755: to readable to receive the reply. 
  756: 
  757: The parameter to this function are:
  758: 
  759: The event. Implicit in this is the watcher and its data.  The data 
  760: contains at least the lond connection object and, if a 
  761: transaction is in progress, the socket attached to the local client.
  762: 
  763: =cut
  764: 
  765: sub LondReadable {
  766: 
  767:     my $Event      = shift;
  768:     my $Watcher    = $Event->w;
  769:     my $Socket     = $Watcher->data;
  770:     my $client     = undef;
  771: 
  772:     &Debug(6,"LondReadable called state = ".$State);
  773: 
  774: 
  775:     my $State = $Socket->GetState(); # All action depends on the state.
  776: 
  777:     SocketDump(6, $Socket);
  778:     my $status = $Socket->Readable();
  779: 
  780:     &Debug(2, "Socket->Readable returned: $status");
  781: 
  782:     if($status != 0) {
  783: 	 # bad return from socket read. Currently this means that
  784: 	# The socket has become disconnected. We fail the transaction.
  785: 
  786: 	Log("WARNING",
  787: 	    "Lond connection lost.");
  788: 	if(exists($ActiveTransactions{$Socket})) {
  789: 	    FailTransaction($ActiveTransactions{$Socket});
  790: 	}
  791: 	$Watcher->cancel();
  792: 	KillSocket($Socket);
  793: 	return;
  794:     }
  795:     SocketDump(6,$Socket);
  796: 
  797:     $State = $Socket->GetState(); # Update in case of transition.
  798:     &Debug(6, "After read, state is ".$State);
  799: 
  800:    if($State eq "Initialized") {
  801: 
  802: 
  803:     } elsif ($State eq "ChallengeReceived") {
  804: 	#  The challenge must be echoed back;  The state machine
  805: 	# in the connection takes care of setting that up.  Just
  806: 	# need to transition to writable:
  807: 
  808: 	$Watcher->cb(\&LondWritable);
  809: 	$Watcher->poll("w");
  810: 
  811:     } elsif ($State eq "ChallengeReplied") {
  812: 
  813: 
  814:     } elsif ($State eq "RequestingKey") {
  815: 	#  The ok was received.  Now we need to request the key
  816: 	#  That requires us to be writable:
  817: 
  818: 	$Watcher->cb(\&LondWritable);
  819: 	$Watcher->poll("w");
  820: 
  821:     } elsif ($State eq "ReceivingKey") {
  822: 
  823:     } elsif ($State eq "Idle") {
  824: 	# If necessary, complete a transaction and then go into the
  825: 	# idle queue.
  826: 	$Watcher->cancel();
  827: 	if(exists($ActiveTransactions{$Socket})) {
  828: 	    Debug(8,"Completing transaction!!");
  829: 	    CompleteTransaction($Socket, 
  830: 				$ActiveTransactions{$Socket});
  831: 	} else {
  832: 	    Log("SUCCESS", "Connection ".$ConnectionCount." to "
  833: 		.$RemoteHost." now ready for action");
  834: 	}
  835: 	ServerToIdle($Socket);	# Next work unit or idle.
  836: 	
  837:     } elsif ($State eq "SendingRequest") {
  838: 	#  We need to be writable for this and probably don't belong
  839: 	#  here inthe first place.
  840: 
  841: 	Deubg(6, "SendingRequest state encountered in readable");
  842: 	$Watcher->poll("w");
  843: 	$Watcher->cb(\&LondWritable);
  844: 
  845:     } elsif ($State eq "ReceivingReply") {
  846: 
  847: 
  848:     } else {
  849: 	 # Invalid state.
  850: 	Debug(4, "Invalid state in LondReadable");
  851:     }
  852: }
  853: 
  854: =pod
  855: 
  856: =head2 LondWritable
  857: 
  858: This function is called whenever a lond connection
  859: becomes writable while there is a writeable monitoring
  860: event.  The action taken is very state dependent:
  861: 
  862: =head3 State = Connected 
  863: 
  864: The connection is in the process of sending the 'init' hailing to the
  865: lond on the remote end.  The connection object''s Writable member is
  866: called.  On error, ConnectionError is called to destroy the connection
  867: and remove it from the ActiveConnections hash
  868: 
  869: =head3 Initialized
  870: 
  871: 'init' has been sent, writability monitoring is removed and
  872: readability monitoring is started with LondReadable as the callback.
  873: 
  874: =head3 ChallengeReceived
  875: 
  876: The connection has received the who are you challenge from the remote
  877: system, and is in the process of sending the challenge
  878: response. Writable is called.
  879: 
  880: =head3 ChallengeReplied
  881: 
  882: The connection has replied to the initial challenge The we switch to
  883: monitoring readability looking for the server to reply with 'ok'.
  884: 
  885: =head3 RequestingKey
  886: 
  887: The connection is in the process of requesting its encryption key.
  888: Writable is called.
  889: 
  890: =head3 ReceivingKey
  891: 
  892: The connection has sent the request for a key.  Switch to readability
  893: monitoring to accept the key
  894: 
  895: =head3 SendingRequest
  896: 
  897: The connection is in the process of sending a request to the server.
  898: This request is part of a client transaction.  All the states until
  899: now represent the client setup protocol. Writable is called.
  900: 
  901: =head3 ReceivingReply
  902: 
  903: The connection has sent a request.  Now it must receive a reply.
  904: Readability monitoring is requested.
  905: 
  906: This function is an event handler and therefore receives as
  907: a parameter the event that has fired.  The data for the watcher
  908: of this event is a reference to a list of one or two elements,
  909: depending on state. The first (and possibly only) element is the
  910: socket.  The second (present only if a request is in progress)
  911: is the socket on which to return a reply to the caller.
  912: 
  913: =cut
  914: 
  915: sub LondWritable {
  916:     my $Event   = shift;
  917:     my $Watcher = $Event->w;
  918:     my $Socket  = $Watcher->data;
  919:     my $State   = $Socket->GetState();
  920: 
  921:     Debug(6,"LondWritable State = ".$State."\n");
  922: 
  923:  
  924:     #  Figure out what to do depending on the state of the socket:
  925:     
  926: 
  927: 
  928: 
  929:     SocketDump(6,$Socket);
  930: 
  931:     if      ($State eq "Connected")         {
  932: 
  933: 	if ($Socket->Writable() != 0) {
  934: 	    #  The write resulted in an error.
  935: 	    # We'll treat this as if the socket got disconnected:
  936: 	    Log("WARNING", "Connection to ".$RemoteHost.
  937: 		" has been disconnected");
  938: 	    FailTransaction($ActiveTransactions{$Socket});
  939: 	    $Watcher->cancel();
  940: 	    KillSocket($Socket);
  941: 	    return;
  942: 	}
  943: 	#  "init" is being sent...
  944: 
  945: 	
  946:     } elsif ($State eq "Initialized")       {
  947: 
  948: 	# Now that init was sent, we switch 
  949: 	# to watching for readability:
  950: 
  951: 	$Watcher->cb(\&LondReadable);
  952: 	$Watcher->poll("r");
  953: 
  954:     } elsif ($State eq "ChallengeReceived") {
  955: 	# We received the challenge, now we 
  956: 	# are echoing it back. This is a no-op,
  957: 	# we're waiting for the state to change
  958: 	
  959: 	if($Socket->Writable() != 0) {
  960: 
  961: 	    $Watcher->cancel();
  962: 	    KillSocket($Socket);
  963: 	    return;
  964: 	}
  965: 	
  966:     } elsif ($State eq "ChallengeReplied")  {
  967: 	# The echo was sent back, so we switch
  968: 	# to watching readability.
  969: 
  970: 	$Watcher->cb(\&LondReadable);
  971: 	$Watcher->poll("r");
  972: 
  973:     } elsif ($State eq "RequestingKey")     {
  974: 	# At this time we're requesting the key.
  975: 	# again, this is essentially a no-op.
  976: 	# we'll write the next chunk until the
  977: 	# state changes.
  978: 
  979: 	if($Socket->Writable() != 0) {
  980: 	    # Write resulted in an error.
  981: 
  982: 	    $Watcher->cancel();
  983: 	    KillSocket($Socket);
  984: 	    return;
  985: 
  986: 	}
  987:     } elsif ($State eq "ReceivingKey")      {
  988: 	# Now we need to wait for the key
  989: 	# to come back from the peer:
  990: 
  991: 	$Watcher->cb(\&LondReadable);
  992: 	$Watcher->poll("r");
  993: 
  994:     } elsif ($State eq "SendingRequest")    {
  995: 	# At this time we are sending a request to the
  996: 	# peer... write the next chunk:
  997: 
  998: 	if($Socket->Writable() != 0) {
  999: 
 1000: 	    if(exists($ActiveTransactions{$Socket})) {
 1001: 		Debug(3, "Lond connection lost, failing transactions");
 1002: 		FailTransaction($ActiveTransactions{$Socket});
 1003: 	    }
 1004: 	    $Watcher->cancel();
 1005: 	    KillSocket($Socket);
 1006: 	    return;
 1007: 	    
 1008: 	}
 1009: 
 1010:     } elsif ($State eq "ReceivingReply")    {
 1011: 	# The send has completed.  Wait for the
 1012: 	# data to come in for a reply.
 1013: 	Debug(8,"Writable sent request/receiving reply");
 1014: 	$Watcher->cb(\&LondReadable);
 1015: 	$Watcher->poll("r");
 1016: 
 1017:     } else {
 1018: 	#  Control only passes here on an error: 
 1019: 	#  the socket state does not match any
 1020: 	#  of the known states... so an error
 1021: 	#  must be logged.
 1022: 
 1023: 	&Debug(4, "Invalid socket state ".$State."\n");
 1024:     }
 1025:     
 1026: }
 1027: =pod
 1028:     
 1029: =cut
 1030: sub QueueDelayed {
 1031:     Debug(3,"QueueDelayed called");
 1032: 
 1033:     my $path = "$perlvar{'lonSockDir'}/delayed";
 1034: 
 1035:     Debug(4, "Delayed path: ".$path);
 1036:     opendir(DIRHANDLE, $path);
 1037:     
 1038:     @alldelayed = grep /\.$RemoteHost$/, readdir DIRHANDLE;
 1039:     Debug(4, "Got ".$alldelayed." delayed files");
 1040:     closedir(DIRHANDLE);
 1041:     my $dfname;
 1042:     my $reqfile;
 1043:     foreach $dfname (sort  @alldelayed) {
 1044: 	$reqfile = "$path/$dfname";
 1045: 	Debug(4, "queueing ".$reqfile);
 1046: 	my $Handle = IO::File->new($reqfile);
 1047: 	my $cmd    = <$Handle>;
 1048: 	chomp $cmd;		# There may or may not be a newline...
 1049: 	$cmd = $cmd."\n";	# now for sure there's exactly one newline.
 1050: 	my $Transaction = LondTransaction->new($cmd);
 1051: 	$Transaction->SetDeferred($reqfile);
 1052: 	QueueTransaction($Transaction);
 1053:     }
 1054:     
 1055: }
 1056: 
 1057: =pod
 1058: 
 1059: =head2 MakeLondConnection
 1060: 
 1061: Create a new lond connection object, and start it towards its initial
 1062: idleness.  Once idle, it becomes elligible to receive transactions
 1063: from the work queue.  If the work queue is not empty when the
 1064: connection is completed and becomes idle, it will dequeue an entry and
 1065: start off on it.
 1066: 
 1067: =cut
 1068: 
 1069: sub MakeLondConnection {     
 1070:     Debug(4,"MakeLondConnection to ".GetServerHost()." on port "
 1071: 	  .GetServerPort());
 1072: 
 1073:     my $Connection = LondConnection->new(&GetServerHost(),
 1074: 					 &GetServerPort());
 1075: 
 1076:     if($Connection == undef) {	# Needs to be more robust later.
 1077: 	Log("CRITICAL","Failed to make a connection with lond.");
 1078: 	$ConnectionRetriesLeft--;
 1079: 	return 0;		# Failure.
 1080:     }  else {
 1081: 	$ConnectionRetriesLeft = $ConnectionRetries; # success resets the count
 1082: 	# The connection needs to have writability 
 1083: 	# monitored in order to send the init sequence
 1084: 	# that starts the whole authentication/key
 1085: 	# exchange underway.
 1086: 	#
 1087: 	my $Socket = $Connection->GetSocket();
 1088: 	if($Socket == undef) {
 1089: 	    die "did not get a socket from the connection";
 1090: 	} else {
 1091: 	    &Debug(9,"MakeLondConnection got socket: ".$Socket);
 1092: 	}
 1093: 	
 1094: 	
 1095: 	$event = Event->io(fd       => $Socket,
 1096: 			   poll     => 'w',
 1097: 			   cb       => \&LondWritable,
 1098: 			   data     => $Connection,
 1099: 			   desc => 'Connection to lond server');
 1100: 	$ActiveConnections{$Connection} = $event;
 1101: 	
 1102: 	$ConnectionCount++;
 1103: 	Debug(4, "Connection count = ".$ConnectionCount);
 1104: 	if($ConnectionCount == 1) { # First Connection:
 1105: 	    QueueDelayed;
 1106: 	}
 1107: 	Log("SUCESS", "Created connection ".$ConnectionCount
 1108: 	    ." to host ".GetServerHost());
 1109: 	return 1;		# Return success.
 1110:     }
 1111:     
 1112: }
 1113: 
 1114: =pod
 1115: 
 1116: =head2 StartRequest
 1117: 
 1118: Starts a lond request going on a specified lond connection.
 1119: parameters are:
 1120: 
 1121: =item $Lond
 1122: 
 1123: Connection to the lond that will send the transaction and receive the
 1124: reply.
 1125: 
 1126: =item $Client
 1127: 
 1128: Connection to the client that is making this request We got the
 1129: request from this socket, and when the request has been relayed to
 1130: lond and we get a reply back from lond it will get sent to this
 1131: socket.
 1132: 
 1133: =item $Request
 1134: 
 1135: The text of the request to send.
 1136: 
 1137: =cut
 1138: 
 1139: sub StartRequest {
 1140:     my $Lond     = shift;
 1141:     my $Request  = shift;	# This is a LondTransaction.
 1142:     
 1143:     Debug(6, "StartRequest: ".$Request->getRequest());
 1144: 
 1145:     my $Socket = $Lond->GetSocket();
 1146:     
 1147:     $Request->Activate($Lond);
 1148:     $ActiveTransactions{$Lond} = $Request;
 1149: 
 1150:     $Lond->InitiateTransaction($Request->getRequest());
 1151:     $event = Event->io(fd      => $Socket,
 1152: 		       poll    => "w",
 1153: 		       cb      => \&LondWritable,
 1154: 		       data    => $Lond,
 1155: 		       desc    => "lond transaction connection");
 1156:     $ActiveConnections{$Lond} = $event;
 1157:     Debug(8," Start Request made watcher data with ".$event->data."\n");
 1158: }
 1159: 
 1160: =pod
 1161: 
 1162: =head2 QueueTransaction
 1163: 
 1164: If there is an idle lond connection, it is put to work doing this
 1165: transaction.  Otherwise, the transaction is placed in the work queue.
 1166: If placed in the work queue and the maximum number of connections has
 1167: not yet been created, a new connection will be started.  Our goal is
 1168: to eventually have a sufficient number of connections that the work
 1169: queue will typically be empty.  parameters are:
 1170: 
 1171: =item Socket
 1172: 
 1173: open on the lonc client.
 1174: 
 1175: =item Request
 1176: 
 1177: data to send to the lond.
 1178: 
 1179: =cut
 1180: 
 1181: sub QueueTransaction {
 1182: 
 1183:     my $requestData   = shift;	# This is a LondTransaction.
 1184:     my $cmd           = $requestData->getRequest();
 1185: 
 1186:     Debug(6,"QueueTransaction: ".$cmd);
 1187: 
 1188:     my $LondSocket    = $IdleConnections->pop();
 1189:     if(!defined $LondSocket) {	# Need to queue request.
 1190: 	Debug(8,"Must queue...");
 1191: 	$WorkQueue->enqueue($requestData);
 1192: 	if($ConnectionCount < $MaxConnectionCount) {
 1193: 	    Debug(4,"Starting additional lond connection");
 1194: 	    if(MakeLondConnection() == 0) {
 1195: 		EmptyQueue();	# Fail transactions, can't make connection.
 1196: 	    }
 1197: 	}
 1198:     } else {			# Can start the request:
 1199: 	Debug(8,"Can start...");
 1200: 	StartRequest($LondSocket,  $requestData);
 1201:     }
 1202: }
 1203: 
 1204: #-------------------------- Lonc UNIX socket handling ---------------------
 1205: 
 1206: =pod
 1207: 
 1208: =head2 ClientRequest
 1209: Callback that is called when data can be read from the UNIX domain
 1210: socket connecting us with an apache server process.
 1211: 
 1212: =cut
 1213: 
 1214: sub ClientRequest {
 1215:     Debug(6, "ClientRequest");
 1216:     my $event   = shift;
 1217:     my $watcher = $event->w;
 1218:     my $socket  = $watcher->fd;
 1219:     my $data    = $watcher->data;
 1220:     my $thisread;
 1221: 
 1222:     Debug(9, "  Watcher named: ".$watcher->desc);
 1223: 
 1224:     my $rv = $socket->recv($thisread, POSIX::BUFSIZ, 0);
 1225:     Debug(8, "rcv:  data length = ".length($thisread)
 1226: 	  ." read =".$thisread);
 1227:     unless (defined $rv && length($thisread)) {
 1228: 	 # Likely eof on socket.
 1229: 	Debug(5,"Client Socket closed on lonc for ".$RemoteHost);
 1230: 	close($socket);
 1231: 	$watcher->cancel();
 1232: 	delete($ActiveClients{$socket});
 1233: 	return;
 1234:     }
 1235:     Debug(8,"Data: ".$data." this read: ".$thisread);
 1236:     $data = $data.$thisread;	# Append new data.
 1237:     $watcher->data($data);
 1238:     if($data =~ /(.*\n)/) {	# Request entirely read.
 1239: 	if($data eq "close_connection_exit\n") {
 1240: 	    Log("CRITICAL",
 1241: 		"Request Close Connection ... exiting");
 1242: 	    CloseAllLondConnections();
 1243: 	    exit;
 1244: 	}
 1245: 	Debug(8, "Complete transaction received: ".$data);
 1246: 	my $Transaction = LondTransaction->new($data);
 1247: 	$Transaction->SetClient($socket);
 1248: 	QueueTransaction($Transaction);
 1249: 	$watcher->cancel();	# Done looking for input data.
 1250:     }
 1251: 
 1252: }
 1253: 
 1254: 
 1255: =pod
 1256: 
 1257: =head2  NewClient
 1258: 
 1259: Callback that is called when a connection is received on the unix
 1260: socket for a new client of lonc.  The callback is parameterized by the
 1261: event.. which is a-priori assumed to be an io event, and therefore has
 1262: an fd member that is the Listener socket.  We Accept the connection
 1263: and register a new event on the readability of that socket:
 1264: 
 1265: =cut
 1266: 
 1267: sub NewClient {
 1268:     Debug(6, "NewClient");
 1269:     my $event      = shift;		# Get the event parameters.
 1270:     my $watcher    = $event->w; 
 1271:     my $socket     = $watcher->fd;	# Get the event' socket.
 1272:     my $connection = $socket->accept();	# Accept the client connection.
 1273:     Debug(8,"Connection request accepted from "
 1274: 	  .GetPeername($connection, AF_UNIX));
 1275: 
 1276: 
 1277:     my $description = sprintf("Connection to lonc client %d",
 1278: 			      $ClientConnection);
 1279:     Debug(9, "Creating event named: ".$description);
 1280:     Event->io(cb      => \&ClientRequest,
 1281: 	      poll    => 'r',
 1282: 	      desc    => $description,
 1283: 	      data    => "",
 1284: 	      fd      => $connection);
 1285:     $ActiveClients{$connection} = $ClientConnection;
 1286:     $ClientConnection++;
 1287: }
 1288: 
 1289: =pod
 1290: 
 1291: =head2 GetLoncSocketPath
 1292: 
 1293: Returns the name of the UNIX socket on which to listen for client
 1294: connections.
 1295: 
 1296: =cut
 1297: 
 1298: sub GetLoncSocketPath {
 1299:     return $UnixSocketDir."/".GetServerHost();
 1300: }
 1301: 
 1302: =pod
 1303: 
 1304: =head2 GetServerHost
 1305: 
 1306: Returns the host whose lond we talk with.
 1307: 
 1308: =cut
 1309: 
 1310: sub GetServerHost {
 1311:     return $RemoteHost;		# Setup by the fork.
 1312: }
 1313: 
 1314: =pod
 1315: 
 1316: =head2 GetServerPort
 1317: 
 1318: Returns the lond port number.
 1319: 
 1320: =cut
 1321: 
 1322: sub GetServerPort {
 1323:     return $perlvar{londPort};
 1324: }
 1325: 
 1326: =pod
 1327: 
 1328: =head2 SetupLoncListener
 1329: 
 1330: Setup a lonc listener event.  The event is called when the socket
 1331: becomes readable.. that corresponds to the receipt of a new
 1332: connection.  The event handler established will accept the connection
 1333: (creating a communcations channel), that int turn will establish
 1334: another event handler to subess requests.
 1335: 
 1336: =cut
 1337: 
 1338: sub SetupLoncListener {
 1339: 
 1340:     my $socket;
 1341:     my $SocketName = GetLoncSocketPath();
 1342:     unlink($SocketName);
 1343:     unless ($socket =IO::Socket::UNIX->new(Local  => $SocketName,
 1344: 					    Listen => 10, 
 1345: 					    Type   => SOCK_STREAM)) {
 1346: 	die "Failed to create a lonc listner socket";
 1347:     }
 1348:     Event->io(cb     => \&NewClient,
 1349: 	      poll   => 'r',
 1350: 	      desc   => 'Lonc listener Unix Socket',
 1351: 	      fd     => $socket);
 1352: }
 1353: 
 1354: =pod 
 1355: 
 1356: =head2 ChildStatus
 1357:  
 1358: Child USR1 signal handler to report the most recent status
 1359: into the status file.
 1360: 
 1361: =cut
 1362: sub ChildStatus {
 1363:     my $event = shift;
 1364:     my $watcher = $event->w;
 1365: 
 1366:     Debug(2, "Reporting child status because : ".$watcher->data);
 1367:     my $docdir = $perlvar{'lonDocRoot'};
 1368:     my $fh = IO::File->new(">>$docdir/lon-status/loncstatus.txt");
 1369:     print $fh $$."\t".$RemoteHost."\t".$Status."\t".
 1370: 	$RecentLogEntry."\n";
 1371: }
 1372: 
 1373: =pod
 1374: 
 1375: =head2 SignalledToDeath
 1376: 
 1377: Called in response to a signal that causes a chid process to die.
 1378: 
 1379: =cut
 1380: 
 1381: 
 1382: sub SignalledToDeath {
 1383:     my $event  = shift;
 1384:     my $watcher= $event->w;
 1385: 
 1386:     Debug(2,"Signalled to death! via ".$watcher->data);
 1387:     my ($signal) = $watcher->data;
 1388:     chomp($signal);
 1389:     Log("CRITICAL", "Abnormal exit.  Child $$ for $RemoteHost "
 1390: 	."died through "."\"$signal\"");
 1391:     LogPerm("F:lonc: $$ on $RemoteHost signalled to death: "
 1392: 	    ."\"$signal\"");
 1393:     exit 0;
 1394: 
 1395: }
 1396: 
 1397: =head2 ToggleDebug
 1398: 
 1399: This sub toggles trace debugging on and off.
 1400: 
 1401: =cut
 1402: 
 1403: sub ToggleDebug {
 1404:     my $Current    = $DebugLevel;
 1405:        $DebugLevel = $NextDebugLevel;
 1406:        $NextDebugLevel = $Current;
 1407: 
 1408:     Log("SUCCESS", "New debugging level for $RemoteHost now $DebugLevel");
 1409: 
 1410: }
 1411: 
 1412: =head2 ChildProcess
 1413: 
 1414: This sub implements a child process for a single lonc daemon.
 1415: 
 1416: =cut
 1417: 
 1418: sub ChildProcess {
 1419: 
 1420: 
 1421:     #
 1422:     #  Signals must be handled by the Event framework...
 1423: #
 1424: 
 1425:     Event->signal(signal   => "QUIT",
 1426: 		  cb       => \&SignalledToDeath,
 1427: 		  data     => "QUIT");
 1428:     Event->signal(signal   => "HUP",
 1429: 		  cb       => \&ChildStatus,
 1430: 		  data     => "HUP");
 1431:     Event->signal(signal   => "USR1",
 1432: 		  cb       => \&ChildStatus,
 1433: 		  data     => "USR1");
 1434:     Event->signal(signal   => "INT",
 1435: 		  cb       => \&ToggleDebug,
 1436: 		  data     => "INT");
 1437: 
 1438:     SetupTimer();
 1439:     
 1440:     SetupLoncListener();
 1441:     
 1442:     $Event::Debuglevel = $DebugLevel;
 1443:     
 1444:     Debug(9, "Making initial lond connection for ".$RemoteHost);
 1445: 
 1446: # Setup the initial server connection:
 1447:     
 1448:      # &MakeLondConnection(); // let first work requirest do it.
 1449: 
 1450: 
 1451:     Debug(9,"Entering event loop");
 1452:     my $ret = Event::loop();		#  Start the main event loop.
 1453:     
 1454:     
 1455:     die "Main event loop exited!!!";
 1456: }
 1457: 
 1458: #  Create a new child for host passed in:
 1459: 
 1460: sub CreateChild {
 1461:     my $sigset = POSIX::SigSet->new(SIGINT);
 1462:     sigprocmask(SIG_BLOCK, $sigset);
 1463:     my $host = shift;
 1464:     $RemoteHost = $host;
 1465:     Log("CRITICAL", "Forking server for ".$host);
 1466:     $pid          = fork;
 1467:     if($pid) {			# Parent
 1468: 	$RemoteHost = "Parent";
 1469: 	$ChildHash{$pid} = $RemoteHost;
 1470: 	sigprocmask(SIG_UNBLOCK, $sigset);
 1471: 
 1472:     } else {			# child.
 1473: 	ShowStatus("Connected to ".$RemoteHost);
 1474: 	$SIG{INT} = DEFAULT;
 1475: 	sigprocmask(SIG_UNBLOCK, $sigset);
 1476: 	ChildProcess;		# Does not return.
 1477:     }
 1478: 
 1479: }
 1480: #
 1481: #  Parent process logic pass 1:
 1482: #   For each entry in the hosts table, we will
 1483: #  fork off an instance of ChildProcess to service the transactions
 1484: #  to that host.  Each pid will be entered in a global hash
 1485: #  with the value of the key, the host.
 1486: #  The parent will then enter a loop to wait for process exits.
 1487: #  Each exit gets logged and the child gets restarted.
 1488: #
 1489: 
 1490: #
 1491: #   Fork and start in new session so hang-up isn't going to 
 1492: #   happen without intent.
 1493: #
 1494: 
 1495: 
 1496: 
 1497: 
 1498: 
 1499: 
 1500: ShowStatus("Forming new session");
 1501: my $childpid = fork;
 1502: if ($childpid != 0) {
 1503:     sleep 4;			# Give child a chacne to break to
 1504:     exit 0;			# a new sesion.
 1505: }
 1506: #
 1507: #   Write my pid into the pid file so I can be located
 1508: #
 1509: 
 1510: ShowStatus("Parent writing pid file:");
 1511: $execdir = $perlvar{'lonDaemons'};
 1512: open (PIDSAVE, ">$execdir/logs/lonc.pid");
 1513: print PIDSAVE "$$\n";
 1514: close(PIDSAVE);
 1515: 
 1516: 
 1517: 
 1518: if (POSIX::setsid() < 0) {
 1519:     print "Could not create new session\n";
 1520:     exit -1;
 1521: }
 1522: 
 1523: ShowStatus("Forking node servers");
 1524: 
 1525: Log("CRITICAL", "--------------- Starting children ---------------");
 1526: 
 1527: my $HostIterator = LondConnection::GetHostIterator;
 1528: while (! $HostIterator->end()) {
 1529: 
 1530:     $hostentryref = $HostIterator->get();
 1531:     CreateChild($hostentryref->[0]);
 1532:     $HostIterator->next();
 1533: }
 1534: $RemoteHost = "Parent Server";
 1535: 
 1536: # Maintain the population:
 1537: 
 1538: ShowStatus("Parent keeping the flock");
 1539: 
 1540: #
 1541: #   Set up parent signals:
 1542: #
 1543: 
 1544: $SIG{INT}  = \&Terminate;
 1545: $SIG{TERM} = \&Terminate; 
 1546: $SIG{HUP}  = \&Restart;
 1547: $SIG{USR1} = \&CheckKids; 
 1548: 
 1549: while(1) {
 1550:     $deadchild = wait();
 1551:     if(exists $ChildHash{$deadchild}) {	# need to restart.
 1552: 	$deadhost = $ChildHash{$deadchild};
 1553: 	delete($ChildHash{$deadchild});
 1554: 	Log("WARNING","Lost child pid= ".$deadchild.
 1555: 	      "Connected to host ".$deadhost);
 1556: 	Log("INFO", "Restarting child procesing ".$deadhost);
 1557: 	CreateChild($deadhost);
 1558:     }
 1559: }
 1560: 
 1561: 
 1562: 
 1563: =pod
 1564: 
 1565: =head1 CheckKids
 1566: 
 1567:   Since kids do not die as easily in this implementation
 1568: as the previous one, there  is no need to restart the
 1569: dead ones (all dead kids get restarted when they die!!)
 1570: The only thing this function does is to pass USR1 to the
 1571: kids so that they report their status.
 1572: 
 1573: =cut
 1574: 
 1575: sub CheckKids {
 1576:     Debug(2, "Checking status of children");
 1577:     my $docdir = $perlvar{'lonDocRoot'};
 1578:     my $fh = IO::File->new(">$docdir/lon-status/loncstatus.txt");
 1579:     my $now=time;
 1580:     my $local=localtime($now);
 1581:     print $fh "LONC status $local - parent $$ \n\n";
 1582:     foreach $pid (keys %ChildHash) {
 1583: 	Debug(2, "Sending USR1 -> $pid");
 1584: 	kill 'USR1' => $pid;	# Tell Child to report status.
 1585: 	sleep 1;		# Wait so file doesn't intermix.
 1586:     }
 1587: }
 1588: 
 1589: =pod
 1590: 
 1591: =head1 Restart
 1592: 
 1593: Signal handler for HUP... all children are killed and
 1594: we self restart.  This is an el-cheapo way to re read
 1595: the config file.
 1596: 
 1597: =cut
 1598: 
 1599: sub Restart {
 1600:     KillThemAll;		# First kill all the children.
 1601:     Log("CRITICAL", "Restarting");
 1602:     my $execdir = $perlvar{'lonDaemons'};
 1603:     unlink("$execdir/logs/lonc.pid");
 1604:     exec("$execdir/lonc");
 1605: }
 1606: 
 1607: =pod
 1608: 
 1609: =head1 KillThemAll
 1610: 
 1611: Signal handler that kills all children by sending them a 
 1612: SIGHUP.  Responds to sigint and sigterm.
 1613: 
 1614: =cut
 1615: 
 1616: sub KillThemAll {
 1617:     Debug(2, "Kill them all!!");
 1618:     local($SIG{CHLD}) = 'IGNORE';      # Our children >will< die.
 1619:     foreach $pid (keys %ChildHash) {
 1620: 	my $serving = $ChildHash{$pid};
 1621: 	Debug(2, "Killing lonc for $serving pid = $pid");
 1622: 	ShowStatus("Killing lonc for $serving pid = $pid");
 1623: 	Log("CRITICAL", "Killing lonc for $serving pid = $pid");
 1624: 	kill 'QUIT' => $pid;
 1625: 	delete($ChildHash{$pid});
 1626:     }
 1627:     my $execdir = $perlvar{'lonDaemons'};
 1628:     unlink("$execdir/logs/lonc.pid");
 1629: 
 1630: }
 1631: 
 1632: =pod
 1633: 
 1634: =head1 Terminate
 1635:  
 1636: Terminate the system.
 1637: 
 1638: =cut
 1639: 
 1640: sub Terminate {
 1641:     KillThemAll;
 1642:     Log("CRITICAL","Master process exiting");
 1643:     exit 0;
 1644: 
 1645: }
 1646: =pod
 1647: 
 1648: =head1 Theory
 1649: 
 1650: The event class is used to build this as a single process with an
 1651: event driven model.  The following events are handled:
 1652: 
 1653: =item UNIX Socket connection Received
 1654: 
 1655: =item Request data arrives on UNIX data transfer socket.
 1656: 
 1657: =item lond connection becomes writable.
 1658: 
 1659: =item timer fires at 1 second intervals.
 1660: 
 1661: All sockets are run in non-blocking mode.  Timeouts managed by the timer
 1662: handler prevents hung connections.
 1663: 
 1664: Key data structures:
 1665: 
 1666: =item RequestQueue
 1667: 
 1668: A queue of requests received from UNIX sockets that are
 1669: waiting for a chance to be forwarded on a lond connection socket.
 1670: 
 1671: =item ActiveConnections
 1672: 
 1673: A hash of lond connections that have transactions in process that are
 1674: available to be timed out.
 1675: 
 1676: =item ActiveTransactions
 1677: 
 1678: A hash indexed by lond connections that contain the client reply
 1679: socket for each connection that has an active transaction on it.
 1680: 
 1681: =item IdleConnections
 1682: 
 1683: A hash of lond connections that have no work to do.  These connections
 1684: can be closed if they are idle for a long enough time.
 1685: 
 1686: =cut

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>