doc/gutshtml/SessionOne.html - view

File: [LON-CAPA] / doc / gutshtml / SessionOne.html
Revision 1.1: download - view: text, annotated - select for diffs
Fri Jun 28 20:30:29 2002 UTC (21 years, 10 months ago) by www
Branches: MAIN
CVS tags: version_0_99_3, version_0_99_2, version_0_99_1, version_0_99_0, version_0_6_2, version_0_6, version_0_5_1, version_0_5, version_0_4, stable_2002_july, conference_2003, STABLE, HEAD

HTML version of GUTS manual. Individual files will still need cleanup.

1: <html> 2: <head> 3: <meta name=Title 4: content="Session One: Intro/Demo, lonc/d, Replication and Load Balancing (Gerd)"> 5: <meta http-equiv=Content-Type content="text/html; charset=macintosh"> 6: <link rel=Edit-Time-Data href="Session%20One_files/editdata.mso"> 7: <title>Session One: Intro/Demo, lonc/d, Replication and Load Balancing (Gerd)</title> 8: <style> 14: </style> 15: </head> 16: <body bgcolor=#FFFFFF link=blue vlink=purple class="Normal" lang=EN-US> 17: <div class=Section1> 18: <h2>Session One: Intro/Demo, lonc/d, Replication and Load Balancing (Gerd)</h2> 19: <img width=432 height=555 20: src="Session%20One_files/image002.jpg" v:shapes="_x0000_i1025"> Fig. 1.1.1 22: Ð Overview of Network 23: <h3><a name="_Toc514840838"></a><a name="_Toc421867040">Overview</a></h3> 24: Physically, the Network consists of relatively inexpensive upper-PC-class 25: server machines which are linked through the commodity internet in a load-balancing, 26: dynamically content-replicating and failover-secure way. Fig. 1.1.1 27: shows an overview of this network. 28: All machines in the Network are connected with each other through two-way 29: persistent TCP/IP connections. Clients (B, F, 31: G and H 33: in Fig. 1.1.1) connect to the 34: servers via standard HTTP. There are two classes of servers, Library Servers 35: (A and E 37: in Fig. 1.1.1) and Access Servers 38: (C, D, I 40: and J in Fig. 1.1.1). Library Servers are used to store all personal records 42: of a set of users, and are responsible for their initial authentication when 43: a session is opened on any server in the Network. For Authors, Library Servers 44: also hosts their construction area and the authoritative copy of the current 45: and previous versions of every resource that was published by that author. 46: Library servers can be used as backups to host sessions when all access servers 47: in the Network are overloaded. Otherwise, for learners, access servers are 48: used to host the sessions. Library servers need to be strong on I/O, while 49: access servers can generally be cheaper hardware. The network is designed 50: so that the number of concurrent sessions can be increased over a wide range 51: by simply adding additional Access Servers before having to add additional 52: Library Servers. Preliminary tests showed that a Library Server could handle 53: up to 10 Access Servers fully parallel. 54: The Network is divided into so-called domains, which are logical boundaries 55: between participating institutions. These domains can be used to limit the 56: flow of personal user information across the network, set access privileges 57: and enforce royalty schemes. 58: <h3><a name="_Toc514840839"></a><a name="_Toc421867041">Example of Transactions</a></h3> 59: Fig. 1.1.1 also depicts examples 60: for several kinds of transactions conducted across the Network. 61: An instructor at client B modifies and publishes a resource on her Home Server A. Server A has a record of all server machines currently subscribed to this resource, 65: and replicates it to servers D and I. However, server 67: D is currently offline, so the update notification gets 69: buffered on A until D comes online again. Servers C and J are currently not 73: subscribed to this resource. 74: Learners F and G have open sessions on server I, and the new resource is immediately available to 77: them. 78: Learner H tries to connect to server 79: I for a new session, however, 80: the machine is not reachable, so he connects to another Access Server J 81: instead. This server currently does not have all necessary resources locally 82: present to host learner H, 83: but subscribes to them and replicates them as they are accessed by H. 85: Learner H solves a problem on server 86: J. Library Server E 87: is HÕs Home Server, so this information gets forwarded 89: to E, where the records of 90: H are updated. 92: <h3><a name="_Toc514840840"></a><a name="_Toc421867042">lonc/lond/lonnet</a></h3> 93: Fig. 1.1.2 elaborates on the details 94: of this network infrastructure. 95: Fig. 1.1.2A depicts three servers 96: (A, B and C, 98: Fig. 1.1.2A) and a client who 99: has a session on server C. 100: As C accesses different resources 101: in the system, different handlers, which are incorporated as modules into 102: the child processes of the web server software, process these requests. 103: Our current implementation uses mod_perl inside of the Apache web server software. As an 105: example, server C currently has four 106: active web server software child processes. The chain of handlers dealing 107: with a certain resource is determined by both the server content resource 108: area (see below) and the MIME type, which in turn is determined by the URL 109: extension. For most URL structures, both an authentication handler and a content 110: handler are registered. 111: Handlers use a common library lonnet 112: to interact with both locally present temporary session data and data across 113: the server network. For example, lonnet 114: provides routines for finding the home server of a user, finding the server 115: with the lowest loadavg, sending simple command-reply sequences, and sending 116: critical messages such as a homework completion, etc. For a non-critical message, 117: the routines reply with a simple Òconnection lostÓ if the message could not 118: be delivered. For critical messages, lonnet tries to re-establish connections, 121: re-send the command, etc. If no valid reply could be received, it answers 122: Òconnection deferredÓ and stores the message in buffer space to be sent at a later point in time. Also, failed critical messages 125: are logged. 126: The interface between lonnet 127: and the Network is established by a multiplexed UNIX domain socket, denoted 128: DS in Fig. 1.1.2A. The rationale behind 129: this rather involved architecture is that httpd processes (Apache children) 130: dynamically come and go on the timescale of minutes, based on workload and 131: number of processed requests. Over the lifetime of an httpd child, however, 132: it has to establish several hundred connections to several different servers 133: in the Network. 134: On the other hand, establishing a TCP/IP connection is resource consuming 135: for both ends of the line, and to optimize this connectivity between different 136: servers, connections in the Network are designed to be persistent on the timescale 137: of months, until either end is rebooted. This mechanism will be elaborated 138: on below. 139: Establishing a connection to a UNIX domain socket is far less resource consuming 140: than the establishing of a TCP/IP connection. lonc is a proxy daemon that forks off 142: a child for every server in the Network. . Which servers are members of the 143: Network is determined by a lookup table, which Fig. 1.1.2B is an example of. In order, the entries denote an 145: internal name for the server, the domain of the server, the type of the server, 146: the host name and the IP address. 147: The lonc parent process maintains 148: the population and listens for signals to restart or shutdown, as well as 149: USR1. Every child establishes a multiplexed 150: UNIX domain socket for its server and opens a TCP/IP connection to the lond 151: daemon (discussed below) on the remote machine, which it keeps alive. If the connection is interrupted, the child dies, whereupon 153: the parent makes several attempts to fork another child for that server. 154: When starting a new child (a new connection), first an init-sequence is carried 155: out, which includes receiving the information from the remote lond 156: which is needed to establish the 128-bit encryption key Ð the key is different 157: for every connection. Next, any buffered (delayed) messages for the server 158: are sent. 159: In normal operation, the child listens to the UNIX socket, forwards requests 160: to the TCP connection, gets the reply from lond, and sends it back to the UNIX socket. 162: Also, lonc takes care to the 163: encryption and decryption of messages. 164: lonc was build by putting 165: a non-forking multiplexed UNIX domain socket server into a framework that 166: forks a TCP/IP client for every remote lond. 168: lond is the remote end of 169: the TCP/IP connection and acts as a remote command processor. It receives 170: commands, executes them, and sends replies. In normal operation, a lonc 172: child is constantly connected to a dedicated lond 173: child on the remote server, and the same is true vice versa (two persistent 174: connections per server combination). 175: lond  listens 176: to a TCP/IP port (denoted P in Fig. 1.1.2A) and forks off enough 177: child processes to have one for each other server in the network plus two 178: spare children. The parent process maintains the population and listens for 179: signals to restart or shutdown. Client servers are authenticated by IP. 180: 182: <img width=432 height=492 183: src="Session%20One_files/image004.jpg" v:shapes="_x0000_i1026"> 184: Fig. 1.1.2A Ð Overview of Network Communication 186: When a new client server comes online, lond 188: sends a signal USR1 to lonc, whereupon lonc tries again to reestablish all lost 191: connections, even if it had given up on them before Ð a new client connecting 192: could mean that that machine came online again after an interruption. 193: The gray boxes in Fig. 1.1.2A denote the entities involved in an example transaction of the Network. 195: The Client is logged into server C, 196: while server B is her Home 197: Server. Server C can be an 198: Access Server or a Library Server, while server B is a Library Server. She submits a solution to a homework 200: problem, which is processed by the appropriate handler for the MIME type ÒproblemÓ. 201: Through lonnet, the 202: handler writes information about this transaction to the local session data. 203: To make a permanent log entry, lonnet 204: establishes a connection to the UNIX domain socket for server B. lonc 206: receives this command, encrypts it, and sends it through the persistent TCP/IP 207: connection to the TCP/IP port of the remote lond. 208: lond decrypts the command, 209: executes it by writing to the permanent user data files of the client, and 210: sends back a reply regarding the success of the operation. If the operation 211: was unsuccessful, or the connection would have broken down, lonc would write the command into a FIFO buffer stack to 213: be sent again later. lonc now 214: sends a reply regarding the overall success of the operation to lonnet via the UNIX domain port, which 216: is eventually received back by the handler. 217: <h3><a name="_Toc514840841"></a><a name="_Toc421867043">Scalability and Performance 218: Analysis</a></h3> 219: The scalability was tested in a test bed of servers between different physical 220: network segments, Fig. 1.1.2B shows the network configuration of this test. 222: <table border=1 cellspacing=0 cellpadding=0> 223: <tr> 224: <td width=443 valign=top class="Normal"> msul1:msu:library:zaphod.lite.msu.edu:35.8.63.51 225: msua1:msu:access:agrajag.lite.msu.edu:35.8.63.68 226: msul2:msu:library:frootmig.lite.msu.edu:35.8.63.69 227: msua2:msu:access:bistromath.lite.msu.edu:35.8.63.67 228: hubl14:hub:library:hubs128-pc-14.cl.msu.edu:35.8.116.34 229: hubl15:hub:library:hubs128-pc-15.cl.msu.edu:35.8.116.35 230: hubl16:hub:library:hubs128-pc-16.cl.msu.edu:35.8.116.36 231: huba20:hub:access:hubs128-pc-20.cl.msu.edu:35.8.116.40 232: huba21:hub:access:hubs128-pc-21.cl.msu.edu:35.8.116.41 233: huba22:hub:access:hubs128-pc-22.cl.msu.edu:35.8.116.42 234: huba23:hub:access:hubs128-pc-23.cl.msu.edu:35.8.116.43 235: hubl25:other:library:hubs128-pc-25.cl.msu.edu:35.8.116.45 236: huba27:other:access:hubs128-pc-27.cl.msu.edu:35.8.116.47</td> 237: </tr> 238: </table> 239: Fig. 1.1.2B Ð Example of Hosts Lookup Table /home/httpd/lonTabs/hosts.tab 242: In the first test, the simple ping command was used. The ping command is used to test connections and yields 246: the server short name as reply.  In this scenario, lonc was expected to be the speed-determining 248: step, since lond at the 250: remote end does not need any disk access to reply.  The graph Fig. 251: 1.1.2C shows number of seconds till completion versus number of processes issuing 253: 10,000 ping commands each against one Library Server (450 MHz Pentium II in 254: this test, single IDE HD). For the solid dots, the processes were concurrently 255: started on the same Access Server and the time was measured till the processes finished Ð all 257: processes finished at the same time. One Access Server (233 MHz Pentium II 258: in the test bed) can process about 150 pings per second, and as expected, 259: the total time grows linearly with the number of pings. 260: The gray dots were taken with up to seven 261: processes concurrently running on different machines and pinging the same server Ð the processes 263: ran fully concurrent, and each process finished as if the other ones were 264: not present (about 1000 pings per second). Execution was fully parallel. 265: In a second test, lond was 266: the speed-determining step Ð 10,000 put 267: commands each were issued first from up to seven concurrent processes on the 268: same machine, and then from up to seven processes on different machines. The 269: put command requires data to be written 271: to the permanent record of the user on the remote server. 272: In particular, one "put" 273: request meant that the process on the Access Server would connect to the UNIX 274: domain socket dedicated to the library server, lonc 275: would take the data from there, shuffle it through the persistent TCP connection, 276: lond on the remote library 277: server would take the data, write to disk (both to a dbm-file and to a flat-text 278: transaction history file), answer "ok", lonc would take that reply and send it 280: to the domain socket, the process would read it from there and close the domain-socket 281: connection. 282: <img width=220 height=190 283: src="Session%20One_files/image005.jpg" v:shapes="_x0000_i1027"> 284: Fig. 1.1.2C Ð Benchmark on Parallelism of Server-Server Communication 286: (no disk access) 287: The graph Fig. 1.1.2D shows the results. 288: Series 1 (solid black diamond) is the result of concurrent processes on the 289: same server Ð all of these are handled by the same server-dedicated lond-child, 290: which lets the total amount of time grow linearly. 291: <img width=432 height=311 292: src="Session%20One_files/image007.jpg" v:shapes="_x0000_i1028"> 293: Fig. 2D Ð Benchmark on Parallelism of Server-Server Communication 295: (with disk access as in Fig. 2A) 296: Series 2 through 8 were obtained from running the processes on different 297: Access Servers against one Library Server, each series goes with one server. 298: In this experiment, the processes did not finish at the same time, which most 299: likely is due to disk-caching on the Library Server Ð lond-children whose datafile was (partly) 301: in disk cache finished earlier. With seven processes from seven different 302: servers, the operation took 255 seconds till the last process was finished 303: for 70,000 put commands (270 304: per second) Ð versus 530 seconds if the processes ran on the same server (130 305: per second). 306: <h3><a name="_Toc514840842"></a><a name="_Toc421867044">Dynamic Resource Replication</a></h3> 307: Since resources are assembled into higher order resources simply by reference, 308: in principle it would be sufficient to retrieve them from the respective Home 309: Servers of the authors. However, there are several problems with this simple 310: approach: since the resource assembly mechanism is designed to facilitate 311: content assembly from a large number of widely distributed sources, individual 312: sessions would depend on a large number of machines and network connections 313: to be available, thus be rather fragile. Also, frequently accessed resources 314: could potentially drive individual machines in the network into overload situations. 315: Finally, since most resources depend on content handlers on the Access Servers 316: to be served to a client within the session context, the raw source would 317: first have to be transferred across the Network from the respective Library 318: Server to the Access Server, processed there, and then transferred on to the 319: client. 320: To enable resource assembly in a reliable and scalable way, a dynamic resource 321: replication scheme was developed. Fig. 1.1.3 shows the details of this mechanism. 323: Anytime a resource out of the resource space is requested, a handler routine 324: is called which in turn calls the replication routine (Fig. 1.1.3A). 325: As a first step, this routines determines whether or not the resource is currently 326: in replication transfer (Fig. 1.1.3A, 327: Step D1a). During replication transfer, the incoming data is 329: stored in a temporary file, and Step D1a checks for the presence of that file. If transfer of a resource is actively 331: going on, the controlling handler receives an error message, waits for a few 332: seconds, and then calls the replication routine again. If the resource is 333: still in transfer, the client will receive the message ÒService currently 334: not availableÓ. 335: In the next step (Fig. 1.1.3A, Step D1b), the replication routine checks if the URL is locally 337: present. If it is, the replication routine returns OK to the controlling handler, 338: which in turn passes the request on to the next handler in the chain. 339: If the resource is not locally present, the Home Server of the resource author 340: (as extracted from the URL) is determined (Fig. 1.1.3A, Step D2). 341: This is done by contacting all library servers in the authorÕs domain (as 342: determined from the lookup table, see Fig. 1.1.2B). 343: In Step D2b a query is sent 344: to the remote server whether or not it is the Home Server of the author (in 345: our current implementation, an additional cache is used to store already identified 346: Home Servers (not shown in the figure)). In Step D2c, the remote server answers the query with True or 348: False. If the Home Server was found, the routine continues, otherwise it contacts 349: the next server (Step D2a). 350: If no server could be found, a ÒFile not FoundÓ error message is issued. In 351: our current implementation, in this step the Home Server is also written into 352: a cache for faster access if resources by the same author are needed again 353: (not shown in the figure). 354: 356: <img width=432 height=581 357: src="Session%20One_files/image009.jpg" v:shapes="_x0000_i1029"> 358: Fig. 1.1.3A Ð Dynamic Resource Replication, subscription 360: 362: <img width=432 height=523 363: src="Session%20One_files/image011.jpg" v:shapes="_x0000_i1030"> 364: Fig. 1.1.3B Ð Dynamic Resource Replication, modification 366: In Step D3a, the routine sends a 367: subscribe command for the URL to the Home Server of the author. The Home Server 368: first determines if the resource is present, and if the access privileges 369: allow it to be copied to the requesting server (Fig. 1.1.3A, Step 370: D3b). If this is true, the requesting 371: server is added to the list of subscribed servers for that resource (Step 372: D3c). The Home Server will reply with 373: either OK or an error message, which is determined in Step D4. 374: If the remote resource was not present, the error message ÒFile not FoundÓ 375: will be passed on to the client, if the access was not allowed, the error 376: message ÒAccess DeniedÓ is passed on. If the operation succeeded, the requesting 377: server sends an HTTP request for the resource out of the /raw 378: server content resource area of the Home Server. 379: The Home Server will then check if the requesting server is part of the network, 380: and if it is subscribed to the resource (Step D5b). If it is, it will send the resource via HTTP to 382: the requesting server without any content handlers processing it (Step 383: D5c). The requesting server will store 384: the incoming data in a temporary data file (Step D5a) Ð this is the file that Step D1a checks for. If the transfer could not complete, and 387: appropriate error message is sent to the client (Step D6). Otherwise, the transferred temporary file is renamed 389: as the actual resource, and the replication routine returns OK to the controlling 390: handler (Step D7). 391: Fig. 1.1.3B  depicts the process 392: of modifying a resource. When an author publishes a new version of a resource, 393: the Home Server will contact every server currently subscribed to the resource 394: (Fig. 1.1.3B, Step U1), as 395: determined from the list of subscribed servers for the resource generated 396: in Fig. 1.1. 3A, Step D3c. 397: The subscribing servers will receive and acknowledge the update message (Step 398: U1c). The update mechanism finishes when the last subscribed 400: server has been contacted (messages to unreachable servers are buffered). 401: Each subscribing server will check if the resource in question had been accessed 402: recently, that is, within a configurable amount of time (Step U2). 403: 404: If the resource had not been accessed recently, the local copy of the resource 405: is deleted (Step U3a) and an unsubscribe 406: command is sent to the Home Server (Step U3b). The Home Server will check if the server had indeed 408: originally subscribed to the resource (Step U3c) and then delete the server from the list of subscribed 410: servers for the resource (Step U3d). 412: If the resource had been accessed recently, the modified resource will be 413: copied over using the same mechanism as in Step D5a through D7 of Fig. 1.1.3A (Fig. 416: 1.1.3B, Steps U4a through U6). 419: Load Balancing 420: lond provides a function to 421: query the serverÕs current loadavg. As a configuration parameter, one can determine 423: the value of loadavg, which 424: is to be considered 100%, for example, 2.00. 425: Access servers can have a list of spare access servers, /home/httpd/lonTabs/spares.tab, 427: to offload sessions depending on own workload. This check happens is done 428: by the login handler. It re-directs the login information and session to the 429: least busy spare server if itself is overloaded. An additional round-robin 430: IP scheme possible. See Fig. 1.1.4 431: for an example of a load-balancing scheme. 432: <img width=241 height=139 433: src="Session%20One_files/image013.jpg" v:shapes="_x0000_i1031"> 434: Fig. 1.1.4 Ð Example 436: of Load Balancing 438: 439: </div> 440: 442: <div class=Section2> </div> 443: </body> 444: </html>