doc/gutshtml/SessionOne.html - view

File: [LON-CAPA] / doc / gutshtml / SessionOne.html
Revision 1.2: download - view: text, annotated - select for diffs
Tue Jul 22 14:47:00 2003 UTC (20 years, 9 months ago) by bowersj2
Branches: MAIN
CVS tags: version_2_9_X, version_2_9_99_0, version_2_9_1, version_2_9_0, version_2_8_X, version_2_8_99_1, version_2_8_99_0, version_2_8_2, version_2_8_1, version_2_8_0, version_2_7_X, version_2_7_99_1, version_2_7_99_0, version_2_7_1, version_2_7_0, version_2_6_X, version_2_6_99_1, version_2_6_99_0, version_2_6_3, version_2_6_2, version_2_6_1, version_2_6_0, version_2_5_X, version_2_5_99_1, version_2_5_99_0, version_2_5_2, version_2_5_1, version_2_5_0, version_2_4_X, version_2_4_99_0, version_2_4_2, version_2_4_1, version_2_4_0, version_2_3_X, version_2_3_99_0, version_2_3_2, version_2_3_1, version_2_3_0, version_2_2_X, version_2_2_99_1, version_2_2_99_0, version_2_2_2, version_2_2_1, version_2_2_0, version_2_1_X, version_2_1_99_3, version_2_1_99_2, version_2_1_99_1, version_2_1_99_0, version_2_1_3, version_2_1_2, version_2_1_1, version_2_1_0, version_2_12_X, version_2_11_X, version_2_11_4_uiuc, version_2_11_4_msu, version_2_11_4, version_2_11_3_uiuc, version_2_11_3_msu, version_2_11_3, version_2_11_2_uiuc, version_2_11_2_msu, version_2_11_2_educog, version_2_11_2, version_2_11_1, version_2_11_0_RC3, version_2_11_0_RC2, version_2_11_0_RC1, version_2_11_0, version_2_10_X, version_2_10_1, version_2_10_0_RC2, version_2_10_0_RC1, version_2_10_0, version_2_0_X, version_2_0_99_1, version_2_0_2, version_2_0_1, version_2_0_0, version_1_99_3, version_1_99_2, version_1_99_1_tmcc, version_1_99_1, version_1_99_0_tmcc, version_1_99_0, version_1_3_X, version_1_3_3, version_1_3_2, version_1_3_1, version_1_3_0, version_1_2_X, version_1_2_99_1, version_1_2_99_0, version_1_2_1, version_1_2_0, version_1_1_X, version_1_1_99_5, version_1_1_99_4, version_1_1_99_3, version_1_1_99_2, version_1_1_99_1, version_1_1_99_0, version_1_1_3, version_1_1_2, version_1_1_1, version_1_1_0, version_1_0_99_3, version_1_0_99_2, version_1_0_99_1, version_1_0_99, version_1_0_3, version_1_0_2, version_1_0_1, version_1_0_0, version_0_99_5, version_0_99_4, loncapaMITrelate_1, language_hyphenation_merge, language_hyphenation, bz6209-base, bz6209, HEAD, GCI_3, GCI_2, GCI_1, BZ4492-merge, BZ4492-feature_horizontal_radioresponse, BZ4492-feature_Support_horizontal_radioresponse, BZ4492-Support_horizontal_radioresponse

Convert GUTs HTML to PROPER line endings.

1: <html> 2: 3: <head> 4: 5: <meta name=Title 6: 7: content="Session One: Intro/Demo, lonc/d, Replication and Load Balancing (Gerd)"> 8: 9: <meta http-equiv=Content-Type content="text/html; charset=macintosh"> 10: 11: <link rel=Edit-Time-Data href="Session%20One_files/editdata.mso"> 12: 13: <title>Session One: Intro/Demo, lonc/d, Replication and Load Balancing (Gerd)</title> 14: 15: <style> 26: 27: </style> 28: 29: </head> 30: 31: <body bgcolor=#FFFFFF link=blue vlink=purple class="Normal" lang=EN-US> 32: 33: <div class=Section1> 34: 35: <h2>Session One: Intro/Demo, lonc/d, Replication and Load Balancing (Gerd)</h2> 36: 37: <img width=432 height=555 38: 39: src="Session%20One_files/image002.jpg" v:shapes="_x0000_i1025"> Fig. 1.1.1 42: 43: � Overview of Network 44: 45: <h3><a name="_Toc514840838"></a><a name="_Toc421867040">Overview</a></h3> 46: 47: Physically, the Network consists of relatively inexpensive upper-PC-class 48: 49: server machines which are linked through the commodity internet in a load-balancing, 50: 51: dynamically content-replicating and failover-secure way. Fig. 1.1.1 52: 53: shows an overview of this network. 54: 55: All machines in the Network are connected with each other through two-way 56: 57: persistent TCP/IP connections. Clients (B, F, 60: 61: G and H 64: 65: in Fig. 1.1.1) connect to the 66: 67: servers via standard HTTP. There are two classes of servers, Library Servers 68: 69: (A and E 72: 73: in Fig. 1.1.1) and Access Servers 74: 75: (C, D, I 78: 79: and J in Fig. 1.1.1). Library Servers are used to store all personal records 82: 83: of a set of users, and are responsible for their initial authentication when 84: 85: a session is opened on any server in the Network. For Authors, Library Servers 86: 87: also hosts their construction area and the authoritative copy of the current 88: 89: and previous versions of every resource that was published by that author. 90: 91: Library servers can be used as backups to host sessions when all access servers 92: 93: in the Network are overloaded. Otherwise, for learners, access servers are 94: 95: used to host the sessions. Library servers need to be strong on I/O, while 96: 97: access servers can generally be cheaper hardware. The network is designed 98: 99: so that the number of concurrent sessions can be increased over a wide range 100: 101: by simply adding additional Access Servers before having to add additional 102: 103: Library Servers. Preliminary tests showed that a Library Server could handle 104: 105: up to 10 Access Servers fully parallel. 106: 107: The Network is divided into so-called domains, which are logical boundaries 108: 109: between participating institutions. These domains can be used to limit the 110: 111: flow of personal user information across the network, set access privileges 112: 113: and enforce royalty schemes. 114: 115: <h3><a name="_Toc514840839"></a><a name="_Toc421867041">Example of Transactions</a></h3> 116: 117: Fig. 1.1.1 also depicts examples 118: 119: for several kinds of transactions conducted across the Network. 120: 121: An instructor at client B modifies and publishes a resource on her Home Server A. Server A has a record of all server machines currently subscribed to this resource, 128: 129: and replicates it to servers D and I. However, server 132: 133: D is currently offline, so the update notification gets 136: 137: buffered on A until D comes online again. Servers C and J are currently not 144: 145: subscribed to this resource. 146: 147: Learners F and G have open sessions on server I, and the new resource is immediately available to 152: 153: them. 154: 155: Learner H tries to connect to server 156: 157: I for a new session, however, 158: 159: the machine is not reachable, so he connects to another Access Server J 160: 161: instead. This server currently does not have all necessary resources locally 162: 163: present to host learner H, 164: 165: but subscribes to them and replicates them as they are accessed by H. 168: 169: Learner H solves a problem on server 170: 171: J. Library Server E 172: 173: is H�s Home Server, so this information gets forwarded 176: 177: to E, where the records of 178: 179: H are updated. 182: 183: <h3><a name="_Toc514840840"></a><a name="_Toc421867042">lonc/lond/lonnet</a></h3> 184: 185: Fig. 1.1.2 elaborates on the details 186: 187: of this network infrastructure. 188: 189: Fig. 1.1.2A depicts three servers 190: 191: (A, B and C, 194: 195: Fig. 1.1.2A) and a client who 196: 197: has a session on server C. 198: 199: As C accesses different resources 200: 201: in the system, different handlers, which are incorporated as modules into 202: 203: the child processes of the web server software, process these requests. 204: 205: Our current implementation uses mod_perl inside of the Apache web server software. As an 208: 209: example, server C currently has four 210: 211: active web server software child processes. The chain of handlers dealing 212: 213: with a certain resource is determined by both the server content resource 214: 215: area (see below) and the MIME type, which in turn is determined by the URL 216: 217: extension. For most URL structures, both an authentication handler and a content 218: 219: handler are registered. 220: 221: Handlers use a common library lonnet 222: 223: to interact with both locally present temporary session data and data across 224: 225: the server network. For example, lonnet 226: 227: provides routines for finding the home server of a user, finding the server 228: 229: with the lowest loadavg, sending simple command-reply sequences, and sending 230: 231: critical messages such as a homework completion, etc. For a non-critical message, 232: 233: the routines reply with a simple �connection lost� if the message could not 234: 235: be delivered. For critical messages, lonnet tries to re-establish connections, 240: 241: re-send the command, etc. If no valid reply could be received, it answers 242: 243: �connection deferred� and stores the message in buffer space to be sent at a later point in time. Also, failed critical messages 248: 249: are logged. 250: 251: The interface between lonnet 252: 253: and the Network is established by a multiplexed UNIX domain socket, denoted 254: 255: DS in Fig. 1.1.2A. The rationale behind 256: 257: this rather involved architecture is that httpd processes (Apache children) 258: 259: dynamically come and go on the timescale of minutes, based on workload and 260: 261: number of processed requests. Over the lifetime of an httpd child, however, 262: 263: it has to establish several hundred connections to several different servers 264: 265: in the Network. 266: 267: On the other hand, establishing a TCP/IP connection is resource consuming 268: 269: for both ends of the line, and to optimize this connectivity between different 270: 271: servers, connections in the Network are designed to be persistent on the timescale 272: 273: of months, until either end is rebooted. This mechanism will be elaborated 274: 275: on below. 276: 277: Establishing a connection to a UNIX domain socket is far less resource consuming 278: 279: than the establishing of a TCP/IP connection. lonc is a proxy daemon that forks off 282: 283: a child for every server in the Network. . Which servers are members of the 284: 285: Network is determined by a lookup table, which Fig. 1.1.2B is an example of. In order, the entries denote an 288: 289: internal name for the server, the domain of the server, the type of the server, 290: 291: the host name and the IP address. 292: 293: The lonc parent process maintains 294: 295: the population and listens for signals to restart or shutdown, as well as 296: 297: USR1. Every child establishes a multiplexed 298: 299: UNIX domain socket for its server and opens a TCP/IP connection to the lond 300: 301: daemon (discussed below) on the remote machine, which it keeps alive. If the connection is interrupted, the child dies, whereupon 304: 305: the parent makes several attempts to fork another child for that server. 306: 307: When starting a new child (a new connection), first an init-sequence is carried 308: 309: out, which includes receiving the information from the remote lond 310: 311: which is needed to establish the 128-bit encryption key � the key is different 312: 313: for every connection. Next, any buffered (delayed) messages for the server 314: 315: are sent. 316: 317: In normal operation, the child listens to the UNIX socket, forwards requests 318: 319: to the TCP connection, gets the reply from lond, and sends it back to the UNIX socket. 322: 323: Also, lonc takes care to the 324: 325: encryption and decryption of messages. 326: 327: lonc was build by putting 328: 329: a non-forking multiplexed UNIX domain socket server into a framework that 330: 331: forks a TCP/IP client for every remote lond. 334: 335: lond is the remote end of 336: 337: the TCP/IP connection and acts as a remote command processor. It receives 338: 339: commands, executes them, and sends replies. In normal operation, a lonc 342: 343: child is constantly connected to a dedicated lond 344: 345: child on the remote server, and the same is true vice versa (two persistent 346: 347: connections per server combination). 348: 349: lond  listens 350: 351: to a TCP/IP port (denoted P in Fig. 1.1.2A) and forks off enough 352: 353: child processes to have one for each other server in the network plus two 354: 355: spare children. The parent process maintains the population and listens for 356: 357: signals to restart or shutdown. Client servers are authenticated by IP. 358: 359: 362: 363: <img width=432 height=492 364: 365: src="Session%20One_files/image004.jpg" v:shapes="_x0000_i1026"> 366: 367: Fig. 1.1.2A � Overview of Network Communication 370: 371: When a new client server comes online, lond 374: 375: sends a signal USR1 to lonc, whereupon lonc tries again to reestablish all lost 380: 381: connections, even if it had given up on them before � a new client connecting 382: 383: could mean that that machine came online again after an interruption. 384: 385: The gray boxes in Fig. 1.1.2A denote the entities involved in an example transaction of the Network. 388: 389: The Client is logged into server C, 390: 391: while server B is her Home 392: 393: Server. Server C can be an 394: 395: Access Server or a Library Server, while server B is a Library Server. She submits a solution to a homework 398: 399: problem, which is processed by the appropriate handler for the MIME type �problem�. 400: 401: Through lonnet, the 402: 403: handler writes information about this transaction to the local session data. 404: 405: To make a permanent log entry, lonnet 406: 407: establishes a connection to the UNIX domain socket for server B. lonc 410: 411: receives this command, encrypts it, and sends it through the persistent TCP/IP 412: 413: connection to the TCP/IP port of the remote lond. 414: 415: lond decrypts the command, 416: 417: executes it by writing to the permanent user data files of the client, and 418: 419: sends back a reply regarding the success of the operation. If the operation 420: 421: was unsuccessful, or the connection would have broken down, lonc would write the command into a FIFO buffer stack to 424: 425: be sent again later. lonc now 426: 427: sends a reply regarding the overall success of the operation to lonnet via the UNIX domain port, which 430: 431: is eventually received back by the handler. 432: 433: <h3><a name="_Toc514840841"></a><a name="_Toc421867043">Scalability and Performance 434: 435: Analysis</a></h3> 436: 437: The scalability was tested in a test bed of servers between different physical 438: 439: network segments, Fig. 1.1.2B shows the network configuration of this test. 442: 443: <table border=1 cellspacing=0 cellpadding=0> 444: 445: <tr> 446: 447: <td width=443 valign=top class="Normal"> msul1:msu:library:zaphod.lite.msu.edu:35.8.63.51 448: 449: msua1:msu:access:agrajag.lite.msu.edu:35.8.63.68 450: 451: msul2:msu:library:frootmig.lite.msu.edu:35.8.63.69 452: 453: msua2:msu:access:bistromath.lite.msu.edu:35.8.63.67 454: 455: hubl14:hub:library:hubs128-pc-14.cl.msu.edu:35.8.116.34 456: 457: hubl15:hub:library:hubs128-pc-15.cl.msu.edu:35.8.116.35 458: 459: hubl16:hub:library:hubs128-pc-16.cl.msu.edu:35.8.116.36 460: 461: huba20:hub:access:hubs128-pc-20.cl.msu.edu:35.8.116.40 462: 463: huba21:hub:access:hubs128-pc-21.cl.msu.edu:35.8.116.41 464: 465: huba22:hub:access:hubs128-pc-22.cl.msu.edu:35.8.116.42 466: 467: huba23:hub:access:hubs128-pc-23.cl.msu.edu:35.8.116.43 468: 469: hubl25:other:library:hubs128-pc-25.cl.msu.edu:35.8.116.45 470: 471: huba27:other:access:hubs128-pc-27.cl.msu.edu:35.8.116.47</td> 472: 473: </tr> 474: 475: </table> 476: 477: Fig. 1.1.2B � Example of Hosts Lookup Table /home/httpd/lonTabs/hosts.tab 482: 483: In the first test, the simple ping command was used. The ping command is used to test connections and yields 490: 491: the server short name as reply.  In this scenario, lonc was expected to be the speed-determining 494: 495: step, since lond at the 498: 499: remote end does not need any disk access to reply.  The graph Fig. 500: 501: 1.1.2C shows number of seconds till completion versus number of processes issuing 504: 505: 10,000 ping commands each against one Library Server (450 MHz Pentium II in 506: 507: this test, single IDE HD). For the solid dots, the processes were concurrently 508: 509: started on the same Access Server and the time was measured till the processes finished � all 512: 513: processes finished at the same time. One Access Server (233 MHz Pentium II 514: 515: in the test bed) can process about 150 pings per second, and as expected, 516: 517: the total time grows linearly with the number of pings. 518: 519: The gray dots were taken with up to seven 520: 521: processes concurrently running on different machines and pinging the same server � the processes 524: 525: ran fully concurrent, and each process finished as if the other ones were 526: 527: not present (about 1000 pings per second). Execution was fully parallel. 528: 529: In a second test, lond was 530: 531: the speed-determining step � 10,000 put 532: 533: commands each were issued first from up to seven concurrent processes on the 534: 535: same machine, and then from up to seven processes on different machines. The 536: 537: put command requires data to be written 540: 541: to the permanent record of the user on the remote server. 542: 543: In particular, one "put" 544: 545: request meant that the process on the Access Server would connect to the UNIX 546: 547: domain socket dedicated to the library server, lonc 548: 549: would take the data from there, shuffle it through the persistent TCP connection, 550: 551: lond on the remote library 552: 553: server would take the data, write to disk (both to a dbm-file and to a flat-text 554: 555: transaction history file), answer "ok", lonc would take that reply and send it 558: 559: to the domain socket, the process would read it from there and close the domain-socket 560: 561: connection. 562: 563: <img width=220 height=190 564: 565: src="Session%20One_files/image005.jpg" v:shapes="_x0000_i1027"> 566: 567: Fig. 1.1.2C � Benchmark on Parallelism of Server-Server Communication 570: 571: (no disk access) 572: 573: The graph Fig. 1.1.2D shows the results. 574: 575: Series 1 (solid black diamond) is the result of concurrent processes on the 576: 577: same server � all of these are handled by the same server-dedicated lond-child, 578: 579: which lets the total amount of time grow linearly. 580: 581: <img width=432 height=311 582: 583: src="Session%20One_files/image007.jpg" v:shapes="_x0000_i1028"> 584: 585: Fig. 2D � Benchmark on Parallelism of Server-Server Communication 588: 589: (with disk access as in Fig. 2A) 590: 591: Series 2 through 8 were obtained from running the processes on different 592: 593: Access Servers against one Library Server, each series goes with one server. 594: 595: In this experiment, the processes did not finish at the same time, which most 596: 597: likely is due to disk-caching on the Library Server � lond-children whose datafile was (partly) 600: 601: in disk cache finished earlier. With seven processes from seven different 602: 603: servers, the operation took 255 seconds till the last process was finished 604: 605: for 70,000 put commands (270 606: 607: per second) � versus 530 seconds if the processes ran on the same server (130 608: 609: per second). 610: 611: <h3><a name="_Toc514840842"></a><a name="_Toc421867044">Dynamic Resource Replication</a></h3> 612: 613: Since resources are assembled into higher order resources simply by reference, 614: 615: in principle it would be sufficient to retrieve them from the respective Home 616: 617: Servers of the authors. However, there are several problems with this simple 618: 619: approach: since the resource assembly mechanism is designed to facilitate 620: 621: content assembly from a large number of widely distributed sources, individual 622: 623: sessions would depend on a large number of machines and network connections 624: 625: to be available, thus be rather fragile. Also, frequently accessed resources 626: 627: could potentially drive individual machines in the network into overload situations. 628: 629: Finally, since most resources depend on content handlers on the Access Servers 630: 631: to be served to a client within the session context, the raw source would 632: 633: first have to be transferred across the Network from the respective Library 634: 635: Server to the Access Server, processed there, and then transferred on to the 636: 637: client. 638: 639: To enable resource assembly in a reliable and scalable way, a dynamic resource 640: 641: replication scheme was developed. Fig. 1.1.3 shows the details of this mechanism. 644: 645: Anytime a resource out of the resource space is requested, a handler routine 646: 647: is called which in turn calls the replication routine (Fig. 1.1.3A). 648: 649: As a first step, this routines determines whether or not the resource is currently 650: 651: in replication transfer (Fig. 1.1.3A, 652: 653: Step D1a). During replication transfer, the incoming data is 656: 657: stored in a temporary file, and Step D1a checks for the presence of that file. If transfer of a resource is actively 660: 661: going on, the controlling handler receives an error message, waits for a few 662: 663: seconds, and then calls the replication routine again. If the resource is 664: 665: still in transfer, the client will receive the message �Service currently 666: 667: not available�. 668: 669: In the next step (Fig. 1.1.3A, Step D1b), the replication routine checks if the URL is locally 672: 673: present. If it is, the replication routine returns OK to the controlling handler, 674: 675: which in turn passes the request on to the next handler in the chain. 676: 677: If the resource is not locally present, the Home Server of the resource author 678: 679: (as extracted from the URL) is determined (Fig. 1.1.3A, Step D2). 680: 681: This is done by contacting all library servers in the author�s domain (as 682: 683: determined from the lookup table, see Fig. 1.1.2B). 684: 685: In Step D2b a query is sent 686: 687: to the remote server whether or not it is the Home Server of the author (in 688: 689: our current implementation, an additional cache is used to store already identified 690: 691: Home Servers (not shown in the figure)). In Step D2c, the remote server answers the query with True or 694: 695: False. If the Home Server was found, the routine continues, otherwise it contacts 696: 697: the next server (Step D2a). 698: 699: If no server could be found, a �File not Found� error message is issued. In 700: 701: our current implementation, in this step the Home Server is also written into 702: 703: a cache for faster access if resources by the same author are needed again 704: 705: (not shown in the figure). 706: 707: 710: 711: <img width=432 height=581 712: 713: src="Session%20One_files/image009.jpg" v:shapes="_x0000_i1029"> 714: 715: Fig. 1.1.3A � Dynamic Resource Replication, subscription 718: 719: 722: 723: <img width=432 height=523 724: 725: src="Session%20One_files/image011.jpg" v:shapes="_x0000_i1030"> 726: 727: Fig. 1.1.3B � Dynamic Resource Replication, modification 730: 731: In Step D3a, the routine sends a 732: 733: subscribe command for the URL to the Home Server of the author. The Home Server 734: 735: first determines if the resource is present, and if the access privileges 736: 737: allow it to be copied to the requesting server (Fig. 1.1.3A, Step 738: 739: D3b). If this is true, the requesting 740: 741: server is added to the list of subscribed servers for that resource (Step 742: 743: D3c). The Home Server will reply with 744: 745: either OK or an error message, which is determined in Step D4. 746: 747: If the remote resource was not present, the error message �File not Found� 748: 749: will be passed on to the client, if the access was not allowed, the error 750: 751: message �Access Denied� is passed on. If the operation succeeded, the requesting 752: 753: server sends an HTTP request for the resource out of the /raw 754: 755: server content resource area of the Home Server. 756: 757: The Home Server will then check if the requesting server is part of the network, 758: 759: and if it is subscribed to the resource (Step D5b). If it is, it will send the resource via HTTP to 762: 763: the requesting server without any content handlers processing it (Step 764: 765: D5c). The requesting server will store 766: 767: the incoming data in a temporary data file (Step D5a) � this is the file that Step D1a checks for. If the transfer could not complete, and 772: 773: appropriate error message is sent to the client (Step D6). Otherwise, the transferred temporary file is renamed 776: 777: as the actual resource, and the replication routine returns OK to the controlling 778: 779: handler (Step D7). 780: 781: Fig. 1.1.3B  depicts the process 782: 783: of modifying a resource. When an author publishes a new version of a resource, 784: 785: the Home Server will contact every server currently subscribed to the resource 786: 787: (Fig. 1.1.3B, Step U1), as 788: 789: determined from the list of subscribed servers for the resource generated 790: 791: in Fig. 1.1. 3A, Step D3c. 792: 793: The subscribing servers will receive and acknowledge the update message (Step 794: 795: U1c). The update mechanism finishes when the last subscribed 798: 799: server has been contacted (messages to unreachable servers are buffered). 800: 801: Each subscribing server will check if the resource in question had been accessed 802: 803: recently, that is, within a configurable amount of time (Step U2). 804: 805: 806: 807: If the resource had not been accessed recently, the local copy of the resource 808: 809: is deleted (Step U3a) and an unsubscribe 810: 811: command is sent to the Home Server (Step U3b). The Home Server will check if the server had indeed 814: 815: originally subscribed to the resource (Step U3c) and then delete the server from the list of subscribed 818: 819: servers for the resource (Step U3d). 822: 823: If the resource had been accessed recently, the modified resource will be 824: 825: copied over using the same mechanism as in Step D5a through D7 of Fig. 1.1.3A (Fig. 830: 831: 1.1.3B, Steps U4a through U6). 836: 837: Load Balancing 838: 839: lond provides a function to 840: 841: query the server�s current loadavg. As a configuration parameter, one can determine 844: 845: the value of loadavg, which 846: 847: is to be considered 100%, for example, 2.00. 848: 849: Access servers can have a list of spare access servers, /home/httpd/lonTabs/spares.tab, 852: 853: to offload sessions depending on own workload. This check happens is done 854: 855: by the login handler. It re-directs the login information and session to the 856: 857: least busy spare server if itself is overloaded. An additional round-robin 858: 859: IP scheme possible. See Fig. 1.1.4 860: 861: for an example of a load-balancing scheme. 862: 863: <img width=241 height=139 864: 865: src="Session%20One_files/image013.jpg" v:shapes="_x0000_i1031"> 866: 867: Fig. 1.1.4 � Example 870: 871: of Load Balancing 874: 875: 876: 877: </div> 878: 879: 882: 883: <div class=Section2> </div> 884: 885: </body> 886: 887: </html> 888: