Annotation of doc/packaging/index.html, revision 1.1
1.1 ! harris41 1: <HTML>
! 2: <HEAD>
! 3: <TITLE>LON-CAPA Packaging</TITLE>
! 4: </HEAD>
! 5: <BODY BGCOLOR=white>
! 6: <H1>LON-CAPA Packaging and Filesystem Management</H1>
! 7: <B>Functional Model Description</B> <I>(Scott Harrison, April 2000)</I>
! 8: <P>
! 9: <OL>
! 10: <LI>Synopsis
! 11: <LI>Summary of Data Store Objects <I>(e.g. CVS, RPM, LCIC)</I>
! 12: <LI>Summary of Processes <I>(e.g. making a CD image, upgrading a system, altering package composition)</I>
! 13: <LI>Summary of Actor Objects <I>(e.g. cron jobs, programmers, head packaging
! 14: developer)</I>
! 15: <LI>Functional Model Diagram
! 16: <LI>Description of the Functional Model Diagram
! 17: <OL>
! 18: <LI>Data Store Objects
! 19: <LI>Processes
! 20: <LI>Actor Objects
! 21: </OL>
! 22: <LI>Frequently Asked Questions
! 23: </OL>
! 24: <P>
! 25: <H1>1. Synopsis</H1>
! 26: Up to this point, dozens of scripts have been written
! 27: to automate various elements of the LON-CAPA <U>installation</U>, <U>upgrade</U>, and <U>software development</U> process. A number of significant issues
! 28: relate to this effort. Security on the system is largely a matter of <B>limiting</B>
! 29: unnecessary network services, <B>monitoring</B> the filesystem of LON-CAPA
! 30: servers to make full account for the presence of every file on the system,
! 31: and securely <B>communicating potential problems</B> to system administrators of a given
! 32: LON-CAPA server machine.
! 33: <P>
! 34: This document is the functional model description which describes the computations
! 35: and their expected results necessary to manage packaging and filesystem management
! 36: for LON-CAPA both now and in the future in a manner that is <B>synchronous</B>
! 37: with software development, installations, and upgrades. Promulgation of coding
! 38: changes, version control, and maintaining a handle on security and network optimization
! 39: issues (such as introducing SAMBA and NetaTalk into the network architecture) have
! 40: been some of the reasons for introducing a more strongly defined characterization
! 41: of all of these processes involved with LON-CAPA software packaging and file system
! 42: management.
! 43: <P>
! 44: <H1>2. Summary of Data Store Objects</H1>
! 45: <UL>
! 46: <LI><B>CVS</B>
! 47: <BR>Concurrent Versions System. A central file repository is set up for LON-CAPA software development.
! 48: <LI><B>RPM</B>
! 49: <BR>Red Hat Package Manager. A system to manage software applications (packages). This system tracks package
! 50: interdependencies, configuration files, and file alterations. The RPM packaging system can (and probably should) reside on a different
! 51: computer than the central file repository computer. The RPM packaging system should probably be on a computer
! 52: accessible to the
! 53: <LI><B>LCIC</B>
! 54: <BR>LON-CAPA Integrity Check. A script-based system which analyzes a computer to determine the RPM composition
! 55: (missing or contrastingly extraneous RPMs), extraneous files which do not belong to any RPM, and RPM file
! 56: alterations.
! 57: </UL>
! 58: <P>
! 59: <H1>3. Summary of Processes</H1>
! 60: <UL>
! 61: <LI><B>Non-automated processes</B>
! 62: <UL>
! 63: <LI><B>implements change</B>
! 64: <BR>Programmer creates a successful change to the LON-CAPA software. To carry out this step, the programmer
! 65: must enter in bug-free code to the CVS file repository. The programmer must also inform the packaging developer
! 66: as to where and how the files need to be placed and configured on LON-CAPA computer systems.
! 67: <LI><B>alter package composition</B>
! 68: <BR>The packaging developer may choose to add/remove different sets of entire RPM packages from LON-CAPA computer
! 69: systems. The packaging developer may also choose to modify the file composition and dependency details of individual
! 70: RPM packages. <I>There are over 700 potential RedHat packages to select from on RedHat 6.1 disks. Selecting an appropriate
! 71: mix of RPMs can GREATLY improve security on LON-CAPA systems (by, for instance, removing unnecessary services) and is
! 72: considered the general start- and end- point of LON-CAPA security.</I>
! 73: <LI><B>change process programs</B>
! 74: <BR>The various automated processes are to be written and modified by the packaging developer. These processes should
! 75: reside on the packaging development system (RPM packaging system) as well the CVS file repository.
! 76: <LI><B>upgrade</B>
! 77: <BR>The packaging developer chooses when to upgrade the packaging system so that the packaging system upgrade passes
! 78: along to other computers in the software development cluster.
! 79: </UL>
! 80: <LI><B>Automated processes</B>
! 81: <UL>
! 82: <LI><B>LON_CAPA_update_RPM_build_tree</B>
! 83: <BR>Retrieves files out of the central software development CVS file repository. Builds any customized packages associated
! 84: with the LON-CAPA system.
! 85: <LI><B>LON_CAPA_make_CD_image</B>
! 86: <BR>Compiles a RedHat installation/upgrade CD image based on the current LON-CAPA RPM packaging descriptions.
! 87: <LI><B>LON_CAPA_post_CD_image</B>
! 88: <BR>Make CD image available for other LON-CAPA development/run-time systems to upgrade from.
! 89: <LI><B>LON_CAPA_upgrade_sys</B>
! 90: <BR>Check to see if a general LON-CAPA RedHat system upgrade should be performed based on the version available from the packaging
! 91: system. Also, monitor current status of system with LCIC (perhaps a "corrupt" system should be re-upgraded/re-installed). If upgrade
! 92: should be performed, then perform a network install/upgrade while preserving all LON-CAPA configuration files.
! 93: </UL>
! 94: </UL>
! 95: <P>
! 96: <H1>4. Summary of Actor Objects</H1>
! 97: <UL>
! 98: <LI><B>Head Packaging Developer</B>
! 99: <BR>Responsible for maintaining and writing scripts which automate the packaging and monitoring of distributed LON-CAPA systems. Also
! 100: responsible for altering overall package composition of LON-CAPA systems.
! 101: <LI><B>Programmer</B>
! 102: <BR>Responsible for introducing software improvements to the LON-CAPA system itself. Submits software additions and modifications to
! 103: the central CVS file repository.
! 104: <LI><B>Cron</B>
! 105: <BR>Cron jobs issue scheduled commands to facilitate the independent distributed processing of the various scripts associated with
! 106: LON-CAPA Packaging and Filesystem Management.
! 107: </UL>
! 108: <P>
! 109: <H1>5. Functional Model Diagram</H1>
! 110: <IMG BORDER=1 SRC=scheme.gif>
! 111: <P>
! 112: <H1>6. Description of the Functional Model Diagram</H1>
! 113: I will now describe a semi-chronological viewpoint of how all the different processes take place
! 114: so as to better understand both the specifics of the functional model, as well as the pertinent
! 115: dynamical and object-oriented characteristics of the algorithm.
! 116: <P>
! 117: Every week on Thursday at 3 in the morning, a <B>cron</B> job associated with the <B>RPM</B> packaging system on
! 118: the main packaging computer updates the build tree of all LON-CAPA source RPMs based on files in the central
! 119: <B>CVS</B> file repository. It does this by initiating the process <B>LON_CAPA_update_RPM_build_tree</B>. This process
! 120: requests and receives files from the central <B>CVS</B> file repository, and updates all the source RPM file trees. RPM packages
! 121: are then "bundled" by issuing build commands to the "Red Hat Package Manager" (<I>rpm -b ...</I>).
! 122: <P>
! 123: The <B>LON_CAPA_update_RPM_build_tree</B> process calls the <B>LON_CAPA_make_CD_image</B> process which writes a
! 124: CD image into a single file on the main packaging system. This is followed by the process <B>LON_CAPA_post_CD_image</B>
! 125: which mounts and posts the CD image file onto a web accessible location.
! 126: <P>
! 127: The <B>LON_CAPA_upgrade_sys</B> process is launched by a <B>cron</B> job specific to every individual
! 128: computer present in the software development cluster. Each individual computer checks the main packaging computer to see
! 129: if a system upgrade is needed. System upgrade information is passed to the <B>LCIC</B> on each specific computer (LON-CAPA
! 130: Integrity Checker) so that integrity checking is done as a function of the most recent system upgrade.
! 131: <P>
! 132: Finally, on Sunday at 3 in the morning, a <B>cron</B> job associated with each computer launches the <B>LCIC</B> integrity checker
! 133: system which resyncs itself based on the RPMs and files present on the system as well as information pertaining to
! 134: the expected status of the system (in terms of upgrade, RPM composition, etc). The results of the <B>LCIC</B> integrity checker
! 135: "re-syncing" are posted under a password controlled directory.
! 136: <P>
! 137: <H1>7. Frequently Asked Questions</H1>
! 138: <P>
! 139: <OL>
! 140: <LI><B>What does LON-CAPA stand for?</B>
! 141: <BR>The LearningOnline Network with a Computer-Assisted Personalized Approach
! 142: <P>
! 143: <LI><B>What is the general structure of the CVS repository?</B>
! 144: <BR>These are the current directories:
! 145: <BR><U>loncom</U> - server configuration and application files associated with LON-CAPA library and access servers
! 146: <BR><U>lonline</U> - lecture online
! 147: <BR><U>CAPA</U> - files associated with CAPA (homework processing, grading, homework generating, exam and quiz administration)
! 148: <BR><U>doc</U> - documentation
! 149: <BR><U>modules</U> - custom built software modules meant for use with the LON-CAPA project
! 150: <BR><U>rat</U> - the "RAT"; resource assembly tool
! 151: <BR><U>packaging</U> - files (including this one) specific to packaging and filesystem management of LON-CAPA
! 152: <P>
! 153: <LI><B>How does a programmer set himself up to use CVS?</B>
! 154: <BR>
! 155: <PRE>
! 156: you will need to have the CVSROOT enviroment variable set to
! 157: :pserver:yourname@zaphod.lite.msu.edu:/home/cvs
! 158: To do this if you are using csh or tcsh put in your .login:
! 159: setenv CVSROOT :pserver:yourname@zaphod.lite.msu.edu:/home/cvs
! 160: For bash put this in .bash_profile
! 161: export CVSROOT=:pserver:yourname@zaphod.lite.msu.edu:/home/cvs
! 162:
! 163: You can add the "cvs login" command to your .login/.bash_profile if you wish to make this simple.
! 164: And you can add "cvs logout" to your .logout/.bash_logout so you rember to logout.
! 165:
! 166: To use CVS, you must first login (cvs login) and enter in your zaphod password.
! 167: To leave CVS, you must logout (cvs logout).
! 168: </PRE>
! 169: <P>
! 170: <LI><B>How does a programmer retrieve files from the CVS repository for the first time?</B>
! 171: <BR><TT>cvs checkout [resourcename]</TT>
! 172: <BR><TT>[resourcename]</TT> is one of the directories above (loncom, rat, doc, and so on)
! 173: <P>
! 174: <LI><B>How does a programmer update his files from the CVS repository?</B>
! 175: <BR>(In other words, CVS server ----to----your---->> client computer.)
! 176: <BR><TT>cvs update -d [resourcename]</TT>
! 177: <BR><TT>[resourcename]</TT> is one of the directories above (loncom, rat, doc, and so on)
! 178: <P>
! 179: <LI><B>How does a programmer send his changes to the central CVS repository?</B>
! 180: <BR>(In other words, client computer ----to----our---->> CVS server.)
! 181: <BR><TT>cvs commit [resourcename]</TT>
! 182: <BR><TT>[resourcename]</TT> is one of the directories above (loncom, rat, doc, and so on)
! 183: <P>
! 184: <LI><B>How does a programmer add or remove files from the central CVS repository?</B>
! 185: <BR>Note: this is different from "updating" or "committing" changes. This is not the alteration
! 186: of existing files. This is the ADDITION or REMOVAL of files.
! 187: <BR>Within "the" directory, use these two commands to add a file:
! 188: <BR><TT>cvs add [filename]</TT>; <TT>cvs commit</TT>
! 189: <BR>Within "the" directory, use these two commands to remove a file:
! 190: <BR><TT>cvs remove [filename]</TT>; <TT>cvs commit</TT>
! 191: <P>
! 192: <LI><B>How does a programmer add or remove directories from the central CVS repository?</B>
! 193: <BR>Within the upper directory, use these two commands to add a directory:
! 194: <BR><TT>cvs add [directoryname]</TT>; <TT>cvs commit</TT>
! 195: <BR>Within "the" directory, use these two commands to remove a directory:
! 196: <BR><TT>cvs remove [directoryname]</TT>; <TT>cvs commit</TT>
! 197: <BR>There is a slight catch that you cannot directly "add" a new major directory like
! 198: loncom, lonline, rat, doc, and so on. You must first "cvs add" it as a subdirectory, then
! 199: move that subdirectory to the top directory structure and "cvs commit".
! 200: <P>
! 201: Better yet, you can <TT>cvs import</TT> your new directory into the top level area of the CVS repository.
! 202: <LI><B>How does a programmer "inform" the packaging developer of changes to LON-CAPA system file composition?</B>
! 203: <BR>By talking to the packaging developer in person.
! 204: <BR>By making well-formatted changes to the <TT>packaging/file_map.txt</TT> file in the CVS repository. See below
! 205: for more information on how to format the <TT>packaging/file_map.txt</TT> file.
! 206: <P>
! 207: <LI><B>What does "run any process" mean in the functional model?</B>
! 208: <BR>Most processes are meant to run <I>automatically</I> and are invoked by cron jobs on various machines.
! 209: It may be the case that, for instance, the processes need to be manually invoked so as to prepare
! 210: a specific CD release.
! 211: <P>
! 212: <LI><B>Where are the automated processes located?</B>
! 213: <BR>In the directory <TT>/usr/local/bin</TT>.
! 214: <BR>By typing <TT>ls /usr/local/bin/LON_CAPA_*</TT>, you can view all the automated LON-CAPA processes associated with the computer systems.
! 215: <P>
! 216: <LI><B>Where are the <U>cron</U> jobs located? What are their names and how reliable are they?</B>
! 217: <BR>On every LON-CAPA computer there is a file <TT>/etc/cron.d/LON_CAPA_filesystem</TT> which launches standard cron jobs.
! 218: <BR>On the LON-CAPA main packaging computer there is a file <TT>/etc/cron.d/LON_CAPA_packager</TT> which launches cron jobs specific to the
! 219: main packaging computer.
! 220: <BR>On LON-CAPA software development computers, there is a file <TT>/etc/cron.d/LON_CAPA_upgrader</TT> which upgrades all computers in the
! 221: LON-CAPA software development cluster.
! 222: <P>
! 223: <LI><B>Can automated processes be run manually? On the packaging computer? On other LON-CAPA computers?</B>
! 224: <BR>Yes. To correctly manually run an automated process, look to see how the process is invoked in the <TT>/etc/cron.d</TT> directory.
! 225: <P>
! 226: <LI><B>What are the specifics associated with the RPM packaging system on the main packaging computer?</B>
! 227: <BR>The RPM packaging system on the main packaging computer introduces the following differences to the main packaging computer compared to
! 228: other computers in the software development cluster:
! 229: <UL>
! 230: <LI>Presence of RPM build tree
! 231: <LI>Presence of CD image
! 232: <LI>More automated processes present on system (LON_CAPA_update_RPM_build_tree, LON_CAPA_make_CD_image, LON_CAPA_post_CD_image)
! 233: </UL>
! 234: <P>
! 235: <LI><B>What are the specifics associated with the process LON_CAPA_update_RPM_build_tree?</B>
! 236: <BR>This process checkouts LON-CAPA directories and files. This process reads from the <TT>packaging/file_map.txt</TT>.
! 237: This process reads RPM <TT>.spec</TT> template files to set up the two LON-CAPA RPMs (LON-CAPA-base and LON-CAPA-auxiliary).
! 238: These updated RPMs are placed into a pool of RPMs from which to generate the customized RedHat installation/upgrade CD.
! 239: A 'comps' file is generated to customize the grouping of packages on the CD image. The CD image is written and made accessible
! 240: on the web.
! 241: <P>
! 242: <LI><B>What are the specifics associated with the writing and web posting of the LON-CAPA system CD image on the main packaging computer?</B>
! 243: <BR>There are a number of shell commands which need to be launched. (Details coming soon in this document.)
! 244: <P>
! 245: <LI><B>What are the specifics associated with the process LON_CAPA_make_CD_image?</B>
! 246: <BR>There are a number of shell commands which need to be launched. (Details coming soon in this document.)
! 247: <P>
! 248: <LI><B>How are upgrades performed on the packaging computer? Is there any way to upgrade/install LON-CAPA on a new machine?</B>
! 249: <BR>Upgrades are performed only based on package composition. New machines can be upgraded via network install or CD install. (More details coming soon.)
! 250: <P>
! 251: <LI><B>What are the specifics associated with the process LON_CAPA_post_CD_image?</B>
! 252: <BR>There are a number of shell commands which need to be launched. (Details coming soon in this document.)
! 253: <P>
! 254: <LI><B>What are the specifics associated with the process LON_CAPA_upgrade_sys?</B>
! 255: <BR>LON_CAPA_upgrade_sys does an automated network upgrade of any out-of-date packages based on the main packaging computer.
! 256: LON_CAPA_upgrade_sys initializes and invokes the LCIC (integrity checker) based on current upgrade information.
! 257: <P>
! 258: <LI><B>What output from the LCIC (integrity checker) warrants taking corrective action?</B>
! 259: <BR>Currently, the output from the LCIC consists of two files, TOTAL_FILE.txt and SUMMARY_FILE.txt. There are no immediate plans for
! 260: automating any form of response to this output. This does provide information as to whether critical system binaries or configuration
! 261: files are being tampered with. Perhaps the SUMMARY_FILE.txt should be e-mailed weekly to a system administrator at the responsible
! 262: institution.
! 263: <P>
! 264: <LI><B>What is the correct format for the <TT>packaging/file_map.txt</TT> file?</B>
! 265: <BR>Every line refers to a file or directory to install/upgrade on a LON-CAPA system. There are six fields to each line which are tab-separated.
! 266: <PRE>
! 267: [computer-type]
! 268: [file location on CVS repository]
! 269: [ultimate location on LON-CAPA system]
! 270: [package name]
! 271: [configuration settings that need to be saved?]
! 272: [description of file and what it does]
! 273: </PRE>
! 274: <UL>
! 275: <LI><TT>[computer-type]</TT> - a list of capitalized letters indicating which computer-types this file is to be installed on. P=main packaging computer. A=LON-CAPA software development access server. L=LON-CAPA software development library server. C=central CVS file repository computer. PAL means this file is to be installed on the main packaging computer and the software development library and access servers. P means this file is to be installed on just the main packaging computer. AL means the file is to be installed on just the software development library and access servers.
! 276: <LI><TT>[file location on CVS repository]</TT> - the file location on the CVS repository. For example, <TT>/loncom/htpasswd</TT>, <TT>loncom/startup.pl</TT>, and
! 277: <TT>/rat/images/1.1.dt.gif</TT> are possible entries.
! 278: <LI><TT>[ultimate location on LON-CAPA system]</TT> - where the file (taken from the CVS repository) is to be installed on the LON-CAPA system. For example,
! 279: <TT>/home/httpd/lonTabs/htpasswd</TT>, <TT>/etc/httpd/conf/startup.pl</TT>, and <TT>/home/httpd/html/adm/rat/1.1.dt.gif</TT> are possible entries.
! 280: <LI><TT>[package name]</TT> - What RPM package should this file be incorporated into for the final RPM install? Current possible entries are
! 281: <TT>LON-CAPA-auxiliary</TT> and <TT>LON-CAPA-base</TT>.
! 282: <LI><TT>[configuration settings that need to be saved?]</TT> - Every machine is anticipated to have a unique identity in terms of the role it
! 283: fills at a given institution. For instance, /etc/httpd/conf/httpd.conf has settings like these:
! 284: <PRE>
! 285: PerlSetVar lonHostID msua3
! 286: # Role of this machine: library, access
! 287: PerlSetVar lonRole access
! 288: # Server Administration
! 289: PerlSetVar lonAdmEMail korte@lite.msu.edu
! 290: # Default domain
! 291: PerlSetVar lonDefDomain msu
! 292: </PRE>
! 293: At some point, these settings need to be saved and used during upgrade processes.
! 294: <LI><TT>[description of file and what it does]</TT> - a brief sentence.
! 295: </UL>
! 296: <P>
! 297: <LI><B>What controls version naming in terms of system upgrades?</B>
! 298: <BR>This information is present in <TT>packaging/version.txt</TT>. This file consists of three tab-separated fields (version number, release number, and version
! 299: name). The release number is automatically incremented each week as a function of LON_CAPA_post_CD_image. The version number and release numbers are used
! 300: to label the LON-CAPA-base and LON-CAPA-auxiliary RPMs as well as the customized RedHat CD release.
! 301: </OL>
! 302: </BODY>
! 303: </HTML>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/back.gif){kind=icon}
Return to index.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [LON-CAPA] / doc / packaging