LON-CAPA Packaging and Filesystem Management

Functional Model Description (Scott Harrison, April 2000)

  1. Synopsis
  2. Summary of Data Store Objects (e.g. CVS, RPM, LCIC)
  3. Summary of Processes (e.g. making a CD image, upgrading a system, altering package composition)
  4. Summary of Actor Objects (e.g. cron jobs, programmers, head packaging developer)
  5. Functional Model Diagram
  6. Description of the Functional Model Diagram
    1. Data Store Objects
    2. Processes
    3. Actor Objects
  7. Frequently Asked Questions

1. Synopsis

Up to this point, dozens of scripts have been written to automate various elements of the LON-CAPA installation, upgrade, and software development process. A number of significant issues relate to this effort. Security on the system is largely a matter of limiting unnecessary network services, monitoring the filesystem of LON-CAPA servers to make full account for the presence of every file on the system, and securely communicating potential problems to system administrators of a given LON-CAPA server machine.

This document is the functional model description which describes the computations and their expected results necessary to manage packaging and filesystem management for LON-CAPA both now and in the future in a manner that is synchronous with software development, installations, and upgrades. Promulgation of coding changes, version control, and maintaining a handle on security and network optimization issues (such as introducing SAMBA and NetaTalk into the network architecture) have been some of the reasons for introducing a more strongly defined characterization of all of these processes involved with LON-CAPA software packaging and file system management.

2. Summary of Data Store Objects

3. Summary of Processes

4. Summary of Actor Objects

5. Functional Model Diagram

6. Description of the Functional Model Diagram

I will now describe a semi-chronological viewpoint of how all the different processes take place so as to better understand both the specifics of the functional model, as well as the pertinent dynamical and object-oriented characteristics of the algorithm.

Every week on Thursday at 3 in the morning, a cron job associated with the RPM packaging system on the main packaging computer updates the build tree of all LON-CAPA source RPMs based on files in the central CVS file repository. It does this by initiating the process LON_CAPA_update_RPM_build_tree. This process requests and receives files from the central CVS file repository, and updates all the source RPM file trees. RPM packages are then "bundled" by issuing build commands to the "Red Hat Package Manager" (rpm -b ...).

The LON_CAPA_update_RPM_build_tree process calls the LON_CAPA_make_CD_image process which writes a CD image into a single file on the main packaging system. This is followed by the process LON_CAPA_post_CD_image which mounts and posts the CD image file onto a web accessible location.

The LON_CAPA_upgrade_sys process is launched by a cron job specific to every individual computer present in the software development cluster. Each individual computer checks the main packaging computer to see if a system upgrade is needed. System upgrade information is passed to the LCIC on each specific computer (LON-CAPA Integrity Checker) so that integrity checking is done as a function of the most recent system upgrade.

Finally, on Sunday at 3 in the morning, a cron job associated with each computer launches the LCIC integrity checker system which resyncs itself based on the RPMs and files present on the system as well as information pertaining to the expected status of the system (in terms of upgrade, RPM composition, etc). The results of the LCIC integrity checker "re-syncing" are posted under a password controlled directory.

7. Frequently Asked Questions

  1. What does LON-CAPA stand for?
    The LearningOnline Network with a Computer-Assisted Personalized Approach

  2. What is the general structure of the CVS repository?
    These are the current directories:
    loncom - server configuration and application files associated with LON-CAPA library and access servers
    lonline - lecture online
    CAPA - files associated with CAPA (homework processing, grading, homework generating, exam and quiz administration)
    doc - documentation
    modules - custom built software modules meant for use with the LON-CAPA project
    rat - the "RAT"; resource assembly tool
    packaging - files (including this one) specific to packaging and filesystem management of LON-CAPA

  3. How does a programmer set himself up to use CVS?
    you will need to have the CVSROOT enviroment variable set to
    :pserver:yourname@zaphod.lite.msu.edu:/home/cvs
    To do this if you are using csh or tcsh put in your .login:
    setenv CVSROOT :pserver:yourname@zaphod.lite.msu.edu:/home/cvs
    For bash put this in .bash_profile
    export CVSROOT=:pserver:yourname@zaphod.lite.msu.edu:/home/cvs
    
    You can add the "cvs login" command to your .login/.bash_profile if you wish to make this simple.
    And you can add "cvs logout" to your .logout/.bash_logout so you rember to logout.
    
    To use CVS, you must first login (cvs login) and enter in your zaphod password.
    To leave CVS, you must logout (cvs logout).
    

  4. How does a programmer retrieve files from the CVS repository for the first time?
    cvs checkout [resourcename]
    [resourcename] is one of the directories above (loncom, rat, doc, and so on)

  5. How does a programmer update his files from the CVS repository?
    (In other words, CVS server ----to----your---->> client computer.)
    cvs update -d [resourcename]
    [resourcename] is one of the directories above (loncom, rat, doc, and so on)

  6. How does a programmer send his changes to the central CVS repository?
    (In other words, client computer ----to----our---->> CVS server.)
    cvs commit [resourcename]
    [resourcename] is one of the directories above (loncom, rat, doc, and so on)

  7. How does a programmer add or remove files from the central CVS repository?
    Note: this is different from "updating" or "committing" changes. This is not the alteration of existing files. This is the ADDITION or REMOVAL of files.
    Within "the" directory, use these two commands to add a file:
    cvs add [filename]; cvs commit
    Within "the" directory, use these two commands to remove a file:
    cvs remove [filename]; cvs commit

  8. How does a programmer add or remove directories from the central CVS repository?
    Within the upper directory, use these two commands to add a directory:
    cvs add [directoryname]; cvs commit
    Within "the" directory, use these two commands to remove a directory:
    cvs remove [directoryname]; cvs commit
    There is a slight catch that you cannot directly "add" a new major directory like loncom, lonline, rat, doc, and so on. You must first "cvs add" it as a subdirectory, then move that subdirectory to the top directory structure and "cvs commit".

    Better yet, you can cvs import your new directory into the top level area of the CVS repository.

  9. How does a programmer "inform" the packaging developer of changes to LON-CAPA system file composition?
    By talking to the packaging developer in person.
    By making well-formatted changes to the packaging/file_map.txt file in the CVS repository. See below for more information on how to format the packaging/file_map.txt file.

  10. What does "run any process" mean in the functional model?
    Most processes are meant to run automatically and are invoked by cron jobs on various machines. It may be the case that, for instance, the processes need to be manually invoked so as to prepare a specific CD release.

  11. Where are the automated processes located?
    In the directory /usr/local/bin.
    By typing ls /usr/local/bin/LON_CAPA_*, you can view all the automated LON-CAPA processes associated with the computer systems.

  12. Where are the cron jobs located? What are their names and how reliable are they?
    On every LON-CAPA computer there is a file /etc/cron.d/LON_CAPA_filesystem which launches standard cron jobs.
    On the LON-CAPA main packaging computer there is a file /etc/cron.d/LON_CAPA_packager which launches cron jobs specific to the main packaging computer.
    On LON-CAPA software development computers, there is a file /etc/cron.d/LON_CAPA_upgrader which upgrades all computers in the LON-CAPA software development cluster.

  13. Can automated processes be run manually? On the packaging computer? On other LON-CAPA computers?
    Yes. To correctly manually run an automated process, look to see how the process is invoked in the /etc/cron.d directory.

  14. What are the specifics associated with the RPM packaging system on the main packaging computer?
    The RPM packaging system on the main packaging computer introduces the following differences to the main packaging computer compared to other computers in the software development cluster:

  15. What are the specifics associated with the process LON_CAPA_update_RPM_build_tree?
    This process checkouts LON-CAPA directories and files. This process reads from the packaging/file_map.txt. This process reads RPM .spec template files to set up the two LON-CAPA RPMs (LON-CAPA-base and LON-CAPA-auxiliary). These updated RPMs are placed into a pool of RPMs from which to generate the customized RedHat installation/upgrade CD. A 'comps' file is generated to customize the grouping of packages on the CD image. The CD image is written and made accessible on the web.

  16. What are the specifics associated with the writing and web posting of the LON-CAPA system CD image on the main packaging computer?
    There are a number of shell commands which need to be launched. (Details coming soon in this document.)

  17. What are the specifics associated with the process LON_CAPA_make_CD_image?
    There are a number of shell commands which need to be launched. (Details coming soon in this document.)

  18. How are upgrades performed on the packaging computer? Is there any way to upgrade/install LON-CAPA on a new machine?
    Upgrades are performed only based on package composition. New machines can be upgraded via network install or CD install. (More details coming soon.)

  19. What are the specifics associated with the process LON_CAPA_post_CD_image?
    There are a number of shell commands which need to be launched. (Details coming soon in this document.)

  20. What are the specifics associated with the process LON_CAPA_upgrade_sys?
    LON_CAPA_upgrade_sys does an automated network upgrade of any out-of-date packages based on the main packaging computer. LON_CAPA_upgrade_sys initializes and invokes the LCIC (integrity checker) based on current upgrade information.

  21. What output from the LCIC (integrity checker) warrants taking corrective action?
    Currently, the output from the LCIC consists of two files, TOTAL_FILE.txt and SUMMARY_FILE.txt. There are no immediate plans for automating any form of response to this output. This does provide information as to whether critical system binaries or configuration files are being tampered with. Perhaps the SUMMARY_FILE.txt should be e-mailed weekly to a system administrator at the responsible institution.

  22. What is the correct format for the packaging/file_map.txt file?
    Every line refers to a file or directory to install/upgrade on a LON-CAPA system. There are six fields to each line which are tab-separated.
    [computer-type]
    [file location on CVS repository]
    [ultimate location on LON-CAPA system]
    [package name]
    [configuration settings that need to be saved?]
    [description of file and what it does]
    

  23. What controls version naming in terms of system upgrades?
    This information is present in packaging/version.txt. This file consists of three tab-separated fields (version number, release number, and version name). The release number is automatically incremented each week as a function of LON_CAPA_post_CD_image. The version number and release numbers are used to label the LON-CAPA-base and LON-CAPA-auxiliary RPMs as well as the customized RedHat CD release.