loncom/build/readme.html - view

File: [LON-CAPA] / loncom / build / readme.html
Revision 1.22: download - view: text, annotated - select for diffs
Mon Feb 3 18:03:52 2003 UTC (21 years, 2 months ago) by harris41
Branches: MAIN
CVS tags: version_2_9_X, version_2_9_99_0, version_2_9_1, version_2_9_0, version_2_8_X, version_2_8_99_1, version_2_8_99_0, version_2_8_2, version_2_8_1, version_2_8_0, version_2_7_X, version_2_7_99_1, version_2_7_99_0, version_2_7_1, version_2_7_0, version_2_6_X, version_2_6_99_1, version_2_6_99_0, version_2_6_3, version_2_6_2, version_2_6_1, version_2_6_0, version_2_5_X, version_2_5_99_1, version_2_5_99_0, version_2_5_2, version_2_5_1, version_2_5_0, version_2_4_X, version_2_4_99_0, version_2_4_2, version_2_4_1, version_2_4_0, version_2_3_X, version_2_3_99_0, version_2_3_2, version_2_3_1, version_2_3_0, version_2_2_X, version_2_2_99_1, version_2_2_99_0, version_2_2_2, version_2_2_1, version_2_2_0, version_2_1_X, version_2_1_99_3, version_2_1_99_2, version_2_1_99_1, version_2_1_99_0, version_2_1_3, version_2_1_2, version_2_1_1, version_2_1_0, version_2_10_0_RC1, version_2_0_X, version_2_0_99_1, version_2_0_2, version_2_0_1, version_2_0_0, version_1_99_3, version_1_99_2, version_1_99_1_tmcc, version_1_99_1, version_1_99_0_tmcc, version_1_99_0, version_1_3_X, version_1_3_3, version_1_3_2, version_1_3_1, version_1_3_0, version_1_2_X, version_1_2_99_1, version_1_2_99_0, version_1_2_1, version_1_2_0, version_1_1_X, version_1_1_99_5, version_1_1_99_4, version_1_1_99_3, version_1_1_99_2, version_1_1_99_1, version_1_1_99_0, version_1_1_3, version_1_1_2, version_1_1_1, version_1_1_0, version_1_0_99_3, version_1_0_99_2, version_1_0_99_1, version_1_0_99, version_1_0_3, version_1_0_2, version_1_0_1, version_1_0_0, version_0_99_5, version_0_99_4, version_0_99_3, version_0_99_2, version_0_99_1, version_0_99_0, conference_2003, bz6209-base, bz6209, bz5969, bz5610, bz2851, PRINT_INCOMPLETE_base, PRINT_INCOMPLETE, HEAD, GCI_3, GCI_2, GCI_1, BZ5971-printing-apage, BZ5434-fox

best wishes to all.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">   <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta> <title>LON-CAPA Software Developer Guide</title> </head> <body> <h1>LON-CAPA Software Developer Guide</h1> January 17, 2001 Last updated, August 14, 2002 <ol> <li><a href="#Using_CVS">Using CVS</a> <ul> <li><a href="#cvslog">Logging in and out (cvs login; cvs logout)</a></li> <li><a href="#cvsupdate">Updating files (cvs update -d)</a></li> <li><a href="#cvssave">Saving files (cvs commit)</a></li> <li><a href="#cvsadd">Adding files (cvs add)</a></li> <li><a href="#cvsadddir">Adding directories (cvs add/import)</a></li> <li><a href="#cvsnotsure">What to do when you're not sure about your files (cvs update)</a></li> </ul></li> <li><a href="#makeHTML">Viewing the software (make HTML)</a></li> <li><a href="#makebuild">Compiling the software (make build)</a></li> <li><a href="#loncapafiles">Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.lpml)</a></li> <li><a href="#configversusnonconfig">Configurable files versus non-configurable files</a></li> <li><a href="#makeinstall">Updating the non-configurable files on your machine (make install)</a></li> <li><a href="#makeconfiginstall">Updating the configurable files on your machine (make configinstall)</a></li> <li><a href="#makeRPM">Building RPMs (make RPM)</a></li> </ol> <ol> <li><a name="Using_CVS" /><h2>Using CVS</h2> These instructions assume that you are using a Linux or UNIX based terminal. <ul> <li><a name="cvslog" /> <h3>Using CVS: Logging in and out (cvs login; cvs logout)</h3> In order to log into CVS, CVS needs to be part of your system environment. You can do this by: <tt>export CVSROOT=:pserver:USERNAME@install.lon-capa.org:/home/cvs</tt> To actually login, you will need to execute the following command: <tt>cvs login</tt> You are then prompted for a password. If you do not have a password, or the password is not working, you should contact <a href="mailto:helen@lon-capa.org">helen@lon-capa.org</a>. The first time you use CVS, you need to CHECKOUT the repository. Generally speaking, you need to checkout <tt>loncapa</tt> only once per machine. To check-out the repository, use the <tt>checkout</tt> command. (Otherwise, just enter your CVS directory, <tt>cd loncapa</tt>.) <tt>cvs checkout loncapa</tt> <tt>cd loncapa</tt> After completing work with the CVS repository, you can log out: <tt>cvs logout</tt> </li> <li><a name="cvsupdate" /> <h3>Using CVS: Updating files (cvs update -d)</h3> After entering your CVS source tree (<tt>cd loncapa</tt>), you should frequently update the software changes that other people have made. This is done with the <tt>update</tt> command. <tt> cvs update -d </tt> The <tt>cvs update</tt> command creates output as it updates your CVS source tree. Common flags are 'U' and 'P'; they indicate that a file in your <tt>loncapa</tt> directory is now updated with changes made by another programmer. <tt>`U FILE'</tt> <blockquote> The file was brought up to date in your <tt>loncapa</tt>. 'U' is done for: * any file that exists in the repository but not in your source, and * files that you have not changed but are not the most recent versions available in the repository. The network behavior of 'U' is that the entire new file is uploaded from the CVS server. </blockquote> <tt> `P FILE' </tt> <blockquote> Like `U', but the CVS server sends a patch instead of an entire file. </blockquote> 'U' and 'P' essentially accomplish the same thing, just in different ways. Usually, when you do not <tt>cvs commit</tt> your code changes, the <tt>update</tt> command will tell you that you have modified your file with the 'M' flag. <tt> `M FILE' </tt> <blockquote> The file is modified in your working <tt>loncapa</tt> directory. This is probably based on changes you made and have not yet "cvs commit"-ed. </blockquote> Sometimes, it will occur that: <ul> <li>you have modified a file and not yet committed it</li> <li>someone else *has* modified a file and *has* committed it</li> </ul> Generally speaking, this is your fault. It is your responsibility to resolve conflicts. <tt>cvs update</tt> informs you of a conflict with the 'C' flag. <tt> `C FILE' </tt> <blockquote> A conflict was detected while trying to merge your changes to FILE with changes from the source repository. </blockquote> You will need to open the file and examine it; CVS will have added in markup tags like "<<<<<<" to tell you about the merging conflicts. (Sometimes, CVS will intelligently merge in other changes and give you the 'M' flag, but many times you will have to manually edit the file as just described.) </li> <li><a name="cvssave" /> <h3>Using CVS: Saving files (cvs commit)</h3> <tt>cvs commit</tt> works to submit changes to an existing file within the repository. If a file does not currently exist, then you will first need to <tt>cvs add</tt> it as described in the following section. Running the <tt>cvs commit</tt> command without additional arguments will commit all of your changes within the current directory and subdirectories. <tt> cvs commit </tt> A more precise approach to using <tt>cvs commit</tt> is to pass it specific file names. (Usually you should do this.) <tt> cvs commit FILENAME </tt> Note that CVS typically invokes the <a href="http://www.eng.hawaii.edu/Tutor/vi.html">vi</a> editor and solicits comments about your latest changes to the software. Your comments should be both short yet uniquely descriptive. For example: <ul> <li>BAD - "made some changes and am drinking soda"</li> <li>GOOD - "implemented syntax checking of perl scripts with -c flag"</li> </ul> </li> <li><a name="cvsadd" /> <h3>Using CVS: Adding files (cvs add)</h3> <tt> cvs add FILENAME </tt> Then you can run <tt>cvs commit FILENAME</tt> and this file will become an "official" part of LON-CAPA. </li> <li><a name="cvsadddir" /> <h3>Using CVS: Adding directories (cvs add/import)</h3> <tt> cvs add DIRECTORYNAME </tt> There is no need to run <tt>cvs commit</tt>. Directories immediately become part of the LON-CAPA CVS source tree by only using the <tt>cvs add</tt> command. You should not ordinarily need to use the <tt>cvs import</tt> command. If misused, <tt>cvs import</tt> can lead to the loss of code within the repository. </li> <li><a name="cvsnotsure" /> <h3>Using CVS: What to do when you're not sure about your files (cvs update -d)</h3> Once in a while, multiple programmers may be working on the same file. Most conflicts are avoidable if everybody regularly commits their changes AND if everybody regularly updates the CVS source tree they are working on. If you are absent from programming for a few days, and fail to run <tt>cvs update -d</tt> on your CVS source repository, you have only yourself to blame if you find yourself writing code in a file that is not up-to-date. </li> </ul></li> <li><a name="makeHTML" /> <h2>Viewing the software (make HTML)</h2> Commands <tt> cd loncom/build rm -Rf HTML (or alternatively, "make clean") make HTML cd HTML (look at the index.html file with a web browser such as Netscape) </tt> General description of what happens This is the actual make target code. <pre>  HTML: install -d HTML cp $(SOURCE)/doc/loncapafiles/*.gif HTML cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \ perl lpml_parse.pl html development default "$(SOURCE)" '$(TARGET)' \ > HTML/index.html  </pre> What basically happens is that specially marked-up data in the LON-CAPA cvs repository file <tt>doc/loncapafiles/loncapafiles.lpml</tt> is parsed into a more viewable format by <tt>loncom/build/lpml_parse.pl</tt>. The resulting file gives a very well organized view of all the files, directories, links, ownerships, permissions, and brief documentation of what each file does. </li> <li><a name="makebuild" /> <h2>Compiling the software (make build)</h2> Commands <tt> cd loncom/build make build </tt> General description of what happens This is the actual make target code. <pre>  build: Makefile.build pod2html.sh pod2man.sh echo -n "" > WARNINGS make -f Makefile.build all make warningnote Makefile.build: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \ perl lpml_parse.pl build $(CATEGORY) $(DIST) "$(SOURCE)" "$(TARGET)" \ > Makefile.build  </pre> <tt>loncom/build/lpml_parse.pl</tt> reads in all the build information out of <tt>doc/loncapafiles/loncapafiles.lpml</tt>. A new Makefile named <tt>loncom/build/Makefile.build</tt> is dynamically constructed. This dynamically generated Makefile is then used to build and compile all the software targets from source. This can take several minutes (it depends on the speed of the machine you compile with). Example Here is information for one file <tt>tth.so</tt> provided in <tt>doc/loncapafiles/loncapafiles.lpml</tt>. <pre> <file> <source>loncom/homework/caparesponse/capa.so</source> <target dist='default'>usr/lib/perl5/site_perl/5.005/capa.so</target> <target dist='redhat7 redhat7.1'>usr/lib/perl5/site_perl/5.6.0/capa.so</target> <categoryname>system file</categoryname> <description> shared library file for dynamic loading and unloading </description> <build trigger='always run'> loncom/homework/caparesponse/commands </build> <dependencies> caparesponse.c; caparesponse.pm; README; Makefile.PL; capa.i; commands </dependencies> </file> </pre> <tt>loncom/build/lpml_parse.pl</tt> sees the build tags and sets up a dynamic file <tt>Makefile.build</tt> to run the command inside the build tags. The files listed inside the dependencies tags are included in the <tt>Makefile.build</tt> so as to determine whether or not there is a need to compile. Here is an example of a dynamically generated <tt>Makefile.build</tt> that builds two LON-CAPA files (one of which is <tt>tth.so</tt>). <pre> all: ../homework/caparesponse/capa.so ../modules/TexConvert/tthperl/tth.so ../homework/caparesponse/capa.so: ../homework/caparesponse/caparesponse.c ../homework/caparesponse/caparesponse.pm alwaysrun cd ../homework/caparesponse/; sh ./commands ../modules/TexConvert/tthperl/tth.so: ../modules/TexConvert/tthperl/../tthdynamic/tthfunc.c ../modules/TexConvert/tthperl/../ttmdynamic/ttmfunc.c cd ../modules/TexConvert/tthperl/; sh ./commands alwaysrun: </pre> </li><li><a name="loncapafiles" /> <h2>Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)</h2> To add and remove (and alter) All that you have to do to alter the behavior of the installation is edit a single file (<tt>doc/loncapafiles/loncapafiles.lpml</tt>). Adding, removing, and altering files requires proper attention to the syntax of file format of course. File Format The preceding <a href="#makebuild">"make build"</a> documentation gives an example of a file entry describing one particular file. All data within <tt>loncapafiles.lpml</tt> is specified according to markup tags. The format and syntax of <tt>loncapafiles.lpml</tt> is currently best described by the HTML documentation code at the beginning of loncapafiles.html (as well as, by example, seeing how various information is coded). All in all, the syntax is quite simple. Philosophy and notes (the thing nobody reads) Packaging the software from CVS onto a machine file system requires many things: <ul> <li>documenting every component of the software,</li> <li>handling CVS source to file system target information,</li> <li>handling (according to a hierarchical scheme of grouping) file ownership and permissions,</li> <li>handling (according to a hierarchical scheme of grouping) directory ownership and permissions,</li> <li>handling symbolic links,</li> <li>providing for multiple options of installation targets (e.g. RedHat versus Debian),</li> <li>providing for different file ownerships and permissions to apply to the same file,</li> <li>allowing system software documentation to be automatically generated (see information on <a href="#makeHTML">"make html"</a>),</li> <li>providing information in an easily adjustable form as new demands are made on the software packaging system,</li> <li>providing software package information (for RPM),</li> <li>having information in a format that allows for troubleshooting the current status of the machine file system,</li> <li>allow for changes to the structure of the CVS repository,</li> <li>and something that is simple enough for any one to immediately work with, without having to learn any specifics (or hidden traps) of complicated Makefile's or a new macro language (m4?).</li> </ul> I looked into, and tried, different ways of accomplishing the above including automake and recursive make. The automake system seemed quite complicated (and needlessly so in terms of this project since, by and large, it works to coordinate many different types of build/compilation parameters whereas we are more concerned with installation parameters). The other alternative, recursive make, has significant deficiencies since not all the information is kept in one place, and there are significant levels of dependency between all the things that must be done to keep software packaging up to date. A particularly convincing article I found when looking into much of this was <a href="http://www.pcug.org.au/~millerp/rmch/recu-make-cons-harm.html"> "Recursive Make Considered Harmful" by Peter Miller</a>. Other complications were that, at the time, it was unclear as to what categories of software files we had, and whether or not the directory structure of CVS would remain constant. With an ever-developing directory structure to CVS, I preferred to organize the information on a per-file basis as opposed to a per-directory basis. Additionally, a standard big Makefile assumes certain "normalcy" to the directory structure of different potential operating system directories (RedHat vs. Debian). If you take time to look at <tt>loncapafiles.lpml</tt> (and perhaps run the <a href="#makeHTML">make HTML</a> command) you will find that the organizing information according to the markup syntax in <tt>loncapafiles.lpml</tt> is simple. Simple is good. <tt>loncom/build/lpml_parse.pl</tt> is the script (invoked automatically by the various targets in <tt>loncom/build/Makefile</tt>) that reads <tt>doc/loncapafiles/loncapafiles.lpml</tt>. <tt>lpml_parse.pl</tt> is capable of reading and returning different types of information from <tt>loncapafiles.lpml</tt> depending on how <tt>lpml_parse.pl</tt> is invoked. <tt>lpml_parse.pl</tt> has yet to have introduced new sources of error, and has been tested in quite a number of ways. As with any parser however, I remain paranoid. Finally, some notes on the development. <tt>lpml_parse.pl</tt> is very fast and styled after a state-based SAX-like approach. I do eventually want to use a real XML/XSLT approach, however I hesitate to make everyone everywhere install something like <a href="http://search.cpan.org/search?dist=XML-Xalan">XML::Xalan</a>. Also note that <tt>loncapafiles.lpml</tt> has a DTD (<tt>loncom/build/lpml.dtd</tt>) against which it is valid. I would also like to use more ENTITY's inside <tt>lpml.dtd</tt> but currently the perl XML modules available at CPAN do not digest complex ENTITY's that well. The <tt>lpml_parse.pl</tt>-<tt>loncapafiles.lpml</tt> combination has been highly efficient and error-free. </li><li><a name="configversusnonconfig" /> <h2>Configurable files versus non-configurable files</h2> Machine-specific information is the difference The current list of configurable files for the LON-CAPA system is <tt>/etc/httpd/conf/loncapa.conf</tt>, <tt>/etc/ntp.conf</tt>, <tt>/etc/krb.conf</tt>, <tt>/etc/ntp/step-tickers</tt>, <tt>/home/httpd/html/res/adm/includes/copyright.tab</tt>, <tt>/home/httpd/html/res/adm/includes/un_keyword.tab</tt>, <tt>/home/httpd/hosts.tab</tt>, and <tt>/home/httpd/spare.tab</tt>. All of these configurable files contain machine-specific information. For instance, the overall LON-CAPA system relies on unique host IDs such as msua3, s1, s2, msul1, and 103a1 (specified as a "PerlSetVar lonHostID" field within <tt>/etc/httpd/conf/loncapa.conf</tt>). Non-configurable files simply do NOT have machine-specific information. The impact on updating software What this means in terms of software updating is that: <ul> <li>non-configurable files can be simply overwritten with newer versions (without "anything" else to worry about),</li> <li>and configurable files must follow these steps to be safely overwritten: <ol> <li>have their machine-specific information saved,</li> <li>be overwritten, and then</li> <li>have their machine-specific information restored.</li> </ol> </li> </ul> </li> <li><a name="makeinstall" /> <h2>Updating the non-configurable files on your machine (make install)</h2> Commands <tt> cd loncom/build make install </tt> General description of what happens This is the actual make target code. <pre>  install: TEST_hosts_tab Makefile.install Makefile echo -n "" > WARNINGS make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" \ directories make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" files make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" links make SOURCE="$(SOURCE)" TARGET="$(TARGET)" \ NORESTORECONF="$(NORESTORECONF)" configinstall make postinstall make warningnote echo "You can run 'make test' to see if your system is ready to go!" Makefile.install: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \ perl lpml_parse.pl install $(CATEGORY) $(DIST) "$(SOURCE)" \ "$(TARGET)" > Makefile.install  </pre> For safety reasons (so as to preserve a machine's configuration), configuration files are NOT installed during this step. This means that files such as <tt>/etc/httpd/conf/loncapa.conf</tt>, <tt>/home/httpd/html/res/adm/includes/copyright.tab</tt>, and <tt>/home/httpd/spare.tab</tt> are not overwritten, but remain as old, non-updated copies. (To automatically update these files and save/restore their encoded machine configuration, you must run "make configinstall"). </li> <li><a name="makeconfiginstall" /> <h2>Updating the configurable files on your machine (make configinstall)</h2> Commands <tt> cd loncom/build make configinstall </tt> General description of what happens This is the actual make target code. <pre>  configinstall: Makefile.configinstall make -f Makefile.configinstall SOURCE="$(SOURCE)" TARGET="$(TARGET)" \ configfiles if (test "0" = $(NORESTORECONF)); then \ perl loncaparestoreconfigurations suffix .lpmlnew; fi Makefile.configinstall: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \ perl lpml_parse.pl configinstall $(CATEGORY) $(DIST) "$(SOURCE)" \ "$(TARGET)" > Makefile.configinstall  </pre> Configuration files are installed during this step. This means that files such as <tt>/etc/httpd/conf/loncapa.conf</tt>, <tt>/home/httpd/html/res/adm/includes/copyright.tab</tt>, and <tt>/home/httpd/spare.tab</tt> are overwritten. Before being overwritten, a backup copy is made though. Information is read out of these backup copies and restored to the new files by the <tt>loncaparestoreconfigurations</tt> script. To ensure that new file permissions and ownerships are installed, a final set of <tt>chown</tt> and <tt>chmod</tt> commands are called for each of the configuration files. For the truly paranoid If you are truly paranoid, you can just make the <tt>Makefile.configinstall</tt> file and then save, copy, and restore all the configuration values yourself. <tt>loncaparestoreconfigurations</tt> is pretty smart though, has yet to fail, and besides, when needed, backup copies are made. </li><li><a name="makeRPM" /> <h2>Building RPMs (make RPM)</h2> LON-CAPA is currently installed through "intelligent tarballs". What I am describing now is part of an earlier (and perhaps future) effort involving RPMs. Commands <tt> cd loncom/build rm -Rf BinaryRoot (or alternatively, "make clean") make RPM (to subsequently install, you can type commands like "rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm") </tt> Configuration files Configuration files are automatically saved with the file suffix ".rpmsave". So <tt>/etc/httpd/conf/loncapa.conf</tt> is saved as <tt>/etc/httpd/conf/loncapa.conf.rpmsave</tt>. The <tt>loncaparestoreconfigurations</tt> script should work to restore configurations in this case. However, please note that if you install an RPM twice without restoring your configuration, you will overwrite the ".rpmsave" files. General description of what happens This is the actual make target code. <pre>  RPM: BinaryRoot base_rpm_file_list cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \ perl lpml_parse.pl make_rpm $(CATEGORY) $(DIST) $(SOURCE) $(TARGET) \ > base_customizerpm.xml cat base_rpm_file_list.txt | perl make_rpm.pl base 3.2 1 '' '' \ BinaryRoot base_customizerpm.xml BinaryRoot: base_rpm_file_list make TARGET='BinaryRoot' NORESTORECONF='1' install base_rpm_file_list: cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \ perl lpml_parse.pl rpm_file_list $(CATEGORY) $(DIST) $(SOURCE) \ 'BinaryRoot' | sort > base_rpm_file_list.txt  </pre> A <tt>BinaryRoot</tt> directory is generated that reflects the locations, ownerships, permissions, and contents for all the CVS source files, compiled binaries, directories, and links as they should eventually occur on the '/' filesystem location. <tt>loncom/build/make_rpm.pl</tt> (also available at <a href="http://www.cpan.org">CPAN</a>) is robust (tested over the span of months) and, unlike other automated RPM-builders, cleanly builds new RPMs without any after-effect of temporary files left on the system. The generated RPM is labeled in the format LON-CAPA-base-(VERSION)-(RELEASE).i386. VERSION is specified inside the Makefile. </li> </ol> </body> </html>