[BACK]Return to readme.html CVS log [TXT] [DIR] Up to [LON-CAPA] / loncom / build

Diff for /loncom/build/readme.html between versions 1.8 and 1.13

version 1.8, 2001/01/17 11:57:59 version 1.13, 2001/01/17 13:40:29
Line 1 Line 1
   <HTML>
   <HEAD>
   <META NAME="GENERATOR" CONTENT="Scott Harrison and Emacs Version 3.14159265358979">
   <TITLE>LON-CAPA Software Developer Instructions</TITLE>
   </HEAD>
   <BODY>
 <H1>LON-CAPA Software Developer Instructions</H1>  <H1>LON-CAPA Software Developer Instructions</H1>
   <BR><I>Written by Scott Harrison, January 17, 2001</I>
   <BR><I>Last updated, January 17, 2001</I>
 <OL>  <OL>
 <LI><A HREF="#Using_CVS">Using CVS</A>  <LI><A HREF="#Using_CVS">Using CVS</A>
 <UL>  <UL>
Line 77  file does. Line 84  file does.
 <FONT COLOR="#008800">  <FONT COLOR="#008800">
 <PRE>  <PRE>
 cd loncom/build  cd loncom/build
 rm -Rf HTML <I>(or alternatively, "make clean")</I>  make build
 make HTML  
 cd HTML  
 <I>(look at the index.html file with a web browser such as Netscape)</I>  
 </PRE>  </PRE>
 </FONT>  </FONT>
 <STRONG>General description of what happens</STRONG>  <STRONG>General description of what happens</STRONG>
Line 88  cd HTML Line 92  cd HTML
 This is the actual make target code.  This is the actual make target code.
 <FONT COLOR="#880000">  <FONT COLOR="#880000">
 <PRE>  <PRE>
 <!-- LONCAPA MAKETARGET=HTML START -->  <!-- LONCAPA MAKETARGET=build START -->
 build:  build:
         perl parse.pl ../../doc/loncapafiles/loncapafiles.html build > Makefile.build          perl parse.pl ../../doc/loncapafiles/loncapafiles.html build > Makefile.build
         make -f Makefile.build all          make -f Makefile.build all
 <!-- LONCAPA MAKETARGET=HTML END -->  <!-- LONCAPA MAKETARGET=build END -->
 </PRE>  </PRE>
 </FONT>  </FONT>
 <TT>loncom/build/parse.pl</TT> reads in all the build information out  <TT>loncom/build/parse.pl</TT> reads in all the build information out
 of <TT>doc/loncapafiles/loncapafiles.html.  A new Makefile named  of <TT>doc/loncapafiles/loncapafiles.html</TT>.  A new Makefile named
 <TT>loncom/build/Makefile.build</TT> is dynamically constructed.  <TT>loncom/build/Makefile.build</TT> is dynamically constructed.
 This dynamically generated Makefile is then run to build/compile  This dynamically generated Makefile is then run to build/compile
 all the software targets from source.  This currently takes 10 minutes  all the software targets from source.  This currently takes 10 minutes
Line 106  all the software targets from source.  T Line 110  all the software targets from source.  T
 <P>  <P>
 Here is information for one file <TT>tth.so</TT> provided in  Here is information for one file <TT>tth.so</TT> provided in
 <TT>doc/loncapafiles/loncapafiles.html</TT>.  <TT>doc/loncapafiles/loncapafiles.html</TT>.
 <FONT COLOR="#660033">  <FONT COLOR="#330066">
 <PRE>  <PRE>
 <BR>&lt;METAGROUP&gt;  <BR>&lt;METAGROUP&gt;
 <BR>&lt;LONCAPA TYPE=LOCATION DIST="redhat6.2" SOURCE="loncom/modules/TexConvert/tthperl/tth.so" TARGET="usr/lib/perl5/site_perl/5.005/tth.so" CATEGORY="system file"&gt;  <BR>&lt;LONCAPA TYPE=LOCATION DIST="redhat6.2" SOURCE="loncom/modules/TexConvert/tthperl/tth.so" TARGET="usr/lib/perl5/site_perl/5.005/tth.so" CATEGORY="system file"&gt;
Line 129  besides documentation). Line 133  besides documentation).
 </P>  </P>
 <P>  <P>
 Here is an example of a dynamically generated <TT>Makefile.build</TT>  Here is an example of a dynamically generated <TT>Makefile.build</TT>
 that builds two LON-CAPA files (one of which is <TT>tth.so</TT>.  that builds two LON-CAPA files (one of which is <TT>tth.so</TT>).
 <FONT COLOR="#660033">  <FONT COLOR="#330066">
 <PRE>  <PRE>
 all: ../homework/caparesponse/capa.so ../modules/TexConvert/tthperl/tth.so   all: ../homework/caparesponse/capa.so ../modules/TexConvert/tthperl/tth.so 
   
Line 148  alwaysrun: Line 152  alwaysrun:
 </P>  </P>
 <LI><A NAME="loncapafiles">  <LI><A NAME="loncapafiles">
     <H2>Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)</H2>      <H2>Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)</H2>
   <STRONG>To add and remove (and alter)</STRONG>
   <P>
   All that you have to do to alter the behavior of the installation is
   edit a single file (<TT>doc/loncapafiles/loncapafiles.html</TT>).
   Adding, removing, and altering files requires proper attention
   to the syntax of file format of course.
   </P>
   <STRONG>File Format</STRONG>
   <P>
   The preceding <A HREF=#"makebuild">"make build"</A> documentation
   gives an example <B>METAGROUP</B> entry describing one particular file.
   All data within <TT>loncapafiles.html</TT> is specified according
   to markup tags.  The format and syntax of <TT>loncapafiles.html</TT>
   is currently best described by the HTML documentation code at the beginning of
   loncapafiles.html (as well as, by example, seeing how various
   information is coded).  All in all, the syntax is quite simple.
   </P>
   <STRONG>Philosophy and notes (the thing nobody reads)</STRONG>
   <P>
   Packaging the software from CVS onto a machine file system requires many
   things:
   <UL>
   <LI>documenting every component of the software,
   <LI>handling CVS <U>source</U> to file system <U>target</U> information
   <LI>handling (according to a hierarchical scheme of grouping) file
   ownership and permissions,
   <LI>handling (according to a hierarchical scheme of grouping) directory
   ownership and permissions,
   <LI>handling symbolic links
   <LI>providing for multiple options of installation targets
   (RedHat versus Debian for instance),
   <LI>providing for different file ownerships and permissions to apply
   to the same file,
   <LI>allowing system software documentation to be automatically generated
   (see information on <A HREF="#makeHTML">"make html"</A>),
   <LI>providing information in an easily adjustable form as new demands
   are made on the software packaging system,
   <LI>providing software package information (for RPM),
   <LI>having information in a format that allows for troubleshooting
   the current status of the machine file system,
   <LI>allow for changes to the structure of the CVS repository,
   <LI>and something that is simple enough for any one to immediately work with,
   without having to learn specifics (or hidden traps) of complicated Makefile's
   or a new macro language (m4?).
   </UL>
   </P>
   <P>
   I looked into, and tried, different ways of accomplishing the above
   including automake and recursive make.  The automake system seemed quite
   complicated (and needlessly so in terms of this project since, by and large,
   it works to coordinate many different types of build/compilation parameters
   whereas we are more concerned with installation parameters).  Recursive make
   has significant deficiencies in the sense that not all the information
   is kept in one place, and there are significant levels of dependency
   between all the things that must be done to keep software packaging
   up to date.  A particularly convincing article I found when looking into
   much of this was
   <A HREF="http://www.pcug.org.au/~millerp/rmch/recu-make-cons-harm.html">
   "Recursive Make Considered Harmful" by Peter Miller</A>.  Complicating
   matters was, at the time, it was unclear as to what categories
   of software files we had, and whether or not the directory structure
   of CVS would remain constant.  With an ever-developing directory structure
   to CVS, I preferred to organize the information on a per-file basis
   as opposed to a per-directory basis (although there is a successful
   implementation of a standard big Makefile in <TT>loncom/Makefile</TT>).
   Additionally, a standard big Makefile assumes certain "normalcy" to
   the directory structure of different potential operating system directories
   (RedHat vs. Debian).
   </P>
   <P>
   If you take time to look at loncapafiles.html
   (and perhaps run the <A HREF="#makeHTML">make HTML</A> command)
   you will find that the organizing information according to the markup
   syntax in <TT>loncapafiles.html</TT> is simple.  Simple is good.
   </P>
   <P>
   <TT>loncom/build/parse.pl</TT> is the script (invoked automatically
   by the various targets in <TT>loncom/build/Makefile</TT>) that reads
   <TT>doc/loncapafiles/loncapafiles.html</TT>.  <TT>parse.pl</TT>
   is capable of reading and returning different types of information
   from <TT>loncapafiles.html</TT> depending on how <TT>parse.pl</TT>
   is invoked.  <TT>parse.pl</TT> has yet to have introduced new sources
   of error, and has been tested in quite a number of ways.  As with
   any parser however, I remain paranoid.
   </P>
   <P>
   My regrets with the current system is that <TT>parse.pl</TT> is
   slow (can take 1 minute to run) and includes a few tidbits of code,
   specific to the make process, that probably should be in
   <TT>loncom/build/Makefile</TT>.  Additionally, <TT>loncapafiles.html</TT>
   should have a DTD and all those other good SGML-ish things (and parsing
   should be done with a real SGML-derived parser).
   </P>
   <P>
   On the plus side, the <TT>parse.pl</TT>-<TT>loncapafiles.html</TT> 
   combination has been working very efficiently and error-free.
   </P>
 <LI><A NAME="configversusnonconfig">  <LI><A NAME="configversusnonconfig">
     <H2>Configurable files versus non-configurable files</H2>      <H2>Configurable files versus non-configurable files</H2>
   <STRONG>Machine-specific information is the difference</STRONG>
   <P>
   The current list of configurable files for the LON-CAPA system is
   /etc/httpd/access.conf, /etc/smb.conf, /etc/ntp.conf, /etc/krb.conf,
   /etc/atalk/config, /etc/ntp/step-tickers, 
   /home/httpd/html/res/adm/includes/copyright.tab, 
   /home/httpd/html/res/adm/includes/un_keyword.tab, 
   /home/httpd/hosts.tab, and
   /home/httpd/spare.tab.
   </P>
   <P>
   All of these configurable files contain machine-specific information.
   For instance, the LON-CAPA system relies on unique host IDs such
   as msua3, s1, s2, msul1, and 103a1 (specified as a "PerlSetVar lonHostID"
   field within /etc/httpd/access.conf).
   Non-configurable files simply do NOT have machine-specific information.
   <STRONG>The impact on updating software</STRONG>
   <P>
   What this means in terms of software updating is that
   <UL>
   <LI>non-configurable files can be simply overwritten with newer versions
   (without "anything" else to worry about),
   <LI>and configurable files must follow these steps to be safely overwritten
   <OL>
   <LI>have their machine specific information saved,
   <LI>be overwritten, and then
   <LI>have their machine specific information restored.
   </UL>
   </P>
 <LI><A NAME="makeinstall">  <LI><A NAME="makeinstall">
     <H2>Updating the non-configurable files on your machine (make install)</H2>      <H2>Updating the non-configurable files on your machine (make install)</H2>
   <STRONG>Commands</STRONG>
   <FONT COLOR="#008800">
   <PRE>
   cd loncom/build
   make install
   </PRE>
   </FONT>
   <STRONG>General description of what happens</STRONG>
   <P>
   This is the actual make target code.
   <FONT COLOR="#880000">
   <PRE>
   <!-- LONCAPA MAKETARGET=install START -->
   install: build
           perl parse.pl ../../doc/loncapafiles/loncapafiles.html install > Makefil
   e.install
           make -f Makefile.install SOURCE="../.." TARGET="" directories
           make -f Makefile.install SOURCE="../.." TARGET="" files
           make -f Makefile.install SOURCE="../.." TARGET="" links
   <!-- LONCAPA MAKETARGET=install END -->
   </PRE>
   </FONT>
   For safety reasons (so as to not mess up a machine's configuration),
   configuration files are NOT installed during this step.  This means
   that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config,
   /home/httpd/html/res/adm/includes/copyright.tab, and
   /home/httpd/spare.tab are not overwritten, but remain as old, non-updated
   copies.  (To automatically update these files and save/restore
   their encoded machine configuration, you must run "make configinstall").
   </P>
 <LI><A NAME="makeconfiginstall">  <LI><A NAME="makeconfiginstall">
     <H2>Updating the configurable files on your machine (make configinstall)</H2>      <H2>Updating the configurable files on your machine (make configinstall)</H2>
   <STRONG>Commands</STRONG>
   <FONT COLOR="#008800">
   <PRE>
   cd loncom/build
   make configinstall
   </PRE>
   </FONT>
   <STRONG>General description of what happens</STRONG>
   <P>
   This is the actual make target code.
   <FONT COLOR="#880000">
   <PRE>
   <!-- LONCAPA MAKETARGET=configinstall START -->
   configinstall: 
           # there is a dependency on having directories in place, but oh well...
           perl parse.pl ../../doc/loncapafiles/loncapafiles.html configinstall > Makefile.configinstall
           make -f Makefile.configinstall SOURCE="../.." TARGET="" configfiles
           perl loncaparestoreconfigurations lasttimestamp
           make -f Makefile.configinstall TARGET="" configpermissions
   <!-- LONCAPA MAKETARGET=configinstall END -->
   </PRE>
   </FONT>
   Configuration files are installed during this step.  This means
   that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config,
   /home/httpd/html/res/adm/includes/copyright.tab, and
   /home/httpd/spare.tab are overwritten.  Before being overwritten,
   a backup copy is made though.  Information is read out of these
   backup copies and restored to the new files by the
   <TT>loncaparestoreconfigurations</TT> script.  To ensure that
   new file permissions and ownerships are installed, a final set of
   <TT>chown</TT> and <TT>chmod</TT> commands are called upon all
   the configuration files.
   </P>
   <STRONG>For the truly paranoid</STRONG>
   <P>
   If you are truly paranoid, you can just make the
   <TT>Makefile.configinstall</TT> file and then save, copy,
   and restore all the configuration values yourself.
   <TT>loncaparestoreconfigurations</TT> is pretty smart though, has yet to
   fail, and besides, a backup copy is always made (time-stamped so that backup
   copies are not overwritten).
   </P>
 <LI><A NAME="makeRPM">  <LI><A NAME="makeRPM">
     <H2>Building RPMs (make RPM)</H2>      <H2>Building RPMs (make RPM)</H2>
   <STRONG>Commands</STRONG>
   <FONT COLOR="#008800">
   <PRE>
   cd loncom/build
   rm -Rf BinaryRootL <I>(or alternatively, "make clean")</I>
   make RPM
   <I>(to subsequently install, you can type commands like
   "rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm")
   </PRE>
   </FONT>
   </P>
   <STRONG>WARNING!!!!!!!!!!!!!!</STRONG>
   <P>
   Never never never never never manually install the
   LON-CAPA-setup-3.1-1.i386.rpm.  This RPM is meant to only be
   installed by the CD installation process (it wipes out
   the existing /etc/passwd file).
   </P>
   <STRONG>Configuration files</STRONG>
   <P>
   Configuration files are automatically saved with the file suffix
   ".rpmsave".  So <TT>/etc/httpd/conf/access.conf</TT> is saved as 
   <TT>/etc/httpd/conf/access.conf.rpmsave</TT>.  You can restore
   the machine-specific configuration information by running
   the <TT>/usr/sbin/loncaparestoreconfigurations</TT>.  However,
   a <B>warning</B> is important here.  If you install an RPM twice
   without restoring your configuration, you will overwrite the
   ".rpmsave" files.
   </P>
   <STRONG>General description of what happens</STRONG>
   <P>
   This is the actual make target code.
   <FONT COLOR="#880000">
   <PRE>
   <!-- LONCAPA MAKETARGET=RPM START -->
   RPM: BinaryRoot
           cat base_file_list.txt | perl make_rpm.pl base 3.1 '' '' BinaryRoot
           cat setup_file_list.txt | perl make_rpm.pl setup 3.1 '' '' BinaryRoot
   
   BinaryRoot:
           perl parse.pl ../../doc/loncapafiles/loncapafiles.html BinaryRoot
   <!-- LONCAPA MAKETARGET=RPM END -->
   </PRE>
   </FONT>
   A <TT>BinaryRoot</TT> directory is generated that reflects the locations,
   ownerships, permissions, and contents for all the CVS source
   files, compiled binaries, directories, and links as they should eventually
   occur on the '/' filesystem location.
   </P>
   <P>
   <TT>loncom/build/make_rpm.pl</TT> is robust (tested over the
   span of months) and, unlike other automated RPM-builders, cleanly
   builds new RPMs without any after-effect of temporary files left
   on the system.  (On the negative side, there are a number of
   LON-CAPA specific customizations inside make_rpm.pl which, for
   the sake of reusability, should eventually be removed).  Two new RPMs
   are generated: LON-CAPA-base-3.1-1.i386 and LON-CAPA-setup-3.1-1.i386.rpm
   (again, never manually install LON-CAPA-setup-3.1-1.i386.rpm).
   </P>
 </OL>  </OL>
   </BODY>
   </HTML>
   
   
   
   
Removed from v.1.8  
changed lines
  Added in v.1.13

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>