Annotation of loncom/build/readme.html, revision 1.16

1.11      harris41    1: <HTML>
                      2: <HEAD>
                      3: <META NAME="GENERATOR" CONTENT="Scott Harrison and Emacs Version 3.14159265358979">
                      4: <TITLE>LON-CAPA Software Developer Instructions</TITLE>
                      5: </HEAD>
                      6: <BODY>
1.1       harris41    7: <H1>LON-CAPA Software Developer Instructions</H1>
1.12      harris41    8: <BR><I>Written by Scott Harrison, January 17, 2001</I>
                      9: <BR><I>Last updated, January 17, 2001</I>
1.2       harris41   10: <OL>
                     11: <LI><A HREF="#Using_CVS">Using CVS</A>
                     12: <UL>
                     13: <LI><A HREF="#cvslog">Logging in and out (cvs login; cvs logout)</A>
                     14: <LI><A HREF="#cvsget">Getting files (cvs update -d)</A>
                     15: <LI><A HREF="#cvsupdate">Updating files (cvs update -d)</A>
                     16: <LI><A HREF="#cvssave">Saving files (cvs commit)</A>
                     17: <LI><A HREF="#cvsadd">Adding files (cvs add)</A>
                     18: <LI><A HREF="#cvsadddir">Adding directories (cvs add/import)</A>
                     19: <LI><A HREF="#cvsnotsure">What to do when you're not sure about your files (cvs update)</A>
                     20: </UL>
                     21: <LI><A HREF="#makeHTML">Viewing the software (make HTML)</A>
                     22: <LI><A HREF="#makebuild">Compiling the software (make build)</A>
                     23: <LI><A HREF="#loncapafiles">Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)</A>
                     24: <LI><A HREF="#configversusnonconfig">Configurable files versus non-configurable files</A>
                     25: <LI><A HREF="#makeinstall">Updating the non-configurable files on your machine (make install)</A>
                     26: <LI><A HREF="#makeconfiginstall">Updating the configurable files on your machine (make configinstall)</A>
                     27: <LI><A HREF="#makeRPM">Building RPMs (make RPM)</A>
                     28: </OL>
                     29: 
                     30: <OL>
                     31: <A NAME="Using_CVS">
                     32: <LI><H2>Using CVS</H2>
1.1       harris41   33: <UL>
1.2       harris41   34: <LI><A NAME="cvslog">
                     35:     <H3>Using CVS: Logging in and out (cvs login; cvs logout)</H3>
                     36: <LI><A NAME="cvsget">
                     37:     <H3>Using CVS: Getting files (cvs update -d)</H3>
                     38: <LI><A NAME="cvsupdate">
1.4       harris41   39:     <H3>Using CVS: Updating files (cvs update -d)</H3>
1.2       harris41   40: <LI><A NAME="cvssave">
1.4       harris41   41:     <H3>Using CVS: Saving files (cvs commit)</H3>
1.2       harris41   42: <LI><A NAME="cvsadd">
1.4       harris41   43:     <H3>Using CVS: Adding files (cvs add)</H3>
1.2       harris41   44: <LI><A NAME="cvsadddir">
1.4       harris41   45:     <H3>Using CVS: Adding directories (cvs add/import)</H3>
1.2       harris41   46: <LI><A NAME="cvsnotsure">
1.4       harris41   47:     <H3>Using CVS: What to do when you're not sure about your files (cvs update)</H3>
1.1       harris41   48: </UL>
1.2       harris41   49: <LI><A NAME="makeHTML">
1.3       harris41   50:     <H2>Viewing the software (make HTML)</H2>
1.5       harris41   51: <STRONG>Commands</STRONG>
                     52: <FONT COLOR="#008800">
                     53: <PRE>
                     54: cd loncom/build
1.8       harris41   55: rm -Rf HTML <I>(or alternatively, "make clean")</I>
1.5       harris41   56: make HTML
                     57: cd HTML
                     58: <I>(look at the index.html file with a web browser such as Netscape)</I>
                     59: </PRE>
                     60: </FONT>
1.6       harris41   61: <STRONG>General description of what happens</STRONG>
                     62: <P>
                     63: This is the actual make target code.
                     64: <FONT COLOR="#880000">
                     65: <PRE>
                     66: <!-- LONCAPA MAKETARGET=HTML START -->
                     67: HTML:
                     68:         install -d HTML
                     69:         cp ../../doc/loncapafiles/*.gif HTML
1.7       harris41   70:         perl parse.pl ../../doc/loncapafiles/loncapafiles.html HTML > HTML/index.html
1.6       harris41   71: <!-- LONCAPA MAKETARGET=HTML END -->
                     72: </PRE>
                     73: </FONT>
1.7       harris41   74: What basically happens is that specially marked-up data in the LON-CAPA
                     75: cvs repository file <TT>doc/loncapafiles.html</TT> is parsed into a more
                     76: viewable format by <TT>loncom/build/parse.pl</TT>.  The resulting
                     77: file gives a very well organized view of all the files, directories,
                     78: links, ownerships, permissions, and brief documentation of what each
                     79: file does.
1.6       harris41   80: </P>
1.2       harris41   81: <LI><A NAME="makebuild">
1.3       harris41   82:     <H2>Compiling the software (make build)</H2>
1.8       harris41   83: <STRONG>Commands</STRONG>
                     84: <FONT COLOR="#008800">
                     85: <PRE>
                     86: cd loncom/build
1.9       harris41   87: make build
1.8       harris41   88: </PRE>
                     89: </FONT>
                     90: <STRONG>General description of what happens</STRONG>
                     91: <P>
                     92: This is the actual make target code.
                     93: <FONT COLOR="#880000">
                     94: <PRE>
1.13      harris41   95: <!-- LONCAPA MAKETARGET=build START -->
1.8       harris41   96: build:
                     97:         perl parse.pl ../../doc/loncapafiles/loncapafiles.html build > Makefile.build
                     98:         make -f Makefile.build all
1.13      harris41   99: <!-- LONCAPA MAKETARGET=build END -->
1.8       harris41  100: </PRE>
                    101: </FONT>
                    102: <TT>loncom/build/parse.pl</TT> reads in all the build information out
1.10      harris41  103: of <TT>doc/loncapafiles/loncapafiles.html</TT>.  A new Makefile named
1.8       harris41  104: <TT>loncom/build/Makefile.build</TT> is dynamically constructed.
                    105: This dynamically generated Makefile is then run to build/compile
                    106: all the software targets from source.  This currently takes 10 minutes
                    107: (depends on the speed of the machine you compile with).
                    108: </P>
                    109: <STRONG>Example</STRONG>
                    110: <P>
                    111: Here is information for one file <TT>tth.so</TT> provided in
                    112: <TT>doc/loncapafiles/loncapafiles.html</TT>.
1.9       harris41  113: <FONT COLOR="#330066">
1.8       harris41  114: <PRE>
                    115: <BR>&lt;METAGROUP&gt;
                    116: <BR>&lt;LONCAPA TYPE=LOCATION DIST="redhat6.2" SOURCE="loncom/modules/TexConvert/tthperl/tth.so" TARGET="usr/lib/perl5/site_perl/5.005/tth.so" CATEGORY="system file"&gt;
                    117: <BR>&lt;DESCRIPTION&gt;
                    118: <BR>shared library file for dynamic loading and unloading of TeX-to-HTML functionality
                    119: <BR>&lt;/DESCRIPTION&gt;
                    120: <BR>&lt;BUILD&gt;
                    121: <BR>loncom/modules/TexConvert/tthperl/commands
                    122: <BR>&lt;/BUILD&gt;
                    123: <BR>&lt;DEPENDENCIES&gt;
                    124: <BR>../tthdynamic/tthfunc.c
                    125: <BR>../ttmdynamic/ttmfunc.c
                    126: <BR>&lt;/DEPENDENCIES&gt;
                    127: </PRE>
                    128: </FONT>
                    129: <TT>loncom/build/parse.pl</TT> sees the <B>BUILD</B> tags and sets up
                    130: a dynamic file <TT>Makefile.build</TT> to run the command inside the
                    131: <B>BUILD</B> tags (currently, <B>DEPENDENCIES</B> is not used for anything
                    132: besides documentation).
                    133: </P>
                    134: <P>
                    135: Here is an example of a dynamically generated <TT>Makefile.build</TT>
1.11      harris41  136: that builds two LON-CAPA files (one of which is <TT>tth.so</TT>).
1.9       harris41  137: <FONT COLOR="#330066">
1.8       harris41  138: <PRE>
                    139: all: ../homework/caparesponse/capa.so ../modules/TexConvert/tthperl/tth.so 
                    140: 
                    141: ../homework/caparesponse/capa.so:  ../homework/caparesponse/caparesponse.c ../ho
                    142: mework/caparesponse/caparesponse.pm alwaysrun
                    143:         cd ../homework/caparesponse/; sh ./commands
                    144: 
                    145: ../modules/TexConvert/tthperl/tth.so:  ../modules/TexConvert/tthperl/../tthdynam
                    146: ic/tthfunc.c ../modules/TexConvert/tthperl/../ttmdynamic/ttmfunc.c
                    147:         cd ../modules/TexConvert/tthperl/; sh ./commands
                    148: 
                    149: alwaysrun:
                    150: </PRE>
                    151: </FONT>
                    152: </P>
1.2       harris41  153: <LI><A NAME="loncapafiles">
1.3       harris41  154:     <H2>Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)</H2>
1.11      harris41  155: <STRONG>To add and remove (and alter)</STRONG>
1.12      harris41  156: <P>
1.11      harris41  157: All that you have to do to alter the behavior of the installation is
                    158: edit a single file (<TT>doc/loncapafiles/loncapafiles.html</TT>).
                    159: Adding, removing, and altering files requires proper attention
                    160: to the syntax of file format of course.
1.12      harris41  161: </P>
1.11      harris41  162: <STRONG>File Format</STRONG>
                    163: <P>
                    164: The preceding <A HREF=#"makebuild">"make build"</A> documentation
                    165: gives an example <B>METAGROUP</B> entry describing one particular file.
                    166: All data within <TT>loncapafiles.html</TT> is specified according
                    167: to markup tags.  The format and syntax of <TT>loncapafiles.html</TT>
                    168: is currently best described by the HTML documentation code at the beginning of
                    169: loncapafiles.html (as well as, by example, seeing how various
                    170: information is coded).  All in all, the syntax is quite simple.
                    171: </P>
                    172: <STRONG>Philosophy and notes (the thing nobody reads)</STRONG>
                    173: <P>
                    174: Packaging the software from CVS onto a machine file system requires many
                    175: things:
                    176: <UL>
                    177: <LI>documenting every component of the software,
                    178: <LI>handling CVS <U>source</U> to file system <U>target</U> information
                    179: <LI>handling (according to a hierarchical scheme of grouping) file
                    180: ownership and permissions,
                    181: <LI>handling (according to a hierarchical scheme of grouping) directory
                    182: ownership and permissions,
                    183: <LI>handling symbolic links
                    184: <LI>providing for multiple options of installation targets
                    185: (RedHat versus Debian for instance),
                    186: <LI>providing for different file ownerships and permissions to apply
                    187: to the same file,
                    188: <LI>allowing system software documentation to be automatically generated
                    189: (see information on <A HREF="#makeHTML">"make html"</A>),
                    190: <LI>providing information in an easily adjustable form as new demands
                    191: are made on the software packaging system,
                    192: <LI>providing software package information (for RPM),
                    193: <LI>having information in a format that allows for troubleshooting
                    194: the current status of the machine file system,
                    195: <LI>allow for changes to the structure of the CVS repository,
                    196: <LI>and something that is simple enough for any one to immediately work with,
                    197: without having to learn specifics (or hidden traps) of complicated Makefile's
                    198: or a new macro language (m4?).
                    199: </UL>
                    200: </P>
                    201: <P>
                    202: I looked into, and tried, different ways of accomplishing the above
                    203: including automake and recursive make.  The automake system seemed quite
                    204: complicated (and needlessly so in terms of this project since, by and large,
                    205: it works to coordinate many different types of build/compilation parameters
                    206: whereas we are more concerned with installation parameters).  Recursive make
                    207: has significant deficiencies in the sense that not all the information
                    208: is kept in one place, and there are significant levels of dependency
                    209: between all the things that must be done to keep software packaging
                    210: up to date.  A particularly convincing article I found when looking into
                    211: much of this was
                    212: <A HREF="http://www.pcug.org.au/~millerp/rmch/recu-make-cons-harm.html">
                    213: "Recursive Make Considered Harmful" by Peter Miller</A>.  Complicating
                    214: matters was, at the time, it was unclear as to what categories
                    215: of software files we had, and whether or not the directory structure
                    216: of CVS would remain constant.  With an ever-developing directory structure
                    217: to CVS, I preferred to organize the information on a per-file basis
                    218: as opposed to a per-directory basis (although there is a successful
                    219: implementation of a standard big Makefile in <TT>loncom/Makefile</TT>).
                    220: Additionally, a standard big Makefile assumes certain "normalcy" to
                    221: the directory structure of different potential operating system directories
                    222: (RedHat vs. Debian).
                    223: </P>
                    224: <P>
                    225: If you take time to look at loncapafiles.html
                    226: (and perhaps run the <A HREF="#makeHTML">make HTML</A> command)
                    227: you will find that the organizing information according to the markup
                    228: syntax in <TT>loncapafiles.html</TT> is simple.  Simple is good.
                    229: </P>
                    230: <P>
                    231: <TT>loncom/build/parse.pl</TT> is the script (invoked automatically
                    232: by the various targets in <TT>loncom/build/Makefile</TT>) that reads
                    233: <TT>doc/loncapafiles/loncapafiles.html</TT>.  <TT>parse.pl</TT>
                    234: is capable of reading and returning different types of information
                    235: from <TT>loncapafiles.html</TT> depending on how <TT>parse.pl</TT>
                    236: is invoked.  <TT>parse.pl</TT> has yet to have introduced new sources
                    237: of error, and has been tested in quite a number of ways.  As with
                    238: any parser however, I remain paranoid.
                    239: </P>
                    240: <P>
                    241: My regrets with the current system is that <TT>parse.pl</TT> is
                    242: slow (can take 1 minute to run) and includes a few tidbits of code,
                    243: specific to the make process, that probably should be in
                    244: <TT>loncom/build/Makefile</TT>.  Additionally, <TT>loncapafiles.html</TT>
                    245: should have a DTD and all those other good SGML-ish things (and parsing
                    246: should be done with a real SGML-derived parser).
                    247: </P>
                    248: <P>
                    249: On the plus side, the <TT>parse.pl</TT>-<TT>loncapafiles.html</TT> 
                    250: combination has been working very efficiently and error-free.
                    251: </P>
1.2       harris41  252: <LI><A NAME="configversusnonconfig">
1.3       harris41  253:     <H2>Configurable files versus non-configurable files</H2>
1.12      harris41  254: <STRONG>Machine-specific information is the difference</STRONG>
                    255: <P>
                    256: The current list of configurable files for the LON-CAPA system is
1.13      harris41  257: /etc/httpd/access.conf, /etc/smb.conf, /etc/ntp.conf, /etc/krb.conf,
                    258: /etc/atalk/config, /etc/ntp/step-tickers, 
                    259: /home/httpd/html/res/adm/includes/copyright.tab, 
                    260: /home/httpd/html/res/adm/includes/un_keyword.tab, 
                    261: /home/httpd/hosts.tab, and
                    262: /home/httpd/spare.tab.
                    263: </P>
                    264: <P>
1.12      harris41  265: All of these configurable files contain machine-specific information.
                    266: For instance, the LON-CAPA system relies on unique host IDs such
                    267: as msua3, s1, s2, msul1, and 103a1 (specified as a "PerlSetVar lonHostID"
                    268: field within /etc/httpd/access.conf).
                    269: Non-configurable files simply do NOT have machine-specific information.
                    270: <STRONG>The impact on updating software</STRONG>
                    271: <P>
                    272: What this means in terms of software updating is that
                    273: <UL>
                    274: <LI>non-configurable files can be simply overwritten with newer versions
                    275: (without "anything" else to worry about),
                    276: <LI>and configurable files must follow these steps to be safely overwritten
                    277: <OL>
                    278: <LI>have their machine specific information saved,
                    279: <LI>be overwritten, and then
                    280: <LI>have their machine specific information restored.
1.14      harris41  281: </OL>
1.12      harris41  282: </UL>
                    283: </P>
1.2       harris41  284: <LI><A NAME="makeinstall">
1.3       harris41  285:     <H2>Updating the non-configurable files on your machine (make install)</H2>
1.13      harris41  286: <STRONG>Commands</STRONG>
                    287: <FONT COLOR="#008800">
                    288: <PRE>
                    289: cd loncom/build
                    290: make install
                    291: </PRE>
                    292: </FONT>
                    293: <STRONG>General description of what happens</STRONG>
                    294: <P>
                    295: This is the actual make target code.
                    296: <FONT COLOR="#880000">
                    297: <PRE>
                    298: <!-- LONCAPA MAKETARGET=install START -->
                    299: install: build
                    300:         perl parse.pl ../../doc/loncapafiles/loncapafiles.html install > Makefil
                    301: e.install
                    302:         make -f Makefile.install SOURCE="../.." TARGET="" directories
                    303:         make -f Makefile.install SOURCE="../.." TARGET="" files
                    304:         make -f Makefile.install SOURCE="../.." TARGET="" links
                    305: <!-- LONCAPA MAKETARGET=install END -->
                    306: </PRE>
                    307: </FONT>
                    308: For safety reasons (so as to not mess up a machine's configuration),
                    309: configuration files are NOT installed during this step.  This means
                    310: that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config,
                    311: /home/httpd/html/res/adm/includes/copyright.tab, and
                    312: /home/httpd/spare.tab are not overwritten, but remain as old, non-updated
                    313: copies.  (To automatically update these files and save/restore
                    314: their encoded machine configuration, you must run "make configinstall").
                    315: </P>
1.2       harris41  316: <LI><A NAME="makeconfiginstall">
1.3       harris41  317:     <H2>Updating the configurable files on your machine (make configinstall)</H2>
1.13      harris41  318: <STRONG>Commands</STRONG>
                    319: <FONT COLOR="#008800">
                    320: <PRE>
                    321: cd loncom/build
                    322: make configinstall
                    323: </PRE>
                    324: </FONT>
                    325: <STRONG>General description of what happens</STRONG>
                    326: <P>
                    327: This is the actual make target code.
                    328: <FONT COLOR="#880000">
                    329: <PRE>
                    330: <!-- LONCAPA MAKETARGET=configinstall START -->
                    331: configinstall: 
                    332:         # there is a dependency on having directories in place, but oh well...
                    333:         perl parse.pl ../../doc/loncapafiles/loncapafiles.html configinstall > Makefile.configinstall
                    334:         make -f Makefile.configinstall SOURCE="../.." TARGET="" configfiles
                    335:         perl loncaparestoreconfigurations lasttimestamp
                    336:         make -f Makefile.configinstall TARGET="" configpermissions
                    337: <!-- LONCAPA MAKETARGET=configinstall END -->
                    338: </PRE>
                    339: </FONT>
                    340: Configuration files are installed during this step.  This means
                    341: that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config,
                    342: /home/httpd/html/res/adm/includes/copyright.tab, and
                    343: /home/httpd/spare.tab are overwritten.  Before being overwritten,
                    344: a backup copy is made though.  Information is read out of these
                    345: backup copies and restored to the new files by the
                    346: <TT>loncaparestoreconfigurations</TT> script.  To ensure that
                    347: new file permissions and ownerships are installed, a final set of
1.15      harris41  348: <TT>chown</TT> and <TT>chmod</TT> commands are called for each of 
1.13      harris41  349: the configuration files.
                    350: </P>
                    351: <STRONG>For the truly paranoid</STRONG>
                    352: <P>
                    353: If you are truly paranoid, you can just make the
                    354: <TT>Makefile.configinstall</TT> file and then save, copy,
                    355: and restore all the configuration values yourself.
                    356: <TT>loncaparestoreconfigurations</TT> is pretty smart though, has yet to
                    357: fail, and besides, a backup copy is always made (time-stamped so that backup
                    358: copies are not overwritten).
                    359: </P>
1.2       harris41  360: <LI><A NAME="makeRPM">
1.3       harris41  361:     <H2>Building RPMs (make RPM)</H2>
1.13      harris41  362: <STRONG>Commands</STRONG>
                    363: <FONT COLOR="#008800">
                    364: <PRE>
                    365: cd loncom/build
1.16    ! harris41  366: rm -Rf BinaryRoot <I>(or alternatively, "make clean")</I>
1.13      harris41  367: make RPM
1.16    ! harris41  368:    <I>(to subsequently install, you can type commands like
        !           369:        "rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm")</I>
1.13      harris41  370: </PRE>
                    371: </FONT>
                    372: </P>
                    373: <STRONG>WARNING!!!!!!!!!!!!!!</STRONG>
                    374: <P>
                    375: Never never never never never manually install the
                    376: LON-CAPA-setup-3.1-1.i386.rpm.  This RPM is meant to only be
                    377: installed by the CD installation process (it wipes out
                    378: the existing /etc/passwd file).
                    379: </P>
                    380: <STRONG>Configuration files</STRONG>
                    381: <P>
                    382: Configuration files are automatically saved with the file suffix
                    383: ".rpmsave".  So <TT>/etc/httpd/conf/access.conf</TT> is saved as 
                    384: <TT>/etc/httpd/conf/access.conf.rpmsave</TT>.  You can restore
                    385: the machine-specific configuration information by running
                    386: the <TT>/usr/sbin/loncaparestoreconfigurations</TT>.  However,
                    387: a <B>warning</B> is important here.  If you install an RPM twice
                    388: without restoring your configuration, you will overwrite the
                    389: ".rpmsave" files.
                    390: </P>
                    391: <STRONG>General description of what happens</STRONG>
                    392: <P>
                    393: This is the actual make target code.
                    394: <FONT COLOR="#880000">
                    395: <PRE>
                    396: <!-- LONCAPA MAKETARGET=RPM START -->
1.16    ! harris41  397: RPM: BinaryRoot
        !           398:         cat base_file_list.txt | perl make_rpm.pl base 3.1 '' '' BinaryRoot
        !           399:         cat setup_file_list.txt | perl make_rpm.pl setup 3.1 '' '' BinaryRoot
1.13      harris41  400: 
1.16    ! harris41  401: BinaryRoot:
        !           402:         perl parse.pl ../../doc/loncapafiles/loncapafiles.html BinaryRoot
        !           403: <!-- LONCAPA MAKETARGET=RPM END -->
1.13      harris41  404: </PRE>
                    405: </FONT>
                    406: A <TT>BinaryRoot</TT> directory is generated that reflects the locations,
                    407: ownerships, permissions, and contents for all the CVS source
                    408: files, compiled binaries, directories, and links as they should eventually
                    409: occur on the '/' filesystem location.
                    410: </P>
                    411: <P>
                    412: <TT>loncom/build/make_rpm.pl</TT> is robust (tested over the
                    413: span of months) and, unlike other automated RPM-builders, cleanly
                    414: builds new RPMs without any after-effect of temporary files left
                    415: on the system.  (On the negative side, there are a number of
                    416: LON-CAPA specific customizations inside make_rpm.pl which, for
                    417: the sake of reusability, should eventually be removed).  Two new RPMs
                    418: are generated: LON-CAPA-base-3.1-1.i386 and LON-CAPA-setup-3.1-1.i386.rpm
                    419: (again, never manually install LON-CAPA-setup-3.1-1.i386.rpm).
                    420: </P>
1.2       harris41  421: </OL>
1.11      harris41  422: </BODY>
                    423: </HTML>
1.12      harris41  424: 
                    425: 
                    426: 
                    427: 

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>