File:
[LON-CAPA] /
loncom /
build /
filecompare.pl
Revision
1.15:
download - view:
text,
annotated -
select for diffs
Mon Jun 9 05:14:12 2014 UTC (9 years, 10 months ago) by
raeburn
Branches:
MAIN
CVS tags:
version_2_12_X,
version_2_11_X,
version_2_11_4_uiuc,
version_2_11_4_msu,
version_2_11_4,
version_2_11_3_uiuc,
version_2_11_3_msu,
version_2_11_3,
version_2_11_2_uiuc,
version_2_11_2_msu,
version_2_11_2_educog,
version_2_11_2,
version_2_11_1,
version_2_11_0,
HEAD
- Bug 6535
- For 2.10.1 -> 2.11.0 transition inhibit generation of unwanted WARNINGS
(and lpmlsave files) for cases where 2.10.1 was installed recently (and
hence files have recent timestamps), by storing old release number
in /etc/loncapa-release.prev and then doing sha1-based checksum comparison
with checksums for files in dns_checksums/2.10.1.tab
1: #!/usr/bin/perl
2:
3: # The LearningOnline Network with CAPA
4: # filecompare.pl - script used to help probe and compare file statistics
5: #
6: # $Id: filecompare.pl,v 1.15 2014/06/09 05:14:12 raeburn Exp $
7: #
8: # Copyright Michigan State University Board of Trustees
9: #
10: # This file is part of the LearningOnline Network with CAPA (LON-CAPA).
11: #
12: # LON-CAPA is free software; you can redistribute it and/or modify
13: # it under the terms of the GNU General Public License as published by
14: # the Free Software Foundation; either version 2 of the License, or
15: # (at your option) any later version.
16: #
17: # LON-CAPA is distributed in the hope that it will be useful,
18: # but WITHOUT ANY WARRANTY; without even the implied warranty of
19: # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
20: # GNU General Public License for more details.
21: #
22: # You should have received a copy of the GNU General Public License
23: # along with LON-CAPA; if not, write to the Free Software
24: # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
25: #
26: # /home/httpd/html/adm/gpl.txt
27: #
28: # http://www.lon-capa.org/
29: #
30: # YEAR=2001
31: # 9/27, 10/24, 10/25, 11/4 Scott Harrison
32: # 11/14 Guy Albertelli
33: # 11/16,11/17 Scott Harrison
34: # 12/3,12/5 Scott Harrison
35: #
36: ###
37:
38: ###############################################################################
39: ## ##
40: ## ORGANIZATION OF THIS PERL SCRIPT ##
41: ## ##
42: ## 1. Invocation ##
43: ## 2. Notes ##
44: ## 3. Dependencies ##
45: ## 4. Process command line arguments ##
46: ## 5. Process file/dir location arguments ##
47: ## 6. Process comparison restrictions ##
48: ## 7. Define output and measure subroutines ##
49: ## 8. Loop through files and calculate differences ##
50: ## 9. Subroutines ##
51: ## 10. POD (plain old documentation, CPAN style) ##
52: ## ##
53: ###############################################################################
54:
55: # ------------------------------------------------------------------ Invocation
56: my $invocation=<<END;
57: filecompare.pl [ options ... ] [FILE1] [FILE2] [ restrictions ... ]
58: or
59: filecompare.pl [ options ... ] [DIR1] [DIR2] [ restrictions ... ]
60: or
61: filecompare.pl [ options ... ] -s TARGET=[target] SOURCE=[source] MODE=[mode]
62: LOC1 LOC2
63:
64: Restrictions: a list of space separated values (after the file/dir names)
65: can restrict the comparison.
66: These values can be: existence, cvstime, age, md5sum, size, lines,
67: and/or diffs.
68:
69: Options (before file/dir names):
70: -p show all files that have the same comparison
71: -n show all files that have different comparisons
72: -a show all files (with comparisons)
73: -q only show file names (based on first file/dir)
74: -v verbose mode (default)
75: -bN buildmode (controls EXIT code of this script; 0 unless...)
76: N=1: md5sum=same --> 1; cvstime<0 --> 2
77: N=2: same as N=1 except without md5sum
78: N=3: md5sum=same --> 1; age<0 --> 2
79: N=4: cvstime>0 --> 2
80: N=5: md5sum=same --> 1; cvstime<0 and sha1sum from dns_checksums=different --> 2;
81: N=6: md5sum=same --> 1; age<0 and sha1sum from dns_checksums=different --> 2;
82:
83: The third way to pass arguments is set by the -s flag.
84: filecompare.pl -s SOURCE=[source] TARGET=[target] MODE=[mode] LOC1 LOC2
85:
86: TARGET corresponds to the root path of LOC2. SOURCE corresponds to
87: the root path of LOC1. MODE can either be file, directory, link, or fileglob.
88:
89: END
90: unless (@ARGV) {
91: print $invocation;
92: exit 1;
93: }
94:
95: # ----------------------------------------------------------------------- Notes
96: #
97: # What are all the different ways to compare two files and how to look
98: # at the differences?
99: #
100: # Ways of comparison:
101: # existence similarity
102: # cvs time similarity (1st arg treated as CVS source; only for buildmode)
103: # age similarity (modification time)
104: # md5sum similarity
105: # size similarity (bytes)
106: # line count difference
107: # number of different lines
108: # sha1sum similarity to checksum for same file in installed version
109: #
110: # Quantities of comparison:
111: # existence (no,yes); other values become 'n/a'
112: # cvstime in seconds
113: # age in seconds
114: # md5sum ("same" or "different")
115: # size similarity (byte difference)
116: # line count difference (integer)
117: # number of different lines (integer)
118: # sha1sum ("same" or "different")
119:
120: # ---------------------------------------------------------------- Dependencies
121: # implementing from unix command line (assuming bash)
122: # md5sum, diff, wc -l, sha1sum
123:
124: # ---------------------------------------------- Process command line arguments
125: # Flags (before file/dir names):
126: # -p show all files the same
127: # -n show all files different
128: # -a show all files (with comparisons)
129: # -q only show file names (based on first file/dir)
130: # -v verbose mode (default)
131: # -bN build/install mode (returns exitcode)
132: # -s status checking mode for lpml
133:
134: my $verbose='1';
135: my $show='all';
136: my $buildmode=0;
137: my $statusmode=0;
138: ALOOP: while (@ARGV) {
139: my $flag;
140: if ($ARGV[0]=~/^\-(\w)/) {
141: $flag=$1;
142: if ($flag eq 'b') {
143: $ARGV[0]=~/^\-\w(\d)/;
144: $buildmode=$1;
145: shift @ARGV;
146: next ALOOP;
147: }
148: shift @ARGV;
149: SWITCH: {
150: $verbose=0, last SWITCH if $flag eq 'q';
151: $verbose=1, last SWITCH if $flag eq 'v';
152: $show='same', last SWITCH if $flag eq 'p';
153: $show='different', last SWITCH if $flag eq 'n';
154: $show='all', last SWITCH if $flag eq 'a';
155: $statusmode=1, last SWITCH if $flag eq 's';
156: print($invocation), exit(1);
157: }
158: }
159: else {
160: last;
161: }
162: }
163: dowarn('Verbose: '.$verbose."\n");
164: dowarn('Show: '.$show."\n");
165:
166: my @files;
167: my $loc1;
168: my $loc2;
169: my $dirmode='directories';
170: # ----------------------------------------- If status checking mode for lpml
171: my ($sourceroot,$targetroot,$mode,$sourceglob,$targetglob);
172: my ($source,$target);
173: if ($statusmode==1) {
174: ($sourceroot,$targetroot,$mode,$sourceglob,$targetglob)=splice(@ARGV,0,5);
175: $targetroot.='/' if $targetroot!~/\/$/;
176: $sourceroot=~s/^SOURCE\=//;
177: $targetroot=~s/^TARGET\=//;
178: $source=$sourceroot.'/'.$sourceglob;
179: $target=$targetroot.''.$targetglob;
180: # print "SOURCE: $source\n";
181: # print "TARGET: $target\n";
182: if ($mode eq 'MODE=fileglob') {
183: $loc1=$source;$loc1=~s/\/[^\/]*$// if length($loc1)>2;
184: $loc2=$target;$loc2=~s/\/[^\/]*$// if length($loc2)>2;
185: @files=map {s/^$loc1\///;$_} glob($source);
186: $dirmode='directories';
187: }
188: elsif ($mode eq 'MODE=file') {
189: $loc1=$source;
190: $loc2=$target;
191: $dirmode='files';
192: @files=($loc1);
193: }
194: }
195: else {
196:
197: # ----------------------------------------- Process file/dir location arguments
198: # FILE1 FILE2 or DIR1 DIR2
199: $loc1=shift @ARGV;
200: $loc2=shift @ARGV;
201: unless ($loc1 and $loc2) {
202: print "LOC1: $loc1\nLOC2: $loc2\n";
203: print($invocation), exit(1);
204: }
205: if (-f $loc1) {
206: $dirmode='files';
207: @files=($loc1);
208: }
209: else {
210: if (-e $loc1) {
211: @files=`find $loc1 -type f`;
212: }
213: else {
214: @files=($loc1);
215: }
216: map {chomp; s/^$loc1\///; $_} @files;
217: }
218: dowarn('Processing for mode: '.$dirmode."\n");
219: dowarn('Location #1: '.$loc1."\n");
220: dowarn('Location #2: '.$loc2."\n");
221: }
222: # --------------------------------------------- Process comparison restrictions
223: # A list of space separated values (after the file/dir names)
224: # can restrict the comparison.
225: my %rhash=('existence'=>0,'cvstime'=>0,'md5sum'=>0,'age'=>0,'size'=>0,
226: 'lines'=>0,'diffs'=>0);
227: my %restrict;
228: while (@ARGV) {
229: my $r=shift @ARGV;
230: if ($rhash{$r}==0) {$restrict{$r}=1;}
231: else {print($invocation), exit(1);}
232: }
233: if (%restrict) {
234: dowarn('Restricting comparison to: '.
235: join(' ',keys %restrict)."\n");
236: }
237:
238: # --------------------------------------- Define output and measure subroutines
239: my %OUTPUT=(
240: 'existence'=>( sub {print 'existence: '.@_[0]; return;}),
241: 'md5sum'=>(sub {print 'md5sum: '.@_[0];return;}),
242: 'cvstime'=>(sub {print 'cvstime: '.@_[0];return;}),
243: 'age'=>(sub {print 'age: '.@_[0];return;}),
244: 'size'=>(sub {print 'size: '.@_[0];return;}),
245: 'lines'=>(sub {print 'lines: '.@_[0];return;}),
246: 'diffs'=>(sub {print 'diffs: '.@_[0];return;}),
247: 'sha1sum'=>(sub {print 'sha1sum: '.@_[0];return;}),
248: );
249:
250: my %MEASURE=(
251: 'existence' => ( sub { my ($file1,$file2)=@_;
252: my $rv1=(-e $file1)?'yes':'no';
253: my $rv2=(-e $file2)?'yes':'no';
254: return ($rv1,$rv2); } ),
255: 'md5sum'=>( sub { my ($file1,$file2)=@_;
256: my ($rv1)=split(/ /,`md5sum $file1`); chop $rv1;
257: my ($rv2)=split(/ /,`md5sum $file2`); chop $rv2;
258: return ($rv1,$rv2); } ),
259: 'cvstime'=>( sub { my ($file1,$file2)=@_;
260: my $rv1=&cvstime($file1);
261: my @a=stat($file2); my $gmt=gmtime($a[9]);
262: my $rv2=&utctime($gmt);
263: return ($rv1,$rv2); } ),
264: 'age'=>( sub { my ($file1,$file2)=@_;
265: my @a=stat($file1); my $rv1=$a[9];
266: @a=stat($file2); my $rv2=$a[9];
267: return ($rv1,$rv2); } ),
268: 'size'=>( sub { my ($file1,$file2)=@_;
269: my @a=stat($file1); my $rv1=$a[7];
270: @a=stat($file2); my $rv2=$a[7];
271: return ($rv1,$rv2); } ),
272: 'lines'=>( sub { my ($file1,$file2)=@_;
273: my $rv1=`wc -l $file1`; chop $rv1;
274: my $rv2=`wc -l $file2`; chop $rv2;
275: return ($rv1,$rv2); } ),
276: 'diffs'=>( sub { my ($file1,$file2)=@_;
277: return (0,0);
278: my $rv1=`diff $file1 $file2 | grep '^<' | wc -l`;
279: chop $rv1; $rv1=~s/^\s+//; $rv1=~s/\s+$//;
280: my $rv2=`diff $file1 $file2 | grep '^>' | wc -l`;
281: chop $rv2; $rv2=~s/^\s+//; $rv2=~s/\s+$//;
282: return ($rv1,$rv2); } ),
283: 'sha1sum'=>( sub { my ($file1,$file2)=@_;
284: if (open(my $fh,"</etc/loncapa-release.prev")) {
285: my $loncaparev = <$fh>;
286: close($fh);
287: chomp($loncaparev);
288: $loncaparev =~ s/^\QLON-CAPA release \E//;
289: $loncaparev =~ s/\-\d{8}$//;
290: my ($rv1)=split(/ /,`sha1sum $file2`); chomp $rv1;
291: my $checksum;
292: if ($loncaparev eq 'CVS_HEAD') {
293: return ($rv1,$checksum);
294: }
295: elsif (open(my $fh,"<../../loncom/dns_checksums/$loncaparev.tab")) {
296: while (<$fh>) {
297: chomp();
298: if (/^\Q$file2\E,[\d\.]+,(\w+)$/) {
299: $checksum = $1;
300: last;
301: }
302: }
303: close($fh);
304: return ($rv1,$checksum);
305: }
306: }
307: return('n/a','n/a'); }),
308: );
309:
310: FLOOP: foreach my $file (@files) {
311: my $file1;
312: my $file2;
313: if ($dirmode eq 'directories') {
314: $file1=$loc1.'/'.$file;
315: $file2=$loc2.'/'.$file;
316: }
317: else {
318: $file1=$loc1;
319: $file2=$loc2;
320: }
321: my ($existence1,$existence2)=&{$MEASURE{'existence'}}($file1,$file2);
322: my $existence=$existence1.':'.$existence2;
323: my ($cvstime,$md5sum,$age,$size,$lines,$diffs,$sha1sum);
324: if ($existence1 eq 'no' or $existence2 eq 'no') {
325: $md5sum='n/a';
326: $age='n/a';
327: $cvstime='n/a';
328: $size='n/a';
329: $lines='n/a';
330: $diffs='n/a';
331: $sha1sum='n/a';
332: }
333: else {
334: if ($buildmode) {
335: my ($cvstime1,$cvstime2)=&{$MEASURE{'cvstime'}}($file1,$file2);
336: $cvstime=$cvstime1-$cvstime2;
337: my ($sha1sumfile,$checksum) = &{$MEASURE{'sha1sum'}}($file1,$file2);
338: $sha1sum='n/a';
339: unless ($checksum eq 'n/a') {
340: if ($sha1sumfile && $checksum) {
341: if ($sha1sumfile eq $checksum) {
342: $sha1sum='same';
343: }
344: else {
345: $sha1sum='different';
346: }
347: }
348: }
349: }
350: else {
351: $cvstime='n/a';
352: $sha1sum='n/a';
353: }
354: my ($age1,$age2)=&{$MEASURE{'age'}}($file1,$file2);
355: $age=$age1-$age2;
356: my ($md5sum1,$md5sum2)=&{$MEASURE{'md5sum'}}($file1,$file2);
357: if ($md5sum1 eq $md5sum2) {
358: $md5sum='same';
359: $size=0;
360: $lines=0;
361: $diffs='0:0';
362: }
363: elsif ($md5sum1 ne $md5sum2) {
364: $md5sum='different';
365: my ($size1,$size2)=&{$MEASURE{'size'}}($file1,$file2);
366: $size=$size1-$size2;
367: my ($lines1,$lines2)=&{$MEASURE{'lines'}}($file1,$file2);
368: $lines=$lines1-$lines2;
369: my ($diffs1,$diffs2)=&{$MEASURE{'diffs'}}($file1,$file2);
370: $diffs=$diffs1.':'.$diffs2;
371: }
372: }
373: my $showflag=0;
374: if ($show eq 'all') {
375: $showflag=1;
376: }
377: if ($show eq 'different') {
378: my @ks=(keys %restrict);
379: unless (@ks) {
380: @ks=('existence','cvstime','md5sum','age','size','lines','diffs');
381: }
382: FLOOP2: for my $key (@ks) {
383: if ($key eq 'existence') {
384: if ($existence ne 'yes:yes') {
385: $showflag=1;
386: }
387: }
388: elsif ($key eq 'md5sum') {
389: if ($md5sum ne 'same') {
390: $showflag=1;
391: }
392: }
393: elsif ($key eq 'cvstime' and $buildmode) {
394: if ($cvstime!=0) {
395: $showflag=1;
396: }
397: }
398: elsif ($key eq 'age') {
399: if ($age!=0) {
400: $showflag=1;
401: }
402: }
403: elsif ($key eq 'size') {
404: if ($size!=0) {
405: $showflag=1;
406: }
407: }
408: elsif ($key eq 'lines') {
409: if ($lines!=0) {
410: $showflag=1;
411: }
412: }
413: elsif ($key eq 'diffs') {
414: if ($diffs ne '0:0') {
415: $showflag=1;
416: }
417: }
418: if ($showflag) {
419: last FLOOP2;
420: }
421: }
422: }
423: elsif ($show eq 'same') {
424: my @ks=(keys %restrict);
425: unless (@ks) {
426: @ks=('existence','md5sum','cvstime','age','size','lines','diffs');
427: }
428: my $showcount=length(@ks);
429: $showcount-- unless $buildmode;
430: FLOOP3: for my $key (@ks) {
431: if ($key eq 'existence') {
432: if ($existence ne 'yes:yes') {
433: $showcount--;
434: }
435: }
436: elsif ($key eq 'md5sum') {
437: if ($md5sum ne 'same') {
438: $showcount--;
439: }
440: }
441: elsif ($key eq 'cvstime' and $buildmode) {
442: if ($cvstime!=0) {
443: $showcount--;
444: }
445: }
446: elsif ($key eq 'age') {
447: if ($age!=0) {
448: $showcount--;
449: }
450: }
451: elsif ($key eq 'size') {
452: if ($size!=0) {
453: $showcount--;
454: }
455: }
456: elsif ($key eq 'lines') {
457: if ($lines!=0) {
458: $showcount--;
459: }
460: }
461: elsif ($key eq 'diffs') {
462: if ($diffs ne '0:0') {
463: $showcount--;
464: }
465: }
466: }
467: if ($showcount==0) {
468: $showflag=1;
469: }
470: }
471: if ($buildmode==1) { # -b1
472: if ($md5sum eq 'same') {
473: exit(1);
474: }
475: elsif ($cvstime<0) {
476: exit(2);
477: }
478: else {
479: exit(0);
480: }
481: }
482: elsif ($buildmode==2) { # -b2
483: if ($cvstime<0) {
484: exit(2);
485: }
486: else {
487: exit(0);
488: }
489: }
490: elsif ($buildmode==3) { # -b3
491: if ($md5sum eq 'same') {
492: exit(1);
493: }
494: elsif ($age<0) {
495: exit(2);
496: }
497: else {
498: exit(0);
499: }
500: }
501: elsif ($buildmode==4) { # -b4
502: if ($existence=~/no$/) {
503: exit(3);
504: }
505: elsif ($cvstime>0) {
506: exit(2);
507: }
508: elsif ($existence=~/^no/) {
509: exit(1);
510: }
511: else {
512: exit(0);
513: }
514: }
515: elsif ($buildmode==5) { # -b5
516: if ($md5sum eq 'same') {
517: exit(1);
518: }
519: elsif ($cvstime<0) {
520: if ($sha1sum eq 'same') {
521: exit(0);
522: }
523: else {
524: exit(2);
525: }
526: }
527: else {
528: exit(0);
529: }
530: }
531: elsif ($buildmode==6) { # -b6
532: if ($md5sum eq 'same') {
533: exit(1);
534: }
535: elsif ($age<0) {
536: if ($sha1sum eq 'same') {
537: exit(0);
538: }
539: else {
540: exit(2);
541: }
542: }
543: else {
544: exit(0);
545: }
546: }
547:
548: if ($showflag) {
549: print "$file";
550: if ($verbose==1) {
551: print "\t";
552: print &{$OUTPUT{'existence'}}($existence);
553: print "\t";
554: print &{$OUTPUT{'cvstime'}}($cvstime);
555: print "\t";
556: print &{$OUTPUT{'age'}}($age);
557: print "\t";
558: print &{$OUTPUT{'md5sum'}}($md5sum);
559: print "\t";
560: print &{$OUTPUT{'sha1sum'}}($sha1sum);
561: print "\t";
562: print &{$OUTPUT{'size'}}($size);
563: print "\t";
564: print &{$OUTPUT{'lines'}}($lines);
565: print "\t";
566: print &{$OUTPUT{'diffs'}}($diffs);
567:
568: }
569: print "\n";
570: }
571: }
572:
573: # ----------------------------------------------------------------- Subroutines
574:
575: sub cvstime {
576: my ($f)=@_;
577: my $path; my $file;
578: if ($f=~/^(.*\/)(.*?)$/) {
579: $f=~/^(.*\/)(.*?)$/;
580: ($path,$file)=($1,$2);
581: }
582: else {
583: $file=$f; $path='';
584: }
585: my $cvstime;
586: if ($buildmode!=3) {
587: my $entry=`grep '^/$file/' ${path}CVS/Entries 2>/dev/null`;
588: # or
589: # die('*** WARNING *** cannot grep against '.${path}.
590: # 'CVS/Entries for ' .$file . "\n");
591: if ($entry) {
592: my @fields=split(/\//,$entry);
593: $cvstime=`date -d '$fields[3] UTC' --utc +"%s"`;
594: chomp $cvstime;
595: }
596: else {
597: $cvstime='n/a';
598: }
599: }
600: else {
601: $cvstime='n/a';
602: }
603: return $cvstime;
604: }
605:
606: sub utctime {
607: my ($f)=@_;
608: my $utctime=`date -d '$f UTC' --utc +"%s"`;
609: chomp $utctime;
610: return $utctime;
611: }
612:
613: sub dowarn {
614: my ($msg)=@_;
615: warn($msg) unless $buildmode;
616: }
617:
618: # ----------------------------------- POD (plain old documentation, CPAN style)
619:
620: =head1 NAME
621:
622: filecompare.pl - script used to help probe and compare file statistics
623:
624: =head1 SYNOPSIS
625:
626: filecompare.pl [ options ... ] [FILE1] [FILE2] [ restrictions ... ]
627:
628: or
629:
630: filecompare.pl [ options ... ] [DIR1] [DIR2] [ restrictions ... ]
631:
632: Restrictions: a list of space separated values (after the file/dir names)
633: can restrict the comparison.
634: These values can be: existence, cvstime, age, md5sum, size, lines,
635: and/or diffs.
636:
637: Options (before file/dir names):
638:
639: -p show all files that have the same comparison
640:
641: -n show all files that have different comparisons
642:
643: -a show all files (with comparisons)
644:
645: -q only show file names (based on first file/dir)
646:
647: -v verbose mode (default)
648:
649: =head1 DESCRIPTION
650:
651: filecompare.pl can work in two modes: file comparison mode, or directory
652: comparison mode.
653:
654: Comparisons can be a function of:
655: * existence similarity
656: * cvs time similarity (first argument treated as CVS source)
657: * age similarity (modification time)
658: * md5sum similarity
659: * size similarity (bytes)
660: * line count difference
661: * number of different lines
662:
663: filecompare.pl integrates smoothly with the LPML installation language
664: (linux packaging markup language). filecompare.pl is a tool that can
665: be used for safe CVS source-to-target installations.
666:
667: =head1 README
668:
669: filecompare.pl integrates smoothly with the LPML installation language
670: (linux packaging markup language). filecompare.pl is a tool that can
671: be used for safe CVS source-to-target installations.
672:
673: The unique identifier is considered to be the file name(s) independent
674: of the directory path.
675:
676: =head1 PREREQUISITES
677:
678: =head1 COREQUISITES
679:
680: =head1 OSNAMES
681:
682: linux
683:
684: =head1 SCRIPT CATEGORIES
685:
686: Packaging/Administrative
687:
688: =cut
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/back.gif){kind=icon}
Return to filecompare.pl CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [LON-CAPA] / loncom / build