source: OFFICIAL/FCM_V1.3/doc/user_guide/extract.html

Last change on this file was 1, checked in by fcm, 15 years ago

creation de larborescence
File size: 47.4 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2
3<html>
4<head>
5  <title>FCM System User Guide: The Extract System</title>
6  <meta name="author" content="FCM development team">
7  <meta name="descriptions" content="User Guide - The Extract System">
8  <meta name="keywords" content="FCM, user guide, extract, build">
9  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
10  <link rel="stylesheet" type="text/css" href="style.css">
11<script type="text/javascript" src="maintain.js">
12</script>
13</head>
14
15<body>
16  <address>
17    <a href="index.html">FCM System User Guide</a> &gt; The Extract System
18  </address>
19
20  <h1>The Extract System</h1>
21
22  <p>The extract system provides an interface between the revision control
23  system (currently Subversion) and the build system. Where appropriate, it
24  extracts code from the repository and other user-defined locations to a
25  directory tree suitable for feeding into the build system. In this chapter, we
26  shall use many examples to explain how to use the extract system. At the end
27  of this chapter, you will be able to extract code from the local file system
28  as well as from different branches of different repository URLs. You will also
29  learn how to mirror code to an alternate destination. Finally, you will be
30  given an introduction on how to specify configurations for the build system
31  via the extract configuration file. (For further information on the build
32  system, please see the next chapter <a href="build.html">The Build
33  System</a>.) The last section of the chapter tells you what you can do in the
34  case when Subversion is not available.</p>
35
36  <h2><a name="command">The Extract Command</a></h2>
37
38  <p>To invoke the extract system, simply issue the command:</p>
39  <pre>
40fcm extract
41</pre>
42
43  <p>By default, the extract system searches for an extract configuration file
44  "ext.cfg" in "$PWD" and then "$PWD/cfg". If an extract configuration file is
45  not found in these directories, the command fails with an error. If an
46  extract configuration file is found, the system will use the configuration
47  specified in the file to perform the current extract.</p>
48
49  <p>If the destination of the extract does not exist, the system performs a new
50  full extract to the destination. If a previous extract already exists at the
51  destination, the system performs an incremental extract, updating any
52  modifications if necessary. If a full (fresh) extract is required for whatever
53  reason, you can invoke the extract system using the "-f" option, (i.e. the
54  command becomes "fcm extract -f"). If you simply want to remove all the items
55  generated by a previous extract in the destination, you can invoke the extract
56  system using the "--clean" option.</p>
57 
58  <p>For further information on the extract command, please see <a href=
59  "command_ref.html#fcm_ext">FCM Command Reference &gt; fcm extract</a>.</p>
60
61  <h2><a name="simple">Simple Usage</a></h2>
62
63  <p>The extract configuration file is the main user interface of the extract
64  system. It is a line based text file. For a complete set of extract
65  configuration file declarations, please refer to the <a href=
66  "annex_ext_cfg.html">Annex: Declarations in FCM extract configuration
67  file</a>.</p>
68
69  <h3><a name="simple_local">Extract from a local path</a></h3>
70
71  <p>A simple example of a basic extract configuration file is given below:</p>
72
73  <table class="pad" summary="extract_example1" border="1" width="100%">
74    <tr>
75      <th>Extract configuration example 1 - extract from a local path</th>
76    </tr>
77
78    <tr>
79      <td>
80        <pre>
81cfg::type         ext       # line 1
82cfg::version      1.0       # line 2
83                            # line 3
84dest              $PWD      # line 4
85                            # line 5
86repos::var::user  $HOME/var # line 6
87                            # line 7
88expsrc::var::user code      # line 8
89</pre>
90      </td>
91    </tr>
92  </table>
93
94  <p>The above demonstrates how to use the extract system to extract code from
95  a local user directory. Here is an explanation of what each line does:</p>
96
97  <ul>
98    <li>line 1: the label "CFG::TYPE" declares the type of the configuration
99    file. The value "ext" tells the system that it is an extract configuration
100    file.</li>
101
102    <li>line 2: the label "CFG::VERSION" declares the version of the extract
103    configuration file. The current default is "1.0". Although it is not
104    currently used, if we have to change the format of the configuration file
105    at a later stage, we shall be able to use this number to determine whether
106    we are reading a file with an older format or one with a newer format.</li>
107
108    <li>line 3: a blank line or a line beginning with a "#" is a comment, and
109    is ignored by the interpreter.</li>
110
111    <li>line 4: the label "DEST" declares the destination root directory of this
112    extract. The value "$PWD" expands to the current working directory.</li>
113
114    <li>line 5: comment line, ignored.</li>
115
116    <li>line 6: the label "REPOS::&lt;pck&gt;::&lt;branch&gt;" declares the top
117    level URL or path of a repository. The package name of the repository is
118    given by &lt;pck&gt;. In our example, we choose "var" as the name of the
119    package. (You can choose any name you like, however, it is usually sensible
120    to use a package name that matches the name of the project or system you are
121    working with.) The branch name in the repository is given by &lt;branch&gt;.
122    (Again, you can choose any name you like, however, it is usually sensible to
123    use a name such as "base", "user" or something that matches your branch
124    name.) In our example, the word "user" is normally used to denote a local
125    user directory. Hence the statement declares that the repository path for
126    the "var" package in the "user" branch can be found at "$HOME/var".</li>
127
128    <li>line 7: comment line, ignored.</li>
129
130    <li>line 8: the label "EXPSRC::&lt;pck&gt;::&lt;branch&gt;" declares an
131    "expandable" source directory for the package &lt;pck&gt; in the branch
132    &lt;branch&gt;. In our example, the package name is "var", and the branch
133    name is "user". These match the package and the branch names of the
134    repository declaration in line 6. It means that the source directory
135    declaration is associated with the path "$HOME/var". The value of the
136    declaration "code" is therefore a sub-directory under "$HOME/var". By
137    declaring a source directory using an "expsrc" label, the system
138    automatically searches for all sub-directories (recursively) under the
139    declared source directory.</li>
140  </ul>
141
142  <p>Invoking the extract system using the above configuration file will
143  "extract" all sub-directories under "$HOME/var/code" to "$PWD/src/var/code".
144  Note: the extract system ignores all hidden files, (i.e. directories and
145  files beginning with a "."). It will write a build configuration file to
146  "$PWD/cfg/bld.cfg". The configuration used for this extract will be
147  written to the configuration file at "$PWD/cfg/ext.cfg".</p>
148
149  <table class="pad" summary="note - incremental extract" border="1" width=
150  "100%">
151    <tr>
152      <th>Note - incremental extract</th>
153    </tr>
154
155    <tr>
156      <td>Suppose you have already performed an extract using the above
157      configuration file. At a later time, you have made some changes to some of
158      the files in the source directory. Re-running the extract system on the
159      same configuration will trigger an incremental extract. In an incremental
160      extract, the system will update only those files that are modified. In
161      exact words, the system checks the modification time of each file in the
162      source and destination. If a source file is newer than its corresponding
163      destination file, it checks whether the content differs.  The destination
164      is only updated if its content differs from the source.</td>
165    </tr>
166  </table>
167
168  <h3><a name="simple_url">Extract from a Subversion URL</a></h3>
169
170  <p>The next example demonstrates how to extract from a Subversion repository
171  URL:</p>
172
173  <table class="pad" summary="extract_example2" border="1" width="100%">
174    <tr>
175      <th>Extract configuration example 2 - extract from a Subversion URL</th>
176    </tr>
177
178    <tr>
179      <td>
180        <pre>
181cfg::type           ext                    # line 1
182cfg::version        1.0                    # line 2
183                                           # line 3
184dest                $PWD                   # line 4
185                                           # line 5
186repos::var::base    svn://server/var/trunk # line 6
187revision::var::base 1234                   # line 7
188                                           # line 8
189expsrc::var::base   code                   # line 9
190</pre>
191      </td>
192    </tr>
193  </table>
194
195  <ul>
196    <li>line 1 to 5: same as example 1.</li>
197
198    <li>line 6: the line declares the repository location of the "base" branch
199    of the "var" package to be the Subversion URL "svn://server/var/trunk".</li>
200
201    <li>line 7: the label "REVISION::&lt;pck&gt;::&lt;branch&gt;" declares the
202    revision of the repository associated with the package &lt;pck&gt; in the
203    branch &lt;branch&gt;. The current line tells the extract system to use
204    revision 1234 of "svn://server/var/trunk". It is worth noting that the
205    declared revision must be a revision when the declared branch exists. The
206    actual revision used is the last changed revision of the declared one. If
207    the revision is not declared, the default is to use the last changed
208    revision at the "HEAD" of the branch.</li>
209
210    <li>line 8: comment line, ignored.</li>
211
212    <li>line 9: the line declares an expandable source directory in the
213    repository "svn://server/var/trunk".</li>
214  </ul>
215
216  <p>Invoking the extract system using the above configuration file will
217  extract all sub-directories under "svn://server/var/trunk/code" to
218  "$PWD/src/var/code". It will write a build configuration file to
219  "$PWD/cfg/bld.cfg". The configuration used for this extract will be
220  written to the configuration file at "$PWD/cfg/ext.cfg".</p>
221
222  <table class="pad" summary=
223  "note - declaration of source directories" border="1" width=
224  "100%">
225    <tr>
226      <th>Note - declaration of source directories</th>
227    </tr>
228
229    <tr>
230      <td>
231        <strong>EXPSRC or SRC?</strong>
232
233        <p>So far, we have only declared source directories using the "EXPSRC"
234        statement, which stands for "expandable source directory". A source
235        directory declared using this statement will trigger the system to
236        search recursively for any sub-directories under the declared one. Any
237        sub-directories containing regular source files will be included in the
238        extract. Empty directories, hidden directories and directories
239        containing only hidden files are ignored.</p>
240
241        <p>If you do not want the system to search for sub-directories
242        underneath your declared source directory, you can declare your source
243        directory using the "SRC" statement. The "SRC" statement is essentially
244        the same as "EXPSRC" except that it does not trigger the automatic
245        recursive search for source directories. In fact, the system implements
246        the "EXPSRC" statement by expanding it into a list of "SRC"
247        statements.</p>
248
249        <p><strong>Package and sub-package</strong></p>
250
251        <p>The second field of a repository, revision or source directory
252        declaration label is the name of the container package. It is a name
253        selected by the user to identify the system or project he/she is
254        working on. (Therefore, it is often sensible to choose an identifier
255        that matches the name of the project or system.) The package name
256        provides a unique namespace for a file container. Source directories
257        are automatically arranged into sub-packages, using the names of the
258        sub-directories as the names of the sub-packages. For example, the
259        declaration at line 9 in example 2 will put the source directory in the
260        "var/code" sub-package automatically.</p>
261
262        <p>Note that, in additional to slash "/", double colon "::" and double
263        underscore "__" (internal only) also act as delimiters for package
264        names. Please avoid using them for naming your files and directories.
265        </p>
266
267        <p>You can declare a sub-package name explicitly in your source
268        directory statement. For example, the following two lines are
269        equivalent:</p>
270        <pre>
271src::var::base                      code/VarMod_Surface
272src::var/code/VarMod_Surface::base  code/VarMod_Surface
273</pre>
274
275        <p>Explicit sub-package declaration should not be used normally, as it
276        requires a lot more typing (although there are some situations where it
277        can be useful, e.g. if you need to re-define the package name).</p>
278
279        <p>Currently, the extract system only supports non-space characters in
280        the package name, as the space character is used as a delimiter between
281        the declaration label and its value. If there are spaces in the path
282        name to a file or directory, you should explicity re-define the package
283        name of that path to a package name with no space using the above
284        method. However, we recommend that only non-space characters are used
285        for naming directories and files to make life simpler.</p>
286      </td>
287    </tr>
288  </table>
289
290  <table class="pad" summary="note - the expanded extract configuration file"
291  border="1" width="100%">
292    <tr>
293      <th>Note - the expanded extract configuration file</th>
294    </tr>
295
296    <tr>
297      <td>
298        At the end of a successful extract, the configuration used by the
299        current extract is written in "cfg/ext.cfg" under the extract
300        destination root. This file is an "expanded" version of the original,
301        with changes in the following declarations:
302
303        <ul>
304          <li>All revision keywords are converted into revision numbers.</li>
305
306          <li>If a revision is not defined for a repository, it is set to the
307          corresponding revision number of the HEAD revision.</li>
308
309          <li>All URL keywords are converted into the full URLs.</li>
310
311          <li>All EXPSRC declarations are expanded into SRC declarations.</li>
312
313          <li>All other variables are expanded.</li>
314        </ul>
315
316        <p>With this file, it should be possible for a later extract to re-create
317        the current configuration even if the contents of the repository have
318        changed. (This applies only to code stored in the repository.)</p>
319      </td>
320    </tr>
321  </table>
322
323  <h3><a name="simple_mirror">Mirror code to an alternate location</a></h3>
324
325  <p>The next example demonstrates how to extract from a repository and mirror
326  the code to an alternate location. It is essentially the same as example 2,
327  except that it has three new lines to describe how the system can mirror the
328  extracted code to an alternate location.</p>
329
330  <table class="pad" summary="extract_example3" border="1" width="100%">
331    <tr>
332      <th>Extract configuration example 3 - mirror code</th>
333    </tr>
334
335    <tr>
336      <td>
337        <pre>
338cfg::type           ext
339cfg::version        1.0
340
341dest                $PWD
342
343rdest::machine      tx01                           # line 6
344rdest::logname      frva                           # line 7
345rdest               /scratch/frva/extract/example3 # line 8
346
347repos::var::base    svn://server/var/trunk
348revision::var::base 1234
349
350expsrc::var::base   code
351</pre>
352      </td>
353    </tr>
354  </table>
355
356  <p>Here is an explanation of what each line does:</p>
357
358  <ul>
359    <li>line 6: "RDEST::MACHINE" declares the target machine to which the code
360    will be mirrored. The example mirrors the code to the machine named
361    "tx01".</li>
362
363    <li>line 7: "RDEST::LOGNAME" declares the user name of the target
364    machine, to which the user has login access. If this is not
365    declared, the system uses the login name of the current user on the local
366    machine.</li>
367
368    <li>line 8: "RDEST" declares the root directory of the alternate
369    destination, where the mirror version of the extract will be sent.</li>
370  </ul>
371
372  <p>Invoking the extract system on the above configuration will trigger an
373  extract similar to that given in example 2, but it will also attempt to
374  mirror the contents at "$PWD/src/var/code" to
375  "/scratch/frva/extract/example3/src" on the alternate destination. It will
376  also mirror the expanded extract configuration file "$PWD/cfg/ext.cfg" to
377  "/scratch/frva/extract/example3/cfg/ext.cfg" and "$PWD/cfg/bld.cfg" to
378  "/scratch/frva/extract/example3/cfg/bld.cfg". It is also worth noting that the
379  content of the build configuration file will be slightly different, since it
380  will include directory names appropriate for the alternate destination.</p>
381
382  <table class="pad" summary="note - mirroring command" border="1" width=
383  "100%">
384    <tr>
385      <th>Note - mirroring command</th>
386    </tr>
387
388    <tr>
389      <td>
390        The extract system currently supports "rdist" and "rsync" as its
391        mirroring tool. The default is "rsync". To use "rdist" instead of
392        "rsync", add the following line to your extract configuration file:
393        <pre>
394rdest::mirror_cmd  rdist
395</pre>
396
397        <p>N.B. If you are going to mirror code to another machine, you need to
398        ensure that your account on the target machine is set up correctly to
399        accept commands from the local machine. In our current settings of both
400        "rdist" and "rsync", all you need to do is set up your "$HOME/.rhosts"
401        file on the target machine. For example, if you are "fred" working on
402        the local machine "eld001", you will need to have the following entry
403        in your "$HOME/.rhosts" on the target machine:</p>
404        <pre>
405eld001  fred
406</pre>
407      </td>
408    </tr>
409  </table>
410
411  <h2><a name="advanced">Advanced Usage</a></h2>
412
413  <h3><a name="advanced_multi">Extract from multiple repositories</a></h3>
414
415  <p>So far, we have only extracted from a single location. The extract system
416  is not much use if that is the only thing it can do. In fact, the extract
417  system supports extract of multiple source directories from multiple
418  branches in multiple repositories. The following configuration file is an
419  example of how to extract from multiple repositories:</p>
420
421  <table class="pad" summary="extract_example4" border="1" width="100%">
422    <tr>
423      <th>Extract configuration example 4 - extract from multiple
424      repositories</th>
425    </tr>
426
427    <tr>
428      <td>
429        <pre>
430cfg::type           ext
431cfg::version        1.0
432
433dest                $PWD
434
435repos::var::base    fcm:var_tr              # line 6
436repos::ops::base    fcm:ops_tr              # line 7
437repos::gen::base    fcm:gen_tr              # line 8
438
439revision::gen::base 2468                    # line 10
440
441expsrc::var::base   src/code                    # line 12
442expsrc::var::base   src/scripts                 # line 13
443expsrc::ops::base   src/code                    # line 14
444src::gen::base      src/code/GenMod_Constants   # line 15
445src::gen::base      src/code/GenMod_Control     # line 16
446src::gen::base      src/code/GenMod_FortranIO   # line 17
447src::gen::base      src/code/GenMod_GetEnv      # line 18
448src::gen::base      src/code/GenMod_ModelIO     # line 19
449src::gen::base      src/code/GenMod_ObsInfo     # line 20
450src::gen::base      src/code/GenMod_Platform    # line 21
451src::gen::base      src/code/GenMod_Reporting   # line 22
452src::gen::base      src/code/GenMod_Trace       # line 23
453src::gen::base      src/code/GenMod_UMConstants # line 24
454src::gen::base      src/code/GenMod_Utilities   # line 25
455</pre>
456      </td>
457    </tr>
458  </table>
459
460  <p>Here is an explanation of what each line does:</p>
461
462  <ul>
463    <li>line 6 to 8: these lines declare the repositories for the "base"
464    branches of the "var", "ops" and "gen" packages respectively. It is worth
465    noting that the values of the declarations are no longer Subversion URLs
466    but are FCM URL keywords. These keywords are normally declared in the
467    central configuration file of the FCM system, and will be expanded into the
468    corresponding Subversion URLs by the FCM system. For further information on
469    URL keywords, please see <a href=
470    "code_management.html#svn_basic_keywords">Code Management System &gt; Using
471    Subversion &gt; Basic Command Line Usage &gt; Repository &amp; Revision
472    Keywords</a>.</li>
473
474    <li>line 10: this line declares the revision number for the "base" branch
475    of the "gen" package, i.e. for the "fcm:gen_tr" repository. It is worth
476    noting that the revision numbers for the "var" and "ops" packages have not
477    been declared. By default, their revision numbers will be set to the last
478    changed revision at the "HEAD".</li>
479
480    <li>line 12 to 14: these line declares the source directories for the
481    "base" branches of the "var" and "ops" packages. For the "var" package, we
482    are extracting everything from the "code" and the "scripts" sub-directory.
483    For the "ops" package, we are extracting everything from the "code"
484    directory.</li>
485
486    <li>line 15 to 25: these line declares the source directories for the
487    "base" branch of the "gen" package. The source directories declared will
488    not be searched for sub-directories underneath the declared
489    directories.</li>
490  </ul>
491
492  <p>We shall end up with a directory tree such as:</p>
493  <pre>
494$PWD
495   |
496   |--- cfg
497   |      |
498   |      |--- bld.cfg
499   |      |--- ext.cfg
500   |
501   |--- src
502          |
503          |--- gen
504          |      |
505          |      |--- code
506          |              |
507          |              |--- GenMod_Constants
508          |              |--- GenMod_Control
509          |              |--- GenMod_FortranIO
510          |              |--- GenMod_GetEnv
511          |              |--- GenMod_ModelIO
512          |              |--- GenMod_ObsInfo
513          |              |--- GenMod_Platform
514          |              |--- GenMod_Reporting
515          |              |--- GenMod_Trace
516          |              |--- GenMod_UMConstants
517          |              |--- GenMod_Utilities
518          |
519          |--- ops
520          |      |
521          |      |--- code
522          |              |
523          |              |--- ...
524          |
525          |--- var
526                 |
527                 |--- code
528                 |       |
529                 |       |--- ...
530                 |
531                 |--- scripts
532                         |
533                         |--- ...
534</pre>
535
536  <table class="pad" summary="note - revision number" border="1" width="100%">
537    <tr>
538      <th>Note - revision number</th>
539    </tr>
540
541    <tr>
542      <td>
543        As seen in the above example, if a revision number is not specified for
544        a repository URL, it defaults to the last changed revision at the "HEAD"
545        of the branch. The revision number can also be declared in other ways:
546
547        <ul>
548          <li>Any revision arguments acceptable by Subversion are allowed. You
549          can use a valid revision number, a date between a pair of curly
550          brackets (e.g. {"2005-05-01 12:00"}) or the keyword "HEAD". However,
551          please do not use the keywords "BASE", "COMMITTED" or "PREV" as these
552          are reserved for working copy only.</li>
553
554          <li>FCM revision keywords are allowed. These must be defined for the
555          corresponding repository URLs in either the central or the user FCM
556          configuration file. For further information on revision keywords,
557          please see <a href="code_management.html#svn_basic_keywords">Code
558          Management &gt; Using Subversion &gt; Basic Command Line Usage &gt;
559          Repository &amp; Revision Keywords</a>.</li>
560
561          <li>Do not use the keyword "USER", as it is used internally by the
562          extract system.</li>
563        </ul>
564
565        <p>If a revision number is specified for a branch, the actual revision
566        used by the extract system is the last changed revision of the branch,
567        which may differ from the declared revision. While this behaviour is
568        useful in most situations, some users may find it confusing to work
569        with. It is possible to alter this behaviour so that extract will fail
570        if the declared revision does not correspond to a changeset of the
571        declared branch. Make the following declaration to switch on this
572        checking:</p>
573
574        <pre>
575revmatch  true
576</pre>
577      </td>
578    </tr>
579  </table>
580
581  <h3><a name="advanced_branches">Extract from multiple branches</a></h3>
582
583  <p>We have so far dealt with a single branch in any package.  The extract
584  system can be used to "combine" changes from different branches of a package.
585  An example is given below:</p>
586
587  <table class="pad" summary="extract_example5" border="1" width="100%">
588    <tr>
589      <th>Extract configuration example 5 - extract from multiple branches</th>
590    </tr>
591
592    <tr>
593      <td>
594        <pre>
595cfg::type               ext
596cfg::version            1.0
597 
598dest                    $PWD
599 
600repos::var::base        fcm:var_tr
601repos::ops::base        fcm:ops_tr
602repos::gen::base        fcm:gen_tr
603 
604revision::gen::base     2468
605 
606expsrc::var::base       src/code
607expsrc::var::base       src/scripts
608expsrc::ops::base       src/code
609src::gen::base          src/code/GenMod_Constants
610src::gen::base          src/code/GenMod_Control
611src::gen::base          src/code/GenMod_FortranIO
612src::gen::base          src/code/GenMod_GetEnv
613src::gen::base          src/code/GenMod_ModelIO
614src::gen::base          src/code/GenMod_ObsInfo
615src::gen::base          src/code/GenMod_Platform
616src::gen::base          src/code/GenMod_Reporting
617src::gen::base          src/code/GenMod_Trace
618src::gen::base          src/code/GenMod_UMConstants
619src::gen::base          src/code/GenMod_Utilities
620
621repos::var::branch1     fcm:var_br/frva/r1234_new_stuff   # line 27
622repos::var::branch2     fcm:var_br/frva/r1516_bug_fix     # line 28
623repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff # line 29
624</pre>
625      </td>
626    </tr>
627  </table>
628
629  <p>The configuration file in example 5 is similar to that of example 4 except
630  for the last three lines. Here is an explanation of what they do:</p>
631
632  <ul>
633    <li>line 27: this line declares a repository URL for the "branch1" branch
634    of the "var" package. From the URL of the branch, we know that the branch
635    was created by the user "frva" based on the trunk at revision at 1234. The
636    description of the branch is "branch1". The following points are worth
637    noting:
638
639      <ul>
640        <li>By declaring a new branch with the same package name to a
641        previously declared branch, it is assumed that both branches reside in
642        the same Subversion repository.</li>
643
644        <li>No revision is declared for this URL, so the default is used which
645        is the last changed revision at the "HEAD" of the branch.</li>
646
647        <li>No source directory is declared for this URL. By default, if no
648        source directory is declared for a branch repository, it will attempt
649        to use the same set of source directories as the first declared branch
650        of the package. In this case, the source directories declared for the
651        "base" branch of the "var" package will be used.</li>
652      </ul>
653    </li>
654
655    <li>line 28: this line declares another branch called "branch2" for the
656    "var" package. No source directory is declared for this URL either, so it
657    will use the same set of source directories declared for the "base"
658    branch.</li>
659
660    <li>line 29: this line declares a branch called "branch1" for the "ops"
661    package. It will use the same set of source directories declared for the
662    "ops" package "base" branch.</li>
663  </ul>
664
665  <p>When we invoke the extract system, it will attempt to extract from the
666  first declared branch of a package, if the last changed revision of the source
667  directory is the same in all the branches. However, if the last changed
668  revision of the source directory differs for different branches, the system
669  will attempt to obtain an extract priority list for each source directory,
670  using the following logic:</p>
671
672  <ol>
673    <li>The system looks for source directory packages from the first declared
674    branch to the last declared branch.</li>
675
676    <li>The branch in which a source directory package is first declared is
677    the "base" branch of the source directory package.</li>
678
679    <li>The last changed revision of a source directory package in a
680    subsequently declared repository branch is compared with that of the base
681    branch. If the last changed revision is the same as that of the base branch,
682    the source directory of this branch is discarded. Otherwise, it is placed at
683    the end of the extract priority list.</li>
684  </ol>
685
686  <p>For the "var" package in the above example, let us assume that we have
687  three source directory packages X, Y and Z under "code", and their last
688  changed revisions under "base" are 100. Let's say we have committed some
689  changes to X and Z in the "branch1" branch at revision 102, and other changes
690  to Y and Z in the "branch2" branch at revision 104, the extract priority lists
691  for X, Y and Z will look like:</p>
692
693  <ul>
694    <li>X: base (100, base), branch1 (102), branch2 (100, discarded)</li>
695
696    <li>Y: base (100, base), branch1 (100, discarded), branch2 (104)</li>
697
698    <li>Z: base (100, base), branch1 (102), branch2 (104)</li>
699  </ul>
700
701  <p>Once we have an extract priority list for a source directory, we can
702  begin extracting source files in the source directory. The source
703  directory of the base branch is extracted first, followed by that in the
704  subsequent branches. If a source file in a subsequent branch has the same
705  content as the that in the base branch, it is discarded. Otherwise, the
706  following logic determines the branch to use:</p>
707
708  <ol>
709    <li>If a source file is modified in only one subsequent branch, the source
710    file in that branch is extracted.</li>
711
712    <li>If a source file is modified in two or more subsequent branches, but
713    their modifications are the same, then the source file in the first
714    modification is used.</li>
715
716    <li>If a source file is modified in two or more subsequent branches and
717    their modifications differ, then the behaviour depends on the "conflict
718    mode" setting, which can be "fail", "merge" (default) and "override". If the
719    conflict mode is "fail", the extract fails. If the conflict mode is "merge",
720    the system will attempt to merge the changes using a tool such as "diff3".
721    The result of the merge will be used to update the destination.  The extract
722    fails only if there are unresolved conflicts in the merge. (In which case,
723    the conflict should be resolved using the version control system before
724    re-running the extract system.) If the conflict mode is "override", the
725    change in the latest declared branch takes precedence, and the changes in
726    all other branches will be ignored. The conflict mode can be changed using
727    the CONFLICT declaration in the extract configuration file. E.g:
728      <pre>
729conflict  fail
730</pre>
731    </li>
732  </ol>
733 
734  <p>Once the system has established which source files to use, it determines
735  whether the destination file is out of date or not. The destination file is
736  out of date if it does not exist or if its content differs from the version of
737  the source file we are using. The system only updates the destination if it is
738  considered to be out of date.</p>
739
740  <p>The extract system can also combine changes from branches in the Subversion
741  repository and the local file system. The limitation is that there can only
742  be one branch from the local file system. (By convention, the branch is named
743  "user".)</p>
744
745  <p>It is also worth bearing in mind that the "user" branch always takes
746  precedence over branches residing in Subversion repositories. Hence, source
747  directories from a "user" branch are always placed at the end of the extract
748  priority list.</p>
749
750  <p>Extracting from a mixture of Subversion repository and local file system
751  is demonstrated in the next example.</p>
752
753  <table class="pad" summary="extract_example6" border="1" width="100%">
754    <tr>
755      <th>Extract configuration example 6 - extract from multiple branches +
756      user paths</th>
757    </tr>
758
759    <tr>
760      <td>
761        <pre>
762cfg::type               ext
763cfg::version            1.0
764 
765dest                    $PWD
766 
767repos::var::base        fcm:var_tr
768repos::ops::base        fcm:ops_tr
769repos::gen::base        fcm:gen_tr
770 
771revision::gen::base     2468
772 
773expsrc::var::base       src/code
774expsrc::var::base       src/scripts
775expsrc::ops::base       src/code
776src::gen::base          src/code/GenMod_Constants
777src::gen::base          src/code/GenMod_Control
778src::gen::base          src/code/GenMod_FortranIO
779src::gen::base          src/code/GenMod_GetEnv
780src::gen::base          src/code/GenMod_ModelIO
781src::gen::base          src/code/GenMod_ObsInfo
782src::gen::base          src/code/GenMod_Platform
783src::gen::base          src/code/GenMod_Reporting
784src::gen::base          src/code/GenMod_Trace
785src::gen::base          src/code/GenMod_UMConstants
786src::gen::base          src/code/GenMod_Utilities
787
788repos::var::branch1     fcm:var_br/frva/r1234_new_stuff
789repos::var::branch2     fcm:var_br/frva/r1516_bug_fix
790repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff
791
792repos::var::user        $HOME/var                         # line 31
793repos::gen::user        $HOME/gen                         # line 32
794</pre>
795      </td>
796    </tr>
797  </table>
798
799  <p>Example 6 is similar to example 5 except that it is also extracting from
800  local directories. Here is an explanation of the lines:</p>
801
802  <ul>
803    <li>line 31 to 32: these line declare the "repositories" for the "user"
804    branches of the "var" and "gen" packages respectively. Both are local paths
805    at the local file system. There are no declarations for source directories
806    for the "user" branches, so they use the same set of source directories of
807    the first declared branches, the "base" branches in both cases.</li>
808  </ul>
809
810  <table class="pad" summary="note - the inc statement" border="1" width=
811  "100%">
812    <tr>
813      <th>Note - the INC declaration</th>
814    </tr>
815
816    <tr>
817      <td>
818        You have probably realised that the above examples have many repeated
819        lines. To avoid having repeated lines in multiple extract configuration
820        files, you can use INC declarations to "include" other extract
821        configuration files. For example, if the configuration file of example 5
822        is stored in the file "$HOME/example5/ext.cfg", line 1 to 29 of
823        example 6 can be replaced with an INC declaration. Example 6 can then be
824        written as:
825        <pre>
826inc                     $HOME/example5/ext.cfg
827
828repos::var::user        $HOME/var
829repos::gen::user        $HOME/gen
830</pre>
831
832        <p>Note: the INC declaration supports the special "environment variable"
833        $HERE. If this variable is already set in the environment, it acts as a
834        normal environment variable. However, if it is not set, it will be
835        expanded into the container directory of the current extract
836        configuration file. This feature is particularly useful if you are
837        including a hierarchy of extract configurations from files in the same
838        container directory in a repository.</p>
839      </td>
840    </tr>
841  </table>
842
843  <h3><a name="advanced_incremental"></a><a name="advanced_inherit">Inherit from
844  a previous extract</a></h3>
845
846  <p>All the examples above dealt with standalone extract, that is, the current
847  extract is independent of any other extract. If a previous extract exists in
848  another location, the extract system can inherit from this previous extract in
849  your current extract. This works like a normal incremental extract, except
850  that your extract will only contain the changes you have specified (compared
851  with the inherited extract) instead of the full source directory tree. This
852  type of incremental extract is useful in several ways. For instance:</p>
853
854  <ul>
855    <li>It is fast, because you only have to extract and mirror files that you
856    have changed.</li>
857
858    <li>The subsequent build will also be fast, since it will use incremental
859    build.</li>
860
861    <li>You do not need write access to the original extract. A system
862    administrator can set up a stable version in a central account, which
863    developers can then inherit from.</li>
864
865    <li>You want an incremental extract, but you need to leave the original
866    extract unmodified.</li>
867  </ul>
868
869  <p>The following example is based on example 4 and 6. The assumption is that
870  an extract has already been performed at the directory "~frva/var/vn22.0"
871  based on the configuration file in example 4.</p>
872
873  <table class="pad" summary="extract_example7" border="1" width="100%">
874    <tr>
875      <th>Extract configuration example 7 - inherit from a previous extract</th>
876    </tr>
877
878    <tr>
879      <td>
880        <pre>
881cfg::type               ext
882cfg::version            1.0
883 
884dest                    $PWD
885
886use                     ~frva/var/vn22.0                  # line 6
887
888repos::var::branch1     fcm:var_br/frva/r1234_new_stuff   # line 8
889repos::var::branch2     fcm:var_br/frva/r1516_bug_fix     # line 9
890repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff # line 10
891
892repos::var::user        $HOME/var                         # line 12
893repos::gen::user        $HOME/gen                         # line 13
894</pre>
895      </td>
896    </tr>
897  </table>
898
899  <ul>
900    <li>line 6: this line replaces line 1 to 25 of example 6. It declares that
901    the current extract should inherit from the previous extract located
902    at "~frva/var/vn22.0".</li>
903  </ul>
904
905  <p>Running the extract system using the above configuration will trigger an
906  incremental extract, as if you are running an incremental extract having
907  modified the configuration file in example 4 to that of example 6. The only
908  difference is that the original extract using the example 4 configuration will
909  be left untouched at "~frva/var/vn22.0", and the new extract will contain only
910  the changes in the branches declared from line 8 to 13.</p>
911
912  <p>Note: extract inheritance allows you to add more branches to a package, but
913  you should not redefine the REPOS, REVISION, EXPSRC or SRC declarations of a
914  branch that is already declared (and already extracted) in the inherited
915  extract. Although the system will not stop you from doing so, you may end up
916  with an extract that does not quite do what it is supposed to do. For example,
917  if the "base" branch in the "foo" package (<tt>repos::foo::base</tt>) is
918  already defined and extracted in an extract you are inheriting from, you
919  should not redefine any of the <tt>*::foo::base</tt> declarations in your
920  current extract. However, you are free to add more branches for the same
921  package with new labels (e.g. <tt>repos::foo::b1</tt>), and indeed new
922  packages that are not already defined in the inherited extract (e.g.
923  <tt>repos::bar::base</tt>).</p>
924
925  <p>If you are setting up an extract to be inherited, you do not have to
926  perform a build. If you don't you will still gain the benefit of incremental
927  file extract, but you will be performing a full build of the code.</p>
928
929  <h3><a name="advanced_build">Extract - Build Configuration</a></h3>
930
931  <p>Configuration settings for feeding into the build system can be declared
932  through the extract configuration file using the "BLD::" prefix. Any line in
933  an extract configuration containing a label with such a prefix will be
934  considered a build system variable. At the end of a successful extract,
935  the system strips out the "BLD::" prefix before writing these variables to
936  the build configuration file. Some example entries are given between line 17
937  and 22 in the following configuration file:</p>
938
939  <table class="pad" summary="extract_example8" border="1" width="100%">
940    <tr>
941      <th>Extract configuration example 8 - setting build configuration</th>
942    </tr>
943
944    <tr>
945      <td>
946        <pre>
947cfg::type           ext
948cfg::version        1.0
949
950dest                $PWD
951
952repos::var::base    fcm:var_tr
953repos::ops::base    fcm:ops_tr
954repos::gen::base    fcm:gen_tr
955
956revision::gen::base 2468
957
958expsrc::var::base   src/code
959expsrc::var::base   src/scripts
960expsrc::ops::base   src/code
961src::gen::base      src/code/GenMod_Constants
962src::gen::base      src/code/GenMod_Control
963src::gen::base      src/code/GenMod_FortranIO
964src::gen::base      src/code/GenMod_GetEnv
965src::gen::base      src/code/GenMod_ModelIO
966src::gen::base      src/code/GenMod_ObsInfo
967src::gen::base      src/code/GenMod_Platform
968src::gen::base      src/code/GenMod_Reporting
969src::gen::base      src/code/GenMod_Trace
970src::gen::base      src/code/GenMod_UMConstants
971src::gen::base      src/code/GenMod_Utilities
972
973bld::target         VarProg_AnalysePF.exe   # line 27
974
975bld::tool::fc       sxmpif90                # line 29
976bld::tool::cc       sxmpic++                # line 30
977bld::tool::ld       sxmpif90                # line 31
978</pre>
979      </td>
980    </tr>
981  </table>
982
983  <p>The above example is essentially the same as example 4, apart from the
984  additional build configuration. The following is a simple explanation of what
985  the lines represent: (For detail of the build system, please see the next
986  chapter on <a href="build.html">The Build System</a>.)</p>
987
988  <ul>
989    <li>Line 27: the line declares a default target of the build.</li>
990
991    <li>Line 29-31: the lines declare the Fortran compiler, the C compiler and
992    the linker respectively.</li>
993  </ul>
994
995  <table class="pad" summary="note - user variables and internal variables"
996  border="1" width=
997  "100%">
998    <tr>
999      <th>Note - user variables and internal variables</th>
1000    </tr>
1001
1002    <tr>
1003      <td>When you start using the extract system to define compiler flags for
1004      the build system, you may end up having to make a lot of long and
1005      repetitive declarations. In this case, you may want to define variables to
1006      replace the repetitive parts of the declarations. In the extract system,
1007      you can define a user variable by making a declaration with a label that
1008      begins with a percent sign "%". For example:
1009
1010        <pre>
1011# Declare a variable %fred
1012%fred                     -Cdebug -eC -Wf,-init heap=nan stack=nan
1013
1014bld::tool::fflags         %fred
1015# bld::tool::fflags       -Cdebug -eC -Wf,-init heap=nan stack=nan
1016
1017bld::tool::fflags::foo    %fred -f0
1018# bld::tool::fflags::foo  -Cdebug -eC -Wf,-init heap=nan stack=nan -f0
1019
1020bld::tool::fflags::bar    -w %fred
1021# bld::tool::fflags::bar  -w -Cdebug -eC -Wf,-init heap=nan stack=nan
1022</pre>
1023
1024      Further to this, each declaration results in an internal variable of the
1025      same name and you can also refer to any of these internal variables in the
1026      same way. So, the example given above could also be written as follows:
1027
1028        <pre>
1029bld::tool::fflags         -Cdebug -eC -Wf,-init heap=nan stack=nan
1030bld::tool::fflags::foo    %bld::tool::fflags -f0
1031bld::tool::fflags::bar    -w %bld::tool::fflags
1032</pre>
1033
1034      Note that, if required, variable names may also be enclosed in curly
1035      brackets "{}" when used in a value.
1036      </td>
1037    </tr>
1038  </table>
1039
1040  <table class="pad" summary="note - as-parsed configuration" border="1" width=
1041  "100%">
1042    <tr>
1043      <th>Note - as-parsed configuration</th>
1044    </tr>
1045
1046    <tr>
1047      <td>If you use a hierarchy of INC declarations or variables, you may end up
1048      with a configuration file that is difficult to understand. To help you with
1049      this, the extract system generates an as-parsed configuration file at
1050      "cfg/parsed_ext.cfg" of the destination. The content of the as-parsed
1051      configuration file is what the extract system actually reads. It should
1052      contain everything in your original extract configuration file, except that
1053      all INC declarations, environment variables and user/internal variables are
1054      expanded.</td>
1055    </tr>
1056  </table>
1057
1058  <h2><a name="verbose">Diagnostic verbose level</a></h2>
1059
1060  <p>The amount of diagnostic messages generated by the extract system is
1061  normally set to a level suitable for normal everyday operation. This is the
1062  default diagnostic verbose level 1. If you want a minimum amount of
1063  diagnostic messages, you should set the verbose level to 0. If you want more
1064  diagnostic messages, you can set the verbose level to 2 or 3. You can modify
1065  the verbose level in two ways. The first way is to set the environment
1066  variable FCM_VERBOSE to the desired verbose level. The second way is to
1067  invoke the extract system with the "-v &lt;level&gt;" option. (If set, the
1068  command line option overrides the environment variable.)</p>
1069
1070  <p>The following is a list of diagnostic output at each verbose level:</p>
1071
1072  <table class="pad" summary="List of diagnostic verbose output" border="1">
1073    <tr>
1074      <th>Verbose level</th>
1075
1076      <th>Possible output</th>
1077    </tr>
1078
1079    <tr>
1080      <th>0</th>
1081
1082      <td>
1083        <ul>
1084          <li>Report the time taken to extract the code.</li>
1085
1086          <li>Report the time taken to mirror the code.</li>
1087
1088          <li>If "rdist" is used to mirror the code, run the command with the
1089          "-q" option.</li>
1090        </ul>
1091      </td>
1092    </tr>
1093
1094    <tr>
1095      <th>1</th>
1096
1097      <td>
1098        <ul>
1099          <li>Everything at verbose level 0.</li>
1100
1101          <li>Report the name of the extract configuration file.</li>
1102
1103          <li>Report the location of the extract destination.</li>
1104
1105          <li>Report date/time at the beginning of the extract step.</li>
1106
1107          <li>If the revision specified for a repository branch is not its last
1108          changed revision, print an information statement to inform the user of
1109          the last changed revision of the branch.</li>
1110
1111          <li>Summarises the destination status and the source status.</li>
1112
1113          <li>Report date/time at the beginning of the mirror step.</li>
1114
1115          <li>Report the location of the alternate destination.</li>
1116
1117          <li>Report total time.</li>
1118        </ul>
1119      </td>
1120    </tr>
1121
1122    <tr>
1123      <th>2</th>
1124
1125      <td>
1126        <ul>
1127          <li>Everything at verbose level 1.</li>
1128
1129          <li>If the revision specified for a repository branch is not current
1130          (i.e. the specified revision number is less than the revision number
1131          of the last commit revision), print an information statement to
1132          inform the user of the last commit revision of the branch.</li>
1133
1134          <li>Report the detail of each change in the destination.</li>
1135
1136          <li>If "rdist" is used to mirror the code, run the command without the
1137          "-q" option.</li>
1138        </ul>
1139      </td>
1140    </tr>
1141
1142    <tr>
1143      <th>3</th>
1144
1145      <td>
1146        <ul>
1147          <li>Everything at verbose level 2.</li>
1148
1149          <li>Report all shell commands invoked by the extract system with
1150          timestamp.</li>
1151
1152          <li>If "rdist" is used to mirror the code, print the "distfile"
1153          supplied to the command.</li>
1154
1155          <li>If "rsync" is used to mirror the code, invoke the command with
1156          the "-v" option.</li>
1157        </ul>
1158      </td>
1159    </tr>
1160  </table>
1161
1162  <h2><a name="nosvn">When Subversion Is Not Available</a></h2>
1163
1164  <p>The extract system can still be used if Subversion is not available.
1165  Clearly, you can only use local repositories. However, you can still do
1166  incremental extract, mirror an extract to an alternate location, or combine
1167  code from multiple local repositories.</p>
1168
1169  <p>If you are using Subversion but your server is down then clearly there is
1170  little you can do. However, if you already have an extract then you can re-run
1171  <tt>fcm extract</tt> as long as the extract configuration file only refers to
1172  fixed revisions. If this is not the case then you can always use the expanded
1173  extract configuration file which can be found in "cfg/ext.cfg" under the
1174  extract destination root. This means that you can continue to makes changes to
1175  local code and do incremental extracts even whilst your Subversion server is
1176  down.</p>
1177
1178  <script type="text/javascript">
1179<!--
1180document.body.appendChild(document.createElement('hr'))
1181document.body.appendChild(displayMaintenanceInfo('doc/user_guide/'))
1182//-->
1183</script>
1184</body>
1185</html>
Note: See TracBrowser for help on using the repository browser.