New URL for NEMO forge!   http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.
extract.html in vendors/fcm/current/doc/user_guide – NEMO

source: vendors/fcm/current/doc/user_guide/extract.html @ 1977

Last change on this file since 1977 was 1977, checked in by flavoni, 14 years ago

importing fcm vendor
File size: 49.6 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
2    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3
4<html xmlns="http://www.w3.org/1999/xhtml">
5<head>
6  <meta name="generator" content=
7  "HTML Tidy for Linux/x86 (vers 1st December 2004), see www.w3.org" />
8
9  <title>FCM User Guide: The Extract System</title>
10  <meta name="author" content="FCM development team" />
11  <meta name="descriptions" content="User Guide - The Extract System" />
12  <meta name="keywords" content="FCM, user guide, extract, build" />
13  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
14  <link rel="stylesheet" type="text/css" href="style.css" />
15<script type="text/javascript" src="fcm.js">
16</script>
17</head>
18
19<body onload=
20"javascript: FCM.load('doc/user_guide/', null, [['.', 'FCM User Guide']]);">
21  <div id="document-info">
22    <address id="fcm-js-trail"></address>
23
24    <address id="fcm-js-maintenance"></address>
25  </div>
26
27  <h1>The Extract System</h1>
28
29  <div id="fcm-content"></div>
30
31  <h2 id="introduction">Introduction</h2>
32
33  <p>The extract system provides an interface between the revision control
34  system (currently Subversion) and the build system. Where appropriate, it
35  extracts code from the repository and other user-defined locations to a
36  directory tree suitable for feeding into the build system. In this chapter,
37  we shall use many examples to explain how to use the extract system. At the
38  end of this chapter, you will be able to extract code from the local file
39  system as well as from different branches of different repository URLs. You
40  will also learn how to mirror code to an alternate destination. Finally, you
41  will be given an introduction on how to specify configurations for the build
42  system via the extract configuration file. (For further information on the
43  build system, please see the next chapter <a href="build.html">The Build
44  System</a>.) The last section of the chapter tells you what you can do in the
45  case when Subversion is not available.</p>
46
47  <h2 id="command">The Extract Command</h2>
48
49  <p>To invoke the extract system, simply issue the command:</p>
50  <pre>
51fcm extract
52</pre>
53
54  <p>By default, the extract system searches for an extract configuration file
55  <samp>ext.cfg</samp> in <samp>$PWD</samp> and then <samp>$PWD/cfg</samp>. If
56  an extract configuration file is not found in these directories, the command
57  fails with an error. If an extract configuration file is found, the system
58  will use the configuration specified in the file to perform the current
59  extract.</p>
60
61  <p>If the destination of the extract does not exist, the system performs a
62  new full extract to the destination. If a previous extract already exists at
63  the destination, the system performs an incremental extract, updating any
64  modifications if necessary. If a full (fresh) extract is required for
65  whatever reason, you can invoke the extract system using the <code>-f</code>
66  option, (i.e. the command becomes <code>fcm extract -f</code>). If you simply
67  want to remove all the items generated by a previous extract in the
68  destination, you can invoke the extract system using the <code>--clean</code>
69  option.</p>
70
71  <p>For further information on the extract command, please see <a href=
72  "command_ref.html#fcm_ext">FCM Command Reference &gt; fcm extract</a>.</p>
73
74  <h2 id="simple">Simple Usage</h2>
75
76  <p>The extract configuration file is the main user interface of the extract
77  system. It is a line based text file. For a complete set of extract
78  configuration file declarations, please refer to the <a href=
79  "annex_ext_cfg.html">Annex: Declarations in FCM extract configuration
80  file</a>.</p>
81
82  <h3 id="simple_local">Extract from a local path</h3>
83
84  <p>A simple example of a basic extract configuration file is given below:</p>
85  <pre id="example_1">
86# Example 1
87# ----------------------------------------------------------------------
88cfg::type         ext       # line 1
89cfg::version      1.0       # line 2
90                            # line 3
91dest              $PWD      # line 4
92                            # line 5
93repos::var::user  $HOME/var # line 6
94                            # line 7
95expsrc::var::user code      # line 8
96</pre>
97
98  <p>The above demonstrates how to use the extract system to extract code from
99  a local user directory. Here is an explanation of what each line does:</p>
100
101  <ul>
102    <li><dfn>line 1</dfn>: the label <code>CFG::TYPE</code> declares the type
103    of the configuration file. The value <samp>ext</samp> tells the system that
104    it is an extract configuration file.</li>
105
106    <li><dfn>line 2</dfn>: the label <code>CFG::VERSION</code> declares the
107    version of the extract configuration file. The current default is
108    <samp>1.0</samp>. Although it is not currently used, if we have to change
109    the format of the configuration file at a later stage, we shall be able to
110    use this number to determine whether we are reading a file with an older
111    format or one with a newer format.</li>
112
113    <li><dfn>line 3</dfn>: a blank line or a line beginning with a
114    <code>#</code> is a comment, and is ignored by the interpreter.</li>
115
116    <li><dfn>line 4</dfn>: the label <code>DEST</code> declares the destination
117    root directory of this extract. The value <samp>$PWD</samp> expands to the
118    current working directory.</li>
119
120    <li><dfn>line 5</dfn>: comment line, ignored.</li>
121
122    <li><dfn>line 6</dfn>: the label
123    <code>REPOS::&lt;pck&gt;::&lt;branch&gt;</code> declares the top level URL
124    or path of a repository. The package name of the repository is given by
125    &lt;pck&gt;. In our example, we choose <samp>var</samp> as the name of the
126    package. (You can choose any name you like, however, it is usually sensible
127    to use a package name that matches the name of the project or system you
128    are working with.) The branch name in the repository is given by
129    &lt;branch&gt;. (Again, you can choose any name you like, however, it is
130    usually sensible to use a name such as <samp>base</samp>, <samp>user</samp>
131    or something that matches your branch name.) In our example, the word
132    <samp>user</samp> is normally used to denote a local user directory. Hence
133    the statement declares that the repository path for the <samp>var</samp>
134    package in the <samp>user</samp> branch can be found at
135    <samp>$HOME/var</samp>.</li>
136
137    <li><dfn>line 7</dfn>: comment line, ignored.</li>
138
139    <li><dfn>line 8</dfn>: the label
140    <code>EXPSRC::&lt;pck&gt;::&lt;branch&gt;</code> declares an
141    <em>expandable</em> source directory for the package &lt;pck&gt; in the
142    branch &lt;branch&gt;. In our example, the package name is
143    <samp>var</samp>, and the branch name is <samp>user</samp>. These match the
144    package and the branch names of the repository declaration in line 6. It
145    means that the source directory declaration is associated with the path
146    <samp>$HOME/var</samp>. The value of the declaration <samp>code</samp> is
147    therefore a sub-directory under <samp>$HOME/var</samp>. By declaring a
148    source directory using an <code>EXPSRC</code> label, the system
149    automatically searches for all sub-directories (recursively) under the
150    declared source directory.</li>
151  </ul>
152
153  <p>Invoking the extract system using the above configuration file will
154  extract all sub-directories under <samp>$HOME/var/code</samp> to
155  <samp>$PWD/src/var/code</samp>. Note: the extract system ignores symbolic
156  links and hidden files, (i.e. file names beginning with a <samp>.</samp>). It
157  will write a build configuration file to <samp>$PWD/cfg/bld.cfg</samp>. The
158  configuration used for this extract will be written to the configuration file
159  at <samp>$PWD/cfg/ext.cfg</samp>.</p>
160
161  <dl>
162    <dt>Note - incremental extract</dt>
163
164    <dd>Suppose you have already performed an extract using the above
165    configuration file. At a later time, you have made some changes to some of
166    the files in the source directory. Re-running the extract system on the
167    same configuration will trigger an incremental extract. In an incremental
168    extract, the system will update only those files that are modified. If the
169    last modified time (or last commit revision) of a source file in the
170    current extract differs from that in the previous extract, the system will
171    attempt a content comparison. The system updates the destination only if
172    the content and/or file access permission of the source differs from that
173    of the destination.</dd>
174  </dl>
175
176  <h3 id="simple_url">Extract from a Subversion URL</h3>
177
178  <p>The next example demonstrates how to extract from a Subversion repository
179  URL:</p>
180  <pre id="example_2">
181# Example 2
182# ----------------------------------------------------------------------
183cfg::type           ext                    # line 1
184cfg::version        1.0                    # line 2
185                                           # line 3
186dest                $PWD                   # line 4
187                                           # line 5
188repos::var::base    svn://server/var/trunk # line 6
189revision::var::base 1234                   # line 7
190                                           # line 8
191expsrc::var::base   code                   # line 9
192</pre>
193
194  <ul>
195    <li><dfn>line 1-5</dfn>: same as <a href="#example_1">example 1</a>.</li>
196
197    <li><dfn>line 6</dfn>: the line declares the repository location of the
198    <samp>base</samp> branch of the <samp>var</samp> package to be the
199    Subversion URL <samp>svn://server/var/trunk</samp>.</li>
200
201    <li><dfn>line 7</dfn>: the label
202    <code>REVISION::&lt;pck&gt;::&lt;branch&gt;</code> declares the revision of
203    the repository associated with the package &lt;pck&gt; in the branch
204    &lt;branch&gt;. The current line tells the extract system to use revision
205    1234 of <samp>svn://server/var/trunk</samp>. It is worth noting that the
206    declared revision must be a revision when the declared branch exists. The
207    actual revision used is the last changed revision of the declared one. If
208    the revision is not declared, the default is to use the last changed
209    revision at the HEAD of the branch.</li>
210
211    <li><dfn>line 8</dfn>: comment line, ignored.</li>
212
213    <li><dfn>line 9</dfn>: the line declares an expandable source directory in
214    the repository <samp>svn://server/var/trunk</samp>.</li>
215  </ul>
216
217  <p>Invoking the extract system using the above configuration file will
218  extract all sub-directories under <samp>svn://server/var/trunk/code</samp> to
219  <samp>$PWD/src/var/code</samp>. It will write a build configuration file to
220  <samp>$PWD/cfg/bld.cfg</samp>. The configuration used for this extract will
221  be written to the configuration file at <samp>$PWD/cfg/ext.cfg</samp>.</p>
222
223  <dl>
224    <dt>EXPSRC or SRC?</dt>
225
226    <dd>
227      <p>So far, we have only declared source directories using the
228      <code>EXPSRC</code> statement, which stands for <em>expandable source
229      directory</em>. A source directory declared using this statement will
230      trigger the system to search recursively for any sub-directories under
231      the declared one. Any sub-directories containing regular source files
232      will be included in the extract. Symbolic links, hidden files and empty
233      directories (or those containing only symbolic links and/or hidden files)
234      are ignored.</p>
235
236      <p>If you do not want the system to search for sub-directories underneath
237      your declared source directory, you can declare your source directory
238      using the <code>SRC</code> statement. The <code>SRC</code> statement is
239      essentially the same as <code>EXPSRC</code> except that it does not
240      trigger the automatic recursive search for source directories. In fact,
241      the system implements the <code>EXPSRC</code> statement by expanding it
242      into a list of <code>SRC</code> statements.</p>
243    </dd>
244
245    <dt>Package and sub-package</dt>
246
247    <dd>
248      <p>The second field of a repository, revision or source directory
249      declaration label is the name of the container package. It is a name
250      selected by the user to identify the system or project he/she is working
251      on. (Therefore, it is often sensible to choose an identifier that matches
252      the name of the project or system.) The package name provides a unique
253      namespace for a file container. Source directories are automatically
254      arranged into sub-packages, using the names of the sub-directories as the
255      names of the sub-packages. For example, the declaration at line 9 in
256      <a href="#example_2">example 2</a> will put the source directory in the
257      <samp>var/code</samp> sub-package automatically.</p>
258
259      <p>Note that, in additional to slash <code>/</code>, double colon
260      <code>::</code> and double underscore <code>__</code> (internal only)
261      also act as delimiters for package names. Please avoid using them for
262      naming your files and directories.</p>
263
264      <p>You can declare a sub-package name explicitly in your source directory
265      statement. For example, the following two lines are equivalent:</p>
266      <pre>
267src::var::base                      code/VarMod_Surface
268src::var/code/VarMod_Surface::base  code/VarMod_Surface
269</pre>
270
271      <p>Explicit sub-package declaration should not be used normally, as it
272      requires a lot more typing (although there are some situations where it
273      can be useful, e.g. if you need to re-define the package name).</p>
274
275      <p>Currently, the extract system only supports non-space characters in
276      the package name, as the space character is used as a delimiter between
277      the declaration label and its value. If there are spaces in the path name
278      to a file or directory, you should explicity re-define the package name
279      of that path to a package name with no space using the above method.
280      However, we recommend that only non-space characters are used for naming
281      directories and files to make life simpler.</p>
282    </dd>
283  </dl>
284
285  <dl>
286    <dt>The expanded extract configuration file</dt>
287
288    <dd>
289      <p>At the end of a successful extract, the configuration used by the
290      current extract is written in <samp>cfg/ext.cfg</samp> under the extract
291      destination root. This file is an <em>expanded</em> version of the
292      original, with changes in the following declarations:</p>
293
294      <ul>
295        <li>All revision keywords are converted into revision numbers.</li>
296
297        <li>If a revision is not defined for a repository, it is set to the
298        corresponding revision number of the HEAD revision.</li>
299
300        <li>All URL keywords are converted into the full URLs.</li>
301
302        <li>All <code>EXPSRC</code> declarations are expanded into
303        <code>SRC</code> declarations.</li>
304
305        <li>All other variables are expanded.</li>
306      </ul>
307
308      <p>With this file, it should be possible for a later extract to re-create
309      the current configuration even if the contents of the repository have
310      changed. (This applies only to code stored in the repository.)</p>
311    </dd>
312  </dl>
313
314  <h3 id="simple_mirror">Mirror code to an alternate location</h3>
315
316  <p>The next example demonstrates how to extract from a repository and mirror
317  the code to an alternate location. It is essentially the same as <a href=
318  "#example_2">example 2</a>, except that it has three new lines to describe
319  how the system can mirror the extracted code to an alternate location.</p>
320  <pre id="example_3">
321# Example 3
322# ----------------------------------------------------------------------
323cfg::type           ext
324cfg::version        1.0
325
326dest                $PWD
327
328rdest::machine      tx01                           # line 6
329rdest::logname      frva                           # line 7
330rdest               /scratch/frva/extract/example3 # line 8
331
332repos::var::base    svn://server/var/trunk
333revision::var::base 1234
334
335expsrc::var::base   code
336</pre>
337
338  <p>Here is an explanation of what each line does:</p>
339
340  <ul>
341    <li><dfn>line 6</dfn>: <code>RDEST::MACHINE</code> declares the target
342    machine to which the code will be mirrored. The example mirrors the code to
343    the machine named <samp>tx01</samp>.</li>
344
345    <li><dfn>line 7</dfn>: <code>RDEST::LOGNAME</code> declares the user name
346    of the target machine, to which the user has login access. If this is not
347    declared, the system uses the login name of the current user on the local
348    machine.</li>
349
350    <li><dfn>line 8</dfn>: <code>RDEST</code> declares the root directory of
351    the alternate destination, where the mirror version of the extract will be
352    sent.</li>
353  </ul>
354
355  <p>Invoking the extract system on the above configuration will trigger an
356  extract similar to that given in <a href="#example_2">example 2</a>, but it
357  will also attempt to mirror the contents at <samp>$PWD/src/var/code</samp> to
358  <samp>/scratch/frva/extract/example3/src</samp> on the alternate destination.
359  It will also mirror the expanded extract configuration file
360  <samp>$PWD/cfg/ext.cfg</samp> to
361  <samp>/scratch/frva/extract/example3/cfg/ext.cfg</samp> and
362  <samp>$PWD/cfg/bld.cfg</samp> to
363  <samp>/scratch/frva/extract/example3/cfg/bld.cfg</samp>. It is also worth
364  noting that the content of the build configuration file will be slightly
365  different, since it will include directory names appropriate for the
366  alternate destination.</p>
367
368  <dl>
369    <dt>Note - mirroring command</dt>
370
371    <dd>
372      <p>The extract system currently supports <code>rdist</code> and
373      <code>rsync</code> as its mirroring tool. The default is
374      <code>rsync</code>. To use <code>rdist</code> instead of
375      <code>rsync</code>, add the following line to your extract configuration
376      file:</p>
377      <pre>
378rdest::mirror_cmd  rdist
379</pre>
380
381      <p>If <code>rsync</code> is used to mirror an extract, the system needs to
382      issue a separate remote shell command to create the container directory of
383      the mirror destination. The default is to issue a shell command in the
384      form <samp>ssh -n -oBatchMode=yes LOGNAME@MACHINE mkdir -p DEST</samp>.
385      The following declarations can be used to modify the command:</p>
386      <pre>
387# Examples using the default settings:
388rdest::rsh_mkdir_rsh         ssh
389rdest::rsh_mkdir_rshflags    -n -oBatchMode=yes
390rdest::rsh_mkdir_mkdir       mkdir
391rdest::rsh_mkdir_mkdirflags  -p
392</pre>
393
394      <p>In addition, the default <code>rsync</code> shell command is
395      <samp>rsync -a --exclude='.*' --delete-excluded --timeout=900 --rsh='ssh
396      -oBatchMode=yes' SOURCE DEST</samp>. The following declarations can be
397      used to modify the command:</p>
398      <pre>
399# Examples using the default settings:
400rdest::rsync       rsync
401rdest::rsyncflags  -a --exclude='.*' --delete-excluded --timeout=900 \
402                   --rsh='ssh -oBatchMode=yes'
403</pre>
404    </dd>
405  </dl>
406
407  <h2 id="advanced">Advanced Usage</h2>
408
409  <h3 id="advanced_multi">Extract from multiple repositories</h3>
410
411  <p>So far, we have only extracted from a single location. The extract system
412  is not much use if that is the only thing it can do. In fact, the extract
413  system supports extract of multiple source directories from multiple branches
414  in multiple repositories. The following configuration file is an example of
415  how to extract from multiple repositories:</p>
416  <pre id="example_4">
417# Example 4
418# ----------------------------------------------------------------------
419cfg::type           ext
420cfg::version        1.0
421
422dest                $PWD
423
424repos::var::base    fcm:var_tr              # line 6
425repos::ops::base    fcm:ops_tr              # line 7
426repos::gen::base    fcm:gen_tr              # line 8
427
428revision::gen::base 2468                    # line 10
429
430expsrc::var::base   src/code                    # line 12
431expsrc::var::base   src/scripts                 # line 13
432expsrc::ops::base   src/code                    # line 14
433src::gen::base      src/code/GenMod_Constants   # line 15
434src::gen::base      src/code/GenMod_Control     # line 16
435src::gen::base      src/code/GenMod_FortranIO   # line 17
436src::gen::base      src/code/GenMod_GetEnv      # line 18
437src::gen::base      src/code/GenMod_ModelIO     # line 19
438src::gen::base      src/code/GenMod_ObsInfo     # line 20
439src::gen::base      src/code/GenMod_Platform    # line 21
440src::gen::base      src/code/GenMod_Reporting   # line 22
441src::gen::base      src/code/GenMod_Trace       # line 23
442src::gen::base      src/code/GenMod_UMConstants # line 24
443src::gen::base      src/code/GenMod_Utilities   # line 25
444</pre>
445
446  <p>Here is an explanation of what each line does:</p>
447
448  <ul>
449    <li><dfn>line 6-8</dfn>: these lines declare the repositories for the
450    <samp>base</samp> branches of the <samp>var</samp>, <samp>ops</samp> and
451    <samp>gen</samp> packages respectively. It is worth noting that the values
452    of the declarations are no longer Subversion URLs but are FCM URL keywords.
453    These keywords are normally declared in the central configuration file of
454    the FCM system, and will be expanded into the corresponding Subversion URLs
455    by the FCM system. For further information on URL keywords, please see
456    <a href="code_management.html#svn_basic_keywords">Code Management System
457    &gt; Using Subversion &gt; Basic Command Line Usage &gt; Repository &amp;
458    Revision Keywords</a>.</li>
459
460    <li><dfn>line 10</dfn>: this line declares the revision number for the
461    <samp>base</samp> branch of the <samp>gen</samp> package, i.e. for the
462    <samp>fcm:gen_tr</samp> repository. It is worth noting that the revision
463    numbers for the <samp>var</samp> and <samp>ops</samp> packages have not
464    been declared. By default, their revision numbers will be set to the last
465    changed revision at the HEAD.</li>
466
467    <li><dfn>line 12-14</dfn>: these line declares the source directories for
468    the <samp>base</samp> branches of the <samp>var</samp> and <samp>ops</samp>
469    packages. For the <samp>var</samp> package, we are extracting everything
470    from the <samp>code</samp> and the <samp>scripts</samp> sub-directory. For
471    the <samp>ops</samp> package, we are extracting everything from the
472    <samp>code</samp> directory.</li>
473
474    <li><dfn>line 15-25</dfn>: these line declares the source directories for
475    the <samp>base</samp> branch of the <samp>gen</samp> package. The source
476    directories declared will not be searched for sub-directories underneath
477    the declared directories.</li>
478  </ul>
479
480  <p>We shall end up with a directory tree such as:</p>
481  <pre>
482$PWD
483   |
484   |--- cfg
485   |      |
486   |      |--- bld.cfg
487   |      |--- ext.cfg
488   |
489   |--- src
490          |
491          |--- gen
492          |      |
493          |      |--- code
494          |              |
495          |              |--- GenMod_Constants
496          |              |--- GenMod_Control
497          |              |--- GenMod_FortranIO
498          |              |--- GenMod_GetEnv
499          |              |--- GenMod_ModelIO
500          |              |--- GenMod_ObsInfo
501          |              |--- GenMod_Platform
502          |              |--- GenMod_Reporting
503          |              |--- GenMod_Trace
504          |              |--- GenMod_UMConstants
505          |              |--- GenMod_Utilities
506          |
507          |--- ops
508          |      |
509          |      |--- code
510          |              |
511          |              |--- ...
512          |
513          |--- var
514                 |
515                 |--- code
516                 |       |
517                 |       |--- ...
518                 |
519                 |--- scripts
520                         |
521                         |--- ...
522</pre>
523
524  <dl>
525    <dt>Note - revision number</dt>
526
527    <dd>
528      <p>As seen in the above example, if a revision number is not specified
529      for a repository URL, it defaults to the last changed revision at the
530      HEAD of the branch. The revision number can also be declared in other
531      ways:</p>
532
533      <ul>
534        <li>Any revision arguments acceptable by Subversion are allowed. You
535        can use a valid revision number, a date between a pair of curly
536        brackets (e.g. <samp>{2005-05-01T12:00}</samp>) or the keyword HEAD.
537        However, please do not use the keywords BASE, COMMITTED or PREV as
538        these are reserved for working copy only.</li>
539
540        <li>FCM revision keywords are allowed. These must be defined for the
541        corresponding repository URLs in either the central or the user FCM
542        configuration file. For further information on revision keywords,
543        please see <a href="code_management.html#svn_basic_keywords">Code
544        Management &gt; Using Subversion &gt; Basic Command Line Usage &gt;
545        Repository &amp; Revision Keywords</a>.</li>
546
547        <li>Do not use the keyword USER, as it is used internally by the
548        extract system.</li>
549      </ul>
550
551      <p>If a revision number is specified for a branch, the actual revision
552      used by the extract system is the last changed revision of the branch,
553      which may differ from the declared revision. While this behaviour is
554      useful in most situations, some users may find it confusing to work with.
555      It is possible to alter this behaviour so that extract will fail if the
556      declared revision does not correspond to a changeset of the declared
557      branch. Make the following declaration to switch on this checking:</p>
558      <pre>
559revmatch  true
560</pre>
561    </dd>
562  </dl>
563
564  <h3 id="advanced_branches">Extract from multiple branches</h3>
565
566  <p>We have so far dealt with a single branch in any package. The extract
567  system can be used to <em>combine</em> changes from different branches of a
568  package. An example is given below:</p>
569  <pre id="example_5">
570# Example 5
571# ----------------------------------------------------------------------
572cfg::type               ext
573cfg::version            1.0
574 
575dest                    $PWD
576 
577repos::var::base        fcm:var_tr
578repos::ops::base        fcm:ops_tr
579repos::gen::base        fcm:gen_tr
580 
581revision::gen::base     2468
582 
583expsrc::var::base       src/code
584expsrc::var::base       src/scripts
585expsrc::ops::base       src/code
586src::gen::base          src/code/GenMod_Constants
587src::gen::base          src/code/GenMod_Control
588src::gen::base          src/code/GenMod_FortranIO
589src::gen::base          src/code/GenMod_GetEnv
590src::gen::base          src/code/GenMod_ModelIO
591src::gen::base          src/code/GenMod_ObsInfo
592src::gen::base          src/code/GenMod_Platform
593src::gen::base          src/code/GenMod_Reporting
594src::gen::base          src/code/GenMod_Trace
595src::gen::base          src/code/GenMod_UMConstants
596src::gen::base          src/code/GenMod_Utilities
597
598repos::var::branch1     fcm:var_br/frva/r1234_new_stuff   # line 27
599repos::var::branch2     fcm:var_br/frva/r1516_bug_fix     # line 28
600repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff # line 29
601</pre>
602
603  <p>The configuration file in <a href="#example_5">example 5</a> is similar to
604  that of <a href="#example_4">example 4</a> except for the last three lines.
605  Here is an explanation of what they do:</p>
606
607  <ul>
608    <li>
609      <dfn>line 27</dfn>: this line declares a repository URL for the
610      <samp>branch1</samp> branch of the <samp>var</samp> package. From the URL
611      of the branch, we know that the branch was created by the user
612      <samp>frva</samp> based on the trunk at revision at 1234. The description
613      of the branch is <samp>branch1</samp>. The following points are worth
614      noting:
615
616      <ul>
617        <li>By declaring a new branch with the same package name to a
618        previously declared branch, it is assumed that both branches reside in
619        the same Subversion repository.</li>
620
621        <li>No revision is declared for this URL, so the default is used which
622        is the last changed revision at the HEAD of the branch.</li>
623
624        <li>No source directory is declared for this URL. By default, if no
625        source directory is declared for a branch repository, it will attempt
626        to use the same set of source directories as the first declared branch
627        of the package. In this case, the source directories declared for the
628        <samp>base</samp> branch of the <samp>var</samp> package will be
629        used.</li>
630      </ul>
631    </li>
632
633    <li><dfn>line 28</dfn>: this line declares another branch called
634    <samp>branch2</samp> for the <samp>var</samp> package. No source directory
635    is declared for this URL either, so it will use the same set of source
636    directories declared for the <samp>base</samp> branch.</li>
637
638    <li><dfn>line 29</dfn>: this line declares a branch called
639    <samp>branch1</samp> for the <samp>ops</samp> package. It will use the same
640    set of source directories declared for the <samp>ops</samp> package
641    <samp>base</samp> branch.</li>
642  </ul>
643
644  <p>When we invoke the extract system, it will attempt to extract from the
645  first declared branch of a package, if the last changed revision of the
646  source directory is the same in all the branches. However, if the last
647  changed revision of the source directory differs for different branches, the
648  system will attempt to obtain an extract priority list for each source
649  directory, using the following logic:</p>
650
651  <ol>
652    <li>The system looks for source directory packages from the first declared
653    branch to the last declared branch.</li>
654
655    <li>The branch in which a source directory package is first declared is the
656    <samp>base</samp> branch of the source directory package.</li>
657
658    <li>The last changed revision of a source directory package in a
659    subsequently declared repository branch is compared with that of the base
660    branch. If the last changed revision is the same as that of the base
661    branch, the source directory of this branch is discarded. Otherwise, it is
662    placed at the end of the extract priority list.</li>
663  </ol>
664
665  <p>For the <samp>var</samp> package in the above example, let us assume that
666  we have three source directory packages X, Y and Z under <samp>code</samp>,
667  and their last changed revisions under <samp>base</samp> are 100. Let's say
668  we have committed some changes to X and Z in the <samp>branch1</samp> branch
669  at revision 102, and other changes to Y and Z in the <samp>branch2</samp>
670  branch at revision 104, the extract priority lists for X, Y and Z will look
671  like:</p>
672
673  <ul>
674    <li>X: base (100, base), branch1 (102), branch2 (100, discarded)</li>
675
676    <li>Y: base (100, base), branch1 (100, discarded), branch2 (104)</li>
677
678    <li>Z: base (100, base), branch1 (102), branch2 (104)</li>
679  </ul>
680
681  <p>Once we have an extract priority list for a source directory, we can begin
682  extracting source files in the source directory. The source directory of the
683  base branch is extracted first, followed by that in the subsequent branches.
684  If a source file in a subsequent branch has the same content as the that in
685  the base branch, it is discarded. Otherwise, the following logic determines
686  the branch to use:</p>
687
688  <ol>
689    <li>If a source file is modified in only one subsequent branch, the source
690    file in that branch is extracted.</li>
691
692    <li>If a source file is modified in two or more subsequent branches, but
693    their modifications are the same, then the source file in the first
694    modification is used.</li>
695
696    <li>If a source file is modified in two or more subsequent branches and
697    their modifications differ, then the behaviour depends on the "conflict
698    mode" setting, which can be <code>fail</code>, <code>merge</code> (default)
699    and <code>override</code>. If the conflict mode is <code>fail</code>, the
700    extract fails. If the conflict mode is <code>merge</code>, the system will
701    attempt to merge the changes using a tool such as <code>diff3</code>. The
702    result of the merge will be used to update the destination. The extract
703    fails only if there are unresolved conflicts in the merge. (In which case,
704    the conflict should be resolved using the version control system before
705    re-running the extract system.) If the conflict mode is
706    <code>override</code>, the change in the latest declared branch takes
707    precedence, and the changes in all other branches will be ignored. The
708    conflict mode can be changed using the <code>CONFLICT</code> declaration in
709    the extract configuration file. E.g:
710      <pre>
711conflict  fail
712</pre>
713    </li>
714  </ol>
715
716  <p>Once the system has established which source files to use, it determines
717  whether the destination file is out of date or not. The destination file is
718  out of date if it does not exist or if its content differs from the version
719  of the source file we are using. The system only updates the destination if
720  it is considered to be out of date.</p>
721
722  <p>The extract system can also combine changes from branches in the
723  Subversion repository and the local file system. The limitation is that there
724  can only be one branch from the local file system. (By convention, the branch
725  is named <samp>user</samp>.)</p>
726
727  <p>It is also worth bearing in mind that the <samp>user</samp> branch always
728  takes precedence over branches residing in Subversion repositories. Hence,
729  source directories from a <samp>user</samp> branch are always placed at the
730  end of the extract priority list.</p>
731
732  <p>Extracting from a mixture of Subversion repository and local file system
733  is demonstrated in the next example.</p>
734  <pre id="example_6">
735# Example 6
736# ----------------------------------------------------------------------
737cfg::type               ext
738cfg::version            1.0
739 
740dest                    $PWD
741 
742repos::var::base        fcm:var_tr
743repos::ops::base        fcm:ops_tr
744repos::gen::base        fcm:gen_tr
745 
746revision::gen::base     2468
747 
748expsrc::var::base       src/code
749expsrc::var::base       src/scripts
750expsrc::ops::base       src/code
751src::gen::base          src/code/GenMod_Constants
752src::gen::base          src/code/GenMod_Control
753src::gen::base          src/code/GenMod_FortranIO
754src::gen::base          src/code/GenMod_GetEnv
755src::gen::base          src/code/GenMod_ModelIO
756src::gen::base          src/code/GenMod_ObsInfo
757src::gen::base          src/code/GenMod_Platform
758src::gen::base          src/code/GenMod_Reporting
759src::gen::base          src/code/GenMod_Trace
760src::gen::base          src/code/GenMod_UMConstants
761src::gen::base          src/code/GenMod_Utilities
762
763repos::var::branch1     fcm:var_br/frva/r1234_new_stuff
764repos::var::branch2     fcm:var_br/frva/r1516_bug_fix
765repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff
766
767repos::var::user        $HOME/var                         # line 31
768repos::gen::user        $HOME/gen                         # line 32
769</pre>
770
771  <p><a href="#example_6">Example 6</a> is similar to <a href=
772  "#example_5">example 5</a> except that it is also extracting from local
773  directories. Here is an explanation of the lines:</p>
774
775  <ul>
776    <li><dfn>line 31-32</dfn>: these line declare the repositories for the
777    <samp>user</samp> branches of the <samp>var</samp> and <samp>gen</samp>
778    packages respectively. Both are local paths at the local file system. There
779    are no declarations for source directories for the <samp>user</samp>
780    branches, so they use the same set of source directories of the first
781    declared branches, the <samp>base</samp> branches in both cases.</li>
782  </ul>
783
784  <dl>
785    <dt>Note - the INC declaration</dt>
786
787    <dd>
788      You have probably realised that the above examples have many repeated
789      lines. To avoid having repeated lines in multiple extract configuration
790      files, you can use <code>INC</code> declarations to include other extract
791      configuration files. For example, if the configuration file of example 5
792      is stored in the file <samp>$HOME/example5/ext.cfg</samp>, line 1 to 29
793      of <a href="#example_6">example 6</a> can be replaced with an
794      <code>INC</code> declaration. <a href="#example_6">Example 6</a> can then
795      be written as:
796      <pre>
797inc                     $HOME/example5/ext.cfg
798
799repos::var::user        $HOME/var
800repos::gen::user        $HOME/gen
801</pre>
802
803      <p>Note: the <code>INC</code> declaration supports the special
804      environment variable <var>$HERE</var>. If this variable is already set in
805      the environment, it acts as a normal environment variable. However, if it
806      is not set, it will be expanded into the container directory of the
807      current extract configuration file. This feature is particularly useful
808      if you are including a hierarchy of extract configurations from files in
809      the same container directory in a repository.</p>
810    </dd>
811  </dl>
812
813  <h3 id="advanced_inherit">Inherit from a previous extract</h3>
814
815  <p>All the examples above dealt with standalone extract, that is, the current
816  extract is independent of any other extract. If a previous extract exists in
817  another location, the extract system can inherit from this previous extract
818  in your current extract. This works like a normal incremental extract, except
819  that your extract will only contain the changes you have specified (compared
820  with the inherited extract) instead of the full source directory tree. This
821  type of incremental extract is useful in several ways. For instance:</p>
822
823  <ul>
824    <li>It is fast, because you only have to extract and mirror files that you
825    have changed.</li>
826
827    <li>The subsequent build will also be fast, since it will use incremental
828    build.</li>
829
830    <li>You do not need write access to the original extract. A system
831    administrator can set up a stable version in a central account, which
832    developers can then inherit from.</li>
833
834    <li>You want an incremental extract, but you need to leave the original
835    extract unmodified.</li>
836  </ul>
837
838  <p>The following example is based on <a href="#example_4">example 4</a> and
839  <a href="#example_6">example 6</a>. The assumption is that an extract has
840  already been performed at the directory <samp>~frva/var/vn22.0</samp> based
841  on the configuration file in <a href="#example_4">example 4</a>.</p>
842  <pre id="example_7">
843# Example 7
844# ----------------------------------------------------------------------
845cfg::type               ext
846cfg::version            1.0
847 
848dest                    $PWD
849
850use                     ~frva/var/vn22.0                  # line 6
851
852repos::var::branch1     fcm:var_br/frva/r1234_new_stuff   # line 8
853repos::var::branch2     fcm:var_br/frva/r1516_bug_fix     # line 9
854repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff # line 10
855
856repos::var::user        $HOME/var                         # line 12
857repos::gen::user        $HOME/gen                         # line 13
858</pre>
859
860  <ul>
861    <li><dfn>line 6</dfn>: this line replaces line 1 to 25 of <a href=
862    "#example_6">example 6</a>. It declares that the current extract should
863    inherit from the previous extract located at
864    <samp>~frva/var/vn22.0</samp>.</li>
865  </ul>
866
867  <p>Running the extract system using the above configuration will trigger an
868  incremental extract, as if you are running an incremental extract having
869  modified the configuration file in <a href="#example_4">example 4</a> to that
870  of <a href="#example_6">example 6</a>. The only difference is that the
871  original extract using the <a href="#example_4">example 4</a> configuration
872  will be left untouched at <samp>~frva/var/vn22.0</samp>, and the new extract
873  will contain only the changes in the branches declared from line 8 to 13.</p>
874
875  <p>Note: extract inheritance allows you to add more branches to a package,
876  but you should not redefine the <code>REPOS</code>, <code>REVISION</code>,
877  <code>EXPSRC</code> or <code>SRC</code> declarations of a branch that is
878  already declared (and already extracted) in the inherited extract. Although
879  the system will not stop you from doing so, you may end up with an extract
880  that does not quite do what it is supposed to do. For example, if the
881  <samp>base</samp> branch in the <samp>foo</samp> package
882  (<tt>repos::foo::base</tt>) is already defined and extracted in an extract
883  you are inheriting from, you should not redefine any of the
884  <tt>*::foo::base</tt> declarations in your current extract. However, you are
885  free to add more branches for the same package with new labels (e.g.
886  <tt>repos::foo::b1</tt>), and indeed new packages that are not already
887  defined in the inherited extract (e.g. <tt>repos::bar::base</tt>).</p>
888
889  <p>If you are setting up an extract to be inherited, you do not have to
890  perform a build. If you don't you will still gain the benefit of incremental
891  file extract, but you will be performing a full build of the code.</p>
892
893  <dl>
894    <dt>Note - inherit and mirror</dt>
895
896    <dd>
897      <p>It is worth bearing in mind that <tt>rdest::*</tt> settings are not
898      inherited. If mirroring is required in the inheriting extract, it will
899      require its own set of <tt>rdest::*</tt> declarations.</p>
900
901      <p>The system will, however, assume that a mirrored version of the
902      inherited extract is available for inheritance from the mirrored
903      destination of the current extract.</p>
904     
905      <p>E.g.: Consider an extract at <samp>/path/to/inherited/</samp> and an
906      inheriting extract at <samp>/path/to/current/</samp>. If the former does
907      not have a mirror, the latter should not have one either. If the former
908      mirrors to <samp>machine@/path/to/inherited/mirror/</samp> and the latter
909      mirrors to <samp>machine@/path/to/current/mirror/</samp>, the system will
910      assume that the subsequent build at
911      <samp>machine@/path/to/current/mirror/</samp> can inherit from the build
912      at <samp>machine@/path/to/inherited/mirror/</samp>. This is illustrated
913      below:</p>
914
915      <pre>
916/path/to/current/       =&gt; at machine: /path/to/current/mirror/
917use /path/to/inherited/ =&gt; at machine: use /path/to/inherited/mirror/
918</pre>
919    </dd>
920  </dl>
921
922  <h3 id="advanced_build">Extract - Build Configuration</h3>
923
924  <p>Configuration settings for feeding into the build system can be declared
925  through the extract configuration file using the <code>BLD::</code> prefix.
926  Any line in an extract configuration containing a label with such a prefix
927  will be considered a build system variable. At the end of a successful
928  extract, the system strips out the <code>BLD::</code> prefix before writing
929  these variables to the build configuration file. Some example entries are
930  given between line 17 and 22 in the following configuration file:</p>
931  <pre id="example_8">
932# Example 8
933# ----------------------------------------------------------------------
934cfg::type           ext
935cfg::version        1.0
936
937dest                $PWD
938
939repos::var::base    fcm:var_tr
940repos::ops::base    fcm:ops_tr
941repos::gen::base    fcm:gen_tr
942
943revision::gen::base 2468
944
945expsrc::var::base   src/code
946expsrc::var::base   src/scripts
947expsrc::ops::base   src/code
948src::gen::base      src/code/GenMod_Constants
949src::gen::base      src/code/GenMod_Control
950src::gen::base      src/code/GenMod_FortranIO
951src::gen::base      src/code/GenMod_GetEnv
952src::gen::base      src/code/GenMod_ModelIO
953src::gen::base      src/code/GenMod_ObsInfo
954src::gen::base      src/code/GenMod_Platform
955src::gen::base      src/code/GenMod_Reporting
956src::gen::base      src/code/GenMod_Trace
957src::gen::base      src/code/GenMod_UMConstants
958src::gen::base      src/code/GenMod_Utilities
959
960bld::target         VarProg_AnalysePF.exe   # line 27
961
962bld::tool::fc       sxmpif90                # line 29
963bld::tool::cc       sxmpic++                # line 30
964bld::tool::ld       sxmpif90                # line 31
965</pre>
966
967  <p>The above example is essentially the same as <a href="#example_4">example
968  4</a>, apart from the additional build configuration. The following is a
969  simple explanation of what the lines represent: (For detail of the build
970  system, please see the next chapter on <a href="build.html">The Build
971  System</a>.)</p>
972
973  <ul>
974    <li><dfn>line 27</dfn>: the line declares a default target of the
975    build.</li>
976
977    <li><dfn>line 29-31</dfn>: the lines declare the Fortran compiler, the C
978    compiler and the linker respectively.</li>
979  </ul>
980
981  <dl>
982    <dt>Note - use of variables</dt>
983
984    <dd>
985      <p>When you start using the extract system to define compiler flags for
986      the build system, you may end up having to make a lot of long and
987      repetitive declarations. In this case, you may want to define variables
988      to replace the repetitive parts of the declarations.</p>
989
990      <p>Environment variables whose names contain only upper case latin
991      alphabets, numbers and underscores can be referenced in a declaration
992      value via the syntax <code>$NAME</code> or <code>${NAME}</code>. For
993      example:</p>
994      <pre>
995repos::um::base    ${HOME}/svn-wc/um
996bld::tool::fflags  $MY_FFLAGS
997</pre>
998     
999      <p>You can define a user variable by making a declaration with a label
1000      that begins with a percent sign <code>%</code>. The value of a user
1001      variable remains in memory until the end of the current file is reached.
1002      You can reference a user variable in a declaration value via the syntax
1003      <code>%NAME</code> or <code>%{NAME}</code>. For example:</p>
1004      <pre>
1005# Declare a variable %fred
1006%fred                     -Cdebug -eC -Wf,-init heap=nan stack=nan
1007
1008bld::tool::fflags         %fred
1009# bld::tool::fflags       -Cdebug -eC -Wf,-init heap=nan stack=nan
1010
1011bld::tool::fflags::foo    %fred -f0
1012# bld::tool::fflags::foo  -Cdebug -eC -Wf,-init heap=nan stack=nan -f0
1013
1014bld::tool::fflags::bar    -w %fred
1015# bld::tool::fflags::bar  -w -Cdebug -eC -Wf,-init heap=nan stack=nan
1016</pre>
1017
1018      <p>Further to this, each declaration results in an internal variable of
1019      the same name and you can also refer to any of these internal variables in
1020      the same way. So, the example given above could also be written as
1021      follows:</p>
1022      <pre>
1023bld::tool::fflags         -Cdebug -eC -Wf,-init heap=nan stack=nan
1024bld::tool::fflags::foo    %bld::tool::fflags -f0
1025bld::tool::fflags::bar    -w %bld::tool::fflags
1026</pre>
1027    </dd>
1028
1029    <dt>Note - as-parsed configuration</dt>
1030
1031    <dd>
1032      <p>If you use a hierarchy of <code>INC</code> declarations or variables,
1033      you may end up with a configuration file that is difficult to understand.
1034      To help you with this, the extract system generates an as-parsed
1035      configuration file at <samp>cfg/parsed_ext.cfg</samp> of the destination.
1036      The content of the as-parsed configuration file is what the extract
1037      system actually reads. It should contain everything in your original
1038      extract configuration file, except that all <code>INC</code>
1039      declarations, environment variables and user/internal variables are
1040      expanded.</p>
1041    </dd>
1042  </dl>
1043
1044  <h2 id="verbose">Diagnostic verbose level</h2>
1045
1046  <p>The amount of diagnostic messages generated by the extract system is
1047  normally set to a level suitable for normal everyday operation. This is the
1048  default diagnostic verbose level 1. If you want a minimum amount of
1049  diagnostic messages, you should set the verbose level to 0. If you want more
1050  diagnostic messages, you can set the verbose level to 2 or 3. You can modify
1051  the verbose level in two ways. The first way is to set the environment
1052  variable <var>FCM_VERBOSE</var> to the desired verbose level. The second way
1053  is to invoke the extract system with the <code>-v &lt;level&gt;</code>
1054  option. (If set, the command line option overrides the environment
1055  variable.)</p>
1056
1057  <p>The following is a list of diagnostic output at each verbose level:</p>
1058
1059  <dl>
1060    <dt>Level 0</dt>
1061
1062    <dd>
1063      <ul>
1064        <li>Report the time taken to extract the code.</li>
1065
1066        <li>Report the time taken to mirror the code.</li>
1067
1068        <li>If <code>rdist</code> is used to mirror the code, run the command
1069        with the <code>-q</code> option.</li>
1070      </ul>
1071    </dd>
1072
1073    <dt>Level 1</dt>
1074
1075    <dd>
1076      <ul>
1077        <li>Everything at verbose level 0.</li>
1078
1079        <li>Report the name of the extract configuration file.</li>
1080
1081        <li>Report the location of the extract destination.</li>
1082
1083        <li>Report date/time at the beginning of the extract step.</li>
1084
1085        <li>If the revision specified for a repository branch is not its last
1086        changed revision, print an information statement to inform the user of
1087        the last changed revision of the branch.</li>
1088
1089        <li>Summarises the destination status and the source status.</li>
1090
1091        <li>Report date/time at the beginning of the mirror step.</li>
1092
1093        <li>Report the location of the alternate destination.</li>
1094
1095        <li>Report total time.</li>
1096      </ul>
1097    </dd>
1098
1099    <dt>Level 2</dt>
1100
1101    <dd>
1102      <ul>
1103        <li>Everything at verbose level 1.</li>
1104
1105        <li>If the revision specified for a repository branch is not current
1106        (i.e. the specified revision number is less than the revision number of
1107        the last commit revision), print an information statement to inform the
1108        user of the last commit revision of the branch.</li>
1109
1110        <li>Report the detail of each change in the destination.</li>
1111
1112        <li>If <code>rdist</code> is used to mirror the code, run the command
1113        without the <code>-q</code> option.</li>
1114      </ul>
1115    </dd>
1116
1117    <dt>Level 3</dt>
1118
1119    <dd>
1120      <ul>
1121        <li>Everything at verbose level 2.</li>
1122
1123        <li>Report all shell commands invoked by the extract system with
1124        timestamp.</li>
1125
1126        <li>If <code>rdist</code> is used to mirror the code, print the
1127        <samp>distfile</samp> supplied to the command.</li>
1128
1129        <li>If <code>rsync</code> is used to mirror the code, invoke the
1130        command with the <code>-v</code> option.</li>
1131      </ul>
1132    </dd>
1133  </dl>
1134
1135  <h2 id="nosvn">When Subversion Is Not Available</h2>
1136
1137  <p>The extract system can still be used if Subversion is not available.
1138  Clearly, you can only use local repositories. However, you can still do
1139  incremental extract, mirror an extract to an alternate location, or combine
1140  code from multiple local repositories.</p>
1141
1142  <p>If you are using Subversion but your server is down then clearly there is
1143  little you can do. However, if you already have an extract then you can
1144  re-run <code>fcm extract</code> as long as the extract configuration file
1145  only refers to fixed revisions. If this is not the case then you can always
1146  use the expanded extract configuration file which can be found in
1147  <samp>cfg/ext.cfg</samp> under the extract destination root. This means that
1148  you can continue to makes changes to local code and do incremental extracts
1149  even whilst your Subversion server is down.</p>
1150</body>
1151</html>
Note: See TracBrowser for help on using the repository browser.