New URL for NEMO forge! http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.

extract.html in vendors/doc/user_guide – NEMO

Context Navigation

source: vendors/doc/user_guide/extract.html @ 10669

Last change on this file since 10669 was 10669, checked in by nicolasmartin, 5 years ago
Import latest FCM release from Github into the repository for testing
File size: 50.3 KB

Line
1	<!DOCTYPE html>
2	<html>
3	<head>
4	<title>FCM: User Guide: Annex: The FCM 1 Extract System</title>
5	<meta name="author" content="FCM team" />
6	<meta name="viewport" content="width=device-width, initial-scale=1.0" />
7	<link rel="icon" href="../etc/fcm-icon.png" type="image/png" />
8	<link rel="shortcut icon" href="../etc/fcm-icon.png" type="image/png" />
9	<link href="../etc/bootstrap/css/bootstrap.min.css" rel="stylesheet" media="screen" />
10	<link href="../etc/fcm.css" rel="stylesheet" media="screen" />
11	</head>
12	<body>
13	<div class="navbar navbar-inverse">
14	<div class="container-fluid">
15	<div class="navbar-header">
16	<a class="navbar-brand" href=".."><span class="fcm-version">FCM</span></a>
17	</div>
18	<div class="collapse navbar-collapse">
19	<ul class="nav navbar-nav">
20	<li><a href="../installation/">Installation</a></li>
21
22	<li><a class="active" href="#">User Guide</a></li>
23	</ul>
24	</div>
25	</div>
26	</div>
27
28	<div class="page-header">
29	<div class="fcm-page-content pull-right well well-sm"></div>
30	<h1>FCM: User Guide: Annex: The FCM 1 Extract System</h1>
31	</div>
32
33	<div class="container">
34	<div class="row">
35	<div class="col-md-12">
36
37	<h2 id="introduction">Introduction</h2>
38
39	<p><em>The FCM 1 extract system is deprecated. The documentation for the
40	current extract system can be found at <a href="make.html">FCM
41	Make</a>.</em></p>
42
43	<p>The extract system provides an interface between the revision control
44	system (currently Subversion) and the build system. Where appropriate, it
45	extracts code from the repository and other user-defined locations to a
46	directory tree suitable for feeding into the build system. In this chapter,
47	we shall use many examples to explain how to use the extract system. At the
48	end of this chapter, you will be able to extract code from the local file
49	system as well as from different branches of different repository URLs. You
50	will also learn how to mirror code to an alternate destination. Finally, you
51	will be given an introduction on how to specify configurations for the build
52	system via the extract configuration file. (For further information on the
53	build system, please see the next chapter <a href="build.html">The Build
54	System</a>.) The last section of the chapter tells you what you can do in the
55	case when Subversion is not available.</p>
56
57	<h2 id="command">The Extract Command</h2>
58
59	<p>To invoke the extract system, simply issue the command:</p>
60	<pre>
61	fcm extract
62	</pre>
63
64	<p>By default, the extract system searches for an extract configuration file
65	<samp>ext.cfg</samp> in <samp>$PWD</samp> and then <samp>$PWD/cfg</samp>. If
66	an extract configuration file is not found in these directories, the command
67	fails with an error. If an extract configuration file is found, the system
68	will use the configuration specified in the file to perform the current
69	extract.</p>
70
71	<p>If the destination of the extract does not exist, the system performs a
72	new full extract to the destination. If a previous extract already exists at
73	the destination, the system performs an incremental extract, updating any
74	modifications if necessary. If a full (fresh) extract is required for
75	whatever reason, you can invoke the extract system using the <code>-f</code>
76	option, (i.e. the command becomes <code>fcm extract -f</code>). If you simply
77	want to remove all the items generated by a previous extract in the
78	destination, you can invoke the extract system using the <code>--clean</code>
79	option.</p>
80
81	<p>For further information on the extract command, please see <a href=
82	"command_ref.html#fcm-extract">FCM Command Reference > fcm extract</a>.</p>
83
84	<h2 id="simple">Simple Usage</h2>
85
86	<p>The extract configuration file is the main user interface of the extract
87	system. It is a line based text file. For a complete set of extract
88	configuration file declarations, please refer to the <a href=
89	"annex_ext_cfg.html">Annex: Declarations in FCM extract configuration
90	file</a>.</p>
91
92	<h3 id="simple_local">Extract from a local path</h3>
93
94	<p>A simple example of a basic extract configuration file is given below:</p>
95	<pre id="example_1">
96	# Example 1
97	# ----------------------------------------------------------------------
98	cfg::type ext # line 1
99	cfg::version 1.0 # line 2
100	# line 3
101	dest $PWD # line 4
102	# line 5
103	repos::var::user $HOME/var # line 6
104	# line 7
105	expsrc::var::user code # line 8
106	</pre>
107
108	<p>The above demonstrates how to use the extract system to extract code from
109	a local user directory. Here is an explanation of what each line does:</p>
110
111	<ul>
112	<li><dfn>line 1</dfn>: the label <code>CFG::TYPE</code> declares the type
113	of the configuration file. The value <samp>ext</samp> tells the system that
114	it is an extract configuration file.</li>
115
116	<li><dfn>line 2</dfn>: the label <code>CFG::VERSION</code> declares the
117	version of the extract configuration file. The current default is
118	<samp>1.0</samp>. Although it is not currently used, if we have to change
119	the format of the configuration file at a later stage, we shall be able to
120	use this number to determine whether we are reading a file with an older
121	format or one with a newer format.</li>
122
123	<li><dfn>line 3</dfn>: a blank line or a line beginning with a
124	<code>#</code> is a comment, and is ignored by the interpreter.</li>
125
126	<li><dfn>line 4</dfn>: the label <code>DEST</code> declares the destination
127	root directory of this extract. The value <samp>$PWD</samp> expands to the
128	current working directory.</li>
129
130	<li><dfn>line 5</dfn>: comment line, ignored.</li>
131
132	<li><dfn>line 6</dfn>: the label
133	<code>REPOS::<pck>::<branch></code> declares the top level URL
134	or path of a repository. The package name of the repository is given by
135	<pck>. In our example, we choose <samp>var</samp> as the name of the
136	package. (You can choose any name you like, however, it is usually sensible
137	to use a package name that matches the name of the project or system you
138	are working with.) The branch name in the repository is given by
139	<branch>. (Again, you can choose any name you like, however, it is
140	usually sensible to use a name such as <samp>base</samp>, <samp>user</samp>
141	or something that matches your branch name.) In our example, the word
142	<samp>user</samp> is normally used to denote a local user directory. Hence
143	the statement declares that the repository path for the <samp>var</samp>
144	package in the <samp>user</samp> branch can be found at
145	<samp>$HOME/var</samp>.</li>
146
147	<li><dfn>line 7</dfn>: comment line, ignored.</li>
148
149	<li><dfn>line 8</dfn>: the label
150	<code>EXPSRC::<pck>::<branch></code> declares an
151	<em>expandable</em> source directory for the package <pck> in the
152	branch <branch>. In our example, the package name is
153	<samp>var</samp>, and the branch name is <samp>user</samp>. These match the
154	package and the branch names of the repository declaration in line 6. It
155	means that the source directory declaration is associated with the path
156	<samp>$HOME/var</samp>. The value of the declaration <samp>code</samp> is
157	therefore a sub-directory under <samp>$HOME/var</samp>. By declaring a
158	source directory using an <code>EXPSRC</code> label, the system
159	automatically searches for all sub-directories (recursively) under the
160	declared source directory.</li>
161	</ul>
162
163	<p>Invoking the extract system using the above configuration file will
164	extract all sub-directories under <samp>$HOME/var/code</samp> to
165	<samp>$PWD/src/var/code</samp>. Note: the extract system ignores symbolic
166	links and hidden files, (i.e. file names beginning with a <samp>.</samp>). It
167	will write a build configuration file to <samp>$PWD/cfg/bld.cfg</samp>. The
168	configuration used for this extract will be written to the configuration file
169	at <samp>$PWD/cfg/ext.cfg</samp>.</p>
170
171	<dl>
172	<dt>Note - incremental extract</dt>
173
174	<dd>Suppose you have already performed an extract using the above
175	configuration file. At a later time, you have made some changes to some of
176	the files in the source directory. Re-running the extract system on the
177	same configuration will trigger an incremental extract. In an incremental
178	extract, the system will update only those files that are modified. If the
179	last modified time (or last commit revision) of a source file in the
180	current extract differs from that in the previous extract, the system will
181	attempt a content comparison. The system updates the destination only if
182	the content and/or file access permission of the source differs from that
183	of the destination.</dd>
184	</dl>
185
186	<h3 id="simple_url">Extract from a Subversion URL</h3>
187
188	<p>The next example demonstrates how to extract from a Subversion repository
189	URL:</p>
190	<pre id="example_2">
191	# Example 2
192	# ----------------------------------------------------------------------
193	cfg::type ext # line 1
194	cfg::version 1.0 # line 2
195	# line 3
196	dest $PWD # line 4
197	# line 5
198	repos::var::base svn://server/var/trunk # line 6
199	revision::var::base 1234 # line 7
200	# line 8
201	expsrc::var::base code # line 9
202	</pre>
203
204	<ul>
205	<li><dfn>line 1-5</dfn>: same as <a href="#example_1">example 1</a>.</li>
206
207	<li><dfn>line 6</dfn>: the line declares the repository location of the
208	<samp>base</samp> branch of the <samp>var</samp> package to be the
209	Subversion URL <samp>svn://server/var/trunk</samp>.</li>
210
211	<li><dfn>line 7</dfn>: the label
212	<code>REVISION::<pck>::<branch></code> declares the revision of
213	the repository associated with the package <pck> in the branch
214	<branch>. The current line tells the extract system to use revision
215	1234 of <samp>svn://server/var/trunk</samp>. It is worth noting that the
216	declared revision must be a revision when the declared branch exists. The
217	actual revision used is the last changed revision of the declared one. If
218	the revision is not declared, the default is to use the last changed
219	revision at the HEAD of the branch.</li>
220
221	<li><dfn>line 8</dfn>: comment line, ignored.</li>
222
223	<li><dfn>line 9</dfn>: the line declares an expandable source directory in
224	the repository <samp>svn://server/var/trunk</samp>.</li>
225	</ul>
226
227	<p>Invoking the extract system using the above configuration file will
228	extract all sub-directories under <samp>svn://server/var/trunk/code</samp> to
229	<samp>$PWD/src/var/code</samp>. It will write a build configuration file to
230	<samp>$PWD/cfg/bld.cfg</samp>. The configuration used for this extract will
231	be written to the configuration file at <samp>$PWD/cfg/ext.cfg</samp>.</p>
232
233	<dl>
234	<dt>EXPSRC or SRC?</dt>
235
236	<dd>
237	<p>So far, we have only declared source directories using the
238	<code>EXPSRC</code> statement, which stands for <em>expandable source
239	directory</em>. A source directory declared using this statement will
240	trigger the system to search recursively for any sub-directories under
241	the declared one. Any sub-directories containing regular source files
242	will be included in the extract. Symbolic links, hidden files and empty
243	directories (or those containing only symbolic links and/or hidden files)
244	are ignored.</p>
245
246	<p>If you do not want the system to search for sub-directories underneath
247	your declared source directory, you can declare your source directory
248	using the <code>SRC</code> statement. The <code>SRC</code> statement is
249	essentially the same as <code>EXPSRC</code> except that it does not
250	trigger the automatic recursive search for source directories. In fact,
251	the system implements the <code>EXPSRC</code> statement by expanding it
252	into a list of <code>SRC</code> statements.</p>
253	</dd>
254
255	<dt>Package and sub-package</dt>
256
257	<dd>
258	<p>The second field of a repository, revision or source directory
259	declaration label is the name of the container package. It is a name
260	selected by the user to identify the system or project he/she is working
261	on. (Therefore, it is often sensible to choose an identifier that matches
262	the name of the project or system.) The package name provides a unique
263	namespace for a file container. Source directories are automatically
264	arranged into sub-packages, using the names of the sub-directories as the
265	names of the sub-packages. For example, the declaration at line 9 in
266	<a href="#example_2">example 2</a> will put the source directory in the
267	<samp>var/code</samp> sub-package automatically.</p>
268
269	<p>Note that, in additional to slash <code>/</code>, double colon
270	<code>::</code> and double underscore <code>__</code> (internal only)
271	also act as delimiters for package names. Please avoid using them for
272	naming your files and directories.</p>
273
274	<p>You can declare a sub-package name explicitly in your source directory
275	statement. For example, the following two lines are equivalent:</p>
276	<pre>
277	src::var::base code/VarMod_Surface
278	src::var/code/VarMod_Surface::base code/VarMod_Surface
279	</pre>
280
281	<p>Explicit sub-package declaration should not be used normally, as it
282	requires a lot more typing (although there are some situations where it
283	can be useful, e.g. if you need to re-define the package name).</p>
284
285	<p>Currently, the extract system only supports non-space characters in
286	the package name, as the space character is used as a delimiter between
287	the declaration label and its value. If there are spaces in the path name
288	to a file or directory, you should explicity re-define the package name
289	of that path to a package name with no space using the above method.
290	However, we recommend that only non-space characters are used for naming
291	directories and files to make life simpler.</p>
292	</dd>
293	</dl>
294
295	<dl>
296	<dt>The expanded extract configuration file</dt>
297
298	<dd>
299	<p>At the end of a successful extract, the configuration used by the
300	current extract is written in <samp>cfg/ext.cfg</samp> under the extract
301	destination root. This file is an <em>expanded</em> version of the
302	original, with changes in the following declarations:</p>
303
304	<ul>
305	<li>All revision keywords are converted into revision numbers.</li>
306
307	<li>If a revision is not defined for a repository, it is set to the
308	corresponding revision number of the HEAD revision.</li>
309
310	<li>All URL keywords are converted into the full URLs.</li>
311
312	<li>All <code>EXPSRC</code> declarations are expanded into
313	<code>SRC</code> declarations.</li>
314
315	<li>All other variables are expanded.</li>
316	</ul>
317
318	<p>With this file, it should be possible for a later extract to re-create
319	the current configuration even if the contents of the repository have
320	changed. (This applies only to code stored in the repository.)</p>
321	</dd>
322	</dl>
323
324	<h3 id="simple_mirror">Mirror code to an alternate location</h3>
325
326	<p>The next example demonstrates how to extract from a repository and mirror
327	the code to an alternate location. It is essentially the same as <a href=
328	"#example_2">example 2</a>, except that it has three new lines to describe
329	how the system can mirror the extracted code to an alternate location.</p>
330	<pre id="example_3">
331	# Example 3
332	# ----------------------------------------------------------------------
333	cfg::type ext
334	cfg::version 1.0
335
336	dest $PWD
337
338	rdest::machine tx01 # line 6
339	rdest::logname frva # line 7
340	rdest /scratch/frva/extract/example3 # line 8
341
342	repos::var::base svn://server/var/trunk
343	revision::var::base 1234
344
345	expsrc::var::base code
346	</pre>
347
348	<p>Here is an explanation of what each line does:</p>
349
350	<ul>
351	<li><dfn>line 6</dfn>: <code>RDEST::MACHINE</code> declares the target
352	machine to which the code will be mirrored. The example mirrors the code to
353	the machine named <samp>tx01</samp>.</li>
354
355	<li><dfn>line 7</dfn>: <code>RDEST::LOGNAME</code> declares the user name
356	of the target machine, to which the user has login access. If this is not
357	declared, the system uses the login name of the current user on the local
358	machine.</li>
359
360	<li><dfn>line 8</dfn>: <code>RDEST</code> declares the root directory of
361	the alternate destination, where the mirror version of the extract will be
362	sent.</li>
363	</ul>
364
365	<p>Invoking the extract system on the above configuration will trigger an
366	extract similar to that given in <a href="#example_2">example 2</a>, but it
367	will also attempt to mirror the contents at <samp>$PWD/src/var/code</samp> to
368	<samp>/scratch/frva/extract/example3/src</samp> on the alternate destination.
369	It will also mirror the expanded extract configuration file
370	<samp>$PWD/cfg/ext.cfg</samp> to
371	<samp>/scratch/frva/extract/example3/cfg/ext.cfg</samp> and
372	<samp>$PWD/cfg/bld.cfg</samp> to
373	<samp>/scratch/frva/extract/example3/cfg/bld.cfg</samp>. It is also worth
374	noting that the content of the build configuration file will be slightly
375	different, since it will include directory names appropriate for the
376	alternate destination.</p>
377
378	<dl>
379	<dt>Note - mirroring command</dt>
380
381	<dd>
382	<p>The extract system currently supports <code>rdist</code> and
383	<code>rsync</code> as its mirroring tool. The default is
384	<code>rsync</code>. To use <code>rdist</code> instead of
385	<code>rsync</code>, add the following line to your extract configuration
386	file:</p>
387	<pre>
388	rdest::mirror_cmd rdist
389	</pre>
390
391	<p>If <code>rsync</code> is used to mirror an extract, the system needs to
392	issue a separate remote shell command to create the container directory of
393	the mirror destination. The default is to issue a shell command in the
394	form <samp>ssh -n -oBatchMode=yes LOGNAME@MACHINE mkdir -p DEST</samp>.
395	The following declarations can be used to modify the command:</p>
396	<pre>
397	# Examples using the default settings:
398	rdest::rsh_mkdir_rsh ssh
399	rdest::rsh_mkdir_rshflags -n -oBatchMode=yes
400	rdest::rsh_mkdir_mkdir mkdir
401	rdest::rsh_mkdir_mkdirflags -p
402	</pre>
403
404	<p>In addition, the default <code>rsync</code> shell command is
405	<samp>rsync -a --exclude='.*' --delete-excluded --timeout=900 --rsh='ssh
406	-oBatchMode=yes' SOURCE DEST</samp>. The following declarations can be
407	used to modify the command:</p>
408	<pre>
409	# Examples using the default settings:
410	rdest::rsync rsync
411	rdest::rsyncflags -a --exclude='.*' --delete-excluded --timeout=900 \
412	--rsh='ssh -oBatchMode=yes'
413	</pre>
414	</dd>
415	</dl>
416
417	<h2 id="advanced">Advanced Usage</h2>
418
419	<h3 id="advanced_multi">Extract from multiple repositories</h3>
420
421	<p>So far, we have only extracted from a single location. The extract system
422	is not much use if that is the only thing it can do. In fact, the extract
423	system supports extract of multiple source directories from multiple branches
424	in multiple repositories. The following configuration file is an example of
425	how to extract from multiple repositories:</p>
426	<pre id="example_4">
427	# Example 4
428	# ----------------------------------------------------------------------
429	cfg::type ext
430	cfg::version 1.0
431
432	dest $PWD
433
434	repos::var::base fcm:var_tr # line 6
435	repos::ops::base fcm:ops_tr # line 7
436	repos::gen::base fcm:gen_tr # line 8
437
438	revision::gen::base 2468 # line 10
439
440	expsrc::var::base src/code # line 12
441	expsrc::var::base src/scripts # line 13
442	expsrc::ops::base src/code # line 14
443	src::gen::base src/code/GenMod_Constants # line 15
444	src::gen::base src/code/GenMod_Control # line 16
445	src::gen::base src/code/GenMod_FortranIO # line 17
446	src::gen::base src/code/GenMod_GetEnv # line 18
447	src::gen::base src/code/GenMod_ModelIO # line 19
448	src::gen::base src/code/GenMod_ObsInfo # line 20
449	src::gen::base src/code/GenMod_Platform # line 21
450	src::gen::base src/code/GenMod_Reporting # line 22
451	src::gen::base src/code/GenMod_Trace # line 23
452	src::gen::base src/code/GenMod_UMConstants # line 24
453	src::gen::base src/code/GenMod_Utilities # line 25
454	</pre>
455
456	<p>Here is an explanation of what each line does:</p>
457
458	<ul>
459	<li><dfn>line 6-8</dfn>: these lines declare the repositories for the
460	<samp>base</samp> branches of the <samp>var</samp>, <samp>ops</samp> and
461	<samp>gen</samp> packages respectively. It is worth noting that the values
462	of the declarations are no longer Subversion URLs but are FCM URL keywords.
463	These keywords are normally declared in the central configuration file of
464	the FCM system, and will be expanded into the corresponding Subversion URLs
465	by the FCM system. For further information on URL keywords, please see
466	<a href="code_management.html#svn_basic_keywords">Code Management System
467	> Using Subversion > Basic Command Line Usage > Repository &
468	Revision Keywords</a>.</li>
469
470	<li><dfn>line 10</dfn>: this line declares the revision number for the
471	<samp>base</samp> branch of the <samp>gen</samp> package, i.e. for the
472	<samp>fcm:gen_tr</samp> repository. It is worth noting that the revision
473	numbers for the <samp>var</samp> and <samp>ops</samp> packages have not
474	been declared. By default, their revision numbers will be set to the last
475	changed revision at the HEAD.</li>
476
477	<li><dfn>line 12-14</dfn>: these line declares the source directories for
478	the <samp>base</samp> branches of the <samp>var</samp> and <samp>ops</samp>
479	packages. For the <samp>var</samp> package, we are extracting everything
480	from the <samp>code</samp> and the <samp>scripts</samp> sub-directory. For
481	the <samp>ops</samp> package, we are extracting everything from the
482	<samp>code</samp> directory.</li>
483
484	<li><dfn>line 15-25</dfn>: these line declares the source directories for
485	the <samp>base</samp> branch of the <samp>gen</samp> package. The source
486	directories declared will not be searched for sub-directories underneath
487	the declared directories.</li>
488	</ul>
489
490	<p>We shall end up with a directory tree such as:</p>
491	<pre>
492	$PWD
493	\|
494	\|--- cfg
495	\| \|
496	\| \|--- bld.cfg
497	\| \|--- ext.cfg
498	\|
499	\|--- src
500	\|
501	\|--- gen
502	\| \|
503	\| \|--- code
504	\| \|
505	\| \|--- GenMod_Constants
506	\| \|--- GenMod_Control
507	\| \|--- GenMod_FortranIO
508	\| \|--- GenMod_GetEnv
509	\| \|--- GenMod_ModelIO
510	\| \|--- GenMod_ObsInfo
511	\| \|--- GenMod_Platform
512	\| \|--- GenMod_Reporting
513	\| \|--- GenMod_Trace
514	\| \|--- GenMod_UMConstants
515	\| \|--- GenMod_Utilities
516	\|
517	\|--- ops
518	\| \|
519	\| \|--- code
520	\| \|
521	\| \|--- ...
522	\|
523	\|--- var
524	\|
525	\|--- code
526	\| \|
527	\| \|--- ...
528	\|
529	\|--- scripts
530	\|
531	\|--- ...
532	</pre>
533
534	<dl>
535	<dt>Note - revision number</dt>
536
537	<dd>
538	<p>As seen in the above example, if a revision number is not specified
539	for a repository URL, it defaults to the last changed revision at the
540	HEAD of the branch. The revision number can also be declared in other
541	ways:</p>
542
543	<ul>
544	<li>Any revision arguments acceptable by Subversion are allowed. You
545	can use a valid revision number, a date between a pair of curly
546	brackets (e.g. <samp>{2005-05-01T12:00}</samp>) or the keyword HEAD.
547	However, please do not use the keywords BASE, COMMITTED or PREV as
548	these are reserved for working copy only.</li>
549
550	<li>FCM revision keywords are allowed. These must be defined for the
551	corresponding repository URLs in either the central or the user FCM
552	configuration file. For further information on revision keywords,
553	please see <a href="code_management.html#svn_basic_keywords">Code
554	Management > Using Subversion > Basic Command Line Usage >
555	Repository & Revision Keywords</a>.</li>
556
557	<li>Do not use the keyword USER, as it is used internally by the
558	extract system.</li>
559	</ul>
560
561	<p>If a revision number is specified for a branch, the actual revision
562	used by the extract system is the last changed revision of the branch,
563	which may differ from the declared revision. While this behaviour is
564	useful in most situations, some users may find it confusing to work with.
565	It is possible to alter this behaviour so that extract will fail if the
566	declared revision does not correspond to a changeset of the declared
567	branch. Make the following declaration to switch on this checking:</p>
568	<pre>
569	revmatch true
570	</pre>
571	</dd>
572	</dl>
573
574	<h3 id="advanced_branches">Extract from multiple branches</h3>
575
576	<p>We have so far dealt with a single branch in any package. The extract
577	system can be used to <em>combine</em> changes from different branches of a
578	package. An example is given below:</p>
579	<pre id="example_5">
580	# Example 5
581	# ----------------------------------------------------------------------
582	cfg::type ext
583	cfg::version 1.0
584
585	dest $PWD
586
587	repos::var::base fcm:var_tr
588	repos::ops::base fcm:ops_tr
589	repos::gen::base fcm:gen_tr
590
591	revision::gen::base 2468
592
593	expsrc::var::base src/code
594	expsrc::var::base src/scripts
595	expsrc::ops::base src/code
596	src::gen::base src/code/GenMod_Constants
597	src::gen::base src/code/GenMod_Control
598	src::gen::base src/code/GenMod_FortranIO
599	src::gen::base src/code/GenMod_GetEnv
600	src::gen::base src/code/GenMod_ModelIO
601	src::gen::base src/code/GenMod_ObsInfo
602	src::gen::base src/code/GenMod_Platform
603	src::gen::base src/code/GenMod_Reporting
604	src::gen::base src/code/GenMod_Trace
605	src::gen::base src/code/GenMod_UMConstants
606	src::gen::base src/code/GenMod_Utilities
607
608	repos::var::branch1 fcm:var_br/frva/r1234_new_stuff # line 27
609	repos::var::branch2 fcm:var_br/frva/r1516_bug_fix # line 28
610	repos::ops::branch1 fcm:ops_br/opsrc/r3188_good_stuff # line 29
611	</pre>
612
613	<p>The configuration file in <a href="#example_5">example 5</a> is similar to
614	that of <a href="#example_4">example 4</a> except for the last three lines.
615	Here is an explanation of what they do:</p>
616
617	<ul>
618	<li>
619	<dfn>line 27</dfn>: this line declares a repository URL for the
620	<samp>branch1</samp> branch of the <samp>var</samp> package. From the URL
621	of the branch, we know that the branch was created by the user
622	<samp>frva</samp> based on the trunk at revision at 1234. The description
623	of the branch is <samp>branch1</samp>. The following points are worth
624	noting:
625
626	<ul>
627	<li>By declaring a new branch with the same package name to a
628	previously declared branch, it is assumed that both branches reside in
629	the same Subversion repository.</li>
630
631	<li>No revision is declared for this URL, so the default is used which
632	is the last changed revision at the HEAD of the branch.</li>
633
634	<li>No source directory is declared for this URL. By default, if no
635	source directory is declared for a branch repository, it will attempt
636	to use the same set of source directories as the first declared branch
637	of the package. In this case, the source directories declared for the
638	<samp>base</samp> branch of the <samp>var</samp> package will be
639	used.</li>
640	</ul>
641	</li>
642
643	<li><dfn>line 28</dfn>: this line declares another branch called
644	<samp>branch2</samp> for the <samp>var</samp> package. No source directory
645	is declared for this URL either, so it will use the same set of source
646	directories declared for the <samp>base</samp> branch.</li>
647
648	<li><dfn>line 29</dfn>: this line declares a branch called
649	<samp>branch1</samp> for the <samp>ops</samp> package. It will use the same
650	set of source directories declared for the <samp>ops</samp> package
651	<samp>base</samp> branch.</li>
652	</ul>
653
654	<p>When we invoke the extract system, it will attempt to extract from the
655	first declared branch of a package, if the last changed revision of the
656	source directory is the same in all the branches. However, if the last
657	changed revision of the source directory differs for different branches, the
658	system will attempt to obtain an extract priority list for each source
659	directory, using the following logic:</p>
660
661	<ol>
662	<li>The system looks for source directory packages from the first declared
663	branch to the last declared branch.</li>
664
665	<li>The branch in which a source directory package is first declared is the
666	<samp>base</samp> branch of the source directory package.</li>
667
668	<li>The last changed revision of a source directory package in a
669	subsequently declared repository branch is compared with that of the base
670	branch. If the last changed revision is the same as that of the base
671	branch, the source directory of this branch is discarded. Otherwise, it is
672	placed at the end of the extract priority list.</li>
673	</ol>
674
675	<p>For the <samp>var</samp> package in the above example, let us assume that
676	we have three source directory packages X, Y and Z under <samp>code</samp>,
677	and their last changed revisions under <samp>base</samp> are 100. Let's say
678	we have committed some changes to X and Z in the <samp>branch1</samp> branch
679	at revision 102, and other changes to Y and Z in the <samp>branch2</samp>
680	branch at revision 104, the extract priority lists for X, Y and Z will look
681	like:</p>
682
683	<ul>
684	<li>X: base (100, base), branch1 (102), branch2 (100, discarded)</li>
685
686	<li>Y: base (100, base), branch1 (100, discarded), branch2 (104)</li>
687
688	<li>Z: base (100, base), branch1 (102), branch2 (104)</li>
689	</ul>
690
691	<p>Once we have an extract priority list for a source directory, we can begin
692	extracting source files in the source directory. The source directory of the
693	base branch is extracted first, followed by that in the subsequent branches.
694	If a source file in a subsequent branch has the same content as the that in
695	the base branch, it is discarded. Otherwise, the following logic determines
696	the branch to use:</p>
697
698	<ol>
699	<li>If a source file is modified in only one subsequent branch, the source
700	file in that branch is extracted.</li>
701
702	<li>If a source file is modified in two or more subsequent branches, but
703	their modifications are the same, then the source file in the first
704	modification is used.</li>
705
706	<li>If a source file is modified in two or more subsequent branches and
707	their modifications differ, then the behaviour depends on the "conflict
708	mode" setting, which can be <code>fail</code>, <code>merge</code> (default)
709	and <code>override</code>. If the conflict mode is <code>fail</code>, the
710	extract fails. If the conflict mode is <code>merge</code>, the system will
711	attempt to merge the changes using a tool such as <code>diff3</code>. The
712	result of the merge will be used to update the destination. The extract
713	fails only if there are unresolved conflicts in the merge. (In which case,
714	the conflict should be resolved using the version control system before
715	re-running the extract system.) If the conflict mode is
716	<code>override</code>, the change in the latest declared branch takes
717	precedence, and the changes in all other branches will be ignored. The
718	conflict mode can be changed using the <code>CONFLICT</code> declaration in
719	the extract configuration file. E.g:
720	<pre>
721	conflict fail
722	</pre>
723	</li>
724	</ol>
725
726	<p>Once the system has established which source files to use, it determines
727	whether the destination file is out of date or not. The destination file is
728	out of date if it does not exist or if its content differs from the version
729	of the source file we are using. The system only updates the destination if
730	it is considered to be out of date.</p>
731
732	<p>The extract system can also combine changes from branches in the Subversion
733	repository and the local file system. This is demonstrated in the next
734	example.</p>
735	<pre id="example_6">
736	# Example 6
737	# ----------------------------------------------------------------------
738	cfg::type ext
739	cfg::version 1.0
740
741	dest $PWD
742
743	repos::var::base fcm:var_tr
744	repos::ops::base fcm:ops_tr
745	repos::gen::base fcm:gen_tr
746
747	revision::gen::base 2468
748
749	expsrc::var::base src/code
750	expsrc::var::base src/scripts
751	expsrc::ops::base src/code
752	src::gen::base src/code/GenMod_Constants
753	src::gen::base src/code/GenMod_Control
754	src::gen::base src/code/GenMod_FortranIO
755	src::gen::base src/code/GenMod_GetEnv
756	src::gen::base src/code/GenMod_ModelIO
757	src::gen::base src/code/GenMod_ObsInfo
758	src::gen::base src/code/GenMod_Platform
759	src::gen::base src/code/GenMod_Reporting
760	src::gen::base src/code/GenMod_Trace
761	src::gen::base src/code/GenMod_UMConstants
762	src::gen::base src/code/GenMod_Utilities
763
764	repos::var::branch1 fcm:var_br/frva/r1234_new_stuff
765	repos::var::branch2 fcm:var_br/frva/r1516_bug_fix
766	repos::ops::branch1 fcm:ops_br/opsrc/r3188_good_stuff
767
768	repos::var::user $HOME/var # line 31
769	repos::gen::user $HOME/gen # line 32
770	</pre>
771
772	<p><a href="#example_6">Example 6</a> is similar to <a href=
773	"#example_5">example 5</a> except that it is also extracting from local
774	directories. Here is an explanation of the lines:</p>
775
776	<ul>
777	<li><dfn>line 31-32</dfn>: these line declare the repositories for the
778	<samp>user</samp> branches of the <samp>var</samp> and <samp>gen</samp>
779	packages respectively. Both are local paths at the local file system. There
780	are no declarations for source directories for the <samp>user</samp>
781	branches, so they use the same set of source directories of the first
782	declared branches, the <samp>base</samp> branches in both cases.</li>
783	</ul>
784
785	<dl>
786	<dt>Note - the INC declaration</dt>
787
788	<dd>
789	You have probably realised that the above examples have many repeated
790	lines. To avoid having repeated lines in multiple extract configuration
791	files, you can use <code>INC</code> declarations to include other extract
792	configuration files. For example, if the configuration file of example 5
793	is stored in the file <samp>$HOME/example5/ext.cfg</samp>, line 1 to 29
794	of <a href="#example_6">example 6</a> can be replaced with an
795	<code>INC</code> declaration. <a href="#example_6">Example 6</a> can then
796	be written as:
797	<pre>
798	inc $HOME/example5/ext.cfg
799
800	repos::var::user $HOME/var
801	repos::gen::user $HOME/gen
802	</pre>
803
804	<p>Note: the <code>INC</code> declaration supports the special
805	environment variable <var>$HERE</var>. If this variable is already set in
806	the environment, it acts as a normal environment variable. However, if it
807	is not set, it will be expanded into the container directory of the
808	current extract configuration file. This feature is particularly useful
809	if you are including a hierarchy of extract configurations from files in
810	the same container directory in a repository.</p>
811	</dd>
812	</dl>
813
814	<h3 id="advanced_inherit">Inherit from a previous extract</h3>
815
816	<p>All the examples above dealt with standalone extract, that is, the current
817	extract is independent of any other extract. If a previous extract exists in
818	another location, the extract system can inherit from this previous extract
819	in your current extract. This works like a normal incremental extract, except
820	that your extract will only contain the changes you have specified (compared
821	with the inherited extract) instead of the full source directory tree. This
822	type of incremental extract is useful in several ways. For instance:</p>
823
824	<ul>
825	<li>It is fast, because you only have to extract and mirror files that you
826	have changed.</li>
827
828	<li>The subsequent build will also be fast, since it will use incremental
829	build.</li>
830
831	<li>You do not need write access to the original extract. A system
832	administrator can set up a stable version in a central account, which
833	developers can then inherit from.</li>
834
835	<li>You want an incremental extract, but you need to leave the original
836	extract unmodified.</li>
837	</ul>
838
839	<p>The following example is based on <a href="#example_4">example 4</a> and
840	<a href="#example_6">example 6</a>. The assumption is that an extract has
841	already been performed at the directory <samp>~frva/var/vn22.0</samp> based
842	on the configuration file in <a href="#example_4">example 4</a>.</p>
843	<pre id="example_7">
844	# Example 7
845	# ----------------------------------------------------------------------
846	cfg::type ext
847	cfg::version 1.0
848
849	dest $PWD
850
851	use ~frva/var/vn22.0 # line 6
852
853	repos::var::branch1 fcm:var_br/frva/r1234_new_stuff # line 8
854	repos::var::branch2 fcm:var_br/frva/r1516_bug_fix # line 9
855	repos::ops::branch1 fcm:ops_br/opsrc/r3188_good_stuff # line 10
856
857	repos::var::user $HOME/var # line 12
858	repos::gen::user $HOME/gen # line 13
859	</pre>
860
861	<ul>
862	<li><dfn>line 6</dfn>: this line replaces line 1 to 25 of <a href=
863	"#example_6">example 6</a>. It declares that the current extract should
864	inherit from the previous extract located at
865	<samp>~frva/var/vn22.0</samp>.</li>
866	</ul>
867
868	<p>Running the extract system using the above configuration will trigger an
869	incremental extract, as if you are running an incremental extract having
870	modified the configuration file in <a href="#example_4">example 4</a> to that
871	of <a href="#example_6">example 6</a>. The only difference is that the
872	original extract using the <a href="#example_4">example 4</a> configuration
873	will be left untouched at <samp>~frva/var/vn22.0</samp>, and the new extract
874	will contain only the changes in the branches declared from line 8 to 13.</p>
875
876	<p>Note: extract inheritance allows you to add more branches to a package,
877	but you should not redefine the <code>REPOS</code>, <code>REVISION</code>,
878	<code>EXPSRC</code> or <code>SRC</code> declarations of a branch that is
879	already declared (and already extracted) in the inherited extract. Although
880	the system will not stop you from doing so, you may end up with an extract
881	that does not quite do what it is supposed to do. For example, if the
882	<samp>base</samp> branch in the <samp>foo</samp> package
883	(<tt>repos::foo::base</tt>) is already defined and extracted in an extract
884	you are inheriting from, you should not redefine any of the
885	<tt>*::foo::base</tt> declarations in your current extract. However, you are
886	free to add more branches for the same package with new labels (e.g.
887	<tt>repos::foo::b1</tt>), and indeed new packages that are not already
888	defined in the inherited extract (e.g. <tt>repos::bar::base</tt>).</p>
889
890	<p>If you are setting up an extract to be inherited, you do not have to
891	perform a build. If you don't you will still gain the benefit of incremental
892	file extract, but you will be performing a full build of the code.</p>
893
894	<dl>
895	<dt>Note - inherit and mirror</dt>
896
897	<dd>
898	<p>It is worth bearing in mind that <tt>rdest::*</tt> settings are not
899	inherited. If mirroring is required in the inheriting extract, it will
900	require its own set of <tt>rdest::*</tt> declarations.</p>
901
902	<p>The system will, however, assume that a mirrored version of the
903	inherited extract is available for inheritance from the mirrored
904	destination of the current extract.</p>
905
906	<p>E.g.: Consider an extract at <samp>/path/to/inherited/</samp> and an
907	inheriting extract at <samp>/path/to/current/</samp>. If the former does
908	not have a mirror, the latter should not have one either. If the former
909	mirrors to <samp>machine@/path/to/inherited/mirror/</samp> and the latter
910	mirrors to <samp>machine@/path/to/current/mirror/</samp>, the system will
911	assume that the subsequent build at
912	<samp>machine@/path/to/current/mirror/</samp> can inherit from the build
913	at <samp>machine@/path/to/inherited/mirror/</samp>. This is illustrated
914	below:</p>
915
916	<pre>
917	/path/to/current/ => at machine: /path/to/current/mirror/
918	use /path/to/inherited/ => at machine: use /path/to/inherited/mirror/
919	</pre>
920	</dd>
921	</dl>
922
923	<h3 id="advanced_build">Extract - Build Configuration</h3>
924
925	<p>Configuration settings for feeding into the build system can be declared
926	through the extract configuration file using the <code>BLD::</code> prefix.
927	Any line in an extract configuration containing a label with such a prefix
928	will be considered a build system variable. At the end of a successful
929	extract, the system strips out the <code>BLD::</code> prefix before writing
930	these variables to the build configuration file. Some example entries are
931	given between line 17 and 22 in the following configuration file:</p>
932	<pre id="example_8">
933	# Example 8
934	# ----------------------------------------------------------------------
935	cfg::type ext
936	cfg::version 1.0
937
938	dest $PWD
939
940	repos::var::base fcm:var_tr
941	repos::ops::base fcm:ops_tr
942	repos::gen::base fcm:gen_tr
943
944	revision::gen::base 2468
945
946	expsrc::var::base src/code
947	expsrc::var::base src/scripts
948	expsrc::ops::base src/code
949	src::gen::base src/code/GenMod_Constants
950	src::gen::base src/code/GenMod_Control
951	src::gen::base src/code/GenMod_FortranIO
952	src::gen::base src/code/GenMod_GetEnv
953	src::gen::base src/code/GenMod_ModelIO
954	src::gen::base src/code/GenMod_ObsInfo
955	src::gen::base src/code/GenMod_Platform
956	src::gen::base src/code/GenMod_Reporting
957	src::gen::base src/code/GenMod_Trace
958	src::gen::base src/code/GenMod_UMConstants
959	src::gen::base src/code/GenMod_Utilities
960
961	bld::target VarProg_AnalysePF.exe # line 27
962
963	bld::tool::fc sxmpif90 # line 29
964	bld::tool::cc sxmpic++ # line 30
965	bld::tool::ld sxmpif90 # line 31
966	</pre>
967
968	<p>The above example is essentially the same as <a href="#example_4">example
969	4</a>, apart from the additional build configuration. The following is a
970	simple explanation of what the lines represent: (For detail of the build
971	system, please see the next chapter on <a href="build.html">The Build
972	System</a>.)</p>
973
974	<ul>
975	<li><dfn>line 27</dfn>: the line declares a default target of the
976	build.</li>
977
978	<li><dfn>line 29-31</dfn>: the lines declare the Fortran compiler, the C
979	compiler and the linker respectively.</li>
980	</ul>
981
982	<dl>
983	<dt>Note - use of variables</dt>
984
985	<dd>
986	<p>When you start using the extract system to define compiler flags for
987	the build system, you may end up having to make a lot of long and
988	repetitive declarations. In this case, you may want to define variables
989	to replace the repetitive parts of the declarations.</p>
990
991	<p>Environment variables whose names contain only upper case latin
992	alphabets, numbers and underscores can be referenced in a declaration
993	value via the syntax <code>$NAME</code> or <code>${NAME}</code>. For
994	example:</p>
995	<pre>
996	repos::um::base ${HOME}/svn-wc/um
997	bld::tool::fflags $MY_FFLAGS
998	</pre>
999
1000	<p>You can define a user variable by making a declaration with a label
1001	that begins with a percent sign <code>%</code>. The value of a user
1002	variable remains in memory until the end of the current file is reached.
1003	You can reference a user variable in a declaration value via the syntax
1004	<code>%NAME</code> or <code>%{NAME}</code>. For example:</p>
1005	<pre>
1006	# Declare a variable %fred
1007	%fred -Cdebug -eC -Wf,-init heap=nan stack=nan
1008
1009	bld::tool::fflags %fred
1010	# bld::tool::fflags -Cdebug -eC -Wf,-init heap=nan stack=nan
1011
1012	bld::tool::fflags::foo %fred -f0
1013	# bld::tool::fflags::foo -Cdebug -eC -Wf,-init heap=nan stack=nan -f0
1014
1015	bld::tool::fflags::bar -w %fred
1016	# bld::tool::fflags::bar -w -Cdebug -eC -Wf,-init heap=nan stack=nan
1017	</pre>
1018
1019	<p>Further to this, each declaration results in an internal variable of
1020	the same name and you can also refer to any of these internal variables in
1021	the same way. So, the example given above could also be written as
1022	follows:</p>
1023	<pre>
1024	bld::tool::fflags -Cdebug -eC -Wf,-init heap=nan stack=nan
1025	bld::tool::fflags::foo %bld::tool::fflags -f0
1026	bld::tool::fflags::bar -w %bld::tool::fflags
1027	</pre>
1028	</dd>
1029
1030	<dt>Note - as-parsed configuration</dt>
1031
1032	<dd>
1033	<p>If you use a hierarchy of <code>INC</code> declarations or variables,
1034	you may end up with a configuration file that is difficult to understand.
1035	To help you with this, the extract system generates an as-parsed
1036	configuration file at <samp>cfg/parsed_ext.cfg</samp> of the destination.
1037	The content of the as-parsed configuration file is what the extract
1038	system actually reads. It should contain everything in your original
1039	extract configuration file, except that all <code>INC</code>
1040	declarations, environment variables and user/internal variables are
1041	expanded.</p>
1042	</dd>
1043	</dl>
1044
1045	<h2 id="verbose">Diagnostic verbose level</h2>
1046
1047	<p>The amount of diagnostic messages generated by the extract system is
1048	normally set to a level suitable for normal everyday operation. This is the
1049	default diagnostic verbose level 1. If you want a minimum amount of
1050	diagnostic messages, you should set the verbose level to 0. If you want more
1051	diagnostic messages, you can set the verbose level to 2 or 3. You can modify
1052	the verbose level in two ways. The first way is to set the environment
1053	variable <var>FCM_VERBOSE</var> to the desired verbose level. The second way
1054	is to invoke the extract system with the <code>-v <level></code>
1055	option. (If set, the command line option overrides the environment
1056	variable.)</p>
1057
1058	<p>The following is a list of diagnostic output at each verbose level:</p>
1059
1060	<dl>
1061	<dt>Level 0</dt>
1062
1063	<dd>
1064	<ul>
1065	<li>Report the time taken to extract the code.</li>
1066
1067	<li>Report the time taken to mirror the code.</li>
1068
1069	<li>If <code>rdist</code> is used to mirror the code, run the command
1070	with the <code>-q</code> option.</li>
1071	</ul>
1072	</dd>
1073
1074	<dt>Level 1</dt>
1075
1076	<dd>
1077	<ul>
1078	<li>Everything at verbose level 0.</li>
1079
1080	<li>Report the name of the extract configuration file.</li>
1081
1082	<li>Report the location of the extract destination.</li>
1083
1084	<li>Report date/time at the beginning of the extract step.</li>
1085
1086	<li>If the revision specified for a repository branch is not its last
1087	changed revision, print an information statement to inform the user of
1088	the last changed revision of the branch.</li>
1089
1090	<li>Summarises the destination status and the source status.</li>
1091
1092	<li>Report date/time at the beginning of the mirror step.</li>
1093
1094	<li>Report the location of the alternate destination.</li>
1095
1096	<li>Report total time.</li>
1097	</ul>
1098	</dd>
1099
1100	<dt>Level 2</dt>
1101
1102	<dd>
1103	<ul>
1104	<li>Everything at verbose level 1.</li>
1105
1106	<li>If the revision specified for a repository branch is not current
1107	(i.e. the specified revision number is less than the revision number of
1108	the last commit revision), print an information statement to inform the
1109	user of the last commit revision of the branch.</li>
1110
1111	<li>Report the detail of each change in the destination.</li>
1112
1113	<li>If <code>rdist</code> is used to mirror the code, run the command
1114	without the <code>-q</code> option.</li>
1115	</ul>
1116	</dd>
1117
1118	<dt>Level 3</dt>
1119
1120	<dd>
1121	<ul>
1122	<li>Everything at verbose level 2.</li>
1123
1124	<li>Report all shell commands invoked by the extract system with
1125	timestamp.</li>
1126
1127	<li>If <code>rdist</code> is used to mirror the code, print the
1128	<samp>distfile</samp> supplied to the command.</li>
1129
1130	<li>If <code>rsync</code> is used to mirror the code, invoke the
1131	command with the <code>-v</code> option.</li>
1132	</ul>
1133	</dd>
1134	</dl>
1135
1136	<h2 id="nosvn">When Subversion Is Not Available</h2>
1137
1138	<p>The extract system can still be used if Subversion is not available.
1139	Clearly, you can only use local repositories. However, you can still do
1140	incremental extract, mirror an extract to an alternate location, or combine
1141	code from multiple local repositories.</p>
1142
1143	<p>If you are using Subversion but your server is down then clearly there is
1144	little you can do. However, if you already have an extract then you can
1145	re-run <code>fcm extract</code> as long as the extract configuration file
1146	only refers to fixed revisions. If this is not the case then you can always
1147	use the expanded extract configuration file which can be found in
1148	<samp>cfg/ext.cfg</samp> under the extract destination root. This means that
1149	you can continue to makes changes to local code and do incremental extracts
1150	even whilst your Subversion server is down.</p>
1151
1152	</div>
1153	</div>
1154	</div>
1155
1156	<hr/>
1157	<div class="container-fluid text-center">
1158	<div class="row"><div class="col-md-12">
1159	<address><small>
1160	© British Crown Copyright 2006-16
1161	<a href="http://www.metoffice.gov.uk">Met Office</a>.
1162	See <a href="../etc/fcm-terms-of-use.html">Terms of Use</a>.<br />
1163	This document is released under the British <a href=
1164	"http://www.nationalarchives.gov.uk/doc/open-government-licence/" rel=
1165	"license">Open Government Licence</a>.<br />
1166	</small></address>
1167	</div></div>
1168	</div>
1169
1170	<script type="text/javascript" src="../etc/jquery.min.js"></script>
1171	<script type="text/javascript" src="../etc/bootstrap/js/bootstrap.min.js"></script>
1172	<script type="text/javascript" src="../etc/fcm.js"></script>
1173	<script type="text/javascript" src="../etc/fcm-version.js"></script>
1174	</body>
1175	</html>

Note: See TracBrowser for help on using the repository browser.

Download in other formats: