The extract system provides an interface between the revision control system (currently Subversion) and the build system. Where appropriate, it extracts code from the repository and other user-defined locations to a directory tree suitable for feeding into the build system. In this chapter, we shall use many examples to explain how to use the extract system. At the end of this chapter, you will be able to extract code from the local file system as well as from different branches of different repository URLs. You will also learn how to mirror code to an alternate destination. Finally, you will be given an introduction on how to specify configurations for the build system via the extract configuration file. (For further information on the build system, please see the next chapter The Build System.) The last section of the chapter tells you what you can do in the case when Subversion is not available.
To invoke the extract system, simply issue the command:
fcm extract
By default, the extract system searches for an extract configuration file "ext.cfg" in "$PWD" and then "$PWD/cfg". If an extract configuration file is not found in these directories, the command fails with an error. If an extract configuration file is found, the system will use the configuration specified in the file to perform the current extract.
If the destination of the extract does not exist, the system performs a new full extract to the destination. If a previous extract already exists at the destination, the system performs an incremental extract, updating any modifications if necessary. If a full (fresh) extract is required for whatever reason, you can invoke the extract system using the "-f" option, (i.e. the command becomes "fcm extract -f"). If you simply want to remove all the items generated by a previous extract in the destination, you can invoke the extract system using the "--clean" option.
For further information on the extract command, please see FCM Command Reference > fcm extract.
The extract configuration file is the main user interface of the extract system. It is a line based text file. For a complete set of extract configuration file declarations, please refer to the Annex: Declarations in FCM extract configuration file.
A simple example of a basic extract configuration file is given below:
Extract configuration example 1 - extract from a local path |
---|
cfg::type ext # line 1 cfg::version 1.0 # line 2 # line 3 dest $PWD # line 4 # line 5 repos::var::user $HOME/var # line 6 # line 7 expsrc::var::user code # line 8 |
The above demonstrates how to use the extract system to extract code from a local user directory. Here is an explanation of what each line does:
Invoking the extract system using the above configuration file will "extract" all sub-directories under "$HOME/var/code" to "$PWD/src/var/code". Note: the extract system ignores all hidden files, (i.e. directories and files beginning with a "."). It will write a build configuration file to "$PWD/cfg/bld.cfg". The configuration used for this extract will be written to the configuration file at "$PWD/cfg/ext.cfg".
Note - incremental extract |
---|
Suppose you have already performed an extract using the above configuration file. At a later time, you have made some changes to some of the files in the source directory. Re-running the extract system on the same configuration will trigger an incremental extract. In an incremental extract, the system will update only those files that are modified. In exact words, the system checks the modification time of each file in the source and destination. If a source file is newer than its corresponding destination file, it checks whether the content differs. The destination is only updated if its content differs from the source. |
The next example demonstrates how to extract from a Subversion repository URL:
Extract configuration example 2 - extract from a Subversion URL |
---|
cfg::type ext # line 1 cfg::version 1.0 # line 2 # line 3 dest $PWD # line 4 # line 5 repos::var::base svn://server/var/trunk # line 6 revision::var::base 1234 # line 7 # line 8 expsrc::var::base code # line 9 |
Invoking the extract system using the above configuration file will extract all sub-directories under "svn://server/var/trunk/code" to "$PWD/src/var/code". It will write a build configuration file to "$PWD/cfg/bld.cfg". The configuration used for this extract will be written to the configuration file at "$PWD/cfg/ext.cfg".
Note - declaration of source directories |
---|
EXPSRC or SRC?
So far, we have only declared source directories using the "EXPSRC" statement, which stands for "expandable source directory". A source directory declared using this statement will trigger the system to search recursively for any sub-directories under the declared one. Any sub-directories containing regular source files will be included in the extract. Empty directories, hidden directories and directories containing only hidden files are ignored. If you do not want the system to search for sub-directories underneath your declared source directory, you can declare your source directory using the "SRC" statement. The "SRC" statement is essentially the same as "EXPSRC" except that it does not trigger the automatic recursive search for source directories. In fact, the system implements the "EXPSRC" statement by expanding it into a list of "SRC" statements. Package and sub-package The second field of a repository, revision or source directory declaration label is the name of the container package. It is a name selected by the user to identify the system or project he/she is working on. (Therefore, it is often sensible to choose an identifier that matches the name of the project or system.) The package name provides a unique namespace for a file container. Source directories are automatically arranged into sub-packages, using the names of the sub-directories as the names of the sub-packages. For example, the declaration at line 9 in example 2 will put the source directory in the "var/code" sub-package automatically. Note that, in additional to slash "/", double colon "::" and double underscore "__" (internal only) also act as delimiters for package names. Please avoid using them for naming your files and directories. You can declare a sub-package name explicitly in your source directory statement. For example, the following two lines are equivalent: src::var::base code/VarMod_Surface src::var/code/VarMod_Surface::base code/VarMod_Surface Explicit sub-package declaration should not be used normally, as it requires a lot more typing (although there are some situations where it can be useful, e.g. if you need to re-define the package name). Currently, the extract system only supports non-space characters in the package name, as the space character is used as a delimiter between the declaration label and its value. If there are spaces in the path name to a file or directory, you should explicity re-define the package name of that path to a package name with no space using the above method. However, we recommend that only non-space characters are used for naming directories and files to make life simpler. |
Note - the expanded extract configuration file |
---|
At the end of a successful extract, the configuration used by the
current extract is written in "cfg/ext.cfg" under the extract
destination root. This file is an "expanded" version of the original,
with changes in the following declarations:
With this file, it should be possible for a later extract to re-create the current configuration even if the contents of the repository have changed. (This applies only to code stored in the repository.) |
The next example demonstrates how to extract from a repository and mirror the code to an alternate location. It is essentially the same as example 2, except that it has three new lines to describe how the system can mirror the extracted code to an alternate location.
Extract configuration example 3 - mirror code |
---|
cfg::type ext cfg::version 1.0 dest $PWD rdest::machine tx01 # line 6 rdest::logname frva # line 7 rdest /scratch/frva/extract/example3 # line 8 repos::var::base svn://server/var/trunk revision::var::base 1234 expsrc::var::base code |
Here is an explanation of what each line does:
Invoking the extract system on the above configuration will trigger an extract similar to that given in example 2, but it will also attempt to mirror the contents at "$PWD/src/var/code" to "/scratch/frva/extract/example3/src" on the alternate destination. It will also mirror the expanded extract configuration file "$PWD/cfg/ext.cfg" to "/scratch/frva/extract/example3/cfg/ext.cfg" and "$PWD/cfg/bld.cfg" to "/scratch/frva/extract/example3/cfg/bld.cfg". It is also worth noting that the content of the build configuration file will be slightly different, since it will include directory names appropriate for the alternate destination.
Note - mirroring command |
---|
The extract system currently supports "rdist" and "rsync" as its
mirroring tool. The default is "rsync". To use "rdist" instead of
"rsync", add the following line to your extract configuration file:
rdest::mirror_cmd rdist N.B. If you are going to mirror code to another machine, you need to ensure that your account on the target machine is set up correctly to accept commands from the local machine. In our current settings of both "rdist" and "rsync", all you need to do is set up your "$HOME/.rhosts" file on the target machine. For example, if you are "fred" working on the local machine "eld001", you will need to have the following entry in your "$HOME/.rhosts" on the target machine: eld001 fred |
So far, we have only extracted from a single location. The extract system is not much use if that is the only thing it can do. In fact, the extract system supports extract of multiple source directories from multiple branches in multiple repositories. The following configuration file is an example of how to extract from multiple repositories:
Extract configuration example 4 - extract from multiple repositories |
---|
cfg::type ext cfg::version 1.0 dest $PWD repos::var::base fcm:var_tr # line 6 repos::ops::base fcm:ops_tr # line 7 repos::gen::base fcm:gen_tr # line 8 revision::gen::base 2468 # line 10 expsrc::var::base src/code # line 12 expsrc::var::base src/scripts # line 13 expsrc::ops::base src/code # line 14 src::gen::base src/code/GenMod_Constants # line 15 src::gen::base src/code/GenMod_Control # line 16 src::gen::base src/code/GenMod_FortranIO # line 17 src::gen::base src/code/GenMod_GetEnv # line 18 src::gen::base src/code/GenMod_ModelIO # line 19 src::gen::base src/code/GenMod_ObsInfo # line 20 src::gen::base src/code/GenMod_Platform # line 21 src::gen::base src/code/GenMod_Reporting # line 22 src::gen::base src/code/GenMod_Trace # line 23 src::gen::base src/code/GenMod_UMConstants # line 24 src::gen::base src/code/GenMod_Utilities # line 25 |
Here is an explanation of what each line does:
We shall end up with a directory tree such as:
$PWD | |--- cfg | | | |--- bld.cfg | |--- ext.cfg | |--- src | |--- gen | | | |--- code | | | |--- GenMod_Constants | |--- GenMod_Control | |--- GenMod_FortranIO | |--- GenMod_GetEnv | |--- GenMod_ModelIO | |--- GenMod_ObsInfo | |--- GenMod_Platform | |--- GenMod_Reporting | |--- GenMod_Trace | |--- GenMod_UMConstants | |--- GenMod_Utilities | |--- ops | | | |--- code | | | |--- ... | |--- var | |--- code | | | |--- ... | |--- scripts | |--- ...
Note - revision number |
---|
As seen in the above example, if a revision number is not specified for
a repository URL, it defaults to the last changed revision at the "HEAD"
of the branch. The revision number can also be declared in other ways:
If a revision number is specified for a branch, the actual revision used by the extract system is the last changed revision of the branch, which may differ from the declared revision. While this behaviour is useful in most situations, some users may find it confusing to work with. It is possible to alter this behaviour so that extract will fail if the declared revision does not correspond to a changeset of the declared branch. Make the following declaration to switch on this checking: revmatch true |
We have so far dealt with a single branch in any package. The extract system can be used to "combine" changes from different branches of a package. An example is given below:
Extract configuration example 5 - extract from multiple branches |
---|
cfg::type ext cfg::version 1.0 dest $PWD repos::var::base fcm:var_tr repos::ops::base fcm:ops_tr repos::gen::base fcm:gen_tr revision::gen::base 2468 expsrc::var::base src/code expsrc::var::base src/scripts expsrc::ops::base src/code src::gen::base src/code/GenMod_Constants src::gen::base src/code/GenMod_Control src::gen::base src/code/GenMod_FortranIO src::gen::base src/code/GenMod_GetEnv src::gen::base src/code/GenMod_ModelIO src::gen::base src/code/GenMod_ObsInfo src::gen::base src/code/GenMod_Platform src::gen::base src/code/GenMod_Reporting src::gen::base src/code/GenMod_Trace src::gen::base src/code/GenMod_UMConstants src::gen::base src/code/GenMod_Utilities repos::var::branch1 fcm:var_br/frva/r1234_new_stuff # line 27 repos::var::branch2 fcm:var_br/frva/r1516_bug_fix # line 28 repos::ops::branch1 fcm:ops_br/opsrc/r3188_good_stuff # line 29 |
The configuration file in example 5 is similar to that of example 4 except for the last three lines. Here is an explanation of what they do:
When we invoke the extract system, it will attempt to extract from the first declared branch of a package, if the last changed revision of the source directory is the same in all the branches. However, if the last changed revision of the source directory differs for different branches, the system will attempt to obtain an extract priority list for each source directory, using the following logic:
For the "var" package in the above example, let us assume that we have three source directory packages X, Y and Z under "code", and their last changed revisions under "base" are 100. Let's say we have committed some changes to X and Z in the "branch1" branch at revision 102, and other changes to Y and Z in the "branch2" branch at revision 104, the extract priority lists for X, Y and Z will look like:
Once we have an extract priority list for a source directory, we can begin extracting source files in the source directory. The source directory of the base branch is extracted first, followed by that in the subsequent branches. If a source file in a subsequent branch has the same content as the that in the base branch, it is discarded. Otherwise, the following logic determines the branch to use:
conflict fail
Once the system has established which source files to use, it determines whether the destination file is out of date or not. The destination file is out of date if it does not exist or if its content differs from the version of the source file we are using. The system only updates the destination if it is considered to be out of date.
The extract system can also combine changes from branches in the Subversion repository and the local file system. The limitation is that there can only be one branch from the local file system. (By convention, the branch is named "user".)
It is also worth bearing in mind that the "user" branch always takes precedence over branches residing in Subversion repositories. Hence, source directories from a "user" branch are always placed at the end of the extract priority list.
Extracting from a mixture of Subversion repository and local file system is demonstrated in the next example.
Extract configuration example 6 - extract from multiple branches + user paths |
---|
cfg::type ext cfg::version 1.0 dest $PWD repos::var::base fcm:var_tr repos::ops::base fcm:ops_tr repos::gen::base fcm:gen_tr revision::gen::base 2468 expsrc::var::base src/code expsrc::var::base src/scripts expsrc::ops::base src/code src::gen::base src/code/GenMod_Constants src::gen::base src/code/GenMod_Control src::gen::base src/code/GenMod_FortranIO src::gen::base src/code/GenMod_GetEnv src::gen::base src/code/GenMod_ModelIO src::gen::base src/code/GenMod_ObsInfo src::gen::base src/code/GenMod_Platform src::gen::base src/code/GenMod_Reporting src::gen::base src/code/GenMod_Trace src::gen::base src/code/GenMod_UMConstants src::gen::base src/code/GenMod_Utilities repos::var::branch1 fcm:var_br/frva/r1234_new_stuff repos::var::branch2 fcm:var_br/frva/r1516_bug_fix repos::ops::branch1 fcm:ops_br/opsrc/r3188_good_stuff repos::var::user $HOME/var # line 31 repos::gen::user $HOME/gen # line 32 |
Example 6 is similar to example 5 except that it is also extracting from local directories. Here is an explanation of the lines:
Note - the INC declaration |
---|
You have probably realised that the above examples have many repeated
lines. To avoid having repeated lines in multiple extract configuration
files, you can use INC declarations to "include" other extract
configuration files. For example, if the configuration file of example 5
is stored in the file "$HOME/example5/ext.cfg", line 1 to 29 of
example 6 can be replaced with an INC declaration. Example 6 can then be
written as:
inc $HOME/example5/ext.cfg repos::var::user $HOME/var repos::gen::user $HOME/gen Note: the INC declaration supports the special "environment variable" $HERE. If this variable is already set in the environment, it acts as a normal environment variable. However, if it is not set, it will be expanded into the container directory of the current extract configuration file. This feature is particularly useful if you are including a hierarchy of extract configurations from files in the same container directory in a repository. |
All the examples above dealt with standalone extract, that is, the current extract is independent of any other extract. If a previous extract exists in another location, the extract system can inherit from this previous extract in your current extract. This works like a normal incremental extract, except that your extract will only contain the changes you have specified (compared with the inherited extract) instead of the full source directory tree. This type of incremental extract is useful in several ways. For instance:
The following example is based on example 4 and 6. The assumption is that an extract has already been performed at the directory "~frva/var/vn22.0" based on the configuration file in example 4.
Extract configuration example 7 - inherit from a previous extract |
---|
cfg::type ext cfg::version 1.0 dest $PWD use ~frva/var/vn22.0 # line 6 repos::var::branch1 fcm:var_br/frva/r1234_new_stuff # line 8 repos::var::branch2 fcm:var_br/frva/r1516_bug_fix # line 9 repos::ops::branch1 fcm:ops_br/opsrc/r3188_good_stuff # line 10 repos::var::user $HOME/var # line 12 repos::gen::user $HOME/gen # line 13 |
Running the extract system using the above configuration will trigger an incremental extract, as if you are running an incremental extract having modified the configuration file in example 4 to that of example 6. The only difference is that the original extract using the example 4 configuration will be left untouched at "~frva/var/vn22.0", and the new extract will contain only the changes in the branches declared from line 8 to 13.
Note: extract inheritance allows you to add more branches to a package, but you should not redefine the REPOS, REVISION, EXPSRC or SRC declarations of a branch that is already declared (and already extracted) in the inherited extract. Although the system will not stop you from doing so, you may end up with an extract that does not quite do what it is supposed to do. For example, if the "base" branch in the "foo" package (repos::foo::base) is already defined and extracted in an extract you are inheriting from, you should not redefine any of the *::foo::base declarations in your current extract. However, you are free to add more branches for the same package with new labels (e.g. repos::foo::b1), and indeed new packages that are not already defined in the inherited extract (e.g. repos::bar::base).
If you are setting up an extract to be inherited, you do not have to perform a build. If you don't you will still gain the benefit of incremental file extract, but you will be performing a full build of the code.
Configuration settings for feeding into the build system can be declared through the extract configuration file using the "BLD::" prefix. Any line in an extract configuration containing a label with such a prefix will be considered a build system variable. At the end of a successful extract, the system strips out the "BLD::" prefix before writing these variables to the build configuration file. Some example entries are given between line 17 and 22 in the following configuration file:
Extract configuration example 8 - setting build configuration |
---|
cfg::type ext cfg::version 1.0 dest $PWD repos::var::base fcm:var_tr repos::ops::base fcm:ops_tr repos::gen::base fcm:gen_tr revision::gen::base 2468 expsrc::var::base src/code expsrc::var::base src/scripts expsrc::ops::base src/code src::gen::base src/code/GenMod_Constants src::gen::base src/code/GenMod_Control src::gen::base src/code/GenMod_FortranIO src::gen::base src/code/GenMod_GetEnv src::gen::base src/code/GenMod_ModelIO src::gen::base src/code/GenMod_ObsInfo src::gen::base src/code/GenMod_Platform src::gen::base src/code/GenMod_Reporting src::gen::base src/code/GenMod_Trace src::gen::base src/code/GenMod_UMConstants src::gen::base src/code/GenMod_Utilities bld::target VarProg_AnalysePF.exe # line 27 bld::tool::fc sxmpif90 # line 29 bld::tool::cc sxmpic++ # line 30 bld::tool::ld sxmpif90 # line 31 |
The above example is essentially the same as example 4, apart from the additional build configuration. The following is a simple explanation of what the lines represent: (For detail of the build system, please see the next chapter on The Build System.)
Note - user variables and internal variables |
---|
When you start using the extract system to define compiler flags for
the build system, you may end up having to make a lot of long and
repetitive declarations. In this case, you may want to define variables to
replace the repetitive parts of the declarations. In the extract system,
you can define a user variable by making a declaration with a label that
begins with a percent sign "%". For example:
# Declare a variable %fred %fred -Cdebug -eC -Wf,-init heap=nan stack=nan bld::tool::fflags %fred # bld::tool::fflags -Cdebug -eC -Wf,-init heap=nan stack=nan bld::tool::fflags::foo %fred -f0 # bld::tool::fflags::foo -Cdebug -eC -Wf,-init heap=nan stack=nan -f0 bld::tool::fflags::bar -w %fred # bld::tool::fflags::bar -w -Cdebug -eC -Wf,-init heap=nan stack=nanFurther to this, each declaration results in an internal variable of the same name and you can also refer to any of these internal variables in the same way. So, the example given above could also be written as follows: bld::tool::fflags -Cdebug -eC -Wf,-init heap=nan stack=nan bld::tool::fflags::foo %bld::tool::fflags -f0 bld::tool::fflags::bar -w %bld::tool::fflagsNote that, if required, variable names may also be enclosed in curly brackets "{}" when used in a value. |
Note - as-parsed configuration |
---|
If you use a hierarchy of INC declarations or variables, you may end up with a configuration file that is difficult to understand. To help you with this, the extract system generates an as-parsed configuration file at "cfg/parsed_ext.cfg" of the destination. The content of the as-parsed configuration file is what the extract system actually reads. It should contain everything in your original extract configuration file, except that all INC declarations, environment variables and user/internal variables are expanded. |
The amount of diagnostic messages generated by the extract system is normally set to a level suitable for normal everyday operation. This is the default diagnostic verbose level 1. If you want a minimum amount of diagnostic messages, you should set the verbose level to 0. If you want more diagnostic messages, you can set the verbose level to 2 or 3. You can modify the verbose level in two ways. The first way is to set the environment variable FCM_VERBOSE to the desired verbose level. The second way is to invoke the extract system with the "-v <level>" option. (If set, the command line option overrides the environment variable.)
The following is a list of diagnostic output at each verbose level:
Verbose level | Possible output |
---|---|
0 |
|
1 |
|
2 |
|
3 |
|
The extract system can still be used if Subversion is not available. Clearly, you can only use local repositories. However, you can still do incremental extract, mirror an extract to an alternate location, or combine code from multiple local repositories.
If you are using Subversion but your server is down then clearly there is little you can do. However, if you already have an extract then you can re-run fcm extract as long as the extract configuration file only refers to fixed revisions. If this is not the case then you can always use the expanded extract configuration file which can be found in "cfg/ext.cfg" under the extract destination root. This means that you can continue to makes changes to local code and do incremental extracts even whilst your Subversion server is down.