CIFXML Overview.

CIFXML supports the CIF syntax and semantics from The International Union of Crystallography (http://www.iucr.org).

The CIF architecture supports the hierarchy:

One or more data files (e.g. a diffraction experiment and its processing)
One or more Dictionaries to which the data file must conform
A Dictionary Defintion Language (DDL) to which the dictionary must conform and which provides semantic information

CIFXML supports documents and dictionaries conformant to DDL1.

This document describes how the CIF specification is implemented in CIFXML. We have copied the text from the CIF syntax[1] and semantics[2] specification documents and annotate it where appropriate. Text in italics is copied verbatim from the specification, while the rest are our comments. The numbers refer to the sections in the CIF specification. Sections irrelevant to this implementation may be omitted.

[1] http://www.iucr.org/iucr-top/cif/spec/version1.1/cifsyntax.html
[2] http://www.iucr.org/iucr-top/cif/spec/version1.1/cifsemantics.html

Definition of Terms (in [1] and [2])

2.1. A CIF is a file conforming to the specification herein stated, containing either information on a crystallographic experiment or its results (or similar scientific content); or descriptions of the data identifiers in such a file.
provided by: CIF
2.2 A data file is understood to convey information relating to a crystallographic experiment.
provided by: CIF
2.3. A dictionary file is understood to contain information about the data items in one or more data files as identified by their data names.
provided by: CIF. CIFXML does not per se support semantics beyond any CIF file, which is done through a further package.
2.4. A data name is a case-insensitive identifier (a string of characters beginning with an underscore character) of the content of an associated data value.
provided by: CIFItem.getName() or CIFLoop.getNames()
2.5. A data value is a string of characters representing a particular item of information. It may represent a single numerical value; a letter, word or phrase; extended discursive text; or in principle any coherent unit of data such as an image, audio clip or virtual-reality object.
provided by: CIFItem.getValue() or CIFLoop.getValues()
2.6. A data item is a specific piece of information defined by a data name and an associated data value.
provided by: CIFItem or CIFLoop
2.8. A data block is the highest-level component of a CIF, containing data items or save frames. A data block is identified by a data block header, which is an isolated character string (that is, bounded by white space and not forming part of a data value) beginning with the case-insensitive reserved characters data_.
provided by: CIFDataBlock
2.9. A block code is the variable part of a data block header, e.g. the string foo in the header data_foo.
provided by: CIFDataBlock.getId()
2.10. A save frame is a partitioned collection of data items within a data block, started by a save frame header, which is an isolated character string beginning with the case-insensitive reserved characters save_, and terminated with an isolated character string containing only the case-insensitive reserved characters save_.
provided by: CIFSaveFrame
2.11. A frame code is the variable part of a save frame header, e.g. the string foo in the header save_foo.
provided by: CIFSaveFrame.getId()

Syntax (from [1])

Supported and enforced.

3-4.
Supported
5.
CIFXML allows save frames but provides little support.
6. Within a single CIF, no two data blocks may have the same name.
Supported.
7. A given data name (tag) (see 2.4 and 2.7) may appear no more than once in a given data block.
Supported.
8.
9.
10. A simple data value may optionally be delimited by any of the same set of delimiting character strings, except for data values that are to be interpreted as numbers.
CIFXML does not provide datatypes, and does NOT store the type of quoting, though it can validate values against CIF dictionary supported (_, #, ', ", ;) and not-supported ($, [, ]). stop_ and global_ are recognised but ignored.
12.
13. The base CIF specification distinguishes between character and numeric values.
We find it difficult to interpret the described semantics in a context-free manner and CIFXML only supports char.
14-18.
19. Matching square bracket characters, '[' and ']', are reserved.
Not supported
20-24.
25. The way in which a line is terminated is operating-system dependent.
CIFXML replaces all EOL by a single "\n" (ASCII 10) character.
26-27.
28. Lines of text may not exceed 2048 characters in length.
Not supported. CIFXML allows unlimited length.
29. Data names may not exceed 75 characters in length.
Not supported. CIFXML allows unlimited length.
30. Data block codes and save frame codes may not exceed 75 characters in length.
Not supported. CIFXML allows unlimited length.
31.
Supported.
32-33.
34. the first 11 bytes of the file should be the string #\#CIF_1.1.
Supported.
35-56
57.
No string LOOP_, STOP_, GLOBAL_ or which starts with DATA_ or SAVE_ is accepted as a non-quoted string.
CIFXML does not support quoting semantics.
58-62.
63. The number of values in the (loop) body must be a multiple of the number of tags in the header.
Supported.

Semantics (from [2])

The IUCr provides a list of accepted, possible and reserved semantics. Some of these are difficult to implement unambiguously and some are simply difficult. Note that dictionaries are supported by layering on top of CIFXML and are generally irrelevant here.

3. [Dictionaries]. In CIF applications, formal catalogues of standard data names and their associated attributes are maintained as external reference files called data dictionaries. These dictionary files share the same structure and syntax rules as data CIFs.
CIFXML itself does not require dictionaries.
4. [DDL] [...](Dictionary Definition Languages or DDL) are supported for detailing the meaning and associated attributes of data names. These are known as DDL1 (Hall &Cook, 1995) and DDL2.
CIFXML only supports DDL1. DDL1 itself is in the CIF dictionary syntax but its semantics are not currently supported
5. While it may be formally possible to define the semantics of the data items in a given data file in both DDL1 and DDL2 data dictionaries, in practice different dictionaries are constructed to define the data names appropriate for particular crystallographic applications, and each such dictionary is written in DDL1 or DDL2 formalism according to which appears better able to describe the data model employed. There is thus in practice a bifurcation of CIF into two dialects according to the DDL used in composing the relevant dictionary file. However, the use of aliases may permit applications tuned to one dialect to import data constructed according to the other.
Only DDL1 is supported.

Data name semantics

6. Data names should be considered as void of semantic content.
Supported.
7. However, it is customary to construct data names as a sequence of components elaborating the classification of the item within the logical structure of its associated dictionary. Hence a data name such as _atom_site_fract_x displays a hierarchical arrangement of components corresponding to membership of nested groupings of data elements. The choice of components readily indicates to a human reader that this data item refers to the fractional x coordinate of an atomic site within a crystal unit cell, but it should be emphasised from a computer programming viewpoint that this is coincidental; the attributes that constrain the value of this data item (and its relationship to others such as _atom_site_fract_y and _atom_site_fract_z) must be obtained from the dictionary and not otherwise inferred.
Supported. CIFXML makes no attempt to parse such context-dependent constructions.
8. In practice data names described in a DDL2 dictionary are constructed with a period character separating their specific function from the name of the category to which they have been assigned. In the absence of a dictionary file, this convention permits the inference that the data item with name _atom_site.fract_x will appear in the same looped list as other items with names beginning _atom_site., and that all such items belong to the same category.
DDL2 is not supported. DDL1 names with an embedded "." are regarded as semantically void.
9. The intention of the maintainers of public CIF dictionaries is to formulate a single authoritative set of data names for each CIF dialect (i.e. DDL1 and DDL2), thus facilitating the reliable archive and interchange of crystallographic data. However, it is also permissible for users to introduce local data names into a CIF. Two mechanisms exist to reduce the danger of collision of data names that are not incorporated into public dictionaries.
Not directly supported in CIFXML.
10. The character string [local] (including the literal bracket characters) is reserved for local use. That is, no public dictionary will define a data name that includes this string. This allows experimentation with data items in a strictly local context, i.e. in cases where the CIF is not intended for interchange with any other user.
Semantics associated with this are ignored.
11. Where CIFs including local data items are expected to enjoy a public circulation, authors may register a reserved prefix for their sole use. The registry is available on the web at

·

·                http://www.iucr.org/iucr-top/cif/spec/reserved.html

A reserved prefix, e.g. foo, must be used in the following ways

    * If the data file contains items defined in a DDL1 dictionary, the local data names assigned under the reserved prefix must contain it as their first component, e.g. _foo_atom_site_my_item.

    * If the data file contains items defined in a DDL2 dictionary, then the reserved prefix must be

          o the first component of data names in a category defined for local use, e.g. _foo_my_category.my_item

          o the first component following the period character in a data name describing a new item in a category already defined in a public dictionary, e.g. _atom_site.foo_my_item

. Semantics ignored by CIFXML.

12. There is no syntactic property identifying such a reserved prefix, so that software validating or otherwise handling such local data names must scan the entire registry and match registered prefixes against the indicated components of data names. Note that reserved prefixes may themselves contain underscore characters, so a maximal matching search must be made.
Semantics ignored by CIFXML.

Note on handling of units

13. The published specification for CIF Version 1.0 permitted data values expressed in different units to be tagged by variant data names (Hall, Allen & Brown, 1991, p. 657:)

·

·                     Many numeric fields contain data for which the units must be known. Each CIF data item has a default units code which is stated in the CIF Dictionary. If a data item is not stored in the default units, the units code is appended to the data name. For example, the default units for a crystal cell dimension are Angstroms. If it is necessary to include this data item in a CIF with the units of picometres, the data name of _cell_length_a is replaced by _cell_length_a_pm. Only those units defined in the CIF Dictionary are acceptable. The default units, except for the Angstrom, conform to the SI Standard adopted by the IUCr.

This approach is deprecated and has not been supported by any official CIF dictionary published subsequent to version 1.0 of the Core. All data values must be expressed in the single unit assigned in the associated dictionary.

A small number of archived CIFs exist with variant data names as permitted by the above clause. If it is necessary to validate them against versions of the Core dictionary subsequent to version 1.0, the formal compatibility dictionary cif_compat.dic may be used for the purpose. No other use should be made of this dictionary.

Semantics ignored by CIFXML.

Data value semantics

14. The STAR syntax permits retrieval of data by simply requesting a specific data name within a specific data block. Prior knowledge about data type (e.g. text or numbers), whether the item is looped, or whether the item exists in the file at all, is unnecessary. However, applications in general need to know data type, valid ranges of values, and relationships between data items; and a program designer needs to know the purpose of the data item (i.e. what physical quantity or internal book-keeping function it represents). While such semantic information may be defined informally for local data items (ones not intended for exchange between different users or software applications), formal descriptions of the semantics associated with data values are catalogued in data dictionary files. Currently two formalisms (dictionary definition languages) for describing data value attributes are supported; full specifications of these formalisms (known as DDL1 and DDL2) are provided elsewhere.

CIFXML provides a clean separation between the data and the application program. CIFXML does allow retrieval of a value or loop of values by requesting a specific data name. It is left to a dictionary validator or application validator to decide whether the value is permissible.

Data typing

15. Four base data types are supported in CIF. These are

·

·                    * numb: a value interpretable as a decimal base number and supplied as an integer, a floating-point number or in scientific notation;

·                    * char: a value to be interpreted as character or text data (where the value contains white-space characters, it must be quoted);

·                    * uchar: a value to be interpreted as character or text data but in a case-insensitive manner (i.e. the values FOO and foo are to be taken as identical);

·                    * null: a special data type associated with items for which no definite value may be stored in computer memory. It is the type associated with the special character literal values ? (query mark) and . (full point) which may appear as values for any data item within a data file (see section on "Special generic values" below). It is also the type assigned to items defined in dictionary files which may not occur in data files.

Without a dictionary there is no knowledge of the datatype, so CIFXML preserves the lexical form of the data value after removal of any quotes. It retains exact whitespace including invisible leading and trailing characters within data values (except that line ends are normalised to #10).

16. Many applications distinguish between multi-line text fields and character string values that fit within a single line of text. While this is a convenient practical distinction for coding purposes, formally both manifestations should be regarded as having the same base type, which might be "char" or "uchar". Applications are at liberty to choose whether to define specific multi-line text subtypes, and whether to permit casting between subtypes of a base type. The examples of character string delimiters in paragraph 20 of the document "Syntax" are predicated on an approach that handles all subtypes of character or text data equivalently.
See above for CIFXML treatment.
17. Where the attributes of a data value are not available in a dictionary listing, it may be assumed that a character string interpretable as a number should be taken to represent an item of type numb. However, an explicit dictionary declaration of type will override such an assumption.
CIFXML makes no attempt to interpret a data value as a numb and will preserve its lexical representation unchanged.

Subtyping

18. The base data types detailed in the previous section are very general, and need to be refined for practical application. Refinement of types is to some extent application-dependent, and different subtypes are supported for data items defined by DDL1 and DDL2 dictionary files. The following notes indicate some considerations, but the relevant dictionary files and documentation should be consulted in each case.
CIFXML per se does not support subtypes.
19. DDL1 dictionaries.
Standard uncertainties: Values of type numb may include a standard uncertainty in the final digit(s) of the number where the associated item definition includes the attribute

·

·                     _type_conditions     esd

For example, a value of 34.5(12) means 34.5 with a standard uncertainty of 1.2; it may also be expressed in scientific notation as 3.45E1(12).
CIFXML supports these semantics and will return values with our without the SUs.

21. The unquoted character literals ? (query mark) and . (full point) are special and are valid expressions for any data type.
CIFXML ignores these semantics.
22. The value ? means that the actual value of a requested data item is unknown.
CIFXML ignores these semantics and processes this value as a text string
23. The value . means that the actual value of a requested data item is inapplicable. This is most commonly used in a looped list where a data value is required for syntactic integrity.
CIFXML ignores these semantics and processes this value as a text string

Embedded data semantics

24. The attributes of data items defined in CIF dictionaries serve to direct crystallographic applications in the retrieval, storage and validation of relevant data. In principle a CIF might include as data items suitably encoded fields representing data suitable for manipulation by text processing, image, spreadsheet, database or other applications. It would be useful to have a formal mechanism allowing a CIF to invoke appropriate content handlers for such data fields, and this is under investigation for the next CIF version specification.
CIFXML ignores these semantics and processes any such values as text strings.

CIF conventions for special characters in text

25. The one existing example of embedded semantics is the text character markup introduced in the CIF Version 1.0 specification and summarised in paragraphs 30-37 below. The specification is silent on which fields should be interpreted according to these markup conventions, but the published examples suggest that they may be used in any character field in a CIF data file except as prohibited by a dictionary directive. It is intended that the next CIF version specification shall formally declare where such markup may be used.

Handling of long lines

· 26. The restriction in line length within CIF requires techniques to handle without semantic loss the content of lines of text exceeding the limit (2048 characters in this revision, 80 characters in the initial CIF specification). The line folding protocol defined here provides a general mechanism for wrapping lines of text within CIFs to any extent within the overall line length limit. A specific application where this would be useful is the conversion of lines longer than 80 characters to the CIF 1.0 limit. This 80-character limit is used in the examples below for illustrative purposes.

These techniques are applied only to the contents of text fields and to comments.

In order to permit such folding a special semantics is defined for use of the backslash. It is important to understand that this does not change the syntax of CIF 1.0. All existing CIFs conforming to the CIF 1.0 specification can be viewed as having exactly the same semantics as they now have. Use of these transformational semantics is optional, but recommended.

[...further discussion omitted...] CIFXML currently does not support this convention

Dictionary compliance

27. Dictionary files containing the definitions and attribute sets for the data items contained in a CIF should be identified within the CIF by some or all of the data items

·

·                    _audit_conform_dict_name

·                    _audit_conform_dict_version

·                    _audit_conform_dict_location

corresponding to DDL1 dictionaries, or

    _audit_conform.dict_name

    _audit_conform.dict_version

    _audit_conform.dict_location

for DDL2 dictionaries. Where no such information is provided, it may be assumed that the file should conform against the core CIF dictionary.
CIFXML does not process these semantics.

28. The _audit_conform data items may be looped in the case where more than one dictionary is used to define the items in a CIF, and they may include dictionaries of local data items provided such dictionary files have been prepared in accordance with the rules of the appropriate DDL.
CIFXML does not process these semantics.
29. A detailed protocol exists for locating, merging and overlaying multiple dictionary files (McMahon, Westbrook & Bernstein, 2000).
CIFXML does not process these semantics.

CIF markup conventions

30. If permitted by the relevant dictionary and if no other indication is present, the contents of a text or character field are assumed to be interpretable as text in English or some other human language. Certain special codes are used to indicate special characters or accented letters not available in the ASCII character set, as listed below.
CIFXML attempts to process some of these into UNICODE. However no explicit conversion is given, and textual representations sometimes have to be used. CIFXML will attempt to create this markup on output of a CIF. However not all conversions are supported
31-37.

CIFXML can be extended to support most of this if required.