Implementing an advanced external source for Adlib
Contents
Below documentation is meant for an Adlib for Windows implementation. Please click here for the documentation relevant to an Axiell Collections implementation.
Normally, when you validate a term or name entered in a linked field, that term will be validated to the Thesaurus or Persons and institutions. However, it is possible as well to set up one or more external sources for that particular purpose in Adlib. This allows you to select an authority database for each linked term to be validated in the Find data for the field window. If you then actually validate to an external source and you select a term which doesn’t occur in your own authority database yet, then that term will added to your database as a new record so that the link in the field will be made on the (new) term in your own authority database. When indeed a new term or name record is created that way, usually only the external term or name will be copied (aka “derived”), any other data from the external record won’t be included.
From 7.2 though, functionality is available allowing you to copy any other details from the external validation record, together with the term itself, to the new record in your own authority database. Moreover, more information from the external record can be displayed during validation (the selection of an appropriate term), so it becomes easier to select the proper variant of a term.
After the upgrade to adlwin.exe 7.2 or higher you won’t see any difference in your existing Adlib application: first the application will have to be modified for an available and appropriate external validation source (returning Adlib XML and having field names identical to the Adlib authority database) before users can reap the benefits of this new functionality. Most of those can probably be gained in combination with an external thesaurus in which terms from different sources have been collected, a kind of multi-thesaurus: a thesaurus of this kind is in the works but is currently (October 2014) not publicly available. In the examples below we therefore use the (largely) Dutch AAT (Art and Architecture Thesaurus, copyright RKD & Getty), which allows us to at least study some of the possibilities of the new functionality.
An external source with advanced settings
Proceed with the following steps to implement the advanced functionality in your Adlib application (each step is explained further in the next paragraph):
| 1. | Here in Designer, set up the desired external source for the desired linked fields in your application. |
| 2. | After setting the basic options for a external source, click the Advanced button to indicate which fields must be derived from the external record when you use the external term. In the advanced settings you must also refer to a custom (yet simple) adapl which will control the display of details from the external records in the Find data for the field window. You may also decide upon the grouping of found terms in the Find data for the field window. |
| 3. | Optionally you can have the date and time of derival of an external term and/or its accompanying data stored in the relevant local validation record. For this to work you’ll have to add two fields to your local authority database (and to a screen) and extend the storage adapl for the relevant database with code to fill these fields in new/adjusted records with the current date and time when applicable. |
ad 1: basic settings for the external source
Here in Axiell Designer you set up external thesauri as follows:
| 1. | In the Application browser, open the \data folder of your Adlib system. |
| 2. | Open the desired database and select the linked field for which you’d like to set up external authority databases, the object_name field (tag OB) in COLLECT for example. |
| 3. | Go to the Linked field properties tab. At the bottom you’ll see the External sources box. |
| 4. | Click the Add button in that box to insert a new external source. |
| 5. | Put the cursor in the Path or URL entry field. You can click the … button to search for the .inf file of the external validation database on your system or on the local network: the path to the selected .inf will then be entered for you. Instead you may fill in a path manually as well, or provide a URL to the web service of an external validation database which returns the search result in Adlib XML format. The RKD for instance, offers a free web service for the Dutch AAT (Art & Architecture Thesaurus), which also contains some English terms by the way. An example of a URL to this web service is: http://service.aat-ned.nl/api/wwwopac.ashx?database=aat-gateway&search=term[nl-NL]="%data%*" sort term[nl-NL]&limit=100 Since the AAT really is multilingual, you’ll probably want to specify the search language in the search statement: the [nl-NL] notation behind the field name in the example makes sure that only Dutch language values will be searched. The %data% part makes sure that the actually executed query always uses the term or name entered by the user in the linked field; the asterisk behind it switches automatic right truncation on. Using sort term[nl-NL], the search result in the Find data for the field window will then be sorted on the Dutch term from the retrieved records. Limit finally, will limit the search result to the indicated number of terms to reduce the run time of the search. If you do not specify a limit, an implicit limit of 500 will be applied to linked fields. If more terms are found than the limit, the user can retrieve the remaining terms in the Find data for the field window by pressing the PageDown key one or more times. You may adjust the query to your liking. If at any time you gain access to a multi-thesaurus (running on a wwwopac.ashx version that supports the group attribute) which contains term collections from different sources (for which the name of the original source has been registered in the records too), then it be-comes interesting to have the search result grouped on the source field, especially since Adlib 7.2 and higher are able to visualize such a grouping in the Find data for the field window (through the advanced settings for the external source, see the next paragraph). In this case, the entire search result (before any limit has been applied) will first be grouped on source, for example, and within those groups any sorting will still be possible: do always sort on the grouping field first though and only after that on another field. In the query you must also use group <field_name1> prior to sort <field_name1>, <field_name2>. For more information about putting together a wwwopac.ashx query, see: http://api.adlibsoft.com/site/api/functions/search. |
| 6. | Click the Description entry field to be able to enter the name or description of the external thesaurus in several languages. |
| 7. | A link screen must be present for any external source being approached by a URL. If you leave the Link screen option empty, by default the link screen as specified on the Link screens properties tab of the current linked field will be used. However, you are free to design a specific link screen and link it here to the external source. |
| 8. | If you want, you can specify even more external thesauri for this linked field, but if you are satisfied for now you can save your changes in the database definition. |
ad 2: Advanced settings for the external source
The Advanced button in the External sources box on the Linked field properties tab opens the External source: advanced properties window, which allows you to set the advanced options for the currently selected external source.
It offers the following options:
| • | Adapl - Add a reference (relative path) to an adapl (by typing it or by using the ... button), which formats each external term or name and its accompanying data for a multi-line presentation in the term list on the View table tab of the Find data for the field window in your local application. Firstly, to display the View table tab at all, a link screen needs to have been specified, specifically for the current external source or for general application on the Link screens properties tab of the current linked field, but all the fields and labels on the selected link screen will be overruled by the adapl you specify here. The relevant adapl will be executed per listed term or name record and has access to all data from the XML document resulting from the query at the basis of this external source, provided that the local, English data dictionary field name of each field tag you use in the adapl is identical to the field name in the XML search result. In the adapl, the data from the field tags must then be assigned to different occurrences of two special reserved variables named &X and &Y. Each occurrence of &X represents a data line on the View table tab in the Find data for the field window, where the smallest occurrence number is the top line of the relevant term data and the biggest occurrence number the bottom line. These occurrences can be assigned strings (of fixed text) and/or the values from variables or field tags, whereby values and strings can be concatenated using the + sign like when building strings in output adapls. Further note that the occurrence numbers do not need to be consecutive to have the same result, so specifying occurrences 1, 2 and 3 has the same result as specifying occurrences 4, 10 and 30 (if you wouldn't find the latter confusing). Each occurrence of &Y represents the font colour of the text displayed through &X. These occurrences must be assigned an HTML hexadecimal colour code as a string value and there must be as many occurrences of &Y as there are of &X and their occurrence numbers need to be identical so that e.g. colour occurrence 4 can be matched to data occurrence 4, etc. You can display as many data lines per term or name as your wish. The full ADAPL code example below displays two data lines per external term, containing the term itself in line 1 and the source in line 2, each text line in a different colour: &X[1]= te; &X[2]= 'Source <' + br + '>'; &Y[1]= '#000000'; &Y[2]= '#A0A0A0'; end |
| • | Group by field (optional) - Use the ... button to select the field which is used in the external source URL to create groups (if any). Each group value will then be displayed as a bold header in a coloured bar above the terms or names that are part of the relevant group. In the adapl specified above, you don't need to program anything for the group-by field. If you use a Group by field while you haven’t specified a grouping in the external source URL (for instance because the wwwopac.ashx version used by the relevant service doesn’t support the group attribute yet), the search result will still be grouped but only after server-side sorting and the limit have been applied (which is not ideal). |
| • | Initial group size - Specify the maximum number of terms or names that should be visible per grouping when the list of terms is opened. This number doesn't limit the actual search result: by double-clicking a group header label with the hook icon in the terms list you can expand that group to display all the terms underneath it. A box icon means that all terms are displayed already: double-click it to collapse that part of the list completely. Setting this option to zero means that all groups will initially be displayed collapsed. |
| • | Mapping (optional) - Even though you might use the adapl set in these advanced properties to display other fields from the external term record as well as the term itself, that doesn't mean the other data is automatically copied to the local record when you actually use the relevant term from the external source: normally only the term or name is copied. To copy more than just the term or name, you must specify a list of mapped fields here. The mapping defines which fields will be written to a newly created local validation record, but also which fields will be updated when the term already exists in the local authority database. In the latter case, mapped fields which have been defined in the local authority database as non-repeated, will be overwritten. To repeatable fields, a new occurrence will be added. It is therefore important to check whether a field has been defined as repeated or not before adding it to the mapping! To map a field, click the Add button to add a new mapping line. Then click the Tag entry field to select the field tag to map. Although it is possible to map many fields, we recommend to map no more than the term itself, the source* and the URI** (if present), and take into account that mapping internally linked fields can possibly create unforeseen and/or conflicting local relations between terms. One (and only one) of the mapped fields can be marked as the Primary key, and this field will then be used to find out if the external term record selected by the user already exists in the local authority database or not: if you select such a field, it must be a field in which all values are unique, like a URI field for example (the term field can often have non-unique values, so that field cannot be used then). If no unique identifying field is present, just select no primary key at all. (If in the running application a selected value from an external term record cannot be found in the primary key field of the local authority database or if no primary key field had been set, then a search on the term or name string in the term/name field will be performed instead to find similar records after which the user can still choose to link to a newly created term record based on the external record or to update the existing local record and then link to it.) * Once data from the external source has been written to the local authority database, it becomes locally managed information, which can be subject to changes. These changes can be different from any changes made by third parties to the description in the external source. Although any locally stored URI can be used to find the most recent description of the term in the external source, we recommend limiting the mapping to the term itself, the source and the URI, to avoid having a local term description that is dependent on an external term description while the local term cannot easily be kept up to date. ** A URI (Uniform Resource Identifier) is a unique character string which serves to identify a web document. Some authority databases have a field that contains such a URI. |
Ad 3: adding the derival date and time to records
Optionally you can have the date and time of derival of an external term and/or its accompanying data saved in the relevant (new or existing) local validation record. If an original input or edit date and time have also been registered per record in the external authority source, then in theory you could compare both dates to find out if it’s time to update certain term definitions in your local validation database: however, there is no tool at hand currently to execute such a comparison, so storing the derival date and time now mostly prepares your authority records for future developments, although Adlib will already allow you to search on the derival dates of local validation records to find out which terms have been added or edited before or after a certain date.
To implement this, add two new, repeatable fields to your local authority database, one for the date, a DateISO field with e.g. tag ad (if that tag doesn’t appear in your validation database yet) and a field with e.g. tag at of data type Time. Further, let’s assume that THESAU is your authority database.
Both fields, together with the tags tn (term.number) and br (source), must be included in a data dictionary field group. Name the field group appropriately.
The idea is that the source, the term number and the date and time should be the only repeatable source fields because they are of external origin while in this case we choose to locally describe the scope note, the content date, the history note and any other notes about the term and do that only once, independent of the source data.
Next we must also add the two new fields to the Thesaurus term (thes.fmt) screen, below the Source and Number fields, so that they can be repeated as a group. The properties of the two new fields must be set to Repeated, Date or Time and Read-only; do not enter a Field group.
Save the changes in the data dictionary and the screen.
Finally you need to extend the storage adapl for THESAU, force.ada in this case, with code to enter the current date and time in new or adjusted term records whenever that is applicable*. If you’ve used the suggested tags, you can insert the following code underneath the change log at the top of the adapl:
* Local variables:
integer maxOcc, occ, THESAUocc, THESAUmaxOcc, found;
* FACS Declarations:
fdstart THESAU '../data+thesau';
%0 is THESAU_priref;
br is THESAU_source;
tn is THESAU_termNo;
fdend;
if (&1 = 11){ /* before storage
/* Add date/time for new/changed source references
maxOcc = repcnt(tn);
if (maxOcc) {
if (&0 = 0) { /* New record
occ = 1;
while (occ <= maxOcc) {
ad[occ] = date$(8);
at[occ] = time$(1);
occ = occ + 1;
}
} else { /* Existing record
/* Check if a new Source occurrence has been added
open THESAU;
if (&E) {
errorm 'Error opening THESAU, code = ' + &E + ~
' [Details: ' + error$(&E) + '].' + ~
chr$(13) + chr$(10) + ~
'The annotation date/time can not be added.';
} else {
/* Retrieve the previous version of the thesaurus record
read THESAU using THESAU_priref = %0;
if (&E) {
errorm 'Error reading THESAU, code = ' + &E + ~
' [Details: ' + error$(&E) + '].' + ~
chr$(13) + chr$(10) + ~
'The annotation date/time can not be added.';
} else {
/* Check each source reference if it is new or changed
found = 0;
THESAUmaxOcc = repcnt(THESAU_source);
occ = 1;
while (occ <= maxOcc) {
THESAUocc = 1;
while (THESAUocc <= THESAUmaxOcc) and not(found) {
if (THESAU_source[THESAUocc] = br[occ]) and ~
(THESAU_termNo[THESAUocc] = tn[occ]) {
found = 1;
}
THESAUocc = THESAUocc + 1;
}
if not(found) {
ad[occ] = date$(8);
at[occ] = time$(1);
}
found = 0;
occ = occ + 1;
}
}
}
close THESAU;
if (&E) {
errorm 'Error closing THESAU, code = ' + &E + ~
' [Details: ' + error$(&E) + '].';
}
}
}
}
Recompile the adapl when you’re done.
* The adapl inserts the current date and time in tags ad and at respectively, just before each new or existing term record is saved, if certain conditions are met: in a new record, each occurrence of the source field group will receive that date and time, while for existing records an occurrence of that field group will only get that date and time if the relevant occurrence is new - in this case an occurrence is considered new if the contents of an occurrence of the Source field in combination with the contents of the same occurrence of the Number field was not present in the previous version of this record (the place of the occurrence in the order is irrelevant in this matter).
Using the advanced validation functionality in Adlib
Please, see the Adlib User Guide for information about how to use the external source when you're actually validating an entered term while editing a record.