NOTE: As of Insight 5.5 you can perform catalog template editing within the Insight Studio client. Luna Recommends using Insight Studio for these actions. The following section is being maintained for reference and lower level editing functions.

Notes on working with Catalog Templates

When making changes to Catalog Templates, it's best to ensure that other users are not importing data using Insight Studio or cataloging data in Inscribe. When you have completed your changes, run "update thumbnail caches" in the Administrator Tools to update the Collection Manager's representation of the schema.

Modifying Catalog Templates (changing field and field group properties)

After a Catalog Template has been created and assigned to a collection, it may no longer be manipulated in Insight Studio. The Administrator Tools can be used to modify existing Catalog Templates and collections.

Understanding the Structure of a Catalog Template

Insight Catalog Templates contains four basic components:

1. Insight Settings: Includes field properties, search and display properties for Insight.
2. Inscribe Settings: validation settings, data grouping, and Inscribe display settings.
3. Cross-Collection Searching Mappings: mappings between fields and the CDWA data Standard.
4. Source Data and Write-back Mappings: mappings between Inscribe/Insight fields and source fields and tables in a traditional relational database.

For a Catalog Template to work, you need settings for at least (1), (2), and (3) for each field. Catalog Templates in Insight Studio simplify the creation of collections and ensure proper configuration for Insight, Inscribe and BrowserInsight.

Modifying Specific Settings within a Catalog Template

Once a collection is built, any modifications to be made to the Catalog Template including data validation settings, Insight display settings, Inscribe settings, adding a field, or removing a field must be completed within the Administrator Tools. Below is a list of each of the features and where in the Administrator Tools to make the appropriate changes.
Table 2: Field Properties in Administrator Tools

• WARNING: Unlike Insight Studio, which has rigid controls to simplify and coordinate configuration of a Catalog Template, the Administrator Tools are designed to be extremely flexible, allowing major changes to be made to Insight components. These changes can possibly cause significant damage to the Catalog Template. Before making any edits, ensure that you have recent backup of the database being modified. This will allow you to recover from any unforeseen errors that may have occurred during the edit.
• It is strongly suggested that you use Insight Studio to do any Catalog Template edits. Only well trained/advised individuals should perform these edits.

Adding a field from a Catalog Template

If you need to add a field to an existing collection after that collection has been published, follow the instructions below:

1. In Administrator Tools, open the Catalog Template for the collection.

1. Unless you are adding the new field to an existing Field Group, you will need to create a new Field Group.

Right-click on the Field Groups node and choose New.

1. Specify a Display Name for the Group. Depending on the option you choose below for Field Group Type, see Choosing Field Group Display Settings on page for an explanation of the options.
2. Choose a Field Group Type.
3. Enter a Display Order.

1. Right-click Fields, add a new field.

1. Specify a Field Name (if this field maps to an external data table, this should be the fieldname in the database).
2. Choose the Field Display Name for the Browser.
3. Specify the Field Group the field should be part of.
4. Specify the Record Type the field should be part of.
5. Specify the other search and display characteristics you would like for the field.
6. Save your changes.

1. To set Data Validation properties, go to the Record Fields section and right click on the data field you just added and choose "Edit."
2. Set any data validation rules or Inscribe display properties necessary for the field. This includes whether the field repeats, is required, etc.
3. Add the field to the field standard:
4. Open the Field Standards node for your Collection Manager.
5. Right-click the local field standard for your collection and choose Add field to standard.
6. Specify field settings for the Insight Java Client.
7. Select the field you added above from the pull-down list.
8. Choose a display name for the field in the Insight Java Client.
9. Specify the other display properties.
10. Choose save.
11. Map the field for cross-collection searching:
12. Right-click the Field Standards node and choose Manage Field Standards.
13. Choose the field standard for your collection from the pull-down on the top left.
14. Select the new field from the list on the left.
15. Select the field(s) in CDWA you would like to map it to in CDWA.

1. Press Save to store your new mappings.
2. Run "Update Thumbnail Caches" or restart your Collection Manager or Personal Insight Manager to finalize the changes. For instructions on updating thumbnail caches, please see Updating a Server's Thumbnail Caches on page .

Removing a Field from a Catalog Template

Though it is possible to remove a field from a Catalog Template, it is strongly recommended that you do not. Removing a field REMOVES ALL DATA associated with that field. You CANNOT GET THAT DATA BACK.

• REMOVING A FIELD WILL CAUSE DATA LOSS

Moving a Field Between Record Types

As Record Types are structural elements in your data, it is strongly suggested you do not move a field between Record Types. Moving a field between Record Types is essentially adding and removing that field, it will REMOVE ALL DATA associated with that field. You CANNOT GET THAT DATA BACK.

Moving a Field Between Field Groups

As field Groups are purely display elements for the data window of the Java Client or BrowserInsight, you can easily move fields between Field Groups.
To move a field between Field Groups, you have two options for making the change in the Administrator Tools:

1. Open the entry for the field in the "Fields" section, and assign it to another Field Group.
2. Open the entry for the Field Group you wish to add the field to in the "Field Groups" section and assign the field to that Field Group.

Modifying the Display of Your Data in the Insight Data Window

Both fields and Field Groups have basic properties which govern their display. These include whether field names are displayed, how the data should be displayed in the data window, whether Field Group names should be displayed, and how repeating values should be handled. Between these properties, data can be displayed quite differently within the data window.

Choosing between the Three Display Types for Data Fields

There are three different display types for data within Insight. These settings are assigned on a per-field basis, allowing you to set different values depending on the type of text in the field.
Non-expandable Short String: A non-expandable short string indents all of the text and wraps it at that indent for each line that follows. (see example below)
Expandable Short String: An expandable short string is similar to the non-expandable, with the exception that after the first line the text is concatenated and end-users can expand the text to see the rest of the value. (see example below)
Long String: A long string does not indent the text after the first line, but instead wraps to the width of the data window. (see example below)

Choosing a Display Type for Field Groups

There are four different display types for Field Groups within Insight. These settings are assigned on a per-Field Group basis, allowing you to mix and match settings as needed.
Type 1: Fields are ungrouped – fields are separated by a horizontal line from each other, Field Names are displayed but the Field Group name is not.
Type 2: Fields are grouped – Field Groups are separated by a horizontal line from each other, Field Names are displayed but the Field Group name is not.
Type 3: Fields are grouped – Field Groups are separated by a horizontal line from each other, the Field Group name is displayed but the Field Names are not.
Type 4: Fields are grouped – Field Groups are separated by a horizontal line from each other, both the Field Group name and the Field Names are displayed.

Display Field Group Names

Displays the Field Group Name, followed by the individual field values (without Field Names), one field per line.
Example: Artist Info:
Pablo Picasso1954Spanish

Display Group & Field Names

Displays the Field Group Name, followed by the individual Named Fields, one field per line.
Example: Artist Info:
Artist:Pablo Picasso Artist Date:1954Artist Nationality:Spanish

Display Field Names

Displays each individual Field Name and groups the fields together.
Example: Artist:Pablo PicassoArtist Date:1954Artist NationalitySpanish

Single – Use Delimiter

Used for a single field that may have multiple repeating values.
NOTE: If more than one field is selected, the delimiter cannot be used. Insight can only use the delimiter type you defined in the field.
Example: Artist: Pablo Picasso; Georges Braque

Catalog Templates and External Databases

A custom Catalog Template is required if Insight's internal tables are to be coordinated with a traditional database structure. There are two main reasons for creating a custom Catalog Template:

1. The template will maintain mappings back to real fields and tables (and Inscribe will attempt to write to these).

1. Catalog Templates which support the Insight Indexer often maintain a different structure than other Catalog Templates.

NOTE: It is STRONGLY suggested that you install a special Collection Manager just for the collection you will use with this source data.

Creating a Catalog Template that Supports the Insight Indexer (but does not write-back to the existing source tables)

Catalog Templates that support indexing content from an existing database, but don't need to write-back changes (i.e. from Inscribe) rely on the existing database & table structure to maintain the structure of the data.
To create a template:

1. Create an object-level Field Group in Insight Studio.

1. Create a field for each field which you wish to import (regardless of what table it comes from).

1. Customize the field settings as needed including, cross-collection searching settings, search & display settings, and Field Type.

1. Publish the template and complete your collection.

1. Open the Administrator Tools and connect to the collection you just created.

1. Define tables for each of the data tables you will be importing from.

1. Map each of the fields to the appropriate table.
   1. For each field, you will need to assign the table from which it originates

1. Designate a Grouping Table and Grouping Field Name.

NOTE: Grouping Table and Grouping Field Name maintain the row position of grouped values. For example: If we consider a table defining people we may have a Name, Date and Nationality. The table that contains this information also has a numeric row identifier. This row identifier is what we are referring to. This way Insight knows to display the values together.

1. Map all of the joins between the tables. Joins define how data is related from one table to another based on the join field name. Edit Joins only applies to a collection that has more than one table. For collections that use a single table, this portion can be ignored. Follow the steps below to create a join:
   1. Right click Edit Joins in the navigation tree.
   2. In the Edit Joins dialog panel, right click Add Join on a table name. You can only edit a join if there are tables already joined together.
   3. Select a Start table name from the drop down list and enter the Start field name of the selected table.
   4. Select the End table name in the drop down list. This is the final table that completes the join function.
   5. Enter the Start and End table field names which link the two tables together.
   6. Click Save to update your joins.

NOTE: Insight is capable of gathering data from many complex data sources. If you need more assistance in connecting insight to an existing data source, please contact our technical support department: LunaSupport@lunaimaging.com.

Creating a Catalog Template that Supports the Insight Indexer and Supports Write-Back to the Existing Source Tables

Catalog Templates that support indexing content from an existing database, but require the functionality of writing those changes (i.e. from Inscribe or Insight Studio's data import module) back to the external source tables, must represent the structure of the data both in the source table structure as well as within Inscribe.
To create a Catalog Template that supports write-back to the existing source tables:

1. Create an object-level Field Group in Insight Studio.

1. Create a Field Group for each table represented in your source-tables.

1. Create a field for each field which you wish to import. Specify the display name for the field (the actual database field name will be specified within the Administrator Tools). Customize the Insight, Inscribe, Cross-Collection Searching and other settings.

1. Add the field to the appropriate Field Group.

1. Specify the Field Group settings.

1. Publish the template and complete your collection.

1. Open the Administrator Tools and connect to the collection you just created.

1. Map each of the fields to the appropriate table.

1. Map all of the joins.
   1. Right-click the Joins folder of the Administrator Tools to access the Manage record Joins panel. Joins define the relationships between Record Types through intermediate mapping tables. Creating join definitions for Inscribe is similar to the method used in the Manage Field Joins, except now you can have multiple root nodes. The example below shows each of the Record Type tables are mapped through an intermediate mapping table that supports the many-to-many relationship that exists between the two record types. In the case of the Object Records relation to People (Artist record type), the relationships between the two independent record types are defined in the ObjectToPeopleMap table. The intermediate mapping tables are two column tables containing the primary keys from each of the related record types. In this case, the ObjectToPeopleMap table contains ObjectID and PeopleID. When defining an Inscribe Join, start from the preferred record type and work outwards to the related record type. These definitions are key in the initial indexing and update source relational data processes used by inscribe to manage data integrity.

NOTE: Insight is capable of gathering data from many complex data sources. If you need more assistance in connecting insight to an existing data source, please contact our technical support department: LunaSupport@lunaimaging.com

Using Insight's Indexer

Insight's indexer is used for the following purposes:

1. to create multi-page documents from document templates
2. to manually regenerate fuzzy date indexes
3. to update thumbnail caches (soft restart)
4. to import data from external data tables

Running the Indexer for Fuzzy Date or Multi-page Document Indexing

If you are just running fuzzy date or Multi-Page Document indexing:

1. Right-click the Invoke Indexer node for your collection (under Collection Settings | Collection Editor) and choose run.

NOTE: Not all users have rights to access the Collection Editor node in the Administrator Tools. You will not see this option if you do not have the appropriate rights.

1. Make sure Value Indexing, Term Indexing and Browser Indexing are NOT selected.
2. Select Fuzzy Date Indexing and/or Multi-Page Document Indexing as needed.
3. Press Start Indexing.
4. A progress panel will report indexing status in real time. Upon completion, press Close.

Updating a Server's Thumbnail Caches

The Insight Collection Server can be "soft restarted" i.e. told to refresh all of it's configurations from the database via special type of indexing.
To tell a collection server to update thumbnail caches:

1. Right-click the Invoke Indexer node for your collection (under Collection Settings | Collection Editor) and choose run.

NOTE: Not all users have rights to access the Collection Editor node in the Administrator Tools. You will not see this option if you do not have the appropriate rights.

1. Make sure Value Indexing, Term Indexing and Browser Indexing are NOT selected.
2. Select Update Thumbnail Caches and specify the Hostname and Port your Collection Manager is using.
3. Press Start Indexing.
4. A progress panel will report indexing status in real time. Upon completion, press Close.

Running the Indexer to Import Data from External Data Tables

If you are using Insight to publish a view of external source data tables, the indexer will need to run each time those data tables change (in order to import the changes into Insight).
NOTE:It is best to run the indexer at off hours as running it will consume both processor and memory.
To run the indexer:

1. Right-click Invoke Indexer under Collection Settings | Collection Editor for your collection, and choose run.

NOTE: Not all users have rights to access the Collection Editor node of the Administrator Tools. You will not see this option if you do not have the appropriate rights.

1. Make sure Value Indexing, Term Indexing, and Browser Indexing are all selected. You may also choose Fuzzy Date Indexing, and Multi-page Document Indexing, if applicable.
2. Choose a data loading method. The indexer uses SQL transactions to post to the database. For complex databases, these transactions can be very time consuming. You can speed up the process on Oracle or Microsoft SQL Server by having the indexer create a set of temporary files which are bulk-loaded into the database during the indexing process. This method is suggested for databases with over 20,000 records, and will significantly increase the speed of the indexing process.

1. Press Start Indexing.
2. A progress panel will report indexing status in real time.
3. If you chose to create bulk load files, the indexer will prompt you when it's ready for you to load the files into the database:

Oracle
Once you are prompted, go to the directory that you specified for these files. You will notice some new files followed by .cmd, .ctl, .txt. You will need to make sure that your environment is set up for the load. Ensure you are using the same character set for the load that your database is using i.e. NLS_LANG=American_America.WE8ISO8859P1
The *.cmd file is an executable file that will run the oracle sqlldr utility. You will need to edit this file to enter the correct username and password for your schema.
The *.ctl files are the Oracle control files used for the bulk load.
The *.txt files contain the data to be loaded.
Execute the appropriate .cmd file. Once the operation completes, return to Administrator Tools and continue the indexing process. If you are prompted again, repeat this action for the second set of data.
MSSQL
Using SQL Server Enterprise Manager, perform the following for each table referenced in the prompt:

1. In Enterprise Manger, right-click on the Tables node for the database you are working on. Go to All Tasks and select Import Data.

1. Follow the wizard through the next steps:
   1. Click Next past the first screen.
   2. In data source, choose text file. Navigate to the Idx_Values.txt file and press Next.
   3. In the Select File Format window, change the row delimiter to {]|#} and select First row has column names. Press Next.
   4. On Specify Column Delimiter, make sure Tab is selected and press Next.
   5. Choose Destination database and press Next.
   6. On Select Source Tables and Views, change the destination table to be the ITValues table. Press Next Verify this by clicking on the transform button. You should see Append rows to destination table is selected and create destination table is not selected.

1. 1. On the next wizard panel, you can save this function as a DTS package for future use. Press Next.
   2. Confirm the process and press Finish.

Repeat these steps for each requested load. When selecting the destination table, replace Idx_ with IT. For example: Idx_Values = ITValues.
Once you have finished the Bulk Load, return to the indexer and continue the process.

Trouble Shooting the Indexer

When attempting to run the indexer, there may be errors that can cause the indexer to fail. This section will describe some of the common errors and will also provide solutions to those errors.
An error you may encounter could cause the indexer to stop in the middle of indexing. In this case, you need to go into the Edit Collection Manager dialog window and turn on (check) the Log Batch Commands check box and then rerun the indexer.
Figure 28 - Enabling Log Batch Commands

When the Log Batch Commands is enabled, the Administrator Tools begins to log queries to the databaseconnector.txt file. You can find this file in the InsightAdministration folder. You can navigate to that folder in your directory and view the databaseconnector.txt file.
Another common error is not having the Primary Object Table and Primary Object Key defined in the Tables Folder.

1. When establishing which table(s) the indexed fields will originate from, the main table must be defined. Then, the indexer will know where to relate the indexed data and on what field to base the relationship.

Figure 29 - Table Where Indexed Fields Originate

For example, if Objects is the only table in a flat database structure, it will be considered the main data table from where the data is derived. Therefore, the Primary Objects Table field must be selected and the Primary Object Key must be set to the field that is chosen to uniquely identify each record. However, if the database is in a relational structure, then the table that is considered to be the main table must be selected as the Primary Object Table, and the Primary Object Key must be the field that is used to relate to these tables.

1. A typical problem that is often encountered is not having the fields in the Grouping Table and Grouping Field Name populated in the Fields table. The Grouping Table tells the indexer which table the row information is being drawn from, and the Group Field Name specifies which field is being used to uniquely identify that row of information.

Figure 30 –Identifying Group Field Name

For example, if the Field Name is Author and the data is being grouped from the Objects table (the main table in a flat database structure), then the Grouping Table would be populated with the table name Objects, and the Grouping Field Name would be populated with the field name ObjectID, because it is the unique value that is being used to identify each row of information in the Objects table.
However, if this was a relational database and the information was coming from two tables, this would be different. The Grouping Table name would be the name of the table in which the row information would be coming from, and the Grouping Field Name would come from the unique identifier that identifies that row of information.
For example, you have an Artist table and a Nationality table. The Artist table contains all of the data that describes the Artist and the Nationality table contains the various types of nationalities that relate to Artists. So, the Artists would be the Grouping Table, because Artist information is what we are looking for; and ArtistID would be the Grouping Field Name, because this is what relates the artist to a nationality.

To transfer indexed data from the IT Tables to the DT Tables manually

On occasion, the insight Indexer will fail when it attempts to transfer the data from the IT(Insight Temporary) to the DT (Destination Tables) tables. This failure is generally caused by a database timeout. Below we have provided the queries used to transfer this data to run manually.

• WARNING: To complete the indexing process you will first need to identify your UniqueCollectionID. You will find this in your DatabaseConnector.txt file located in the Administrator Tools root directory. Starting from the end of the file, search for "UniqueCollectionID". Once Located, populate the following script with the appropriate value. Next you will need to execute each line of the script separately. Once you have completed this the live data should be ready to use once you restart your Collection Manager or run an update thumbnail cache. If you have any questions please contact the Luna Imaging support department at LunaSupport@lunaimaging.com.

1) Open a database script editor and connect to your Collection Manager database.
Microsoft SQL

• SQL Query Analyzer or isql
  Oracle
• SQLPlus or SQLPlus Worksheet
  MySQL
• MySQL Query Browser
  2) Run the following SQL Script, replacing <UCID> with your Unique Collection CID.
  DELETE FROM DTTermToEntityMap WHERE UniqueCollectionID = <UCID>
  DELETE FROM DTTermObjectMap WHERE UniqueCollectionID = <UCID>
  DELETE FROM DTTerms WHERE TermID NOT IN (SELECT TermID FROM DTTermObjectMap)
  DELETE FROM DTValueToEntityMap WHERE UniqueCollectionID = <UCID>
  DELETE FROM DTValueToObject WHERE UniqueCollectionID = <UCID>
  DELETE FROM DTValues WHERE UniqueCollectionID = <UCID>
  DELETE FROM DTEntityToEntityMap WHERE UniqueCollectionID = <UCID>
  DELETE FROM ISCollectionEntityMap WHERE UniqueCollectionID = <UCID>
  DELETE FROM IDTempOld
  DELETE FROM IDTempNew
  INSERT INTO DTTermToEntityMap SELECT Distinct * FROM ITTermToEntityMap
  INSERT INTO DTTermObjectMap SELECT Distinct * FROM ITTermObjectMap
  INSERT INTO DTTerms SELECT * FROM ITTerms
  INSERT INTO DTValueToEntityMap SELECT Distinct * FROM ITValueToEntityMap
  INSERT INTO DTValueToObject SELECT Distinct * FROM ITValueToObject
  INSERT INTO DTValues SELECT * FROM ITValues
  INSERT INTO DTEntityToEntityMap SELECT Distinct * FROM ITEntityToEntityMap
  INSERT INTO ISCollectionEntityMap (UniqueCollectionID, EntityTypeID, EntityID) SELECT Distinct UniqueCollectionID, EntityTypeID, EntityID FROM DTValueToEntityMap WHERE UniqueCollectionID = <UCID>