This tutorial describes how to set up a large data page for an Android or an iOS offline-enabled custom mobile app. Using large data pages to store reference data in your custom mobile app improves the performance of the custom mobile app. Only individual records of the large data page that have actually changed are synced, not the entire content. Furthermore, for efficient memory management, only records needed to render a specific screen on the mobile device are loaded to memory, leaving the remaining parts of reference data in the device's permanent store.

• Requirements
• Define a data page
• Declare a data page as large
• Create custom JavaScript function
• Add JavaScript code as a text file
• Attach text file to Pega Platform application
• Best practices

Requirements

Before you start this tutorial, review the following prerequisites and perform the following tasks:

• You should have experience creating and using list-structured data pages in the Pega^® Platform. For more information, see Understanding data pages.
• Create a simple Pega Platform offline-enabled custom mobile app that uses a list-structured data page that is linked to a user interface control such as a drop-down list.
• Familiarize yourself with the JavaScript language. You should have experience creating JavaScript code.
• Read Custom populator JavaScript functions for Large Data Pages, which describes the populator JavaScript functions to use for large data pages.
• You should be familiar with SQL, and you should be able to use the SQLite syntax to make queries.

In this tutorial, the large data page is sourced from a report definition, but it can also be sourced from a connector or an activity. You define a large data page in the Pega Platform just like you would a regular data page. However, after the data page is created, you have to mark it as a large data page in the Pega Platform.

Defining a data page

You first need to create a data page to hold a large amount of data. After it has been created, you must configure the large data page for use in an offline-enabled custom mobile app. Keep in mind that you must:

• Add the pyModificationDateTime column to the result (for example, a report definition). In Pega 7.2.1, you must create the ModificationDateTime property manually, but in Pega 7.2.2 it is automatically defined in the data page class.
  This column is expected to return the time of previous synchronization of the large data page to the client. The data source uses it to return only the records that were modified (added, updated, or removed) after this date. The values in this column are expressed as a DateTime format (yyyyMMdd'T'hhmmss.SSS z) starting from midnight, January 1, 1970 UTC (19700101T000000.000 GMT). When a date of midnight, January 1, 1970 UTC or earlier is provided, the data source returns all records. When a future date or no date is provided, for example, the pyModificationDateTime parameter is empty or null, the data source returns no records. For more information, see pyModificationDateTime optimization.
• Add the pyIsRecordDeleted Boolean column to the result (for example, a report definition), because this is automatically defined in the data page class.
  This is expected to return true only for records that have been removed. Correct values for all key columns should be returned for such removed records, as well. The data source should always maintain removed records tracking in order to return appropriate values. Manual removal of rows is still possible; however, it always has to be accompanied by forcing a full sync for all the access groups that use the data source in offline-enabled apps.

In the following example, a large data page is created named D_Movie that holds movie information in the following columns: title, release date, whether it is a sequel, description, modification date/time, and short description. It is sourced from a report definition.

1. Create a data type called Movie that includes several fields: Title, ReleaseDate, IsSequel, Description, ModificationDateTime, and MovieShortDescription.
   
2. Create a data page called D_Movie.
3. In the Data page definition section, from the Structure list, select List.
4. In the Object type field, select or enter the name of the new data type that was created in step 1: Cars-Dinoco-Data-Movie.
   
5. In the Data sources section, select Report Definition from the Source list.
6. In the Name field, enter a name for the report definition for the data page: MovieRD.
   
7. Click the Open icon to the right of the Name field.
8. Make sure that the following columns are defined in the Edit columns section. Note that you must manually add the .pyIsRecordDeleted column to the list:
   
9. Click the Parameters tab and add a new parameter called pyModificationDateTime of Date Time type.
   
10. Click the Query tab and define a filter condition that uses the pyModificationDateTime column.
    
11. Click Save.

You have now successfully created a list-structured data page in the Pega Platform that is sourced from a report definition.

Declaring a data page as large

After you create a data page that holds large data records in your custom mobile app, you must declare it as a large data page. Edit the pyDataPageWhiteListForOffline rule for each large data page to be used in the Pega Platform by inserting a new line in the following format:
D_myDatapageName;large;[["index1_column1","index1_column2","index1_column3"],["index2_column1","index1_column2","index2_column3 COLLATE NOCASE"]]

The list of column indexes for the client store database is optional; however, for performance reasons, using them is strongly advised. Notice that each column index name is always specified in double quotation marks. Remember that the column index names should match the SQL queries that are performed on a specific large data page. Adding COLLATE NOCASE after an indexed column makes queries with LIKE statements be faster and case insensitive. See the section entitled Index optimization in Best Practices below for more details.

1. From Designer Studio, click Mobile > Offline.
2. Click Modify whitelist to edit the pyDataPageWhiteListForOffline rule.
3. In the HTML source field, declare the data page called D_Movie as large.
   
4. Click Save.

The data page that was previously created is now marked as a large page in the Pega Platform.

Creating custom JavaScript function

After the report data page is declared as a large data page in Pega Platform, you need to define a custom populator for the JavaScript function that performs some operation on the large data page by using query statements, such as:

• Filtering its contents
• Joining data from two separate large data pages
• Preprocessing or postprocessing the result

For the large data page that was created in the previous sections in this tutorial, you now need to create a JavaScript custom populator function called moviesCustomFunction(). The target data page for our example is D_Movie. The query operation selects the Title, Release date, Description, and Is sequel columns from this existing large data page. No query parameters are used because no parameters need to be passed to the SQL WHERE clause.

The following JavaScript code is needed:

var moviesCustomFuction = function(parametersMap, clientStore, onSuccess, onFailure) {
  var targetDatapageName = "D_Movie";
  var queryParameters = [];

  var query = "SELECT Title, ReleaseDate, Description, IsSequel FROM D_Movie";
  clientStore.runQuery(query, queryParameters, targetDatapageName, onSuccess, onFailure);
}
pega.ui.ClientCache.registerLargeDatapage("D_Movie", moviesCustomFuction);

Adding JavaScript code as a text file

To use the new custom populator function called moviesCustomFunction() in your Pega Platform application, insert the JavaScript code that we created as a text file.

1. From the App Explorer, right-click the name of your case and click Create > Technical > Text File.
2. In the Label field, enter movies_datapage_api as the name for the new text file.
3. In the App Name (Directory) field, enter the name for the application: webwb.
4. In the File Type (extension) field, enter: js.
   
5. Create and open the new text file, and then paste your JavaScript code into the File source field.
   
6. Click Save.

Your JavaScript code is now saved as a text file in the Pega Platform application for later use.

Attaching a text file to the Pega Platform application

Now include the JavaScript code text file called movies_datapage_api in the pypega_ui_userscripts_offline Static Content Bundle rule. This action ensures that the JavaScript code that you created is accessible within the Pega Platform offline-enabled custom mobile application.

1. In Designer Studio, search for the pypega_ui_userscripts_offline rule.
2. Edit the pypega_ui_userscripts_offline UI-Kit rule for the current application.
3. In the App name field, enter: webwb.
4. In the File extension field, enter: js.
5. In the File name field, enter: movies_datapage_api.
6. Click the arrow next to the Private edit button and click Check out to branch to make sure that the pypega_ui_userscripts_offline rule is checked out to the branch for the current application.
   
7. Click Save.

Now that the custom populator JavaScript function is available in the Pega Platform, you can improve performance by rendering large data page records in your offline-enabled custom mobile app.

Best practices

To use large data records in your custom mobile app, keep in mind the following information:

• The large data page that you define in the Pega Platform must always contain at least one record.
• All refresh strategies on the Load Management tab of the Data Page rule form are supported. The only exception is the Do Not Reload When refresh strategy when it is used with the pyModificationDateTimetime stamp.
• It is recommended for the large data page to have node scope in its data page definition. No use case exists for a large data page with a thread scope.
• On the Android platform, do not define your property as softflagINTERNAL. Under the iOS platform, do not define your property as softflag.INTERNAL@.
• If present, pzInsKey is always used as the large data page key on the client database. Otherwise, the keys defined on the data page class are used.
• For performance reasons, do not reuse the same data pages if they are expected to hold different data. This does not mean that you must always define a separate data page for each user interface element. However, define separate data pages more often than in Pega Platform desktop applications.
• If multiple elements of the user interface exist on the same screen and interact with the same data source, but use different query parameters, they should all have separate target data pages defined. An example is two lists of elements filtered by two separate categories are those that can be fixed or selected individually.
  The reason for separate target data pages is that if the multiple user interface elements all use the same target data page, it is not be possible to cache the data properly; only the data for the last control that accessed the data page is cached (the last set of parameters used), while the data for others is overwritten. Then the next time the page needs to be rendered on refresh, all of the data is fetched from the database again. Target data pages are designed to be lightweight. Therefore, for this particular use case, a single-source data page with dedicated target data pages is the preferred solution: one for each of these user interface elements. For technical reasons, each target data page must contain at least one record (it can contain dummy content).
• Alternatively, if you source a data page by using the same parameters for multiple user interface elements, they should reuse the target data page. An example is for each element list to have a couple of columns displaying different properties of the elements.

Index optimization

When you define indexes for the TEXT column of a large data page in the pyDataPageWhiteListForOffline rule, you can optionally add one of the collating sequences (namely: COLLATE NOCASE, COLLATE BINARY, COLLATE RTRIM). Collating sequences optimize the indexes by making the LIKE statements in a query faster. The following example shows how to define a collating sequence for an index column of the data page in this rule:

D_myDatapage;large;[["firstName"], ["firstName COLLATE NOCASE"]]

When you add COLLATE NOCASE after an indexed column, the query with the LIKE statement is faster and case insensitive. If you want to use SQL queries with the equal operator, you need to add another index without any addition. The following example shows two ways of creating indexes:
CREATE INDEX myIndex ON D_myDatapage (firstName); // This makes equal operator queries fast and case sensitive
CREATE INDEX myIndex ON D_myDatapage (firstName COLLATE NOCASE); // This makes LIKE queries fast and case insensitive

Based on your need, you can create one index or both indexes to speed up queries. This takes into account that the add operation takes more time because of the cost of creating the second index. For more details, see LIKE optimization.

pyModificationDateTime optimization

The main use of the pyModificationDateTime parameter by the synchronization process is to limit the time that is needed to update the data stored on a user's device. The time spent fetching the source data should depend on the amount of data that is returned rather than the size of the entire source database. Therefore, you should strive to make filtering by the pyModificationDateTime parameter more efficient. In particular, if your source is a database table and you decide to implement filtering by using a column that keeps the modification, creation, and deletion time of each record, do not forget to create an index for it. You implement the pyModificationDateTime parameter filtering by using a column with the same name as the parameter. This column keeps the modification, creation, and deletion time of each record.