Skip to main content

         This documentation site is for previous versions. Visit our new documentation site for current releases.      

This content has been archived and is no longer being updated.

Links may not function; however, this content may be relevant to outdated versions of the product.

Creating and sourcing large data pages to store large reference data in offline-enabled applications for Pega 7.4

Updated on June 24, 2020

This tutorial describes how to set up a large data page for offline-enabled applications. Using large data pages to store reference data in your custom mobile app improves the performance of the custom mobile app. Only individual records of the large data page that have actually changed are synchronized, not the entire content. Furthermore, for efficient memory management, only records that are needed to render a specific screen on the mobile device are loaded to memory; the remaining parts of the reference data stay in the device's permanent store.

Large data pages can be automatically sourced from report definitions. You define them in Pega® Platform just like you would define any other data page. After the data page is created, you mark it as a large data page in Pega Platform. Large data pages can also be sourced from a connector, activity, or data transform, but you must create a custom JavaScript populator function as described in the following steps. If you are setting up a large data page that uses any of the unsupported functionality listed in Additional considerations, you must also create a custom JavaScript populator function.

To create and source large data pages, complete the following tasks:

This tutorial takes approximately 1 hour to complete.


Before you start this tutorial, review the following prerequisites and perform the following tasks:

  • Make sure that you have experience creating and using list-structured data pages in Pega Platform. For more information, see Understanding data pages.
  • Create a simple Pega Platform offline-enabled custom mobile app that uses a list-structured data page that links to a user interface control such as a drop-down list.
  • Make sure that the datapages/addReportDefinitionToDatapage Dynamic System Setting that is owned by the Pega-Engine ruleset is set to true or is not set. Otherwise, you must create a custom JavaScript function.

If you plan to use a custom populator, review the following prerequisites:

Creating a data page automatically

First, create a data page to hold a large amount of data. After you create the data page, you must configure the large data page for use in an offline-enabled custom mobile app. In the following example, a large data page, named D_Movie, contains columns that provide the following movie information: title, release date, whether it is a sequel, description, modification date and time, and short description. The data page is sourced from a report definition.

  1. In your report definition, add the pyModificationDateTime column to the results. The column is already predefined in the base class (@baseclass), which is the topmost class in the class hierarchy. All other classes are derived from the base class.
    The pyModificationDateTime column returns the time when the record was last modified in the data source. The data source must update the value of the pyModificationDateTime column at each add, update, or delete operation. The values in this column are expressed in a DateTime format (yyyyMMdd'T'hhmmss.SSS z) starting from midnight, January 1, 1970 UTC (19700101T000000.000 GMT).
  2. Add the pyIsRecordDeleted column to the result (for example, a report definition). This column is already predefined in the base class (@baseclass).
    The pyIsRecordDeleted column returns true only for records that were removed. The data source should always maintain removed records tracking in order to return appropriate values. You can remove rows manually; however, you must force synchronization of all the access groups that use the data source in offline-enabled apps.
  3. Create a data type called Movie that includes the Title, ReleaseDate, IsSequel, Description, and MovieShortDescription fields.
    Movie data model
  4. Optional: If the source of the data type is local data storage, in the Records tab click Configure source to create a dedicated table for the class, and add the pyModificationDateTime and pyIsRecordDeleted properties to the list. When you do this, the pyModificationDateTime and pyIsRecordDeleted properties get optimized.
    Movie data type
  5. Create a data page called D_Movie.
  6. In the Data page definition section, select List from the Structure list.
  7. In the Object type field, select or enter the name of the new data type that you created in step 1, for example, Cars-Dinoco-Data-Movie.
  8. In the Scope field, select Node.
    Edit data page - Movie
  9. In the Data sources section, select Report Definition from the Source list.
  10. In the Name field, enter a name for the report definition for the data page, for example: MovieRD.
    Data Sources section
  11. Click the Open icon to the right of the Name field to configure the report definition:
    1. In the Edit columns section, make sure that the following columns are present. You must add the .pyIsRecordDeleted column and the .pyModificationDateTime column manually.
      Edit report definition page
    2. Click the Parameters tab and add a new DateTime type parameter called pyModificationDateTime.
      Parameters section
    3. Return to the Query tab and define a filter condition that uses the pyModificationDateTime column. Define at least one other condition and additional filters, if necessary.
      During the initial synchronization, the clients receive all the data after it is filtered by the data source on the server side, regardless of the value in the pyModificationDateTime column. Subsequent synchronizations return only rows that are newer than the value of the pyModificationDateTime parameter. You must not modify the pyModificationDateTime > param.pyModificationDateTime filter condition that is required by a large data page; therefore the use of Use Null if Empty or default values for the pyModificationDateTime parameter is not supported. For more information, see pyModificationDateTime parameter optimization.
      All parameter-based filter conditions are ignored on the server. On the client, the pyModificationDateTime > param.pyModificationDateTime filter condition is ignored, while all other filter conditions are applied (either parameter-based or hardcoded by means of providing a string value).

      Edit filters section
    4. Optional: To bind your data with data from another class, click the Data Access tab and enter a data page name in the Prefix field to declare a class join.
      You can add only one join for a specific data page. Other options on the Data Access tab are ignored on the client.
      Data Access tab
    5. Optional: Click Edit conditions to configure a filter to apply to the join.
      Class join filter
  12. Click Save.

Declaring a data page as large

After you create a data page that holds large data records in your custom mobile app, declare it as a large data page.

In the pyDataPageWhiteListForOffline rule, you must declare both the target large page, as well as any source large data pages, if applicable. You must create a separate entry in this rule for each large data page. Every large data page must always contain at least one record.
  1. In Designer Studio, click Mobile > Offline.
  2. Click the first Modify whitelist link to edit the pyDataPageWhiteListForOffline rule.
  3. In the HTML source field, declare the D_Movie data page as large.
    Data page whitelist
  4. Optional: To improve system performance, for each large data page that is used in Pega Platform, add a list of column indexes after the declaration made in the HTML source field, as shown in the following example:
    D_Movie;large;[["index1_column1","index1_column2","index1_column3"],["index2_column1","index1_column2","index2_column3 COLLATE NOCASE"]]
    Individual column index names must be specified within double quotation marks and must match the SQL queries that are performed on a specific large data page. Adding COLLATE NOCASE after an indexed column name makes queries with LIKE statements faster and case insensitive. For more information, see the Index optimization.
  5. Click Save.

Optional: Creating a data page based on a custom JavaScript populator function

If your implementation requires functionality that is unsupported (see the Additional considerations section), or requires that a large data page be sourced from a connector, activity, or data transform, you must create a data page. You declare this data page as large and define a custom populator for the JavaScript function that performs an operation on the large data page by using query statements. For details about populator JavaScript functions and examples for the above use cases, see Custom populator JavaScript functions for large data pages.

A custom populator takes precedence over the automated process.

Initially, you need to create a large data page and then create a JavaScript custom populator function called moviesCustomFunction(). The target data page for our example is D_Movie. The query operation selects the Title, Release date, Description, and Is sequel columns from this existing large data page. No query parameters are used because there is no need to pass any parameters to the SQL WHERE clause. To use the new custom populator function called moviesCustomFunction() in your Pega Platform application, you must save the JavaScript code as a text file. Then you include the text file called movies_datapage_api in the pypega_ui_userscripts_offline Static Content Bundle rule. This action ensures that the JavaScript code that you created is accessible from the Pega Platform offline-enabled custom mobile application.

  1. Follow the Create a data page procedure. Skip steps 8 through 10.
  2. Follow the Declare a data page as large procedure.
  3. Add a JavaScript code as a text file:
    1. In the App Explorer, right-click the name of your case and click Create > Technical > Text File.
    2. In the Label field, enter movies_datapage_api as the name for the new text file.
    3. In the App Name (Directory) field, enter webwb.
    4. In the File Type (extension) field, enter js.
      Create Text File page
    5. Create and open the new text file.
    6. In the File source field, paste your JavaScript code, similar to the following snippet:

      var moviesCustomFunction = function(parametersMap, clientStore, onSuccess, onFailure) {
      var targetDatapageName = "D_Movie";
      var queryParameters = [];

      var query = "SELECT Title, ReleaseDate, Description, IsSequel FROM D_Movie";
      clientStore.runQuery(query, queryParameters, targetDatapageName, onSuccess, onFailure);
      pega.ui.ClientCache.registerLargeDatapage("D_Movie", moviesCustomFuction);

    7. Click Save.
      Edit Text File
  4. Attach the text file to the Pega Platform application:
    1. In Designer Studio, search for pypega_ui_userscripts_offline.
    2. Click pypega_ui_userscripts_offline to edit the UI-Kit rule.
    3. In the App name field, enter webwb.
    4. In the File extension field, enter js.
    5. In the File name field, enter movies_datapage_api.
    6. Click the arrow next to the Private edit button and click Check out to branch to make sure that the pypega_ui_userscripts_offline rule is checked out to the branch for the current application.
      Editing a Static Content Bundle rule
  5. Click Save.

Additional considerations

To use large data records in your custom mobile app, keep in mind the following information:

  • Always add at least one record when you define a large data page in Pega Platform (it can contain placeholder content). All key columns must always be returned.
  • For performance reasons, do not reuse the same data pages if they are expected to hold different data. This does not mean that you must always define a separate data page for each user interface element. It is a good practice to define separate data pages more often than you would for Pega Platform desktop applications.
  • On the Android platform, do not define your property as softflagINTERNAL. Under the iOS platform, do not define your property as [email protected].
  • Do not use complex type parameters (objects, pages, and so on that have their properties) as large data page parameters.
  • Do not set large data page parameters as required.
  • Do not use the Do Not Reload When refresh strategy when it is used with the pyModificationDateTime time stamp. However, the other refresh strategies on the Load Management tab of the Data Page rule form are supported.
  • Do not use target data pages as source data pages for other target data pages.
  • You can use the following functionality in your report definition, but these items are ignored on the client:
    • Select values button when defining your filter logic
    • Aliases (columns are accessible by a name defined in the report definition's Column source field)
    • Calculations
    • Default parameter values
    • Use Null if Empty strategy
  • If you create on one screen multiple elements of the user interface that interact with the same data source, but use different query parameters, make sure that they all have separate target data pages defined. An example is two lists of elements filtered by two separate categories are those that can be fixed or selected individually.
  • If you source a data page by using the same parameters for multiple user interface elements, make sure that they reuse the target data page. An example is for each element list to have a couple of columns displaying different properties of the elements.
  • If you use SQL queries, keep in mind that the function that generates SQL queries is cached on the client, the generated SQL queries themselves are not cached. Also, the WHERE clause of an SQL query might differ depending on the number of provided parameters. If a parameter is not provided, a part of the filtering logic that uses this parameter is ignored.

Index optimization

When declaring a data page as large, you can define indexes for the TEXT column of a large data page in the pyDataPageWhiteListForOffline rule and add one of the collating sequences, for example, COLLATE NOCASE, COLLATE BINARY, or COLLATE RTRIM. Collating sequences optimizes the indexes by making the LIKE statements in a query faster. The following example shows how to define a collating sequence for an index column of the data page in this rule:

D_myDatapage;large;[["firstName"], ["firstName COLLATE NOCASE"]]

When you add COLLATE NOCASE after an indexed column, the query with the LIKE statement is faster and case insensitive. To use SQL queries with the equal operator, you need to add another index. The following example shows two ways of creating indexes:
CREATE INDEX myIndex ON D_myDatapage (firstName); // This makes equal operator queries fast and case sensitive
CREATE INDEX myIndex ON D_myDatapage (firstName COLLATE NOCASE); // This makes LIKE queries fast and case insensitive

Based on your requirements, you can create one or both indexes to speed up queries. Keep in mind that the add operation takes more time because of the cost of creating the second index. For more details, see LIKE optimization.

Consider the following information when you use the source properties in a report definition in filters:

  • You must create indexes manually in list of offline-supported data pages.
  • The filtering relationship and the filter order determine whether a specific index works or not.
  • If you do not provide a report definition parameter that is used in a filter and no default or null value is set, the filter is ignored.

pyModificationDateTime parameter optimization

The main use of the pyModificationDateTime parameter by the synchronization process is to limit the time that is needed to update the data stored on a user's device. The time spent fetching the source data should depend on the amount of data that is returned rather than the size of the entire source database.

Consider the following information about the pyModificationDateTime parameter:

  • If your source is a database table, create an index for the pyModificationDateTime column.
  • The Filter conditions to apply field of filters that you edit in the Query tab must contain the pyModificationDateTime filter and another condition that contains any supported filter logic.
  • When editing the conditions of a join, you cannot use the pyModificationDateTime parameter as a filter condition or source column.

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best. is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us