You can now define File data sets for more advanced scenarios more flexibly with an additional set of customization options.

Custom stream processing

To improve customer data security, apply encryption and decryption algorithms to write and read that data by implementing your own Java classes. This way you can also add compression methods, in addition to the basic .gzip and .zip formats.

Manifest file support

Gain more control over source data by listing the files to read in an .xml manifest file, instead of using a fixed filepath, as in the following example:

<manifest>
    <files>  
          <file>  
              <name>file0001.csv</name> 
          </file>  
          <file>  
              <name>file0002.csv</name> 
          </file>
    </files>  
</manifest> 

Meta files created on save

Retrieve detailed information on every file that you save. For every file, the system creates a dedicated meta file which includes the saved file name, size, and the number of records.

Date and time pattern support

To facilitate data management processes, automatically extend the filename of every file with a date and time of creation by adding a Java SimpleDateFormat string to the file path.

Parser configuration on demand

Choose if you want to automatically configure the parser settings for the data in your File data set. This way, even if you update the file path, your parser configuration and the .csv file column mapping do not change.

For more information, see Creating a File data set for files on repositories and Requirements for custom stream processing in File data sets.