File Data Set file path pattern
To process data in parallel by using multiple threads, the File Data Set operates on
a collection of files instead of a single file. The file path pattern includes tokens that
match existing files to the pattern or generate new files. When you specify the file path pattern for a File Data Set, you can use the following
file path pattern tokens: All files that the system generates in a file repository by using a File Data Set,
include a unique ID in the file name. This ID is a string that guarantees the
uniqueness of the file names. For example, if the file name pattern for a File Data
Set is If the file name pattern includes a wildcard character (*), for example,
The system does not generate any file names without a unique ID. For example,
To better understand the different tokens, refer to the examples provided in the
following table:File path pattern tokens
%{FORMAT}
where
FORMAT is any date time format accepted by
java.time.format.DateTimeFormatter with the
exception of timezone specifications "zzzz",
"V", "X",
"x", "Z", and
"O".*
(asterisk)
character.-
(hyphen) character
followed by any string.-
(hyphen), followed by a
string that guarantees file name uniqueness.file.json
, a file called
file.json
is never generated.File name uniqueness when generating files
file.json
, the system generates files names with unique
IDs after a hyphen (-) and before the file extension:file-vlmmssaprdnd7421164395985556020230321.csv
file-vlmmssaprdnd7421164395986666420230321.csv
file-vlmmssaprdnd7421164395983234420230321.csv
file*.json
, the system generates file names by replacing
the wildcard character with a unique ID:filevlmmssaprdnd7421164395985556020230321.csv
filevlmmssaprdnd7421164395986666420230321.csv
filevlmmssaprdnd7421164395983234420230321.csv
file.json
is not a valid file name for new files and is
never generated.Examples of file path patterns
Examples of file path patterns
Pattern Allowed Matched files Not matched files Generated files file.json
Yes file.json
file-abc11658845611263.json
file-abc.json
fileabc11658845611263.json
(no
hyphen)file-abc11658845611263.json
file
Yes file
file-abc11658845611263
file-abc
fileabc11658845611263
(no hyphen)file-abc11658845611263
dir/file.json
Yes dir/file.json
dir/file-abc11658845611263.json
dir/file-abc.json
dir/fileabc11658845611263.json
(no
hyphen)wrongdir/file-abc11658845611263.json
(dir does not match pattern)dir/file-abc11658845611263.json
file_*_123.json
Yes file_abc11658845611263_123.json
file_abc_123.json
file___123.json
(no wildcard token
value)file_abc11658845611263_123.json
file%{yyyyMMdd}_123.json
Yes file20220726_123.json
file19800102_123.json
file20220726010203_123.json
(wrong date
time token)file_123.json
(missing date time
token)fileabc_123.json
(not a date time token
value)file20220726_123-abc11658845611263.json
file%{yyyy-MM-dd-HH-mm-ss}_123.json
Yes file2022-07-26-16-43-21_123.json
file1980-01-02-03-04-05_123.json
file20220726164321_123.json
(wrong date time
token format)file2022-07-26-16-43-21_123-abc11658845611263.json
file_*_123_%{yyyyMMdd}_456.json
Yes file_abc11658845611263_123_20220726_456.json
file_abc11658845611263_123_19800102_456.json
file_a_123_20220726_456.json
file__123_20220726_456.json
(no
wildcard token value)file_abc11658845611263_123_2022_456.json
(wrong date time token format)file_abc11658845611263_123_20220726_456.json
file*abc*xyz.json
No (two wildcard tokens) Not applicable Not applicable Not applicable file%{yyyyMMdd}abc%{yyyy-MM-ddHH:mm:ss}xyz.json
No (two date time tokens) Not applicable Not applicable Not applicable dir*/file.json
No (directory has a token) Not applicable Not applicable Not applicable
Previous topic Creating a File data set record for files on repositories Next topic Requirements for custom stream processing in File data sets