Filtering incoming data

Filter incoming data to reduce the number of records that your data flow needs to process. Specify filter conditions to make sure that you get the data that is applicable to your use case. Reducing the number of records that your data flow needs to process, decreases the processing time and hardware utilization.

  1. In a data flow, click the Plus icon on a shape, and select Filter.
  2. Double-click the Filter shape to configure it.
  3. In the Name field, enter a name for the shape.
  4. In the Filter conditions section, click Add condition.
  5. In the left field, enter the name of a property that is evaluated by the filter.
  6. From the dropdown list, select a comparison operator.
  7. In the right field, enter a value for your filter condition.
  8. Click Add condition to add more conditions.
A data record that enters the Filter shape is compared against the filter conditions. When the record matches the conditions, the Filter shape outputs the record for further processing in the remaining data flow shapes. For example, to filter out customers who are younger than 18 years old and are unemployed, your filter conditions can look like this: .CustomerAge > 18.IsEmployed = false