Mulesoft – Salesforce Integration Using Batch Processing

Mulesoft has the capability to process messages in batches. It splits the large messages into individual records that are processed asynchronously within batch jobs. Batch Processing can be used for integrating large or small datasets and process the records in parallel. Further, one can set or remove variables on individual records so that during batch processing, Mule can route or otherwise act upon records in a batch according to a record variable. With the batch approach, large volumes of incoming data from any upstream system can be extracted, transformed, and loaded (ETL) into any destination system in real time. In this paper upstream system used is Oracle database and destination system used profoundly is Salesforce. Salesforce is a cloud computing platform which stores data in the form of data objects. This paper identifies challenges that are encountered when upstream systems have complex data storage formats and hence the conversions that are necessary to perform efficacious data transfers are discussed. To help provide a deeper insight, this paper discusses many components that are very specific to batch processing and can be used to implement business logic along with some general scenarios that form the basis of any batch flow. Uses Cases wherein up to 52 million records were retrieved from database, transformed and upserted successfully to Salesforce along with appropriate error handling mechanisms are discussed. Also, the recent news of Salesforce acquiring Mulesoft opens up vast opportunities to integrate data with Salesforce using powerful Mulesoft capabilities like the Batch Processing.