In today’s enterprise infrastructure, system and application integration is more and more frequently a mission critical concern. There are number of Enterprise Service Bus (ESB) products available in the market today. These products can help you remove basic dependencies between applications by eliminating the need for one application to be aware of the other’s location, but connectivity is not the only issue. In reality, most systems do not speak same language so Data transformation is one of the most important topic in Integration space.
Data transformation is the process of converting data or information from one format to another, usually from the format of a source system into the required format of a new destination system. MuleSoft has developed “DataWeave” – a new language and module for querying and transforming data. DataWeave is a full-featured and fully native framework for querying and transforming data on Anypoint Platform. Fully integrated with the graphical interface of Anypoint Studio and DataSense, DataWeave makes even the most complex data integration simple.
The DataWeave language supports a variety of transformations, from simple one-to-one mappings to more elaborate mappings including normalization, grouping, joins, deduplication, pivoting and filtering. It also supports XML, JSON, CSV, Java and EDI out of the box. The DataWeave Language is a powerful template engine that allows you to transform data to and from any kind of format (XML, CSV, JSON, Pojos, Maps, etc). It is fully integrated with Anypoint Studio, making on-ramp and continued development easy. It includes full integration with DataSense, allowing payload-aware development with auto-completion, auto-scaffolding of transforms, and live previews.
Let’s learn little bit more about the basics of this elegant and lightweight expression language. DataWeave files are divided into two main sections: 1) The Header, which defines directives (optional) and 2) The Body, which describes the output structure. Both sections are delimited by a separator, which is not required if no header is present. The separator consists of three dashes: “—“.
Header
The DataWeave header contains the directives, these define high level information about your transformation. The structure of the Header is a sequence of lines, each with its own Directives. Through directives you can define:
- DataWeave version
- Input types and sources
- Output type
- Namespaces to import into your transform
- Constants that can be referenced throughout the body
- Functions that can be called throughout the body
Body
The body contains the expression that generates the output structure. Regardless of the types of the input and output, the data model for the output is always described in the standard DataWeave language, and this model that the transform executes. The data model of the produced output can consist of three different types of data:
- Objects: represented as collection of key value pairs
- Arrays: represented as a sequence of comma separated values
- Simple literals
Let’s take a look at a simple example to understand more about DataWeave. In this example, we will first transform JSON input data to Java object and finally Java object to XML format.
Step 1. Import JSON Schema
Employee JSON Schema:
Step 2. Import XML Schema
Employee XML Schema:
Step 3. Create Java Classes
Employee.java
Contact.java
Address.java
Email.java
Step 4. Write Mule Flow in Anypoint Studio
Step 5. Write JSON to Java Transformation
DW Transformation (JSON to Java)
Step 6. Write Java to XML Transformation
DW Transformation (Java to XML)
Step 7. Input and Output
Input Data (JSON Format)
Output Data (XML Format)
References
- http://mulesoft.github.io/data-weave/
- https://www.mulesoft.com/integration-solutions/dataweave-integration
- https://docs.mulesoft.com/mule-user-guide/v/3.7/dataweave-reference-documentation
- https://docs.mulesoft.com/mule-user-guide/v/3.7/dataweave-tutorial
Advantages
- Mapping and transforming with DataWeave eliminates error-prone custom code
- Rules, lookups, and editing capabilities enable advanced transformations
- DataSense™ discovers end-point meta-data for intelligent design
- Delivers both batch and real-time event-driven data integration capabilities
- Supports XML, JSON, CSV, POJOs, Excel, and more