Interfacing Virtual MDM through DataStage

Introduction

MDM Connector stage is a key to open the door of IBM Virtual MDM. Yes, we can manipulate the data in MDM (MDM refers to IBM Virtual MDM in this post) using the MDM Connector stage which was introduced in IBM DataStage v11.3.

We know that loading data into MDM is not an easy task since it involves many tables and the relationship among the tables should be maintained properly, otherwise will end up dealing with junk not with the data. MDM Connector stage makes this task simpler by allowing us to configure everything in the single configuration window.

This blog post details on how the basic operations (read/write) on data can be performed using the Connector stage in v11.5.

DataStage and MDM Server Connectivity

To connect to the MDM server, the following should be specified in the stage properties


Figure 1: Specifying Stage Properties

  • Host name – IP Address/Host Name of the MDM Server
  • Port number – http Port number of the MDM Server. This can be found in the app.properties file of MDM
Writing Data into MDM

The Interaction mode should be specified as Member write as shown in the following figure if we want to write anything into MDM

Figure 2: Write Data to MDM

Model identifier specifies the data model of the MDM which should be imported into IIS using the IBM Metadata Asset Manager. The steps to import IBM Infosphere MDM assets into an IBM Infosphere Information Server is given in the tutorial https://developer.ibm.com/recipes/tutorials/importing-ibm-infosphere-mdm-assets-into-ibm-infosphere-information-server-repository/

The Member attributes will be listed in the Configuration Window based on this Model identifier.

Source

Following is the source data which we would like to load into MDM:

Member write Configuration

Required Member type and attributes can be selected and the column wise mapping can be performed by clicking the next button which is at the bottom of the configuration window

Figure 3: Member write Configuration

Target

Tables corresponding to the selected Member attributes can be queried to check the data. Rejections can also be captured by adding the output link from MDM Connector stage and configuring the properties required for the Reject link as shown below.

Reading Data from MDM

The Interaction mode should be specified as Member read as shown in the following figure if we want to read anything from MDM

Figure 4: Data Read from MDM

Source

To read the records from MDM we have to provide one of the following values as an input:

  • MEMIDNUM and SRCCODE
  • ENTRECNO
  • MEMRECNO

The input data is MEMIDNUM and SRCCODE

Figure 5: Input Data

Member read Configuration

Required Member type and attributes can be selected and the column wise mapping can be performed by clicking the next button which is present in the configuration window

Figure 6: Member read Configuration

The output link can be chosen and the required columns can be selected across the member attributes. We have two output links – one for Name data and another for Address data

Target

Data can be written to the file or table or it can be processed further. Data in the address output link is shown as below:

Issues faced and the solution
Version Compatibility

DataStage job was failing and the log has below warning message

‘DicStore could not be created; Possibly due to the userId and/or password not being valid.
Underlying IOException: java.io.IOException: ErrCode.EVERSION’

By default, the version of the MDM API library (madapi.jar) provided with IIS is 11.3. This should be replaced with the correct version of a jar which is present in MDM

Location of madapi.jar in MDM server –

<WAS_INSTALL_LOC>/profiles/<NODE>/installedApps/<CELL>/MDM-native-*.ear/native.war/lib/

Location of madapi.jar in IIS server –

/opt/IBM/InformationServer/Server/DSComponents/bin

Multiple Rows returned for single MEMIDNUM

The DataStage job was returning multiple records if more than one table was being queried for a single Entity ID in v11.5.

The issue was resolved after applying the “fix pack2”.

About the Author

Kanimozhi is a Senior Consultant at Mastech InfoTrellis and has good knowledge on IBM DataStage. She has over 4 years of experience in implementing EDI solutions.

Access Virtual MDM usingDataStage, DataStage and MDM Connectivity, DataStage versions, MDM Connector Stage, Member read configuration, Member write configuration, Version compatibility

3 responses to “Interfacing Virtual MDM through DataStage”

  1. Abhi says:

    Hi Kani,

    Nice tutorial! Let’s keep our fingers crossed that this works. I would like to put this all to rest.
    I am moving data from source to target table. Target table particular column has less length defined than source and because of it, truncation happens and tasks fails. We have defined the recovery strategy on failure to resume from last check point. On failure, we fix the source data, and start the task in recovery but it fails again pointing to the same source record which we fixed it. Not sure why it’s not taking the updated data from source.
    Anyways great write up, your efforts are much appreciated.

    Many Thanks,
    Abhi

    • Asha Shyam Sundar, Mastech InfoTrellis says:

      Hi Abhi,

      Thanks for posting your query. Please shoot your query at info@mastechinfotrellis.com to get the guidance from our technical team.
      Regards
      Asha

  2. Irene Hynes says:

    Hi,

    Awesome article. Thanks for making that available. I’ve been using your help to build my own POC and will publish the steps in another blog soon.

    I agree with your logic but it might be that Informatica MDM Training PT is considering the system columns also as normal columns and giving the functionality (just blind guess) .

    And yes I tried replicating your issue its same even for me.

    Are you using IDD to search records?

    Very useful article, if I run into challenges along the way, I will share them here.
    Thanks,

    Irene Hynes

Leave a Reply

Your email address will not be published. Required fields are marked *