KScope 18 Speaker Award

Posted in ACE, Career, DEVEPM, EPM, Kscope, Kscope 18, ODI, ODI Architecture, ODTUG, PBCS with tags , , , , , , on September 17, 2018 by RZGiampaoli

Hey guys how are you?

It has been awhile since last time I wrote anything here…. and surprise, surprise, it’s because I’m crazy working in a project that was sized small but turn out huge and the size didn’t change…. 🙂 never happened before heheheh 😉

This is just a small post to tell how grateful and happy we are for receiving the EPM Data Integration Speaker Award in Kscope 18 with the presentation: How to Use Your ODI On-Premise to Seamlessly Integrate PBCS.

We start this blog in 2012 and we have been presenting at Kscope since 2013 and it has been very rewarding, not only because we become Oracle ACEs because of this, but because every single post or presentation we learn a lot with it.

When you do a presentation you need to stop to think in a solution for a specif project and start to thinking in a solution that can be used to all projects. This alone is a challenge, but the amount of thing we learn is a great deal. We can easily said that our code improved a lot since 2012 when we began with this blog and it’s in great part because of this blog and our presentations.

Then we thank you all of you that read our blog (even if we don’t post as much as we would like), to everybody that goes to KScope and decide to watch our presentations and to ODTUD that provide this bi-lateral learning platform.

Thank you all of you for supporting us and see you soon.

Advertisements

Building dynamic ODI code using Oracle metadata dictionary

Posted in Data Warehouse, ODI, ODI 12c with tags , , , on July 27, 2018 by radk00

Hi all, today’s post will be about how ODI can be used to generate any kind of SQL statements using Oracle metadata tables. We always like to say that ODI is way more than just an ETL tool and that people needs to start to think about ODI as being a full development platform, where you may create any kind of code that you wish there. Today I’ll describe how we may create a simple (but dynamic) merge statement between two similar tables using an ODI procedure that will read from ALL_CONSTRAINTS, ALL_CONS_COLUMNS and ALL_TAB_COLS Oracle tables to figure out what to do.

This is the scenario that we will be working on: we have several stage tables that are truncated and loaded everyday with daily records from a source system. We have another set of tables that are used to store all the historical information and the process uses the first stage tables as sources, merging its data against the historical tables using their primary key. This is very common in a lot of places where we have a stage data layer that stores daily data pulls and then a “base” data layer that stores the historical data. In this scenario that we will describe here, both source and target set of tables have very similar structures, with the same column names, data types and so on.

Using the conventional ODI development process, we would need to create one mapping object for each set of source->target tables, so if we have 50 sources that needs to be merged against 50 targets, we would need to create 50 ODI mappings. Since the set of source->target tables are similar in this case, we may be smarter and create an ODI process that will receive a table name as a input parameter (in this case the target table name) and it will create a merge statement against those two tables in a dynamic way using Oracle metadata dictionary.

For those that are not familiar with Oracle metadata dictionary, its nothing more than a set of tables that exists in Oracle database that contains information about its existing components like, information about its tables, what are the columns that they have, which is their data type and so on. This is a great resource place that ODI may read from it and build generic code using its results. Let’s see how it looks like with a real example.

Imagine that you have two tables with the following structure:

As you can see, our base table is almost the same as our stage table and the only difference is that it contains 2 additional columns named INSERT_DTTM and UPDATE_DTTM that will be used as “control columns” to identify when that data was inserted/updated in our base table. For ODI to figure out which columns are presented in which table, we may query ALL_TAB_COLS in Oracle filtering its table name, as below:

3

This is showing us all the table columns that this table contains. Similarly, if we query ALL_CONSTRAINTS and ALL_CONS_COLUMNS, we may get all the table constraints (like Primary Key) with all its associated columns:

4

With those two sets of data, we may create a SQL that will build our dynamic merge statement. To make it easier, I’ll show you the final SQL statement now, that is divided in two pieces, and then I’ll explain each of them:

WITH TABLE_PARAMS AS
(
SELECT 
    'BASE_TABLE_A' AS TABLE_NAME,
    'SCHEMA_A' AS TABLE_OWNER
FROM DUAL
),
TABLE_PK AS
(
SELECT
    ACC.OWNER, 
    ACC.TABLE_NAME, 
    ACC.COLUMN_NAME
FROM  ALL_CONSTRAINTS AC, 
      ALL_CONS_COLUMNS ACC,
      TABLE_PARAMS 
WHERE 1=1
AND AC.OWNER = ACC.OWNER
AND AC.TABLE_NAME = ACC.TABLE_NAME
AND AC.OWNER = TABLE_PARAMS.TABLE_OWNER
AND AC.TABLE_NAME = TABLE_PARAMS.TABLE_NAME
AND AC.CONSTRAINT_NAME = ACC.CONSTRAINT_NAME
AND AC.CONSTRAINT_TYPE = 'P'
)
,
MAIN_TAB_COLS AS
(
SELECT 
    ATC.OWNER,
    ATC.TABLE_NAME,
    ATC.COLUMN_NAME
FROM ALL_TAB_COLS ATC,
     TABLE_PARAMS 
WHERE 1=1
AND ATC.TABLE_NAME = TABLE_PARAMS.TABLE_NAME
AND ATC.OWNER = TABLE_PARAMS.TABLE_OWNER
AND ATC.COLUMN_NAME NOT IN ('INSERT_DTTM','UPDATE_DTTM')
AND ATC.COLUMN_NAME NOT IN (SELECT COLUMN_NAME FROM TABLE_PK)
)
SELECT
    MTC.TABLE_NAME AS TARGET_TABLE,
    REPLACE(MTC.TABLE_NAME,'BASE_','STG_') AS SOURCE_TABLE,
    PK_ST_LIST,
    PK_S_LIST||','||(LISTAGG('S.'||MTC.COLUMN_NAME ,',') WITHIN GROUP (ORDER BY MTC.COLUMN_NAME)) || ',SYSDATE,SYSDATE' AS TABLE_S,
    PK_T_LIST||','||(LISTAGG('T.'||MTC.COLUMN_NAME ,',') WITHIN GROUP (ORDER BY MTC.COLUMN_NAME)) || ',T.INSERT_DTTM,T.UPDATE_DTTM' AS TABLE_T,
    LISTAGG ('T.'||MTC.COLUMN_NAME||'=S.'||MTC.COLUMN_NAME , ',')  WITHIN GROUP (ORDER BY MTC.COLUMN_NAME ) AS ST_COLS
FROM MAIN_TAB_COLS MTC, 
    (SELECT 
        TP.OWNER,
        TP.TABLE_NAME,
        LISTAGG ('T.'||TP.COLUMN_NAME||'=S.'||TP.COLUMN_NAME , ' AND ')  WITHIN GROUP (ORDER BY TP.COLUMN_NAME ) PK_ST_LIST,
        LISTAGG ('S.'||TP.COLUMN_NAME, ',')  WITHIN GROUP (ORDER BY TP.COLUMN_NAME ) PK_S_LIST,
        LISTAGG ('T.'||TP.COLUMN_NAME, ',')  WITHIN GROUP (ORDER BY TP.COLUMN_NAME ) PK_T_LIST
    FROM TABLE_PK TP
    GROUP BY 
        TP.OWNER,
        TP.TABLE_NAME
    ) TP
WHERE 1=1
AND MTC.OWNER = TP.OWNER
AND MTC.TABLE_NAME = TP.TABLE_NAME
GROUP BY
    MTC.OWNER,
    MTC.TABLE_NAME,
    PK_ST_LIST,
    PK_S_LIST,
    PK_T_LIST;

The first piece of the SQL contains a WITH clause with three sections:

  • TABLE_PARAMS: used to “receive” the table name parameter that will be sent by ODI;
  • TABLE_PK: used to figure out which is the table PK that will be used to do a “merge” in our target table;
  • MAIN_TAB_COLS: used to retrieve all columns that exists in a table removing any kind of control columns (in this case INSERT_DTTM and UPDATE_DTTM) and any column that is already presented in the PK columns;

The second piece is the main one where we will use the three sub selects from the WITH section to build what we need. In this case, it will return the following columns:

  • TARGET_TABLE: name of the target table that will be merged;
  • SOURCE_TABLE: name of the source table that will be used as a source of the merge process;
  • PK_ST_LIST: PK columns list that will be used on merge process;
  • TABLE_S: column names from the source table;
  • TABLE_T: column names from target table;
  • ST_COLS: combination of source and target columns for update process inside the merge;

When we run the SQL for our tables in this example, this is the result:

5

Now we have all information that we need to create a dynamic merge statement for any set of similar tables, but how do we use it in ODI? This is very simple with one of the best features that ODI has (if you read our blog, you know that we just love it): command on source/target. Let’s create a procedure and add our SQL statement in the command on source tab:

6.png

In our command on target tab, we will add the following code there:

7

As you can see, this SQL contains a lot of variables in it. These variables will be used at runtime to receive the return values from the SQL in command on source. In this way, we don’t need to worry about creating 50 mappings to do 50 merge processes. Instead, we have one procedure that will receive a table name as a parameter and will build the necessary SQL accordingly. Let’s see how it looks like in an ODI package:

8

As you can see, it’s a very simple package that is receiving a table name as a parameter and then building/running a dynamic merge SQL. This package can be called by an external package that may run it N times with different table names (like doing 50 table mergers with one single procedure). Of course, that this was just one example of a simple merge task, but it shows you the main idea of having ODI building the code for you. You may add more tasks to your procedure to create temp tables, run gather statistics and so on. There are almost no limits on what you may do using this kind of technique.

I hope you have enjoyed! See ya!

KScope 18! It’s a wrap.

Posted in ACE, DEVEPM, EPM, Kscope, Kscope 18, PBCS, Uncategorized with tags , , on June 21, 2018 by RZGiampaoli

That’s it guys, one more year of KScope finished successfully.

This year was a big one for us since we had 3 sessions, one lunch and panel and one lip-sync battle…. that we lost… but was a lot of fun (way better than I thought it would be).

The sessions were great and we are very proud to receive the Top Speaker Awards for EPM Data Integration track with the session How to Use Your ODI On-Premise to Seamlessly Integrate PBCS.

This means a lot to us since we are always worried about our speeches because our marvelous English and our subtle accent (I sound like a famous robot from the future “Come with me if you want to live….”), then we always try to compensate with the content.

And this year I think we made it. We’ll try very hard to keep the content this interesting. We always try, but some times what is interesting for us is not for others. Would be very nice if you guys leave comments with thing you would like to see in our presentation or blogs.

Thank you very much for all people that attended our sessions and look forward to see you next year!

Thanks

 

Kscope 18 is coming fast and DevEPM will be all around the place!

Posted in Uncategorized on June 4, 2018 by RZGiampaoli

Hi guys how are you?

It has been so long that I almost forgot about the blog :). We have being very busy lately, with a lot of projects and Kscope. This year we’ll be all around the place in Kscope.

Yes we’ll be presenting 3 sessions, one lunch and learn panel and a lip-sync battle….

On Monday 11 at 1:15 PM, we will present the session How to Use Your Existing ODI On-Premise to Seamlessly Integrate PBCS where we will be talking about some nice ways to use you current environment to integrate your new PBCS app.

And at night, 8 PM, I’ll be lip-syncing for my life in the EPM Community Night Event. I think this will be really fun (or funny to watch at least).

On Tuesday 1 pm, I’ll be in the Lunch and Learn Essbase Panel at and I’m planning to put some twists in our regular essbase talk.

On Wednesday 13 we also have two presentations: Dynamic Metadata Integrations for Multiple ASO Applications at 11:45 AM where we’ll show a way to load metadata to any number of ASO apps in a very easy and dynamic way and a updated version of Incredible ODI Tips for Working with Hyperion Tools 2.0 at 3:30 PM this time we add some demos to make easier for everybody see how things happens.

I hope I can see you all there and that you guys have a blast in Kscope 18 🙂 I know I will.

See you soon guys!

DEVEPM will be at Kscope18!

Posted in ACE, DEVEPM, Essbase, Kscope 18, ODI, ODTUG with tags , on March 14, 2018 by radk00

Hi all, how are you doing? With some delay (same as last year, I was on vacation 🙂 ) we are very happy to announce that DEVEPM will be once again at KScope! We are very honored to be selected to present on the best EPM conference in the word! We got two presentations in, so here is what we are going present at Kscope18:

Dynamic Metadata Integrations for Multiple ASO Applications

Metadata management is a challenge on complex environments, since it is always changing over time. If the metadata process is not build in a robust and dynamic way, we will end up having a lot of rework every time that the business changes.

One of the best ways to maintain metadata on ASO cubes is using ODI, which gives us great flexibility to create real complex enterprise ETL processes. But if we use ODI in its standard way, we will end up having multiple similar ODI objects, since each ASO application/dimension is tied to a particular ODI data store. In other words, the higher the number of ASO applications we have, the higher is the number of ODI objects, increasing the possible failure points and code rework if something changes, which will make us lose a lot of time, money and trust on those systems.

This study case describes how to implement a smart EPM environment that uses Oracle Data Integrator with Oracle Essbase and take full advantage of its potential. This session will show how to create dynamic processes that changes automatically for any number of Essbase applications, allowing metadata maintenance that meets the business needs with low development costs.

Incredible ODI tips to work with Hyperion tools that you ever wanted to know

ODI is an incredible and flexible development tool that goes beyond simple data integration. But most of its development power comes from outside the box ideas.

  • Did you ever wanted to dynamically run any number of “OS” commands using a single ODI component?
  • Did you ever wanted to have only one datastore and loop different sources without the need of different ODI contexts?
  • Did you ever wanted to have only one interface and loop any number of ODI Objects with a lot of control?
  • Did you ever need to have a “Third Command Tab” in your procedures or KMs to improve ODI powers?
  • Did you still use an old version of ODI and miss a way to know the values of the Variables in a scenario execution?
  • Did you know that ODI has 4 “Substitution Tags”? And do you know how useful they are?
  • Do you use “Dynamic Variables” and know how powerful they can be?
  • Do you know how to have control over you ODI priority jobs automatically? (Stop, Start and Restart scenarios)

If you want to know the answer of all this questions please join us in this session to learn the special secrets of ODI that will take your development skills to the next level.

ys2header

Kscope is the largest EPM conference in the world and it will be held in Orlando on June 2018. It will feature more than 300 technical sessions, five symposiums, deep dive sessions, and hands on labs over the course of five days.

Got interested? If you register by March 29th you’ll take advantage of the Kscope early bird rates. Don’t waste more time and let’s be part of the greatest EPM event in the world. If you are still unsure about it, read our post about how Kscope/ODTUG changed our lives! Kscope is indeed a life changer event!

 

All about Meetups!

Posted in ACE, MeetUp, ODTUG, Oracle, Uncategorized with tags , , on January 23, 2018 by RZGiampaoli

Hey guys how are you doing? It has been a long long LONG time since our last post…. and we are sorry for that. We having been very busy with a lot of projects (both work and personal) and we had to get some time for us!

But we are finally getting back to action! And I’ll do that posting something about one thing that everybody loves…. Meetups..

First of all I would like to talk about the next ODTUG meetup that will happens in 2 days in Orlando, inside the Walt Disney World Dolphin Resort…. I think I don’t need to say anything else about it other that it’ll be awesome…. but anyway, here’s the link for you guys to attend:

All Community Social Meetup with the ODTUG Board of Directors

Thursday, Jan 25, 2018, 5:30 PM

Walt Disney World Dolphin Resort- Phins Lounge (Lobby Bar)
1500 Epcot Resorts Blvd Orlando, fl

8 ODTUGers Attending

Join us on Thursday, January 25, 2018, at 5:30 p.m. for an ODTUG social Meetup in the Phins Lounge at the Walt Disney World Dolphin Resort. Come enjoy a drink with the ODTUG Board of Directors and mingle with fellow ODTUGers! Date: Thursday, January 25, 2018, at 5:30 – 7:00 p.m. Location: Walt Disney World Dolphin Resort – Phins Lounge (lobby bar) …

Check out this Meetup →

Now that this is sort out, do you know that if you want to do your own Meetup you can do it together with ODTUG? And you can even get financial support to do it????

Well it’s true. You just need to plan your meet up and then fill out the Meetup Planning form. This will provide ODTUG with all the information they need in order to move forward.

It’s also helpful to provide them if you are seeking financial support and if so, how much, and also your estimated attendee count as well. Another advantage is that there’s an list of services that ODTUG will provide as a sponsor along with the qualifications for sponsorship.

  • Promotion for your local Meetup:
    • ODTUG will post your event on the ODTUG Meetup page.
    • Email Campaigns: ODTUG will invite members to your event
    • Social Media: ODTUG will promote your event on Twitter, Facebook, and LinkedIn
    • ODTUG will publish the success of your meeting in your community’s quarterly newsletter
  • Source an Oracle speaker if needed
  • Ship ODTUG swag for you to distribute at your event.

Off coarse there’re a few Sponsorship Requirements: Communicate ODTUG Resources and Events:

  • Educational resources

o   Provide an overview ODTUG’s memberships and benefits

o   Provide ODTUG Kscope18 deadlines

  • Early bird rate deadline – March 29, 2018

If you get exited with this piece of information and want to do you own Meetup, here are some useful tips and strategies for planning a Community Meetup.

 1- Choose a Date and Location

Be mindful of holidays and tech conference dates when planning a meetup. A free venue is best—workplaces and universities tend to be free, while private rooms at restaurants and hotels usually have a rental fee.  When securing a space, be sure to ask what the room capacity is and monitor your RSVP’s accordingly. If you need assistance with securing a venue please be sure to fill out the Meetup Planning form.

 2- Schedule Speaker(s)

Education and content should be a part of a Meetup. Even if presentations are brief, education helps to engage members. If you need assistance with securing presenters, please be sure to fill out the Meetup Planning form.

 3- Include Time for Networking with Food and/or Drinks

This could be as simple as having pizza delivered to an office after a speaker presentation. If there is a nearby bar/ restaurant the group may want to go for a Happy Hour.

 4- Promote ODTUG Membership and ODTUG Kscope18 Conference

Share the benefits of ODTUG membership, encourage others to volunteer with the community, and promote ODTUG Kscope18.

 Tips for a successful Meetup:

 Scheduling

o   Schedule quarterly Meetups if possible. Five-week months result in higher attendance and commitment. (Customers have an extra week to plan for maintenance, testing, and some downtime.)

o   Wednesdays and Thursdays are the best days for scheduling meetings; whereas Fridays you will be competing with families and/or social lives.

 Location is important

o   Provide an environment that is fun and inviting. Seek popular local bars or restaurants that provide private rooms for formal meetings.

 Budget

o   Everything is negotiable! Find creative ways for customizing menus for your meetings. Order appetizers for a small group or buffets for larger group.

o   The best option for savings is scheduling meetings in the workplace if possible.

 Greeting your guests

o   Have someone available to greet your guests. Provide a sign-up sheet, name tags, and pass out ODTUG/Kscope18 swag.

 Engaging your guests is KEY!

o   Creative ice breaker activity is a great way for people to engage and find common ground.

 Follow-up is vital for success

o   Feedback is important – this gives you an opportunity to thank your guests and provide a questionnaire (limited to 10 questions or less) to help improve and plan future Meetups.

 Non-solicitation

o   The user community is interested in learning how other organizations are leveraging the tools today within their environment, not a sales pitch. Solicitation is not allowed.

 

I hope you guys enjoy all this info and let’s start to plan our meetups 🙂

DEVEPM on Oracle Magazine

Posted in ACE, Career, DEVEPM, ETL, ODI with tags , , on November 16, 2017 by radk00

Hi all!

Very quick post today! Rodrigo was interviewed by Oracle Magazine for the November/December issue. Check it out!

DEVEPM on Oracle Magazine

Thanks!