Hey guys how are you? Continuing our SQL series (S01EP13), today I’ll share a very hand little query that I use very often for check data duplication. In fact, this would be an upgrade version of ODI’s pk check.
An upgrade version because in ODI, if you enable PK check, if he finds duplication, he eliminate both data. This code I’ll show you, you would choose if you want to keep the last created duplication or the oldest one, but only one will be eliminated.
I have a test table with this values:
If I want to check for duplicate PK, I can just run this query here:
The Idea here is, we have 2 queries. The first one will check if the ROWID it has is bigger or smaller (you choice) than the MIN or MAX ROWID (depending of your previous choice) than the second sub query by any joins you want to check.
In this case, we wanted to check only if the PK column had duplicated values, but we could check any other column by just replace it in the join. In fact, we could have any amount of columns in the join and that would check if there’s any duplications in all columns you inserted there.
Then you can select the first using > and MIN or the last by using < and MAX as well you can select what column you want to check in the where clause.
One important thing to mention is that this query is meant to work as a delete because it’ll keep what was not in the select. What I mean is, if you have more than one duplication, it’ll bring, in this case, all the rows that has the ROWID > then the one selected in the first query:
Then if I have multiple duplications, the query will return everything that needs to be deleted and the only one remaining was the first one inserted (3, Chuck, Giampaoli).
I hope you enjoy this little trick and see you soon.
Hey guys how are you doing? Today I’ll continue with the Oracle always free cloud offering and we’ll finally start to provisioning a VM in our environment. If you want to know more about how it works (Part 1) or the overview about the Dashboard (Part 2) please check my previous posts.
The first thing we need to do is check for the best practices and see if everything in our environment is adequate.
IP Addresses Reserved for Use by Oracle:
Certain IP addresses are reserved for Oracle Cloud Infrastructure use and may not be used in your address numbering scheme (169.254.0.0/16).
These addresses are used for iSCSI connections to the boot and block volumes, instance metadata, and other services.
Three IP Addresses in Each Subnet
The first IP address in the CIDR (the network address)
The last IP address in the CIDR (the broadcast address)
The first host address in the CIDR (the subnet default gateway address)
For example, in a subnet with CIDR 192.168.0.0/24, these addresses are reserved:
192.168.0.0 (the network address)
192.168.0.255 (the broadcast address)
192.168.0.1 (the subnet default gateway address)
The remaining addresses in the CIDR (192.168.0.2 to 192.168.0.254) are available for use.
Essential Firewall Rules
All Oracle-provided images include rules that allow only “root” on Linux instances or “Administrators” on Windows Server instances to make outgoing connections to the iSCSI network endpoints (169.254.0.2:3260, 169.254.2.0/24:3260) that serve the instance’s boot and block volumes.
Oracle recommends that you do not reconfigure the firewall on your instance to remove these rules. Removing these rules allows non-root users or non-administrators to access the instance’s boot disk volume.
Oracle recommends that you do not create custom images without these rules unless you understand the security risks.
Running Uncomplicated Firewall (UFW) on Ubuntu images might cause issues with these rules. Because of this, Oracle recommends that you do not enable UFW on your instances.
System Resilience
Oracle Cloud Infrastructure runs on Oracle’s high-quality Sun servers. However, any hardware can experience a failure:
Design your system with redundant compute nodes in different availability domains to support failover capability.
Create a custom image of your system drive each time you change the image.
Back up your data drives, or sync to spare drives, regularly.
If you experience a hardware failure and have followed these practices, you can terminate the failed instance, launch your custom image to create a new instance, and then apply the backup data.
Uninterrupted Access to the Instance
Make sure to keep the DHCP client running so you can always access the instance. If you stop the DHCP client manually or disable NetworkManager (which stops the DHCP client on Linux instances), the instance can’t renew its DHCP lease and will become inaccessible when the lease expires (typically within 24 hours). Do not disable NetworkManager unless you use another method to ensure renewal of the lease.
Stopping the DHCP client might remove the host route table when the lease expires. Also, loss of network connectivity to your iSCSI connections might result in loss of the boot drive.
User Access
If you created your instance using an Oracle-provided Linux image, you can use SSH to access your instance from a remote host as the opc user. After logging in, you can add users on your instance.
If you created your instance using an Oracle-provided Windows image, you can access your instance using a Remote Desktop client as the opc user. After logging in, you can add users on your instance.
NTP Service
Oracle Cloud Infrastructure offers a fully managed, secure, and highly available NTP service that you can use to set the date and time of your Compute and Database instances from within your virtual cloud network (VCN).
We recommend that you configure your instances to use the Oracle Cloud Infrastructure NTP service.
Fault Domains
A fault domain is a grouping of hardware and infrastructure that is distinct from other fault domains in the same availability domain. Each availability domain has three fault domains. By properly leveraging fault domains you can increase the availability of applications running on Oracle Cloud Infrastructure.
Your application’s architecture will determine whether you should separate or group instances using fault domains.
Customer-Managed Virtual Machine (VM) Maintenance
When an underlying infrastructure component needs to undergo maintenance, you are notified before the impact to your VM instances. You can control how and when your applications experience maintenance downtime by proactively rebooting (or stopping and starting) your instances at any time before the scheduled maintenance event.
A maintenance reboot is different from a normal reboot. When you reboot an instance for maintenance, the instance is stopped on the physical VM host that needs maintenance, and then restarted on a healthy VM host.
If you choose not to reboot before the scheduled time, then Oracle Cloud Infrastructure will reboot and migrate your instances before proceeding with the planned infrastructure maintenance.
When you work with Oracle Cloud Infrastructure, one of the first steps is to set up a virtual cloud network (VCN) for your cloud resources. I was thinking to do a more detail explanation here but this topic is very big. Then I decide to try do a simple step by step in how to set you Network for you to access your resources from your computer.
This is not the best way to create an complex network or anything like that, is just a way to quick start using your always free components and test your VM and DB.
To start we will click in the “Setup a network with wizard” quick link:
After you click there you have 2 options:
Select VCN with Internet Connectivity, and then click Start VNC Wizard. In the next page, just insert the name of your VCN and leave averything else as it is (unless you have a reason to change). Click Next.
In the next page, it’ll show everything that will be create by the Wizard. Note that you can create manually piece by piece of it, but for simplicity, the wizard should be enough.”Click in Create.
Next screen will show the installation of what was requested:
And that’s it for the network. Now we can start to create our databases and VM’s all inside our network, and they all going to “see” each-other.
That’s it for the network. Again, this is a very simple way to set your Network and every single step above can be setup individually with greater complexity but I’m for sure, but that will be impossible to be done in the always free since a lot of the complexity stuff needs to be paid for.
You can get a lot more information in the Jumpstart your Cloud Skills on the Start Explore. There are a lot of videos there explaining a lot of things. For simplicity, I’ll post here all links available there just for people that wants to see the videos before they subscribe to the OCI.
Next thing we can do is create a load balancing. To do that, we just have to click in the Create Load Balancer in the Quick Actions and then fill the new page like this:
The most important thing here is to make sure you selected the Micro in the Bandwidth selection. This one is free (you can also see the Always free Eligible logo there. Click Next after this.
In the next page we need to choose the load balance policy, and for that, depending of your application you’ll select one specific one. We have 3 options:
Weighted Round Robin: This one distribute the load sequentially in the servers (one each)
IP Hash: This one guarantee that the request from one specific client always go to the same server
Least Connections: this one always select the server with less connections
Next you need to add Back-ends. We don’t have any create now, but we can add this later. And finally we can change the Health Check policy, but for what we are doing we can just leave as it is. Click Next. In this screen we have to create a listener:
Here we have 3 options of traffic listener, HTTPS, HTTP and TCP. I’ll going to select TCP without SSL for simplicity, but if you select HTTPS you’ll need to have SSL certificate files and private keys. It’s safer but if you want just o play around its better to select HTTP or TCP.
For TCP we just have this options:
If you select USE SSL you also need to provide the Digital Certificate and private keys.
After you select yours, just finish the process. You’ll be taking to the Load Balance Monitoring page where’ll see something like this:
And that’s it for the network. Next time we’ll provisioning a VM and we’ll set our machine to connect into the VM.
Hey guys how are you? Today I’ll continue to talk about the Oracle Always free cloud offering and I’ll try to summarize what you can do after your account is set up. If you want to know how to setup you account you can find it HERE.
After you receive an email saying everything is set you can login in your account and you’ll see a screen like this:
This is the main dashboard. Here’s where you’ll create your Database, your VM’s, convert your account to paid, manage your account, ask for help, etc… Let’s start with the main dashboard:
(1) Quick Actions: Here you’ll find the most important links as quick actions.
(2)Compute: This is where you can create a VM to be used with your databases. You can use it to install tools and develop whatever you want inside the your environment.
(3)Networking: Here’s where you set up your cloud network. This is the first step you must do to ensure your VM and databases will be in the same network and reaching each other.
(4)Autonomous Transaction Processing: This is where you create a transaction database.
(5)Autonomous Data Warehouse: This is where you can create your Data Warehouse database.
(6)Search: A quick way to view all your resources.
(7)Account Center: Here’s a quick place to manage your account and see how many credits you have and billing information
(8)Main Menu: This is the main menu where you have access to everything that you can do inside your Cloud.
(9)Top Bar: Where you can change regions, in case you have more than one region, access the Cloud Shell (for OS commands), see the help, ask for help in the chat, change language and see your profile.
(10)Start Exploring: Here’s a place where you can find articles to help you start setting up your environment.
(11)What’s new: And finally here’s where you can see news about Oracle cloud, like releases and things that will be added.
One important thing to add here is that before you add anything or create anything, look for the “Always Free Eligible” logo or description to be sure you’ll not buying anything by mistake. Now about the main menu:
Core Infrastructure: Here’s where you can set your VM’s, networks and storage options.
Database: Here’s where you can Set your databases options, backups and Servers (VM or Bare metal).
Data and AI: Here’s where you can set your Big Data and AI environment.
Solution and Platform: Here’s where you can set your Analytics cloud services, Integrations, monitoring and marketplace.
More Oracle Cloud Services: Here’s where you have other cloud services.
Governance and Administration: And here is where you can administrate your environment like provisioning security, Account Management, Identity and Governance.
As you can see there’s a lot that can be done, but we’ll concentrate in the “Always Free” content, but the following list summarizes the Oracle Cloud Always Free-eligible resources that you can provision in your tenancy:
Compute (up to two instances)
Autonomous Database (up to two database instances)
Load Balancing (one load balancer)
Block Volume (up to 100 GB total storage)
Object Storage (up to 20 GiB)
Vault (up to 20 keys and up to 150 secrets)
In the next post we’ll setup our environment. See you soon guys.
I decide to do some posts about Oracle Always free offering, how it works, how you setup things, a few things we can do with that and maybe more. I think is fair for us to start by what’s it and what you need to do to get one.
Basically Always Free is a services for anyone that wants to try the world’s first self-driving database and Oracle Cloud Infrastructure for an unlimited time. The ideas is let people explore the full functionality of Oracle Autonomous Database and Oracle Cloud Infrastructure, including Compute VMs, Block and Object Storage, and Load Balancer, all of the essentials for developers to build complete applications on Oracle Cloud.
Oracle’s Free Tier program has two components:
Always Free services, which provide access to Oracle Cloud services for an unlimited time
Free Trial, which provides $300 in credits for 30 days to try additional services and larger shapes
The new Always Free program includes the essentials users need to build and test applications in the cloud: Oracle Autonomous Database, Compute VMs, Block Volumes, Object and Archive Storage, and Load Balancer. Specifications include:
2 Autonomous Databases (Autonomous Data Warehouse or Autonomous Transaction Processing), each with 1 OCPU and 20 GB storage
2 Compute VMs, each with 1/8 OCPU and 1 GB memory
2 Block Volumes, 100 GB total, with up to 5 free backups
10 GB Object Storage, 10 GB Archive Storage, and 50,000/month API requests
1 Load Balancer, 10 Mbps bandwidth
10 TB/month Outbound Data Transfer
500 million ingestion Datapoints and 1 billion Datapoints for Monitoring Service
1 million Notification delivery options per month and 1000 emails per month
Well, if you ask me this is far better than install an Oracle XE in your machine and configure everything there for you to learn or to create some small app. In fact, if you want to learn, it’ll far better if you start learning in an cloud environment since everyday we have more and more companies migrating to cloud.
Ok, but what do you need to do to get one? In fact is very easy, you just need to access this link and click in the Start for Free button. After that you have to fill a short form where you need to inform:
Your email and user information like address and cellphone
You need to validate your cellphone through message (Oracle will send a code to your cell)
You need to choose the region you’ll going to have you OCI (Oracle Cloud Infrastructure)
This needs to be as close as possible as your real region to decrease latency and improve network performance
Some regions are not available for always free (it’s written next to the region name if is available or not)
And you need to add a credit card to your account
You’ll not be charged but you may see 1 Dollar/Euro/… getting charged but it’ll be return
Also, Revolut card don’t work, you need a proper credit card.
And that’s it, Oracle will create your account (in fact takes around 15 minutes until you receive a email with further instructions [Bare in mind that because the COVID-19, it’s taking several days to create a new account]). After you receive your email, you can login in your dashboard and start to create your network, Disk, Database, VM’s and more.
We’ll see how to configure a database in my next post. I hope you enjoy this and see you soon.
Just a quick one today, Oracle is offering free access to online learning content and certifications for a broad array of users for Oracle Cloud Infrastructure and Oracle Autonomous Database, and will be available until May 15, 2020.
This is a great opportunity and if you want to learn more, you can find it here.
Hey guys how are you? Let’s take a look today in the opposite of S01EP12 situation, in fact we’ll use the same example again to show how can we convert a string in a list of values in a easy and dynamic way, starting with this query here:
I’ll transform this query in a with and I’ll use REGEXP to put this back into a list of values. This is very useful when we extract metadata from essbase for example, because essbase exports the UDA’s as a list of values. Of coarse this has many uses other than this but let’s keep this one in mind.
Now what we need to do is to split the strings by comma in this case, then the idea is to count the amount of commas we have in a row and split the strings by that amount.
The idea here is to use the REGEXP_COUNT to count how many words we have in between the commas and then use it to multiply the rows in the CONNECT BY LEVEL. For example, if we have 3 words, the connect by will create 3 rows of the same row, one with the LEVEL = 1 another with the LEVEL =2 and the last one with LEVEL=3.
With that we just need to use the REGEXP_SUBSTR to extract the words based in the LEVEL, this way we’ll have the REGEXP_SUBSTR(STR, ‘[^,]+’, 1, LEVEL (that will be 1 for the first row, 2 for the second and 3 for the third one).
Hey guys how are you keeping? I hope everybody is healthy and keep this way in this difficult times.
And to make our life less complicate, here’s another tip. Let’s talk about how to concatenate stuff in Oracle.
Imagine a simple case, we want to query the Planning repository to get the list of UDA’s a member have. We can easily do that by query the HSP_OBJECT, HSP_MEMBER_TO_UDA and HSP_UDA tables.
I’m filtering just 3 products to make it easier for us to see. The results shows that each project has a different number of UDA’s, and we never know how many it’ll be, then the easiest way to concatenate them is to use the command LISTAGG (or WM_CONCAT if you are in a DB version prior to 11.1).
The command is very simple LISTAGG(Column, Separator) WITHIN GROUP (ORDER BY column). As we can see the command allow us to select the separator we want (can be comma or any string really) as well to order the results by another column). Let’s take a look in the example above.
As you can see, it easily create a list split my comma (as specified) for me, and the nice thing about it is that I don’t need to do any string treatment if return null or if I have just one string on it and things like that.
This is an extremely good Function and we heavily use it in ODI to generate dynamic code because its simplicity, for example, we can generate a SQL statement on the fly using the command on source and command on target:
With this results we can easily pass this info to the command on target to generate a dynamic query where ODI will replace the columns we got in the target as well the table name and will also loop for each row we have in the source. This is very handy.
And for the ones that are not in the ORACLE 11.2 and ahead, we can still do that using WM_CONCAT. Is not as powerful as LISTAGG, but works pretty well. Let’s try the first example again:
I cannot show you the results since WM_CONCAT was decommissioned in the 12c (my version), but it’ll work like this. We don’t have the option to choose the separator and to make the string unique and to order by it we need to add DISTINCT in the command WM_CONCAT(DISTINCT column).
Today I’ll post something that is very simple but very useful specially when working with ODI.
When we work with partitioned table we know that if we filter that table by the partitioned column Oracle will use that partition as source of data. But what if we are doing an Insert, Update or Merge?
There’s another way to explicit refer to a partition and make sure Oracle will be working inside that one and is by defining it in the From clause.
For example if I want to query the Partition “DELL_BALANCES_FY20_FEB” I can query:
As we can see, after the table name I specified the PARTITION (DELL_BALANCES_FY20_FEB) and put inside the parentheses the partition name (don’t specify as string) and that makes oracle distinct all the rows in that partition, and my Distinct of the PARTITION_KEY shows only one results as expected. (this command needs to come before the table alias).
If we are doing an Insert, Update or Merge the idea is the same:
This way we can, specially in the MERGE, make sure Oracle will be working in the right partition in the target table.
And it’s specially useful with ODI because we always know the partition we want to query or insert data when we use ODI, then we can always bind Oracle to a specific partition and make sure he’ll stay there.
Today a quick tip that I think is very useful. From time to time the business ask us to validate if a table has data or not before we load it. It’s fare, specially if you use a truncate and insert approach.
The problem is, sometimes, the table/view they are asking for has millions of rows, and there’s no other safe way to validate if a table has data or not than querying it.
I just fixed a case where an interface had a validation that basically counts 3 different tables that together had 40 million rows per period. This validations were taking around 1000 sec to happens.
The data load that happens before that took 1200 sec. Then, basically the validation process were taking as much time as the load process.
After some changes, the query now is validating the 3 tables in 0.3 seconds. Way better than before. Basically I just used 3 things:
The hint /*+ FIRST_ROWS(1) */ that makes oracle prepare the best plan to query just one row (in my case since I used 1 as parameter.
The filter ROWNUM = 1 to make sure oracle just return 1 row, if we don’t use that, the hint can make everything very slow because oracle will be planning for just one row, but without filtering it’ll bring more (using the best plan possible for 1 row).
And UNION ALL instead of UNION, because there’s a huge difference between them. when you use UNION, oracle matches the sets of data to make sure you have unique rows after that. UNION ALL in other case, just bring everything each set return without any extra process to validate anything. UNION ALL is always faster than UNION.
In the end I have an query like this:
As you can see, the query is very simple and for this example I just had the name of the table there, then we know the table is not empty for that period. We can do other approach like summing then all together and validate if the results is = 3 for example or any other logic we need can be implemented on top of this query.
I hope this is helpful for you guys and see you in the next post.
Today it’ll be a quick tip for you guys that like/need to query the Planning repository.
The Planning repository stores both the Plan Type and the Consolidation in a very particular way, in fact this is true for a lot of other things like security, form properties etc… but I’ll focus these 2, that are the more often used and the solution is the same for all of them anyways.
If we take a look in the HSP_PLAN_TYPE table we’ll have something similar to this (depending in how many plan types you have in your app).
As we can see Planning stores in this table all the plan types that were created when we setup the application. In my case I have 4 plan types and we can have up to 5 BSO plan types in a Planning app. Now, if we join the HSP_OBJECT and the HSP_MEMBER filtering the OBJECT_TYPE = 2 we can take a look in all the dimensions we set in the repository.
The USED_IN columns is the column that says to planning in witch plan type that member will exists. The interesting thing here is that, you don’t see the PLAN_TYPE ID that you supposed to right? And that is because a member can exists in more than 1 plan type right, and if we use the PLAN_TYPE ID straight, we would need one row for each plan type right?
Instead, we have just one row but we also have the ability to tell Planning where that member should exists, and we can do that by summing the PLAN_TYPE ID’s together. In the example above, the Account dimension exists in all 4 plan types (1+2+4+8 = 15). Now the Products dimension exists only in one plan type (1), and by the number you can say that is the Pnl Plan type.
you seen the idea here is to check if a PLAN_TYPE ID exists inside that number we have here in the USED_IN column. Another example is the Employee dimension that has the USED_IN set as 8. The only number that will fit in here is the 8 itself (1+2+4 = 7, 1+8 = 9…) meaning the Plan type is WrkForce.
I think the most used way for us to figure out if a number exists inside another number is to use MOD.
CASE WHEN MOD(USED_IN,2)>=1 THEN 1 ELSE 0 END PT_1
CASE WHEN MOD(USED_IN,4)>=2 THEN 1 ELSE 0 END PT_2
CASE WHEN MOD(USED_IN,8)>=4 THEN 1 ELSE 0 END PT_3
CASE WHEN MOD(USED_IN,16)>=8 THEN 1 ELSE 0 END PT_4
CASE WHEN MOD(USED_IN,32)>=16 THEN 1 ELSE 0 END PT_5
The Oracle MOD(N,M) is used to return the remainder of a dividend divided by a divisor where:
N
Dividend.
M
Divisor.
Pictorial Presentation of MOD() function
Then in our case, we need to test if the USED_IN number contains the PLAN_TYPE ID on it, and for that we need to MOD it by rolling sum of the plan types + 1. To make it easier I’ll put that in numbers:
N = USED_IN = 31 (max number possible)
M = PLAN_TYPE ID = 1 (Pnl) what I want to test) + 1 = 2
MODE (31, 2) = 1
31/2 = 15 Reminder = 1
MOD = 1
What that is telling us is that if the MOD is = 1, the Plan type 1 exists in that number. I run a simulation just to show us when the Plan Type 1 does not exists in the USED_IN:
As we can see, the Plan Type 1 only exists in the odd possible results (as expected) what means in any other possible combination of the other 4 plan types he doesn’t exists (2, 4, 2+4=6, 8, 8+2=10, 8+4=12, 8+2+4=14, 16, 16+2=18, 16+4=20, 16+2+4=22, 16+8=24, 16+8+2=26, 16+8+4=28, 16+8+2+4=30).
The same is true for the other Plan types, you can try then out using the MOD. Now, this work well but there’s a way easier and clean way to do exactly the same thing using the function BITAND.
The BITAND function treats its inputs and its output as vectors of bits, the output is the bitwise AND of the inputs. Basically it performs below steps.
Converts the inputs into binary.
Performs a standard bitwise AND operation on these two strings.
Converts the binary result back into decimal and returns the value.
Ok, it looks more complicated now, but the good news is that to use is simpler than it sounds like. The main difference between this function and MOD is that MOD returns a boolean, BITAND return the value you asked if it’s true. Expanding my previous test using BITAND:
As you can see, with BITAND returning the number you asked for instead of 0 or 1 make it possible for us to Join the HSP_PLAN_TYPE with HSP_MEMBER using the USED_IN and the PLAN_TYPE in the BITAND Function as a Join:
As you can see, this is a far better way to split the members by Plan Type. And now we can see that the Dimension Products only exists in the Plan Type Pnl and that Entity exists in 4 different plan types. We don’t need to worry about any mathematics formula to create all our MODs, we just need to Join our Plan Type table with the BITAND of USED_IN by PLAN_TYPE.
The Consolidation is another place where you can use the exactly same thing. Instead of using something like this:
DEV EPM 