Sunday, November 29, 2009

Defining a Constant PL/SQL Lookup Table the Hard Way

The situation: we have a Large Company, with an American and a European division. Each division is divided into multiple regions. In the U.S., each region is assigned a state code. In the European division this code should probably be called something else, but to promote code reusability, it still is called a state code.

The problem: Given two regions, to determine if they have been assigned the same state code.

The bigger problem: We don't have a proper database lookup table. I won't go into why in detail (I grade the reason as "defensible" but not "conclusive"). So, the problem becomes, how to put such a lookup table into PL/SQL code, but not into a real database table.

Now, an associative array with a varchar2 index is a very nice object to use in place of a lookup table. Suppose we have a list of sales regions, and we want to know their state code. We could define an array as follows:

TYPE lookup_tbl_type IS TABLE OF VARCHAR2(2) INDEX BY VARCHAR2(4);
lookup_tbl lookup_tbl_type;

And then in the code section, assign some values:

lookup_tbl('REG1') := 'MI';
lookup_tbl('REG2') := 'MI';
lookup_tbl('REG3') := 'OH';
lookup_tbl('REG4') := 'KY';
lookup_tbl('REG5') := 'MI';

Given this, if we have a variable, l_region say, and want to do something if this region is in Michigan, but something else if it is in Ohio, we could then code:

IF (lookup_tbl(l_region) = 'MI')
THEN do something
ELSIF (lookup_tbl(l_region) = 'OH')
THEN do something else
END IF;

All one has to do is to get the values loaded into such an array. Which
is normally done from a database table. Which if we could have one, we wouldn't be having this discussion.

As it happens, I already have a package containing global types and constants. This is the natural place to put the lookup table. Unfortunately, one cannot define a constant PL/SQL table. As the PL/SQL User's Guide states (regarding associative arrays):

An initialization clause is not allowed. There is no constructor notation for associative arrays.

Include the 'CONSTANT' clause without initialization, and you get a PLS-00322 error. Try some kind of initialization and you get a PLS-00320. Leave off the CONSTANT -- and the table is no longer constant.

Well, I really want to have this "table" defined entirely in the package heading. But, there doesn't seem to be any way to do it using PL/SQL tables. Finally, the Dumb Bell rang, and I realized the solution was to use a PL/SQL Nested Table instead.

My first thought here was to define a record type, then a table type, and then a table:

TYPE look_me_up IS RECORD (
region varchar2(4),
state varchar2(2)
);
TYPE lookup_tbl_type IS TABLE OF look_me_up;
lookup_tbl lookup_tbl_type;

But this puts me back where I started: there is no constructor method for records, so one cannot assign them in their definition, so one cannot define them as constant (I suspect that the issue with PL/SQL tables is related).

The solution is to use a table of tables:

create or replace package testtypes as
TYPE lookup_tbl_type IS TABLE OF varchar2(4);
TYPE look_me_up_type IS TABLE OF lookup_tbl_type;

Then we can define as constant some nested tables. The plan is to put all the regions in one state in the same table:

lookup_tblMI constant lookup_tbl_type := lookup_tbl_type (
'REG1', 'REG2', 'REG5');
lookup_tblOH constant lookup_tbl_type := lookup_tbl_type
('REG3');
lookup_tblKY constant lookup_tbl_type := lookup_tbl_type
('REG4');

Then we can define our final "lookup table" as a constant table of these tables:

look_me_up constant look_me_up_type
:= look_me_up_type (lookup_tblMI, lookup_tblOH,
lookup_tblKY);

Now, if my primary question was to determine the state code for an arbitrary region, I would still be up a creek, and would have to create yet another table to indicate which index of look_me_up to use. Fortunately, the precise question I have is to determine if two arbitrary regions have the same state code. And this is a somewhat
easier question to answer. First, I do need yet another table:

tester_tbl lookup_tbl_type := lookup_tbl_type(null,null);

And then assign the two regions I wish to test:

tester_tbl(1) := 'REG1';
tester_tbl(2) := 'REG5';

Finally, I can test whether they are both in the same state by using the SUBMULTISET operator:

IF (tester_tbl SUBMULTISET OF look_me_up(1) )
THEN Do Something;
END IF:

And yes, fighting the Corporate reasons for not creating the natural lookup table would be more trouble than the above coding.

For reference, here is a package defining the nested lookup table, and a demonstration stub using it:


create or replace package testtypes as
TYPE lookup_tbl_type IS TABLE OF varchar2(4);
TYPE look_me_up_type IS TABLE OF lookup_tbl_type;
lookup_tbl1 constant lookup_tbl_type := lookup_tbl_type (
'REG1', 'REG2', 'REG5');
lookup_tbl2 constant lookup_tbl_type := lookup_tbl_type
('REG3');
lookup_tbl3 constant lookup_tbl_type := lookup_tbl_type
('REG4');
look_me_up constant look_me_up_type
:= look_me_up_type (lookup_tbl1, lookup_tbl2,
lookup_tbl3);
end;


declare
tester_tbl testtypes.lookup_tbl_type
:= testtypes.lookup_tbl_type(null,null);
begin
tester_tbl(1) := 'REG1';
tester_tbl(2) := 'REG2';
FOR i in 1 .. testtypes.look_me_up.count LOOP
IF (tester_tbl SUBMULTISET OF testtypes.look_me_up(i) )
THEN dbms_output.put_line('Found a Match at '||i );
END IF;
END LOOP;
end;

Thursday, October 15, 2009

Two Precompilier Surprises

A few more considerations when working with precompiliers. And I keep using the term "precompiliers" instead of Pro*C, Pro*Fortran, Pro*Cobol, Pro*whatever, because this will affect any and all of them. But first ...

If you have ever an ORA-0600, or had to read a trace file, you will have seen many references to data structures beginning with the letter 'K'. These represent Oracle Kernel processes and data structures, and the documentation
for them is very poor. Orafaq has a list of them
here
A little more information is available on Metalink and on a few blogs.

The relevance to this story is that access to all of these kernel processes is through a top-level process known as the Oracle Program Interface, or OPI. Which is not available to user programs. Instead, the OPI must be called through another undocumented API known as the User Program Interface (UPI). Which again, user programs are not supposed to ever call. Instead, user programs access the UPI through a documented API such as sql*plus, or OCI calls, or ODBC, or JDBC, or the SQLLIB library.

Now, the OCI is well documented. And I used to think that it was the only way to access the UPI. Or, rather, I am certain that I would have thought that it was the only way to access the UPI if I had known about the UPI. And, in particular, I always thought that what the precompiliers did was to translate embedded SQL statements into OCI calls.

And I was wrong.

It turns out that precompiliers translate into SQLLIB calls. And SQLLIB is not another name for the OCI. Instead, these SQLLIB functions access the UPI directly, without any OCI action at all. And this explains some strange behaviour I have been seeing. For example:

Here, I was just trying to use the CALL statement to execute a stored procedure. Of course, if I had been able to properly compile the program (for reasons too complex to list, I was unable to connect to the database for the compilation, so object validity could not be done) this would probably have had a different ending. But
I was getting this error:

ORA-00942: table or view does not exist

So, after checking my spelling, I rechecked all grants, and was mystified. Finally, I came up with a way to invoke the stored procedure using sql*plus (if it had been trivial I would not be in this spot) and found

ERROR at line 2:
ORA-06550: line 2, column 6:
PLS-00201: identifier 'PACKAGE.PROCEDURE' must be declared
ORA-06550: line 2, column 6:
PL/SQL: Statement ignored
And after I granted execute on the procedure to the Id doing the execution all was well.

We know that the DBA_TAB_PRIVS includes stored procedures as if they were tables. Perhaps SQLLIB does something similar. Would love to be able to be more definite, but, this is undocumented territory.

Here is another case where SQLLIB based clients have a different experience than OCI based clients. Consider stored procedures with OUT or IN/OUT parameters, and what happens when the procedure terminates with an unhandled exception. The PL/SQL documentation states for both kinds of parameters:

If you exit a subprogram successfully, PL/SQL assigns values to the actual parameters. If you exit with an unhandled exception, PL/SQL does not assign values to the actual parameters
In this case, it is supposed to assign a NULL to an OUT parameter. What happens to an IN/OUT parameter is not clearly stated.

So, to see what happens with Sql*Plus, I wrote a simple(minded) stored procedure:

       -----------Case 1-------------------
       CREATE OR REPLACE PROCEDURE JUNK(P_EMPNO  IN OUT NUMBER,
                                        P_ENAME OUT VARCHAR2)
       IS
       BEGIN
          P_EMPNO := 42;
          P_ENAME := 'DODAH';
          RAISE TOO_MANY_ROWS;
       EXCEPTION
        WHEN OTHERS THEN
          P_EMPNO := 4242;
          P_ENAME := 'DDT';
       END;
       /
In other words, do something to those parameters, raise an exception, and handle it. I wrote three variations on this theme:
       --------Case Two-------
       CREATE OR REPLACE PROCEDURE JUNK
                         (P_EMPNO  IN OUT NUMBER,
                          P_ENAME OUT VARCHAR2)
       IS
       BEGIN
          P_EMPNO := 42;
          P_ENAME := 'DODAH';
          RAISE NO_DATA_FOUND;
       EXCEPTION
        WHEN OTHERS THEN
          P_EMPNO := 4242;
          P_ENAME := 'DDT';
       END;
       /

--------Case Three-------
       CREATE OR REPLACE PROCEDURE JUNK
                         (P_EMPNO  IN OUT NUMBER,
                          P_ENAME OUT VARCHAR2)
       IS
       BEGIN
          P_EMPNO := 42;
          P_ENAME := 'DODAH';
          RAISE NO_DATA_FOUND;
       EXCEPTION
        WHEN OTHERS THEN
          P_EMPNO := 4242;
          P_ENAME := 'DDT';
          RAISE;
       END;
       /

       --------Case Four-------
       CREATE OR REPLACE PROCEDURE JUNK
                         (P_EMPNO  IN OUT NUMBER,
                          P_ENAME OUT VARCHAR2)
       IS
       BEGIN
          P_EMPNO := 42;
          P_ENAME := 'DODAH';
          RAISE TOO_MANY_ROWS;
       EXCEPTION
        WHEN OTHERS THEN
          P_EMPNO := 4242;
          P_ENAME := 'DDT';
          RAISE;
       END;
       /
Two different exceptions, handled and unhandled. I tested them with this bit of clever (?) code:
       declare
        v_ename varchar2(10);
        v_empno number;
       begin
        v_empno := 4;
        v_ename := 'FOUR';
        junk(v_empno, v_ename);
        dbms_output.put_line('For '||v_empno||' the name is
'||v_ename);
       exception
        when others then
          dbms_output.put_line(sqlerrm);
          dbms_output.put_line('For '||v_empno||' the name is
'||v_ename);
       end;
       /
[blockquote]
Yes, normally it is crazy to store anything in a variable which is going to be passed as an actual OUT parameter. But, here I am trying to distinguish between Oracle returning a NULL and Oracle leaving the parameter unchanged.
[/blockquote]
I'm going to interject here that the NO_DATA_FOUND exception was deliberate. This was a routine which was going to be called to scroll up and down a screen. The program it was going to replace was looking for NO_DATA_FOUND. Consequently, I wanted to continue to return this exception at end-of-screen.

To continue, I found that for cases 1 and 2, the values of p_empno and p_ename were 4242 and DDT respectively, and there was no exception reported (the pl/sql block completed without error). Which is just what I had anticipated; upon return from the stored procedure, these parameters contained the values which the procedure had assigned to them, and the exception was not reraised. In cases 3 and 4, the re-raised exception was returned to the calling block and was reported back to the command line. p_empno had a value of 4 (which was what the calling block had put there) and p_ename was 'FOUR'. In other words, even though the stored procedure had assigned values to those parameters, raising the exception wiped out those values. To further confirm this, I reran the test with my pl/sql block not assigning anything to ename and empno. When I did this, the values returned were null.

Still, just what I had anticipated after I RFTM'd. Oracle only populates variables passed as OUT parameters if the subroutine completes without raising any exception, and whatever was there in the first place is still there. In other words, if you insist on using an OUT parameter as if it were an IN/OUT parameter, you can't conclude that a non-null return means the procedure completed without error.

Next, I ran those cases through a program with embedded sql and null indicators. Everyone always includes null indicators for their Pro* programs, right?. The results for cases 1, 2, and 4 were identical to those with the pl/sql test harness. In case 3, where no_data_found is being raised, p_empno was returned with a value of 42, p_empname had a value of 'FOUR', and no exception was reported. Which was most unexpected, as I had clearly raised an exception, and the return from the parameters suggested that p_empno had never seen the exception handler.

But wait, there's more!. I re-ran these cases without using any null indicators. Cases 1,2, and 4 were unchanged. Case 3 returned the same parameter values, but this
time the error number was -1405: fetched column value is NULL.

Wild Guess: OCI and SQLLIB have some control as to what goes back to the client. In the case of precompilier applications, NO_DATA_FOUND is not an error in the same way as other errors. Hence, SQLLIB wants to return (at least to IN/OUT parameters) the state when the error was raised. Moreover (and this is the surprising part) it still wants to inform that a null is being returned. A null with no null indicator raises a -1405. So ... two errors need to be returned: NO_DATA_FOUND and -1405. And the -1405 takes priority.

Sunday, August 16, 2009

DAYS360 Redux

First, I want to apologize for the horrible look this Blog has taken on. I made one little change to the settings, and everything went crazy. I am working on fixing it.


When this whole business started, it was supposed to be about the
Excel DAYS360 function, wasn't it?


The whole purpose of DAYS360 is to realize a calendar of 360 days, consisting of 12 months of 30 days each. The idea is, if the month happens to consist of 30 days, leave it alone. If not, adjust it so that it does. Let's restate exactly what this means.


Define the starting date (Dt1) and the ending date (Dt2) as follows:


Dt1 = MM1/DD1/YYYY1
and
Dt2 = MM2/DD2/YYYY2


For example, If the starting date were March 5, 2007, then MM1 = 03, DD1 = 05, and YYY1 = 2007.


Then, every 30/360 calendar ever devised attempts to define the days between Dt1 and Dt2 (Days) as


Days = (DD2 - DD1) + 30*(MM2 - MM1) + 360*(YYYY2 - YYYY1)


All of the differences and tricks and complications come from adjusting the days and months to fit the requirements of 30 days per month, regardless of what the Gregorian Calendar might suggest. Different systems use different conventions, but they all have one thing in common: after all the conventions are applied, neither DD1 nor DD2
ever equal 31.


Here is an example of a series of dates, each falling on the first of the month (all values as calculated with the Office 2003 version of Excel, set for the US calculation). We expect that each month should have 30 days:


Date 1 Date 2 Days
--------- -------- -----
1/1/2009 2/1/2009 30
2/1/2009 3/1/2009 30
3/1/2009 4/1/2009 30
4/1/2009 5/1/2009 30
5/1/2009 6/1/2009 30
6/1/2009 7/1/2009 30
7/1/2009 8/1/2009 30
8/1/2009 9/1/2009 30
9/1/2009 10/1/2009 30
10/1/2009 11/1/2009 30
11/1/2009 12/1/2009 30
12/1/2009 1/1/2010 30
======
360


And we see that we do. And you get the same result if you try the 2nd, 3rd, ... up to the 27th of the month. And, we would expect the same result for the 28th. But, when we plug these dates into the formula we get:


Date 1 Date 2 Days
========= ========= ====
1/28/2009 2/28/2009 30
2/28/2009 3/28/2009 28
3/28/2009 4/28/2009 30
4/28/2009 5/28/2009 30
5/28/2009 6/28/2009 30
6/28/2009 7/28/2009 30
7/28/2009 8/28/2009 30
8/28/2009 9/28/2009 30
9/28/2009 10/28/2009 30
10/28/2009 11/28/2009 30
11/28/2009 12/28/2009 30
12/28/2009 1/28/2010 30
===
360

No, I have not forgotten how to add. DAYS360(1/28/2009, 1/28/2010) happens to equal 360 and not 358. Here, the sum of the individual months does not equal the year.


Which brings up two questions:


  1. Is this a bug?

  2. What is the correct value for Days360('2/28/2009','3/28/2009') ?


Let's put this aside for a moment, and ask the question: what happens if we stipulate that the dates are to be the last of the month rather than the first. Security traders do not like to stipulate that last of the month (for reasons that will become obvious) and lenders greatly prefer to specify a date, rather than just "the last of the month". However, this is not an impossible question, and we should
expect a definite answer.


One would expect a 30/360 calendar to give a result similar to that from the 1st of the month. But, we find instead:


Date 1 Date 2 Days
========= ========= ====
1/31/2009 2/28/2009 28
2/28/2009 3/31/2009 30
3/31/2009 4/30/2009 30
4/30/2009 5/31/2009 30
5/31/2009 6/30/2009 30
6/30/2009 7/31/2009 30
7/31/2009 8/31/2009 30
8/31/2009 9/30/2009 30
9/30/2009 10/31/2009 30
10/31/2009 11/30/2009 30
11/30/2009 12/31/2009 30
12/31/2009 1/31/2010 30
====
360


And you know by now where the 360 came from. Examining the 1/31/2009 to 2/28/2009 case more closely, the formula works out as


Days = (28 - 30) + 30*(2 - 1) + 360*(2009 - 2009) = 28


where DD1 was changed from 31 to 30 (we are supposed to have 30 day months, remember?)


But, 2/28 is the last day of the month. And we are supposed to have 30 day months (remember?). Which suggests that DD2 should also be changed to 30. In which case, Days becomes


Days = (30 - 30) + 30*(2 - 1) + 360*(2009 - 2009) = 30


Except that then 1/28/2009 to 2/28/2009 becomes


Days = (30 - 28) + 30*(2 - 1) + 360*(2009 - 2009) = 32


There are other anomalies around Feb. 28 as well:


Date 1 Date 2 Days
========= ========= ====
2/1/2009 2/26/2009 25
2/1/2009 2/27/2009 26
2/1/2009 2/28/2009 27
2/1/2009 3/1/2009 30
1/30/2009 2/26/2009 26
1/30/2009 2/27/2009 27
1/30/2009 2/28/2009 28
1/30/2009 3/1/2009 31


The transition from Feb. 28 to Mar. 1 doesn't "seem right", no matter what approach one takes.


But we are supposed to be talking about the Excel DAYS360 function. We may as well start with the Microsoft documentation for Office 2003 (which can be found at
http://office.microsoft.com/en-us/excel/HP052090471033.aspx
:

U.S. (NASD) method. If the starting date is the last day of a month, it becomes equal to the 30th of the same month. If the ending date is the last day of a month and the starting date is earlier than the 30th of a month, the ending date becomes equal to the 1st of the next month; otherwise the ending date becomes equal to the 30th of the same month.

If this is correct, then DD2 should be changed to 30 from 28 in the month of February.


While Microsoft must be given that last word for what their DAYS360 function is supposed to do, they also mention that it is supposed to emulate the NASD day counting method. Exactly what that method is supposed to do should not be open for question -- except that I have not been able to find any conclusive documentation about it. I have found a couple of references which attempt to get into this level of detail:

http://www.six-swiss-exchange.com/download/trading/products/bonds/accrued_interest_en.pdf
from the Swiss Exchange, and
http://www.eclipsesoftware.biz/DayCountConventions.html?
from the Eclipse foundation. But neither seems conclusive.


In addition, each refers to securities. My interest sprang from the small loan industry, which, in the U.S., is State and not Federal regulated. It is quite common (or, at least it used to be) for an identical loan pattern to pay a different number of days interest in different states. So, depending on the use one is making of these formulae, none of them may be correct.

So, I'm not going to offer any corrections. My assumption is that anyone who has read this far must have gained the ability to think for themselves. You, dear reader, know what the constraints of your loan regulations far better than I, and you now have everything you need (or at least everything I can give you) to adapt them to your unique situation.

Monday, May 25, 2009

Collections, Part V

It is now time to introduce Varrays and Nested Tables into the discussion. Later, I will compare and contrast the three kinds of collections in more detail (in the sort of way which OCP examinations seem to love). As an introduction here, I will just list some of the ways they are like Associative Arrays, and some of the ways they differ.


Associative Arrays, Varrays, and Nested Tables all


- Require that the object be first defined somewhere in a TYPE statement


- Can be used in PL/SQL anywhere a built-in data type can be used


- Use subscript notation to identify individual items.


- Have the same methods


- Can be used for bulk inserts and selects


They differ in that


- Varrays and Nested Tables can be defined in the database (and thus become SQL types)


- Varrays and Nested Tables can be stored as columns in relational tables


- Varrays and Nested Tables can thus appear in ordinary SQL operations


- Varrays and Nested Tables cannot have a varchar2 index.


- Varrays must have an upper size limit specified in their TYPE statement, and this limit cannot be extended


- Varrays must always be 'dense'; there can be no gaps in their index structure (if v is a varray, you cannot have a situation where v(1), v(2), and v(4) exist, but there is no v(3)).


- Nested Tables must be created and loaded dense, but it is possible after the fact to delete internal elements. Thus, one must insert v(1), v(2), v(3), and v(4) into a nested table, but one can then delete v(3).


They also have one other difference with Associative Arrays, which I will illustrate by changing Listing1 to use a Varray instead of an Associative Array. The obvious way to go is to replace lines 8 and 9 with



8 type amortTableType is varray(362) of amortTableLine;



where all we do is replace the clause 'table' with 'varray(362) and remove the 'index by binary_integer'. When we then try to run thus we get



ERROR at line 1:
ORA-06531: Reference to uninitialized collection
ORA-06512: at line 32



Which reminds us that Varrays and Nested Tables have to first be 'initialized'. And we do that by calling the 'constructor function', which is provided to us as a function with the same name as the collection type. So, we add a new line (between lines 31 and 33) as follows:



30 v_monthlyRate := p_InterestRate / 1200;
31
32 amortTable := amortTableType();
33
34 amortTable(1).pmtdate := add_months(p_effectiveDate,1);



and try again:



ERROR at line 1:
ORA-06533: Subscript beyond count
ORA-06512: at line 34

For reference, line 34 is the first time we try to access a specific element:



34* amortTable(1).pmtdate := add_months(p_effectiveDate,1);



Let's pause a bit. Oracle refers to an unitialized collection as being "atomically null". To understand the significance of this, let us first ask a few related questions (how they are related will soon be made evident).


Suppose we have 2 varrays (or Nested Tables), A and B. What do the following assertions mean?


1) A = B


2) A < B 3) A IS NULL


Try to decide for yourselves before I continue. For myself, I would answer (1) as follows: A = B if (and only if) for every element A(i), there is j such that A(i)=B(j). Some others might want to add, " and none of the A(i) or B(i) are null. Others might want to add that the two collections contain the same number of elements. And others might insist that the elements are stored in the exact same order.


I could make a really long list of possibilites around question (2). Some people might insist that all A(i) be <= B(i), with at least one element being greater. Me, I would take another tack, and define it such that for each A(i) there is an identical B(j), and there is at least one B(j) not found in A(i).


For question (3), I would like it to mean that every element of A is null. But Oracle does not adopt any of these. Instead, any attempt to compare two collections results in a PLS-0306 error: wrong number or types of elements in call to '=' (or '<' or '>' or whatever). The only comparison operator one can use against an entire collection is the IS NULL operator. Before a collection X is initialized, 'X IS NULL' evaluates to TRUE. Afterwards, it evaluates to FALSE. And before a collection X is initialized, one cannot refer to its individual elements. Afterwards, one can. So, to be atomicaly null, is to have meaningless elements, and to have the IS NULL operator return TRUE.


To return to our immediate issue, we have not yet allocated any space in the amortTable collection. Let us now allocate some space, and get this block working:
add

32 amortTable := amortTableType();
33 amortTable.extend (p_loanTerm * 12 + 1);

we run it and get:

25-MAY-09 541.67 90.4 99909.6
25-JUN-09 541.18 90.89 99818.71
25-JUL-09 540.68 91.39 99727.32
25-AUG-09 540.19 91.88 99635.44

The trick was line 33, where we actually created 361 slots in the Varray.




The sharp-eyed observer will have noticed that everything has thus far been defined as a record. Since we will eventually want to have these types declared in the database, we need to make sure that this will work there also. So, let's try defining our record in the database:

1 create or replace type amortTableLine is record (
2 pmtDate date,
3 ToInterest number(12,2),
4 ToPrincipal number(12,2),
5 EndingBalance number(12,2)
6 );
7/


Warning: Type created with compilation errors.

SQL> show errors
Errors for TYPE AMORTTABLELINE:
1/24 PLS-00103: Encountered the symbol "RECORD" when expecting one of
the following:
array varray table object fixed varying opaque sparse
The symbol "object was inserted before "RECORD" to continue.



It appears we cannot define a record as a SQL type. Fortunately, there is an alternative; define an object instead:

1 create or replace type amortTableLine is object (
2 pmtDate date,
3 ToInterest number(12,2),
4 ToPrincipal number(12,2),
5 EndingBalance number(12,2)
6 );
7/

Type Created

Much as we would like to remain with anonymous PL/SQL blocks, we cannot, for if we try to change the type of the type from 'record' to 'object', we find


 type amortTableLine is object (
*
ERROR at line 2:
ORA-06550: line 2, column 2:
PLS-00540: object not supported in this context.
So, let us keep amortTableLine defined as a SQL object, and try our routine. We find that it no longer works:
ERROR at line 1:
ORA-06530: Reference to uninitialized composite
ORA-06512: at line 29

SQL> List 1,29
1 DECLARE
2 type amortTableType is table of amortTableLine;
3
4 amortTable amortTabletype;
5
6 p_LoanAmount number(10,2);
7 p_InterestRate number;
8 p_loanTerm number(2,0);
9 p_effectiveDate date;
10 p_payment number(7,2);
11
12 v_pmtNumber number(4,0);
13 v_monthlyRate number;
14 v_totalInt number(13,2);
15 v_totalPmt number(13,2);
16 v_currentPrinc number(8,2);
17 BEGIN
18 p_LoanAmount := 10326.05;
19 p_InterestRate := 2.875;
20 p_loanTerm := 30;
21 p_effectiveDate := to_date('20090425','yyyymmdd');
22 p_payment := 62.22;
23 v_pmtNumber := 2;
24 v_monthlyRate := p_InterestRate / 1200;
25
26 amortTable := amortTableType();
27 amortTable.extend(362);
28
29 amortTable(1).pmtdate := add_months(p_effectiveDate,1);

SQL>

Which comes as a great surprise to someone is familiar with the various writings on the matter. All (well, almost all) suggest that the above should work.


To be fair, the examples usually look something like this:
  1  declare
2 type v_type is table of varchar2(10);
3 v_tbl v_type;
4 begin
5 v_tbl := v_type();
6 v_tbl.extend(30);
7 v_tbl(1) := 'aaaaaa';
8 v_tbl(2) := 'bbbbb';
9 dbms_output.put_line(v_tbl(1)||v_tbl(2));
10 end;
11/
which, when run, gives the expected (and desired) results:
SQL> /
aaaaaabbbbb

PL/SQL procedure successfully completed.

The working example's lines 5 and 6 looks similar to our lines 26 and 27. So, what is wrong?


The answer is that the working example's nested table is a table of varchar2, which is something like a one-field record, while our failed example is a table of objects. And objects have to be initialized. We can break our working example by changing the type of the type from varchar2 to object. First, we must create an object that looks like the varchar2 above:
SQL> create type v_obj is object (dodah varchar2(10));
2 /

Type created.

Next, we revise the snippit accordingly:

1 declare
2 type v_type is table of v_obj;
3 v_tbl v_type;
4 begin
5 v_tbl := v_type();
6 v_tbl.extend(30);
7 v_tbl(1).dodah := 'aaaaaa';
8 v_tbl(2).dodah := 'bbbbb';
9 dbms_output.put_line(v_tbl(1).dodah||v_tbl(2).dodah);
10 end;
11/

and we get the ORA-06530 error.


To make the snippit work, we must recognize that v_tbl is a nested table of objects, and initialize those objects. The following works:

1 declare
2 type v_type is table of v_obj;
3 v_tbl v_type;
4 begin
5 v_tbl := v_type();
6 v_tbl.extend(30);
7 v_tbl(1) := v_obj(null);
8 v_tbl(1).dodah := 'aaaaaa';
9 v_tbl(2) := v_obj(null);
10 v_tbl(2).dodah := 'bbbbb';
11 dbms_output.put_line(v_tbl(1).dodah||v_tbl(2).dodah);
12 end;
13/

But it is less than satisfactory, mostly because while we can extend the nested table in a single statement, we are initializing the v_obj object one element at a time. We are able to take care of records in a single statement, can we do the same for objects? The answer is, almost. The solution is to call the object constructor and pass it to the Collection constructor. This creates and initializes one element of the Collection. Then, we can call the EXTEND method, passing it this well-formed first element. A glance at our snippit may help:

1 declare
2 type v_type is table of v_obj;
3 v_tbl v_type;
4 begin
5 v_tbl := v_type(v_obj(null));
6 v_tbl.extend(30,1);
7 v_tbl(1).dodah := 'aaaaaa';
8 v_tbl(2).dodah := 'bbbbbb';
9 dbms_output.put_line(v_tbl(1).dodah||v_tbl(2).dodah);
10 end;
11/

Pay particular attention to lines 5 and 6. Line 5 initializes the nested table and also initializes the first element of it. Then line 6 extends and initializes 30 more elements.


So, let us conclude by applying this to our mortgage amortization routine (where we have modified things a bit by putting the initialization into the DECLARE section in line 4:

Listing 6
1 DECLARE
2 type amortTableType is table of amortTableLine;
3
4 amortTable amortTableType := amortTableType(amortTableLine(null,null,null,null));
5
6 p_LoanAmount number(10,2);
7 p_InterestRate number;
8 p_loanTerm number(2,0);
9 p_effectiveDate date;
10 p_payment number(7,2);
11
12 v_pmtNumber number(4,0);
13 v_monthlyRate number;
14 v_totalInt number(13,2);
15 v_totalPmt number(13,2);
16 v_currentPrinc number(8,2);
17 BEGIN
18
19 p_LoanAmount := 100000.00;
20 p_InterestRate := 6.5;
21 p_loanTerm := 30;
22 p_effectiveDate := to_date('20090425','yyyymmdd');
23 p_payment := 632.07;
24 v_pmtNumber := 2;
25 v_monthlyRate := p_InterestRate / 1200;
26
27 amortTable.extend(p_loanTerm * 12 + 1,1);
28
29 amortTable(1).pmtdate := add_months(p_effectiveDate,1);
30 amortTable(1).ToInterest := p_LoanAmount * v_monthlyRate;
31 amortTable(1).ToPrincipal := p_payment - amortTable(1).ToInterest;
32 amortTable(1).EndingBalance := p_LoanAmount - amortTable(1).ToPrincipal;
33 v_totalInt := amortTable(1).ToInterest;
34 v_totalPmt := p_payment;
35
36 WHILE (v_pmtNumber < p_loanTerm * 12) LOOP
37 amortTable(v_pmtNumber).pmtdate := add_months(amortTable(v_pmtNumber-1).pmtdate,1);
38 amortTable(v_pmtNumber).ToInterest := amortTable(v_pmtNumber - 1).EndingBalance * v_monthlyRate;
39 amortTable(v_pmtNumber).ToPrincipal := p_payment - amortTable(v_pmtNumber).ToInterest;
40 amortTable(v_pmtNumber).EndingBalance := amortTable(v_pmtNumber - 1).EndingBalance - amortTable(v_pmtNumber).ToPrincipal;
41 v_totalInt := v_totalInt + amortTable(v_pmtNumber).ToInterest;
42 v_totalPmt := v_totalPmt + p_payment;
43
44 v_pmtNumber := v_pmtNumber + 1;
45 END LOOP;
46 amortTable(v_pmtNumber).pmtdate := add_months(amortTable(v_pmtNumber-1).pmtdate,1);
47 amortTable(v_pmtNumber).ToInterest := amortTable(v_pmtNumber - 1).EndingBalance * v_monthlyRate;
48 amortTable(v_pmtNumber).ToPrincipal := amortTable(v_pmtNumber -1).EndingBalance;
49 amortTable(v_pmtNumber).EndingBalance := 0;
50 v_totalInt := v_totalInt + amortTable(v_pmtNumber).ToInterest;
51 v_totalPmt := v_totalPmt + amortTable(v_pmtNumber).ToPrincipal;
52
53 v_pmtNumber := 1;
54 WHILE (v_pmtNumber <= p_loanTerm * 12) LOOP
55 dbms_output.put_line(amortTable(v_pmtNumber).pmtdate||' '||
56 amortTable(v_pmtNumber).ToInterest||' '||
57 amortTable(v_pmtNumber).ToPrincipal||' '||
58 amortTable(v_pmtNumber).EndingBalance);
59 v_pmtNumber := v_pmtNumber + 1;
60 END LOOP;
61
62 END;
63/





So, let's summarize the housekeeping necessary when using Varrays and Nested Tables:


1) The collection must be defined in a TYPE statement somewhere (either in the database, a package definition, or in the PL/SQL block itself). The syntax is as follows:


Varray: TYPE varraytype IS VARRAY(n) OF datatype; where 'n' is the maximum size of the Varray


Nested Table: TYPE nestedtype IS TABLE OF datatype;


The datatype can be a built-in data type, a record type, or an object type.


2) Declare and initialize the collection. The syntax varies slightly, depending on whether the datatype is a built-in datatype, a record (which is just a generalization of a built-in datatype) or an object. The syntax for records:


Records: CollectionName varraytype := varraytype(data);
or CollectionName nestedtype := nestedtype(data);


Objects: CollectionName varraytype := varraytype(datatype(data));
or CollectionName nestedtype := nestedtype(datatype(data));


The data can be default data, or null.


3) If you know at the start how many elements will be needed, construct them at the beginning of your code as follows:


CollectionName.EXTEND(m,1);


where 'm' is one less that the needed number of elements. If the number is unknown, then extend the collection whenever a new element is required.

Collections, Part V

It is now time to introduce Varrays and Nested Tables into the discussion. Later, I will compare and contrast the three kinds of collections in more detail (in the sort of way which OCP examinations seem to love). As an introduction here, I will just list some of the ways they are like Associative Arrays, and some of the ways they differ.

Associative Arrays, Varrays, and Nested Tables all

- Require that the object be first defined somewhere in a TYPE statement

- Can be used in PL/SQL anywhere a built-in data type can be used

- Use subscript notation to identify individual items.

- Have the same methods

- Can be used for bulk inserts and selects

They differ in that

- Varrays and Nested Tables can be defined in the database (and thus become SQL types)

- Varrays and Nested Tables can be stored as columns in relational tables

- Varrays and Nested Tables can thus appear in ordinary SQL operations

- Varrays and Nested Tables cannot have a varchar2 index.

- Varrays must have an upper size limit specified in their TYPE statement, and this limit cannot be extended

- Varrays must always be 'dense'; there can be no gaps in their index structure (if v is a varray, you cannot have a situation where v(1), v(2), and v(4) exist, but there is no v(3)).

- Nested Tables must be created and loaded dense, but it is possible after the fact to delete internal elements. Thus, one must insert v(1), v(2), v(3), and v(4) into a nested table, but one can then delete v(3).

They also have one other difference with Associative Arrays, which I will illustrate by changing Listing1 to use a Varray instead of an Associative Array. The obvious way to go is to replace lines 8 and 9 with


8 type amortTableType is varray(362) of amortTableLine;


where all we do is replace the clause 'table' with 'varray(362) and remove the 'index by binary_integer'. When we then try to run thus we get


ERROR at line 1:
ORA-06531: Reference to uninitialized collection
ORA-06512: at line 32


Which reminds us that Varrays and Nested Tables have to first be 'initialized'. And we do that by calling the 'constructor function', which is provided to us as a function with the same name as the collection type. So, we add a new line (between lines 31 and 33) as follows:


30 v_monthlyRate := p_InterestRate / 1200;
31
32 amortTable := amortTableType();
33
34 amortTable(1).pmtdate := add_months(p_effectiveDate,1);


and try again:


ERROR at line 1:
ORA-06533: Subscript beyond count
ORA-06512: at line 34

For reference, line 34 is the first time we try to access a specific element:


34* amortTable(1).pmtdate := add_months(p_effectiveDate,1);


Let's pause a bit. Oracle refers to an unitialized collection as being "atomically null". To understand the significance of this, let us first ask a few related questions (how they are related will soon be made evident).

Suppose we have 2 varrays (or Nested Tables), A and B. What do the following assertions mean?

1) A = B

2) A < B

3) A IS NULL

Try to decide for yourselves before I continue.

For myself, I would answer (1) as follows: A = B if (and only if) for every element A(i), there is j such that A(i)=B(j). Some others might want to add, " and none of the A(i) or B(i) are null. Others might want to add that the two collections contain the same number of elements. And others might insist that the elements are stored in the exact same order.

I could make a really long list of possibilites around question (2). Some people might insist that all A(i) be <= B(i), with at least one element being greater. Me, I would take another tack, and define it such that for each A(i) there is an identical B(j), and there is at least one B(j) not found in A(i).

For question (3), I would like it to mean that every element of A is null. But Oracle does not adopt any of these. Instead, any attempt to compare two collections results in a PLS-0306 error: wrong number or types of elements in call to '=' (or '<' or '>' or whatever). The only comparison operator one can use against an entire collection is the IS NULL operator. Before a collection X is initialized, 'X IS NULL' evaluates to TRUE. Afterwards, it evaluates to FALSE. And before a collection X is initialized, one cannot refer to its individual elements. Afterwards, one can. So, to be atomicaly null, is to have meaningless elements, and to have the IS NULL operator return TRUE.

To rerturn to our immediate issue, we have not yet allocated any space in the amortTable collection. Let us now allocate some space, and get this block working:


add

32 amortTable := amortTableType();
33 amortTable.extend (p_loanTerm * 12 + 1);

we run it and get:

25-MAY-09 541.67 90.4 99909.6
25-JUN-09 541.18 90.89 99818.71
25-JUL-09 540.68 91.39 99727.32
25-AUG-09 540.19 91.88 99635.44


The trick was line 33, where we actually created 361 slots in the Varray.


The sharp-eyed observer will have noticed that everything has thus far been defined as a record. Since we will eventually want to have these types declared in the database, we need to make sure that this will work there also. So, let's try defining our record in the database:


1 create or replace type amortTableLine is record (
2 pmtDate date,
3 ToInterest number(12,2),
4 ToPrincipal number(12,2),
5 EndingBalance number(12,2)
6 );
7/


Warning: Type created with compilation errors.

SQL> show errors
Errors for TYPE AMORTTABLELINE:
1/24 PLS-00103: Encountered the symbol "RECORD" when expecting one of
the following:
array varray table object fixed varying opaque sparse
The symbol "object was inserted before "RECORD" to continue.


It appears we cannot define a record as a SQL type. Fortunately, there is an alternative; define an object instead:


1 create or replace type amortTableLine is object (
2 pmtDate date,
3 ToInterest number(12,2),
4 ToPrincipal number(12,2),
5 EndingBalance number(12,2)
6 );
7/

Type Created


Much as we would like to remain with anonymous PL/SQL blocks, we cannot, for if we try to change the type of the type from 'record' to 'object', we find


type amortTableLine is object (
*
ERROR at line 2:
ORA-06550: line 2, column 2:
PLS-00540: object not supported in this context.


So, let us keep amortTableLine defined as a SQL object, and try our routine. We find that it no longer works:


ERROR at line 1:
ORA-06530: Reference to uninitialized composite
ORA-06512: at line 29

SQL> List 1,29
1 DECLARE
2 type amortTableType is table of amortTableLine;
3
4 amortTable amortTabletype;
5
6 p_LoanAmount number(10,2);
7 p_InterestRate number;
8 p_loanTerm number(2,0);
9 p_effectiveDate date;
10 p_payment number(7,2);
11
12 v_pmtNumber number(4,0);
13 v_monthlyRate number;
14 v_totalInt number(13,2);
15 v_totalPmt number(13,2);
16 v_currentPrinc number(8,2);
17 BEGIN
18 p_LoanAmount := 10326.05;
19 p_InterestRate := 2.875;
20 p_loanTerm := 30;
21 p_effectiveDate := to_date('20090425','yyyymmdd');
22 p_payment := 62.22;
23 v_pmtNumber := 2;
24 v_monthlyRate := p_InterestRate / 1200;
25
26 amortTable := amortTableType();
27 amortTable.extend(362);
28
29 amortTable(1).pmtdate := add_months(p_effectiveDate,1);

SQL>


Which comes as a great surprise to someone is familiar with the various writings on the matter. All (well, almost all) suggest that the above should work.

To be fair, the examples usually look something like this:


1 declare
2 type v_type is table of varchar2(10);
3 v_tbl v_type;
4 begin
5 v_tbl := v_type();
6 v_tbl.extend(30);
7 v_tbl(1) := 'aaaaaa';
8 v_tbl(2) := 'bbbbb';
9 dbms_output.put_line(v_tbl(1)||v_tbl(2));
10 end;
11/

which, when run, gives the expected (and desired) results:


SQL> /
aaaaaabbbbb

PL/SQL procedure successfully completed.


The working example's lines 5 and 6 looks similar to our lines 26 and 27. So, what is wrong?

The answer is that the working example's nested table is a table of varchar2, which is something like a one-field record, while our failed example is a table of objects. And objects have to be initialized. We can break our working example by changing the type of the type from varchar2 to object. First, we must create an object that looks like the varchar2 above:


SQL> create type v_obj is object (dodah varchar2(10));
2 /

Type created.

Next, we revise the snippit accordingly:

1 declare
2 type v_type is table of v_obj;
3 v_tbl v_type;
4 begin
5 v_tbl := v_type();
6 v_tbl.extend(30);
7 v_tbl(1).dodah := 'aaaaaa';
8 v_tbl(2).dodah := 'bbbbb';
9 dbms_output.put_line(v_tbl(1).dodah||v_tbl(2).dodah);
10 end;
11/

and we get the ORA-06530 error.

To make the snippit work, we must recognize that v_tbl is a nested table of objects, and initialize those objects. The following works:


1 declare
2 type v_type is table of v_obj;
3 v_tbl v_type;
4 begin
5 v_tbl := v_type();
6 v_tbl.extend(30);
7 v_tbl(1) := v_obj(null);
8 v_tbl(1).dodah := 'aaaaaa';
9 v_tbl(2) := v_obj(null);
10 v_tbl(2).dodah := 'bbbbb';
11 dbms_output.put_line(v_tbl(1).dodah||v_tbl(2).dodah);
12 end;
13/

But it is less than satisfactory, mostly because while we can extend the nested table in a single statement, we are initializing the v_obj object one element at a time. We are able to take care of records in a single statement, can we do the same for objects? The answer is, almost. The solution is to call the object constructor and pass it to the Collection constructor. This creates and initializes one element of the Collection. Then, we can call the EXTEND method, passing it this well-formed first element. A glance at our snippit may help:


1 declare
2 type v_type is table of v_obj;
3 v_tbl v_type;
4 begin
5 v_tbl := v_type(v_obj(null));
6 v_tbl.extend(30,1);
7 v_tbl(1).dodah := 'aaaaaa';
8 v_tbl(2).dodah := 'bbbbbb';
9 dbms_output.put_line(v_tbl(1).dodah||v_tbl(2).dodah);
10 end;
11/


Pay particular attention to lines 5 and 6. Line 5 initializes the nested table and also initializes the first element of it. Then line 6 extends and initializes 30 more elements.

So, let us conclude by applying this to our mortgage amortization routine (where we have modified things a bit by putting the initiallization into the DECLARE section in line 4:


Listing 6
1 DECLARE
2 type amortTableType is table of amortTableLine;
3
4 amortTable amortTableType := amortTableType(amortTableLine(null,null,null,null));
5
6 p_LoanAmount number(10,2);
7 p_InterestRate number;
8 p_loanTerm number(2,0);
9 p_effectiveDate date;
10 p_payment number(7,2);
11
12 v_pmtNumber number(4,0);
13 v_monthlyRate number;
14 v_totalInt number(13,2);
15 v_totalPmt number(13,2);
16 v_currentPrinc number(8,2);
17 BEGIN
18
19 p_LoanAmount := 100000.00;
20 p_InterestRate := 6.5;
21 p_loanTerm := 30;
22 p_effectiveDate := to_date('20090425','yyyymmdd');
23 p_payment := 632.07;
24 v_pmtNumber := 2;
25 v_monthlyRate := p_InterestRate / 1200;
26
27 amortTable.extend(p_loanTerm * 12 + 1,1);
28
29 amortTable(1).pmtdate := add_months(p_effectiveDate,1);
30 amortTable(1).ToInterest := p_LoanAmount * v_monthlyRate;
31 amortTable(1).ToPrincipal := p_payment - amortTable(1).ToInterest;
32 amortTable(1).EndingBalance := p_LoanAmount - amortTable(1).ToPrincipal;
33 v_totalInt := amortTable(1).ToInterest;
34 v_totalPmt := p_payment;
35
36 WHILE (v_pmtNumber < p_loanTerm * 12) LOOP
37 amortTable(v_pmtNumber).pmtdate := add_months(amortTable(v_pmtNumber-1).pmtdate,1);
38 amortTable(v_pmtNumber).ToInterest := amortTable(v_pmtNumber - 1).EndingBalance * v_monthlyRate;
39 amortTable(v_pmtNumber).ToPrincipal := p_payment - amortTable(v_pmtNumber).ToInterest;
40 amortTable(v_pmtNumber).EndingBalance := amortTable(v_pmtNumber - 1).EndingBalance - amortTable(v_pmtNumber).ToPrincipal;
41 v_totalInt := v_totalInt + amortTable(v_pmtNumber).ToInterest;
42 v_totalPmt := v_totalPmt + p_payment;
43
44 v_pmtNumber := v_pmtNumber + 1;
45 END LOOP;
46 amortTable(v_pmtNumber).pmtdate := add_months(amortTable(v_pmtNumber-1).pmtdate,1);
47 amortTable(v_pmtNumber).ToInterest := amortTable(v_pmtNumber - 1).EndingBalance * v_monthlyRate;
48 amortTable(v_pmtNumber).ToPrincipal := amortTable(v_pmtNumber -1).EndingBalance;
49 amortTable(v_pmtNumber).EndingBalance := 0;
50 v_totalInt := v_totalInt + amortTable(v_pmtNumber).ToInterest;
51 v_totalPmt := v_totalPmt + amortTable(v_pmtNumber).ToPrincipal;
52
53 v_pmtNumber := 1;
54 WHILE (v_pmtNumber <= p_loanTerm * 12) LOOP
55 dbms_output.put_line(amortTable(v_pmtNumber).pmtdate||' '||
56 amortTable(v_pmtNumber).ToInterest||' '||
57 amortTable(v_pmtNumber).ToPrincipal||' '||
58 amortTable(v_pmtNumber).EndingBalance);
59 v_pmtNumber := v_pmtNumber + 1;
60 END LOOP;
61
62 END;
63/


So, let's summarize the housekeeping necessary when using Varrays and Nested Tables:

1) The collection must be defined in a TYPE statement somewhere (either in the database, a package definition, or in the PL/SQL block itself). The syntax is as follows:

Varray: TYPE varraytype IS VARRAY(n) OF datatype; where 'n' is the maximum size of the Varray

Nested Table: TYPE nestedtype IS TABLE OF datatype;

The datatype can be a built-in data type, a record type, or an object type.

2) Declare and initialize the collection. The syntax varies slightly, depending on whether the datatype is a built-in datatype, a record (which is just a generalization of a built-in datatype) or an object. The syntax for records:

Records: CollectionName varraytype := varraytype(data);
or CollectionName nestedtype := nestedtype(data);

Objects: CollectionName varraytype := varraytype(datatype(data));
or CollectionName nestedtype := nestedtype(datatype(data));

The data can be default data, or null.


3) If you know at the start how many elements will be needed, construct them at the beginning of your code as follows:

CollectionName.EXTEND(m,1);

where 'm' is one less that the needed number of elements. If the number is unknown, then extend the collection whenever a new element is required.

Sunday, May 10, 2009

Collections, Part IV (Associative Arrays as Parameters)

The code segments I have presented so far just illustrate the use of Associative Arrays in an anonymous PL/SQL block. Which would be more useful if PL/SQL were useful as a stand alone programming language. Which it is not. Instead, PL/SQL is primarily useful as a database interface. Which means functions, procedures, and packages. Which means, that we need a way for these arrays to get loaded and unloaded other than by being calculated anew each time the program executes. Which (finally) means either passing the array into or out of a routine as a parameter, or storing it in the database. In this posting, I am going to concentrate on the first of these: passing Associative Arrays as parameters of a stored program unit.

But first, I want to clarify some facts which often are glossed over. The first fact is that Oracle SQL and PL/SQL are executed by "different engines". The corollary is that the data types defined for the SQL engine are not identical with PL/SQL datatypes. Confusion arises, because most of the time the only datatypes used are the built-in ones (NUMBER, DATE, VARCHAR2, etc.) and these are available to both engines As a general rule, PL/SQL can use any SQL datatype (built in or user created), but the SQL engine cannot use PL/SQL datatypes.

As it happens, when I first suggest to someone that there is a distinction to be made between SQL data types and PL/SQL data types, their eyes glaze over. And I've had the same reaction to some Oracle employees. So it it probably best to turn to an example, and it makes most sense to me to start with something which works, and then show why making changes will cause it to fail. The example will be simply to convert Listing1 into two separate listings: a calculation routine and a print routine.

Here is the print routine:

        Listing 4a 

1 Create or Replace Package amortTablePkg is
2 type amortTableLine is record (
3 pmtDate date,
4 ToInterest number(12,2),
5 ToPrincipal number(12,2),
6 EndingBalance number(12,2)
7 );

8 type amortTableType is table of amortTableLine
9 index by binary_integer;

10 procedure amortTablePrt (p_amortTable amortTableType);
11 end amortTablePkg;
12 /

13 Create or Replace Package Body amortTablePkg is

14 procedure amortTablePrt (p_amortTable amortTableType) IS

15 v_pmtNumber number(4,0);
16 BEGIN
17 v_pmtNumber := 1;
18 WHILE (v_pmtNumber <= p_amortTable.count) LOOP
19 dbms_output.put_line(p_amortTable(v_pmtNumber).pmtdate||' '||
20 p_amortTable(v_pmtNumber).ToInterest||' '||
21 p_amortTable(v_pmtNumber).ToPrincipal||' '||
22 p_amortTable(v_pmtNumber).EndingBalance);
23 v_pmtNumber := v_pmtNumber + 1;
24 END LOOP;
25 END amortTablePrt;
26 END amortTablePkg;
27 /



My opinion is that package variables are the pons asinorum of PL/SQL, so I will go over this slowly. We have seen the "Type amortTableLine" (lines 2 - 6) and the "type amortTableType" (lines 8 - 9) declarations before. What is different is that they are part of a package specification. By doing this, they are exposed to everyone with privileges on the package. Thus, in the package body (line 14) we don't need to describe what an amortTableType is, because it has already be defined. These package datatypes are known to any piece of PL/SQL because they are defined in the database, however, they still are PL/SQL datatypes and not SQL datatypes.

The other interesting thing about the procedure definition (line 10 or 14) is that the input parameter is typed just as one would type a varchar2 or number parameter. This demonstrates that a PL/SQL type can be used as a PL/SQL parameter, providing that it is exposed as a package variable.

Let's look at a function which can create (but not print) the array:
     Listing4b

1 create or replace function amortTableFunc (
2 p_LoanAmount number,
3 p_InterestRate number,
4 p_loanTerm number,
5 p_effectiveDate date,
6 p_payment number
7 )
8 return amortTablePkg.amortTabletype
9 IS
10
11 amortTable amortTablePkg.amortTabletype;
12
13 v_pmtNumber number(4,0);
14 v_monthlyRate number;
15 v_totalInt number(13,2);
16 v_totalPmt number(13,2);
17 v_currentPrinc number(8,2);
18 BEGIN
19 v_pmtNumber := 2;
20 v_monthlyRate := p_InterestRate / 1200;
21
22 amortTable(1).pmtdate := add_months(p_effectiveDate,1);
23 amortTable(1).ToInterest := p_LoanAmount * v_monthlyRate;
24 amortTable(1).ToPrincipal := p_payment - amortTable(1).ToInterest;
25 amortTable(1).EndingBalance := p_LoanAmount - amortTable(1).ToPrincipal;
26 v_totalInt := amortTable(1).ToInterest;
27 v_totalPmt := p_payment;
28
29 WHILE (v_pmtNumber < p_loanTerm * 12) LOOP
30 amortTable(v_pmtNumber).pmtdate := add_months(amortTable(v_pmtNumber-1).pmtdate,1);
31 amortTable(v_pmtNumber).ToInterest := amortTable(v_pmtNumber - 1).EndingBalance * v_monthlyRate;
32 amortTable(v_pmtNumber).ToPrincipal := p_payment - amortTable(v_pmtNumber).ToInterest;
33 amortTable(v_pmtNumber).EndingBalance := amortTable(v_pmtNumber - 1).EndingBalance - amortTable(v_pmtNumber).ToPrincipal;
34 v_totalInt := v_totalInt + amortTable(v_pmtNumber).ToInterest;
35 v_totalPmt := v_totalPmt + p_payment;
36
37 v_pmtNumber := v_pmtNumber + 1;
38 END LOOP;
39 amortTable(v_pmtNumber).pmtdate := add_months(amortTable(v_pmtNumber-1).pmtdate,1);
40 amortTable(v_pmtNumber).ToInterest := amortTable(v_pmtNumber - 1).EndingBalance * v_monthlyRate;
41 amortTable(v_pmtNumber).ToPrincipal := amortTable(v_pmtNumber -1).EndingBalance;
42 amortTable(v_pmtNumber).EndingBalance := 0;
43 v_totalInt := v_totalInt + amortTable(v_pmtNumber).ToInterest;
44 v_totalPmt := v_totalPmt + amortTable(v_pmtNumber).ToPrincipal;
45
46 return amortTable;
47
48 END;
49/


So, we now have a procedure (amortTablePkg.amortTablePrt) which can print out the contents of an amortTablePkg.amortTableType array, and a function, amortTableFunc, which can return such an array. It would seem logical that if we put them together, we ought to end up where we were in back at the start of this Let's see if this actually works.

A SQL*Plus guru might try:

exec amortTablePkg.amortTablePrt(amortTableFunc( 100000, 6.5, 30, to_date('20090315','yyyymmdd'), 632.07))

With results (only the first few lines shown):


15-APR-09 541.67 90.4 99909.6
15-MAY-09 541.18 90.89 99818.71
15-JUN-09 540.68 91.39 99727.32
etc.


But, this is ugly. A better guru might try instead:


exec amortTablePkg.amortTablePrt( -
amortTableFunc( -
p_LoanAmount => 100000, -
p_InterestRate => 6.5, -
p_loanTerm => 30, -
p_effectiveDate => to_date('20090315','yyyymmdd'), -
p_payment => 632.07 -
) -
)


Which also works. But what are those dashes at the end of each line? And where is the semicolon?

The answer of course, is that 'exec' is not a SQL command, but a SQL*Plus command. So, the end-of-line character starts execution, not a semicolon. And the dashes are needed as line continuation characters. In fact, 'exec' is just a way for SQL*Plus to generate the following bit of PL/SQL:

     Listing 4c
1 begin
2 amortTablePkg.amortTablePrt(
3 amortTableFunc(
4 p_LoanAmount => 100000,
5 p_InterestRate => 6.5,
6 p_loanTerm => 30,
7 p_effectiveDate => to_date('20090315','yyyymmdd'),
8 p_payment => 632.07
9 )
10 );
11* end;


So, we actually have not executed any SQL at all. We can always try to execute the function by selecting from DUAL:


SQL> select amortTableFunc(
p_LoanAmount => 100000,
p_InterestRate => 6.5,
p_loanTerm => 30,
p_effectiveDate => to_date('20090315','yyyymmdd'),
p_payment => 632.07
)
from dual
/


and if we try this we find:


SQL> select amortTableFunc(
*
ERROR at line 1:
ORA-00902: invalid datatype


Which proves that PL/SQL datatypes cannot be used where a SQL datatype is required.

Let's create a new function, which returns a SQL datatype. At least, then it has a chance of being used in a SQL query:


create or replace function amorttableDemo (p_tbl amortTablePkg.amorttabletype)
return number
is
begin
return p_tbl(5).toInterest;
end;
/


What this function lacks in usefulness, it makes up by returning a SQL datatype. Let's prove that it actually works:


Declare
v_result number;
begin
v_result := amortTableDemo(amortTableFunc( 100000, 6.5, 30, to_date('20090315','yyyymmdd'), 632.07));
dbms_output.put_line(v_result);
end;
/

SQL>539.69

PL/SQL procedure successfully completed.


Indeed, it returns the number we expect. But, when we try to call it from SQL:


select amortTableDemo(amortTableFunc( 100000, 6.5, 30, to_date('20090315','yyyymmdd'), 632.07))
*
ERROR at line 1:
ORA-00902: invalid datatype


We still have a problem.

Clearly, we need something else, and those somethings are Varrays and Nested Tables.

Saturday, April 25, 2009

Collections, Part III

In this series, I am demonstrating the use of PL/SQL collections with an case study of a mortgage amortization table, with each line of the table being an element of the collection. In the previous installment, I used an 8 byte character string (the date of the payment, in yyyymmdd format) as the index to an associative array, and illustrated some of the problems with that. In it I used the PRIOR method to find the previous element (since subtracting 1 from the current index no longer worked). Having brought up the subject of these collection methods (or pseudo operators, or functions), I may as well list all of the available ones. I put this table together as a quick summary:


Method                    Description                           Array    Varray      Nest. Tbl

EXISTS(n) Returns TRUE if element 'n' No Yes[1],[2] Yes[1],[2]
exists, otherwise FALSE

COUNT Returns the current number of Yes[3] Yes[3] Yes[3]
elements in a collection

LIMIT Returns the maximum size of a Yes[5] Yes[4] Yes[5]
collection. Useful if one is
using a TYPE stored in the
Database, and one is not sure
what the limit for this Varray
may be.

FIRST Returns the first or last Yes[6] Yes[7] Yes[8]
LAST subscript value in a collection,
or NULL if the collection is empty.


PRIOR(sub) Returns the item whose subscript
NEXT(sub) is right before 'sub' (PRIOR) Yes[9] Yes[9] Yes[9]
or right after 'sub' (NEXT), or
NULL if there is no next or prior
element.

TRIM TRIM removes one element from No Yes[10] Yes[10]
TRIM(n) the (high) end of a collection.
TRIM(n) removes 'n' elements from
the end of a collection.

DELETE DELETE removes all elements Yes[10] Yes[10] Yes[10]
DELETE(n) a collection. DELETE(n) removes
DELETE(n,m) the element with subscript 'n' from
a collection. DELETE(n,m) deletes
element 'n' (low) through 'm' (high).
If 'm' is greater than 'n', nothing
is deleted, and no error is raised.

EXTEND Add a null element to the (high) No Yes[10] Yes{10]
EXTEND(n) end of the collection. EXTEND(n)
EXTEND(n,i) adds 'n' null elements to the end
of the collection. And EXTEND(n,i)
adds 'n' copies of the 'i-th' element.

--------------------------
[1] One way to skip around missing elements in a sparse nested table.

[2] Another way to avoid the SUBSCRIPT_OUTSIDE_LIMIT error.

[3] For Varrays, COUNT always equals LAST. With Nested Tables, if you TRIM
or DELETE an element, the COUNT is reduced accordingly. With Associative
Arrays, you don't even have to do a DELETE (since you can load an array up
sparsely). In other words, if you create an array V, and assign V(1), V(3),
V(5), the COUNT is 3.

[4] A Varray has an absolute maximum size, which is set by it's TYPE statement.
Thus, once a PL/SQL block starts executing, the value returned by LIMIT is known.
The number of elements actually in the Varray will often be less than this absolute
LIMIT, so COUNT will usually be less than LIMIT.

[5] But really only useful for Varrays, as Associative Arrays and Nested
tables always return NULL.

[6] If an Associative Array is indexed by a VARCHAR variable, what is first
and last depend on the character set and sort order. Since numeric
Associative Arrays can be entered with any subscript value, LAST can be
greater than COUNT, even if nothing has been deleted.

[7] For Varrays, FIRST always = 1 and LAST always = COUNT.

[8] For Nested Tables, FIRST can be greater than 1 if beginning elements are
deleted. If nothing has been deleted (in the middle) LAST = COUNT. If
something has been deleted, then LAST < COUNT

[9] As with FIRST and LAST, which of all the subscripts of the collection is
the previous or the subsequent may depend on the sort order of the character
set. For integer subscripts, there is no such dependency, but exactly what
subscripts are in the collection can cause some surprises. For example, if
one loads up a PL/SQL table with subscripts 2, 8, 3, 12, 15, 19, 1 (in that
order), then PRIOR(12) is the element whose subscript is 8, and not the
element whose subscript is 3.

[10] The TRIM, DELETE, and EXTEND methods are complicated, especially when
one uses them in tandem. A complete discussion of these is being defered to a
later posting.


Well, let's now look at some more of these methods, as applied to Associative Arrays. We are adding an additional complication to the requirements of the previous post: We only want the first value, the year-ending values, and the very last value. So, we do this the quick-and-dirty way: we take what we already have, and then go and delete what we no longer want.

If not very elegant, it does provide an excuse for using the DELETE method:


Listing-3

1 alter session set nls_date_format = 'yyyymmdd';
2
3 DECLARE
4 type amortTableLine is record (
5 ToInterest number(12,2),
6 ToPrincipal number(12,2),
7 EndingBalance number(12,2)
8 );
9 type amortTableType is table of amortTableLine
10 index by varchar2(9);
11
12 amortTable amortTabletype;
13
14 p_LoanAmount number(10,2);
15 p_InterestRate number;
16 p_loanTerm number(2,0);
17 p_effectiveDate date;
18 p_payment number(7,2);
19 v_pmtNumber number;
20
21 v_pmtDate date;
22 v_pmtDateIdx varchar2(9); /* Because we are using it as an index */
23 v_pmtDateJan varchar2(9); /* Lower range of dates we are going to*/
24 /* delete (typically, will be in January*/
25 v_pmtDateNov varchar2(9); /* November of the year of v_pmtDateJan*/
26 v_monthlyRate number;
27 v_totalInt number(13,2);
28 v_totalPmt number(13,2);
29 v_currentPrinc number(8,2);
30 BEGIN
31 p_LoanAmount := 100000;
32 p_InterestRate := 6.5;
33 p_loanTerm := 30;
34 p_effectiveDate := to_date('20090315','yyyymmdd');
35 p_payment := 632.07;
36 v_pmtDateIdx := to_char(add_months(p_effectiveDate,1)); /* Deliberately using the session's date mask */
37 v_monthlyRate := p_InterestRate / 1200;
38
39 amortTable(v_pmtDateIdx).ToInterest := p_LoanAmount * v_monthlyRate;
40 amortTable(v_pmtDateIdx).ToPrincipal := p_payment - amortTable(v_pmtDateIdx).ToInterest;
41 amortTable(v_pmtDateIdx).EndingBalance := p_LoanAmount - amortTable(v_pmtDateIdx).ToPrincipal;
42 v_totalInt := amortTable(v_pmtDateIdx).ToInterest;
43 v_totalPmt := p_payment;
44 v_pmtDate := add_months(p_effectiveDate,2);
45 v_pmtDateIdx := to_char(v_pmtDate);
46
47 WHILE (v_pmtDate < add_months(p_effectiveDate,p_loanTerm * 12)) LOOP
48 amortTable(v_pmtDateIdx).ToInterest := amortTable(amortTable.prior(v_pmtDateIdx)).EndingBalance * v_monthlyRate;
49 amortTable(v_pmtDateIdx).ToPrincipal := p_payment - amortTable(v_pmtDateIdx).ToInterest;
50 amortTable(v_pmtDateIdx).EndingBalance := amortTable(amortTable.prior(v_pmtDateIdx)).EndingBalance - amortTable(v_pmtDateIdx).ToPrincipal;
51 v_totalInt := v_totalInt + amortTable(v_pmtDateIdx).ToInterest;
52 v_totalPmt := v_totalPmt + p_payment;
53 v_pmtDate := add_months(v_pmtDate,1);
54 v_pmtDateIdx := to_char(v_pmtDate);
55 END LOOP;
56 amortTable(v_pmtDateIdx).ToInterest := amortTable(amortTable.prior(v_pmtDateIdx)).EndingBalance * v_monthlyRate;
57 amortTable(v_pmtDateIdx).ToPrincipal := amortTable(amortTable.prior(v_pmtDateIdx)).EndingBalance;
58 amortTable(v_pmtDateIdx).EndingBalance := 0;
59 v_totalInt := v_totalInt + amortTable(v_pmtDateIdx).ToInterest;
60 v_totalPmt := v_totalPmt + amortTable(v_pmtDateIdx).ToPrincipal + amortTable(v_pmtDateIdx).ToInterest;
61
62 /*Just to Illustrate the DELETE method, we will here delete */
63 /*Everything except the 1st line, the last line, and the */
64 /*end of the month. */
65 v_pmtDateJan := amortTable.next(amortTable.first);
66 v_pmtDateNov := substr(v_pmtDateJan,1,4)||'11'||substr(v_pmtDateJan,7,2);
67 WHILE (v_pmtDateNov <= amortTable.last) LOOP
68 amortTable.delete(v_pmtDateJan, v_pmtDateNov);
69 v_pmtDateJan := amortTable.next(amortTable.next(v_pmtDateNov));
70 v_pmtDateNov := substr(v_pmtDateJan,1,4)||'11'||substr(v_pmtDateJan,7,2);
71 END LOOP;
72 v_pmtDateNov := amortTable.prior(amortTable.last);
73 amortTable.delete(v_pmtDateJan, v_pmtDateNov);
74
75 v_pmtDateIdx := amortTable.first;
76 WHILE (v_pmtDateIdx <= amortTable.last) LOOP
77 dbms_output.put_line(v_pmtDateIdx||' '||
78 amortTable(v_pmtDateIdx).ToInterest||' '||
79 amortTable(v_pmtDateIdx).ToPrincipal||' '||
80 amortTable(v_pmtDateIdx).EndingBalance);
81 v_pmtDateIdx := amortTable.next(v_pmtDateIdx);
82 END LOOP;
83
84* END;


Lines 23-25: The plan is to use the DELETE(n,m) form of the delete function.
Since our index is a character string of the form 'yyyymmdd', 'n' will have
usually have the form 'yyyy01dd' (will be in January, except for the first set)
and 'm' will usually be of the form 'yyyy11dd' (will be in November) except
for the final year, where it could be any month.

Line 66: Here we start to have something different from the last posting. We
want to keep the first subscript, and start deleting from the second to the
end of the year.

Lines 68-72: We have set up our loop variables, and use them here.

Here is the output:

20090415 541.67 90.4 99909.6
20091215 537.67 94.4 99168.51
20101215 531.35 100.72 97995.06
20111215 524.61 107.46 96743.03
20121215 517.41 114.66 95407.15
20131215 509.73 122.34 93981.82
20141215 501.54 130.53 92461.03
20151215 492.8 139.27 90838.38
20161215 483.47 148.6 89107.05
20171215 473.52 158.55 87259.77
20181215 462.9 169.17 85288.79
20191215 451.57 180.5 83185.81
20201215 439.48 192.59 80941.99
20211215 426.58 205.49 78547.89
20221215 412.82 219.25 75993.44
20231215 398.14 233.93 73267.96
20241215 382.47 249.6 70359.91
20251215 365.75 266.32 67257.11
20261215 347.92 284.15 63946.51
20271215 328.89 303.18 60414.21
20281215 308.58 323.49 56645.32
20291215 286.92 345.15 52624.03
20301215 263.8 368.27 48333.43
20311215 239.14 392.93 43755.48
20321215 212.82 419.25 38870.92
20331215 184.74 447.33 33659.22
20341215 154.79 477.28 28098.51
20351215 122.82 509.25 22165.38
20361215 88.72 543.35 15834.9
20371215 52.33 579.74 9080.48
20381215 13.5 618.57 1873.67
20390315 3.39 626.46 0

Thursday, April 16, 2009

Collections, Part II

In the last installment I went over the high-level specifications for our case study (a mortgage amortization schedule) using PL/SQL collections. It is time to actually view some code. So, without further ado, here is Listing-1:

          Listing-1

1 DECLARE
2 type amortTableLine is record (
3 pmtDate date,
4 ToInterest number(12,2),
5 ToPrincipal number(12,2),
6 EndingBalance number(12,2)
7 );
8 type amortTableType is table of amortTableLine
9 index by binary_integer;
10
11 amortTable amortTabletype;
12
13 p_LoanAmount number(10,2);
14 p_InterestRate number;
15 p_loanTerm number(2,0);
16 p_effectiveDate date;
17 p_payment number(7,2);
18
19 v_pmtNumber number(4,0);
20 v_monthlyRate number;
21 v_totalInt number(13,2);
22 v_totalPmt number(13,2);
23 v_currentPrinc number(8,2);
24 BEGIN
25 p_LoanAmount := 100000;
26 p_InterestRate := 6.5;
27 p_loanTerm := 30;
28 p_effectiveDate := to_date('20090315','yyyymmdd');
29 p_payment := 632.07;
30 v_pmtNumber := 2;
31 v_monthlyRate := p_InterestRate / 1200;
32
33 amortTable(1).pmtdate := add_months(p_effectiveDate,1);
34 amortTable(1).ToInterest := p_LoanAmount * v_monthlyRate;
35 amortTable(1).ToPrincipal := p_payment - amortTable(1).ToInterest;
36 amortTable(1).EndingBalance := p_LoanAmount - amortTable(1).ToPrincipal;
37 v_totalInt := amortTable(1).ToInterest;
38 v_totalPmt := p_payment;
39
40 WHILE (v_pmtNumber <>

What Listing-1 does is to put each line of the amortization table into a separate
Associative Array record. Some comments on the program:

Lines 2 - 11: Here we define the array in which we will store each line of the
amortization table. This is done in the standard way, by first (lines 2 - 6)
defining a record type (to define the fields of the line), then an array type
(lines 8-9) and finally (line 11) declaring the array (amortTable) as an instance
of that array type. This could be done in one step if Oracle supplied a suitable
built-in data type, but it doesn't.

Lines 13 - 23: Now we define the variables to be used in the routine. Although
this is going to be run as an anonymous PL/SQL block, we define 5 variables to sort
of be input parameters, and indicate as much by prefixing them with 'p_'

Lines 25 - 331: Here we initialize those variables which require initialization.
Yes, this could have been combined with lines 13 - 23.

Lines 33 - 36: The first line of the amortization table requires special treatment,
so we do that here.

Lines 37 & 38: We are going to calculate the total of payments and the total
interest paid by accumulating the values, so we start the process here. Back when
I was more of a mathematician, I would have skipped this and calculated them at the
end. Since then, I have been perverted by the Finance people, and insist on doing
these calculations the hard way.

Lines 40 - 49: The body of the loop where all but the last payment are calculated.

Lines 50 - 55: Here we calculate the final line in the table.

Lines 57 - 67: And here we print out the results.

Remember the Sherlock Holmes story about the barking dog? The dog didn't bark. And
that was the key to the mystery.

What is most significant about using associative arrays with a dense integer
subscript is what does not have to be done. There is no need to do anything special
to create a row in the array; one just adds it. There is no need to do anything
special to navigate the array, one just increments the index.

The results of running Listing-1 are as follows:













MonthInterestPrincipalBalance
15-APR-09 541.67 90.40 99909.60
15-MAY-09 541.18 90.89 99818.71
15-JUN-09 540.68 91.39 99727.32
15-JUL-09 540.19 91.88 99635.44
15-AUG-09 539.69 92.38 99543.06
....
....
....
15-JAN-39 10.15 621.92 1251.75
15-FEB-39 6.78 625.29 626.46
15-MAR-39 3.39 626.46 0


Nothing surprising here. So, let's now try something new. We can now index a PL/SQL table with a VARCHAR2 index (which is why they are now called 'associative arrays'). Let us try to do this by using the payment date as the index, while making as few changes as necessary to our program. The result is Listing-2.

Listing-2

1 DECLARE
2 type amortTableLine is record (
3 ToInterest number(12,2), /* No more payment date field in the record */
4 ToPrincipal number(12,2),
5 EndingBalance number(12,2)
6 );
7 type amortTableType is table of amortTableLine
8 index by varchar2(9); /* This will become the payment date, but it */
9 /* cannot be a date type */
10 amortTable amortTabletype;
11
12 p_LoanAmount number(10,2);
13 p_InterestRate number;
14 p_loanTerm number(2,0);
15 p_effectiveDate date;
16 p_payment number(7,2);
17 v_pmtNumber number;
18
19 v_pmtDate date; /* We track the payment date as a date (to use date arithmetic) */
20 v_pmtDateIdx varchar2(9); /* Because we are using it as an index */
21 v_monthlyRate number;
22 v_totalInt number(13,2);
23 v_totalPmt number(13,2);
24 v_currentPrinc number(8,2);
25 BEGIN
26 p_LoanAmount := 100000;
27 p_InterestRate := 6.5;
28 p_loanTerm := 30;
29 p_effectiveDate := to_date('20090315','yyyymmdd');
30 p_payment := 632.07;
31 v_pmtDateIdx := to_char(add_months(p_effectiveDate,1)); /* Deliberately using the session's date mask */
32 v_monthlyRate := p_InterestRate / 1200;
33
34 amortTable(v_pmtDateIdx).ToInterest := p_LoanAmount * v_monthlyRate;
35 amortTable(v_pmtDateIdx).ToPrincipal := p_payment - amortTable(v_pmtDateIdx).ToInterest;
36 amortTable(v_pmtDateIdx).EndingBalance := p_LoanAmount - amortTable(v_pmtDateIdx).ToPrincipal;
37 v_totalInt := amortTable(v_pmtDateIdx).ToInterest;
38 v_totalPmt := p_payment;
39 v_pmtDate := add_months(p_effectiveDate,2);
40 v_pmtDateIdx := to_char(v_pmtDate);
41
42 WHILE (v_pmtDate <>

The first change is at Line 8, where the 'index by' variable becomes a varchar2(9). This induces a change at Line 3, where we take the payment date out of the record (we could have left it in, but the result would have been very confusing.

But this leads to a problem: how do we generate new indexes? Back in the old days, we would have kept the date in a numeric variable, and written routines to increment it as needed (and hoped we didn't have to cross the year 2100 on February 29th!). But this is Oracle, and we have a real Date Type, with Real Date Arithmetic. So, we keep the payment date variable as a date, and use standard date arithmetic on it. Then, we convert the incremented date to the varchar2 variable we use as the actual index (this can be seen in lines 39 - 40 and 48 - 49).

Lines 43 and 45 illustrate the next problem. With an integer subscript, the predecessor of row 'I' was simply 'I - 1'. To do this here would require us to subtract 1 from a varchar2 variable, which is an exercise in frustration. But PL/SQL is a modern, somewhat object-oriented language. So we have a 'prior' method (or function), which returns the previous row, as needed.

So, now when we print out the results we get the following:














MonthInterestPrincipalBalance
15-APR-09 541.67 90.4 99909.6
15-MAY-09 541.18 90.89 99818.71
15-JUN-09 541.18 90.89 99818.71
15-JUL-09 541.18 90.89 99818.71
15-AUG-09 541.18 90.89 99818.71
....
....
....
15-JAN-39 524.61 107.46 96743.03
15-FEB-39 524.61 107.46 96743.03
15-MAR-39 96957.37
0


Which is not even close to what we want.

Here is what went wrong: when we calculated the second row, we calculated the index as May 15, 2009. This was converted using the default date mask to '15-MAY-09'. The prior function returned the loan balance as of the only other index present, which was '15-APR-09'.

Next we calculated the third row. Now the index is '15-JUN-09' The prior function grabbed the loan balance for the index which sorts just before '15-JUN-09'. This is not '15-MAY-09', but '15-APR-09' all over again. And the same thing happened with July 15, 2009. Indeed, it is not until we get to August that we again reference '15-MAY-09', by which time we were intending to reference '15-JUL-09'.

The fix is set the default date mask to something which coverts to a varchar2 value which sorts in the order we need:

SQL> alter session set nls_date_format = 'yyyymmdd';

Then, when we run the program, we get what we want:














MonthInterestPrincipalBalance
15-APR-09 541.67 90.4 99909.6
15-MAY-09 541.18 90.89 99818.71
15-JUN-09 540.68 91.39 99727.32
15-JUL-09 540.19 91.88 99635.44
15-AUG-09 539.69 92.38 99543.06
....
....
....
15-JAN-39 10.15 621.92 1251.75
15-FEB-39 6.78 625.29 626.46
15-MAR-39 3.39 626.46 0


The alternative is to obey Singer's first law of Oracle Date Types:

"Never convert (implicitly or explicitly) an Oracle Date to a number or a character string without wrapping it explicitly in a TO_CHAR function with a format mask you supply"

But this did nicely illustrate what can go wrong with a varchar2 index on an associative array where the sort order is not what one expects.

Monday, April 13, 2009

Collections, Part I

By "Collections", Oracle means the use of Varrays, Nested Tables, or Associative Arrays (formerly Index-By Tables (formerly PL/SQL Tables)) in PL/SQL. There are many examples extant, most of which use an (almost) meaningless code fragment to illustrate one, or perhaps two, points. My goal here is to illustrate the use of Collections through a meaningful example. Which is more complicated than one might assume at first glance.

For the first question one must ask when considering the appropriateness of an array in a PL/SQL program, is, "why is a relational table not being used instead"? Many of the examples I have come across in existing programs appear to have been written because the programmer was afraid to write a join. Indeed, when I first was introduced to PL/SQL, my immediate thought was "now I won't have to become a SQL expert." Today, I argue that any proper use of collections should be to do something which cannot be done with SQL.

In 8i and 9i, one standard use was for bulk fetches or inserts. Today, Oracle will automatically optimize DML to use bulk fetches or inserts (the problem is to prevent it when one-row-at-a-time is more efficient.) So, for this series of posts, I want to illustrate other uses for them.

I am going to develop a case study, specifically, that of a mortgage amortization schedule. As an illustration I am going to use what Lending Tree was displaying at the time I went looking. The exact numbers are not important at the moment; here I just want to show the basics of the calculations (only the first 5 and the last two lines are shown):


Amortization Schedule



















































Month Interest Principal Balance
Mar, 2009 $541.67 $90.40 $99,909.60
Apr, 2009 $541.18 $90.89 $99,818.71
May, 2009 $540.68 $91.38 $99,727.32
Jun, 2009 $540.19 $91.88 $99,635.45
Jul, 2009 $539.69 $92.38 $99,543.07
Jan, 2039 $6.79 $625.28 $628.66
Feb, 2039 $3.41 $628.66 $0.00



Elsewhere, we are told that the annual interest rate is 6.5%, the loan amount is $100,000, and the monthly payment is $632.07. The table then gives us the date of each payment, the amount of the payment allocated to interest, the amount allocated to principal reduction, and the loan balance after the payment is made.

If we were given this as a programming assignment, these would be the proverbial "napkin specs." Before we can proceed, we must bring them up to at least "one-pager" status. We can start by looking at how each column gets calculated:

Month: [initial] = Loan Origination Date + one month
[subsequent] = Previous value + one month

Balance [initial] = Loan Amount - Current Principal
[subsequent] = Previous value - Current Principal

Principal = Payment - Current Interest

Interest: = Previous Balance * I
where I = (Annual Interest Rate) / 1200

To make sure we really understand what is happening, let's look at a few lines. The monthly interest rate is 6.5/1200, or 0.005416667. Multiply this by the original loan balance, and we get the interest for the first month: $541.67. Subtracting this from the payment ($632.07) we find that the principal reduction is $90.40. Subtract $90.40 from $100,000 and we get the balance after 1 month, $99,909.60.

Just to make sure that this isn't a fluke, let us repeat this with the third payment. The balance after two payments is $99,818.71. Multiplying this by 0.005416667 we get the interest for this payment, $540.68. Subtracting this from $632.07 we get the amount devoted to principal, $91.39. Subtract this from last month's balance, and we get the new loan balance, $99.727.32. And the published chart has $91.38 for this month's principal reduction. Payment 3 and we are already out of synch.

The reader who has stayed with me this far with the numbers will have already noted a number of other places where the published chart is wrong. What is happening is that the program computing the numbers is not rounding at each step. The actual payment used is $632.0680238 (which we see rounded as $632.07). The actual interest charge for the first payment is $541.666667 and the balance after one payment is actually $99,909.59868, the balance after two payments $99,818.70765, and the balance after three $99,727.32429. Since the unrounded interest is 540.68467 and the unrounded principal reduction is $91.38335 we see that things really are in balance.

It is, of course, illegal for a financial institution to actually use the example numbers. This gives us another specification for our routine: make sure all intermediate results are properly rounded.

And there is yet another item to be added. Because no intermediate results were rounded, the last line happens to balance perfectly. In general this will not be the case, instead, the last payment will be a little larger or smaller than the others. So, the finished routine will have to adjust the last payment.

Before any specification can be considered complete, the boundary conditions must be examined. We have already examined the case where the final payment must be examined. What happens if the specified payment is less than the interest due? At first glance, all seems well, the principal reduction will become negative and get added to the outstanding principal. But there is a problem here: if we are not careful, we could end up breaking the law.

The "law" allows for two approaches to this problem, where the payment is less than the interest due. One approach, known as the "actuarial method" (because it is more mathematically elegant) allows for the interest to be added to the principal, and, in the next period, for interest to be charged on this interest. The other, termed the U.S. Rule, has the interest added to the principal, but no interest is then collected on this interest. Which rule applies depends on the loan documents. For simplicity, I am going to be applying the actuarial rule, but this complication must be mentioned at least in passing.

Already at the end of the first post, and not a line of PL/SQL has yet been written. Expect quite a bit of it in the next post.