Automatic Oracle Sequence Regeneration
Most of our Oracle tables have a surrogate primary key (SPK) whose value is derived in the normal way from an Oracle sequence and inserted via a trigger tied to the “BEFORE INSERT” condition.
It takes some effort to maintain all the sequences and triggers. For every new table an associated sequence and trigger must be created, and if any sequence or trigger is accidentally dropped during development the development system will eventually fail in an unobvious way. We also frequently copy table data between systems – eg from the live to the staging system, so that candidate releases can be tested on live data. When data is copied, all the relevant sequences must be reset to be higher than the new maximum SPK value.
We therefore developed a PL/SQL function ResetSequences provided as part of an Oracle package that would
- Reset the current value of each sequence to be the maximum value of the SPK in the associated table
- Detect and recreate a missing sequence or trigger (or both)
ResetSequences takes as parameters the name of the tablespace in which to operate and either a table prefix which all affected tables should match or a comma separated list of table names. If both the table prefix and list of names are null, it attempts to reset sequences for every table in the given tablespace.
ResetSequences returns as a CLOB a text list of what sequences it reset or created, and what triggers it created.
It can be called from SQL like this:
SELECT APEXTRAS_RESET_TRIG_SEQ.ResetSequences(‘TABLESPACE’,’TABLE_PREFIX’,’TABLE_LIST’) FROM dual
Or from a PL/SQL procedure:
DECLARE
l_return CLOB;
BEGIN
l_return := APEXTRAS_RESET_TRIG_SEQ.ResetSequences(p_tablespace => 'TABLESPACE',
p_tableprefix => 'TABLE_PREFIX',
p_tablelist => 'TABLE_LIST');
END;
Tested with Oracle 10.2g.
The usual caveats apply about using ResetSequences on a production system – test extensively first. In particular, ResetSequences assumes that any primary key on a numeric column should be derived from a sequence and associated trigger and will create both if they don’t exist. Don’t use it on tables for which that is not true.
Technical Details
The Oracle views all_constraints, all_ind_columns, all_triggers, all_trigger_cols and all_dependencies contain the required information and when joined and filtered give us a list of tables, primary key columns, sequences, and triggers.
all_constraints filtered on all_constraints.constraint_type = ‘P’ gives the name of tables with primary keys.
all_ind_columns gives the referenced column name.
all_dependencies, suitably filtered, gives the names of the trigger and sequence.
If the trigger and sequence exist then the next value of the sequence is reset to be 1 greater than the current maximum value of the SPK column (the sequence is not dropped and recreated). If either or both of the sequence or trigger are missing, they are created from scratch.
The name given to newly created sequences and triggers is based on the table name minus the table prefix concatenated with the SPK column name. If the table and column names start with more than 3 characters in common then only the column name is used.
Eg for a table PREFIX_TEST_TABLE with SPK column TEST_TABLE_ID, the trigger and sequence would both be called TEST_TABLE_ID.
For a table PREFIX_TEST_TABLE with SPK column TAB_ID the trigger and sequence would both be called TEST_TABLE_TAB_ID.
DDL issued by ResetSequences
The parts of the DDL statements inside square brackets ([]) are each replaced by the relevant parameter or calculated number before execution.
Sequence Creation
CREATE SEQUENCE [TABLESPACE.SEQUENCE_NAME] START WITH [START_VALUE] INCREMENT BY 1 CACHE 20
Trigger Creation
CREATE OR REPLACE TRIGGER [TABLESPACE.TRIGGER_NAME] BEFORE INSERT ON [TABLE_NAME] FOR EACH ROW WHEN (NEW.[COLUMN_NAME] IS NULL) BEGIN SELECT [SEQUENCE_NAME.NEXTVAL] INTO :NEW.[COLUMN_NAME] FROM DUAL; END;
Sequence Resetting
This must be done with a series of commands:
ALTER SEQUENCE [TABLESPACE.SEQUENCE_NAME] INCREMENT BY 1 NOCACHE;
SELECT [TABLESPACE.SEQUENCE_NAME.NEXTVAL] FROM DUAL INTO nextval;
IF (nextval < desired_value) THEN
IF (nextval < (desired_value - 1)) THEN
ALTER SEQUENCE [TABLESPACE.SEQUENCE_NAME] INCREMENT BY [desired_value - nextval - 1] MINVALUE 1
SELECT [TABLESPACE.SEQUENCE_NAME.NEXTVAL] FROM DUAL INTO nextval;
ALTER SEQUENCE [TABLESPACE.SEQUENCE_NAME] INCREMENT BY 1';
ELSIF (nextval = (desired_value - 1)) THEN
SELECT [TABLESPACE.SEQUENCE_NAME.NEXTVAL] FROM DUAL INTO nextval;
END IF;
END IF;
ALTER SEQUENCE [TABLESPACE.SEQUENCE_NAME] INCREMENT BY 1 CACHE [ORIGINAL_CACHE_SIZE];
Using the Oracle scheduler in APEX
We often give APEX users the opportunity to have displayed reports emailed to them as PDF attachments. It may take minutes of processing to produce such an email, and it would be completely unacceptable to expect a web user to wait for the processing to complete. The solution is to use the Oracle scheduler, available in versions 10g and above. The call from APEX to schedule the report then takes at most a few seconds.
The Oracle scheduler is designed to run pre-defined scheduler jobs, which are PL/SQL procedures or anonymous blocks of PL/SQL. However all our report production is done by procedures that are defined as part of PL/SQL packages, and we are constantly writing new ones. We didn’t want the overhead of defining and maintaining a separate job for each procedure. Instead, we wrote a generalized interface to the scheduler that allows the creation and scheduling of a new job on demand. From APEX, an emailed report can then be scheduled for immediate or future production from a single PL/SQL call.
More details:
Calling interface
For each procedure that is to be called by the scheduler, we create a wrapper procedure with the same parameters. The wrapper procedure is the one that is called from APEX. We also declare a type arglist_t as a table of records to hold the details of each parameter.
In the wrapper procedure we assign the name, type, default value, and actual value of each parameter to elements of a variable of type arglist_t, then call the single procedure ScheduleSPJob.
The generalized form of the wrapper procedure.
A specific example – scheduling a call to a procedure ResendEmail(p_email_id IN NUMBER, p_recipients IN VARCHAR2).
Ensuring scheduler jobs are enabled
Scheduler jobs with arguments must be created as disabled, then be explicitly enabled after the arguments have been set. Our PL/SQL scheduling procedure ScheduleSPJob tries to enable each job after it is created. But if many jobs are scheduled in a short time – eg from a loop that emails several users – it is likely that some of them will not have been been available for enabling at the time they were initially created. They must then be enabled by a separate process call. We use an On Demand Application Process that calls a PL/SQL procedure to do this: APEXTRAS_SCHEDULER.EnableJobsByCreator(‘APEX_PUBLIC_USER’);. In our system, we have found it is sufficient to call this On Demand process once from an Apex Page Process immediately after the Page Process that creates the scheduled jobs.
The current state of all scheduled jobs can be seen in the view user_scheduler_jobs. Jobs created from Apex have JOB_CREATOR set to ‘APEX_PUBLIC_USER’.
PL/PDF version 2.1.0
We use the PL/PDF package to produce PDF documents that can be mailed on request to APEX web site users. The PDF contents will usually be a version of something that was displayed on screen, for example a list of answers to a questionnaire. Version 2.1.0 of PL/PDF has recently become available and we’ve just upgraded our development and production systems.
PL/PDF does quite a good job of converting simple text or tabular data into reasonable-looking PDFs, but it
takes some work to produce more complicated documents, and it lacks the ability to convert an HTML document directly into a PDF.
Pound sign (£)
The FAQs on the PL/PDF web site state that the default decoding cp1252 does not include a pound sign, but that one can be displayed using unistr(‘0a3’) or chr(49827).
We found those characters display a capital L with a line crossing it, rather than a real pound sign. I wrote a procedure to print every Unicode character just to make sure there wasn’t some other encoding for a real pound sign – but there wasn’t. Instead we had to use an TrueType embedded font rather than one of the built-in fonts.
TrueType Fonts
One of the least friendly aspects of previous PL/PDF versions was the procedure for using TrueType embedded fonts, which involved running a command line program to generate an Adobe Font Metrics (AFM) file, then using SQL*Loader with custom control files to load two tables, and finally running a stored procedure. It took me most of a day’s work to do this for a single TrueType font file, and the resulting embedded font was unusable – it produced a blank document. I never had the time or the inclination to try again. Version 2.1.0 has fixed that – it includes a graphical application (PL/PDF v2.1.0 TTF Loader) that does all the work automatically. In 5 minutes I managed to install all 4 Arial TrueType fonts in a usable form. I had to specify utf16 encoding and unicode in the graphical interface (our Oracle NLS_CHARACTERSET=AL32UTF8, NLS_NCHAR_CHARACTERSET=AL16UTF16).
It’s not immediately obvious how to use the embedded TrueType fonts. They first have to be loaded with a user-defined alias using the numerical value of an ID field in a PL/PDF table; plpdf_ttf_add.ID. The value of the ID is derived from an Oracle sequence so it can vary with the server it is running on – in order to avoid hard-coding it we had to write a procedure to find the ID corresponding to a given font name.
Once loaded, TrueType fonts can be referred to by their aliases. However, the different font styles are each in different TrueType files, and it seems they must therefore have different font names when used in PL/PDF, so the built-in PL/PDF method for specifying font styles won’t work. Eg to print in ArialMT bold, instead of specifying font ‘ArialMT’ and style ‘B’, you need to specify font ‘Arial-BoldMT’ and style NULL.
Other PL/SQL anomalies
These were seen in earlier versions – we’ve not yet had time to retest with version 2.1.0:
- The left margin setting procedure appears not to work; to produce a left indent we have to add spaces to the output text.
- The expected width of text to be printed as reported by the GetTextWidth function is sometimes wrong (by a constant factor) – we need to declare a variable to hold an empirically determined “fudge factor” that we multiply every result from GetTextWidth by.
SQLPLUS fails to start with error ORA-12560 or ORA-12557
SQL*Plus would not run on one of our Windows Oracle/APEX servers, although it was running fine on all the other apparently identical servers. ORACLE_HOME and ORACLE_SID were correctly set.
We were getting one or other of these errors:
ORA-12560: TNS:protocol adapter error
ORA-12557: TNS:protocol adapter not loadable
The problem was the Windows “path” environment variable. The \bin directory associated with each of the two Oracle homes was correctly in the path, but in the wrong order. The path associated with the Oracle/Apache HTTP server preceeded the path associated with the Oracle database.
Ie we had: path=C:\oracle\product\10.2.0\http_1\bin;C:\oracle\product\10.2.0\db_1\bin
when we should have had: path=C:\oracle\product\10.2.0\db_1\bin;C:\oracle\product\10.2.0\http_1\bin
(ignoring all the other non-Oracle elements of the path).
Presumably SQL*Plus looks for the configuration files LISTENER.ORA, SQLNET.ORA and TNSNAMES.ORA in the first Oracle home it finds.
With the path correctly set, SQL*Plus would start correctly without environment variables ORACLE_HOME or ORACLE_SID explicitly set – probably because these were also set in the registry keys
HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_OraDb10g_home1\ORACLE_SID
HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_OraDb10g_home1\ORACLE_HOME
Software Versions:
Oracle 10.2
SQL*Plus 10.1.0.3.0
Windows Server 2003
Integrating YubiKey two-factor authentication with APEX login
To integrate the low-cost YubiKey device we use a custom APEX authentication scheme in which we manage user names and passwords in a table under our control.
To enable this you need to enter “return name_of_your_custom_authentication_function” in the setup screen for the APEX authentication scheme in the “Login Processing/Authentication Function” box. The APEX popup help for this field gives an excellent guide to the requirements for that function. In our demonstration application we called the function APEXTRAS_YUBICO.auth.
The YubiKey outputs a unique string every time it is activated. This is the one-time password (OTP). Its first 12 characters constitiute the YubiKey ID, the unique identifier for each YubiKey. In other words, the first 12 characters of a YubiKey’s output are always the same, and are always different from those of any other YubiKey.
It is quite simple to incorporate the YubiKey into an APEX Authentication Scheme.
1. Add a column to the user table to hold the YubiKey ID of the YubiKey issued to each user.
2. Then add a third field to the login screen to take the YubiKey output as well as the login name and password.
3. Perform this pseudocode to log in:
Verify the YubiKey OTP
IF the YubiKey OTP passes verification THEN
Retrieve the user name corresponding to the YubiKey ID
IF the retrieved user name matches the one entered THEN
Make the call to APEX_CUSTOM_AUTH.LOGIN to check the hash
of the entered password against the stored hash of the user's password
END IF
END IF
If this succeeds, then the logged-in user is
a) in possession of a valid YubiKey that is assigned to their account
and
b) knows the password assigned to that account
To verify the YubiKey OTP you need to submit it (via an HTTP GET) to a YubiKey authentication service. You can either run this somewhere on your own network, or you can use Yubico’s public authentication service.
If you use the Yubico service you will also need two pieces of information about your Yubico account: your API key and your Yubico-issued company user ID: both are available from the Yubico management site. Within our demonstration client the API Key, User ID and the URI of the authentication service need to be entered into the APEXTRAS_YUBICO package as package constants.
The call to the authentication service looks something like this:
http://api.yubico.com/wsapi/verify?id=NNNN&otp=XXX…XXX&h=ZZZZZZZ
id is your company user ID
otp is the one time password you want to verify
h is a hashed MAC (using hashed SHA-1 keyed on your API key) of the query string
The authentication service will respond with a status such as OK, BAD_OTP, REPLAYED_OTP, BAD_SIGNATURE etc
Demonstration of YubiKey two-factor authentication integrated with APEX login.
Evaluating the YubiKey
Sometimes we develop applications that really require more security than is provided by the standard combination of a username and password. We like two-factor/strong authentication which typically consists of something you know (a password) and something you have (a token). In the past we’ve used RSA’s SecureID but we’ve recently been evaluating the Yubikey from Yubico.
The Yubikey is a slim device that plugs into the USB port on any computer. To the computer it appears to be a USB keyboard which means it works across all operating systems and doesn’t require any drivers to be installed.
When you touch the button on the top of a Yubikey it generates a 32 character one-time password (OTP) which appears at the current cursor position: remember the host computer sees the Yubikey as a keyboard. You can validate this OTP by passing it in a call to a restful web service provided by Yubico or you can pass it to your own authentication servers. Yubico provides Java and PHP versions of the authentication server software under an Open Source license.
We like the concept of the Yubikey because, compared to other two-factor solutions, it is very cost effective. Each Yubikey costs about $25 (if you were to buy hundreds that would drop to under $10) and both public authentication service and the authentication server software are free.
We often develop systems that consist of two APEX applications on top of the same database — a public application and an administration application that provides access to the back-end of the system and to table maintenance tasks. The administration application is typically used by just a handful of users and would really benefit from two-factor authentication. The Yubikey would allow us to incorporate strong authentication into the administration application at a common-sense price.
Roger has successfully integrated the Yubikey into APEX.
Demonstration of YubiKey two-factor authentication integrated with APEX login.
APEX 3.2 upgrade
Our first test upgrade from APEX 3.1 to 3.2 was 100% error-free, but our applications showed up all sorts of strange errors. Then I remembered that we were using Patrick Wolf’s wrapped versions of the v, nv, and dv functions – see here for more details. We needed to replace “FLOWS_030100.V” with “APEX_030200.V” throughout – then everything worked again.
Page visit logging
APEX has the ability to log and analyse user activity in some detail. However, the unit of the built-in APEX logs is the APEX page, and the data collected about what a user does on a particular page is limited. In many of our APEX applications, a single APEX page may correspond to multiple web pages. For example we have implemented a dynamic questionnaire system in which each screen of the questionnaire is rendered on the same APEX page. A PL/SQL routine is called with a parameter representing the page of the questionnaire that should be displayed which outputs the complete questionnaire screen by htp.p calls. We still need to collect information about “page” visits although only one APEX page is involved. In addition, there is often a requirement to collect more detailed information about how people are using the system – search terms entered, number of checkboxes ticked, etc.
The solution is to build your own page logging system that inserts a row into a log table for every page visited. You can then record any sort of detail you like. We call the logging code as an Application Process at Process Point “On Load: Before Header (page template header)”. Because APEX doesn’t provide a built-in variable that records the current page alias (and we use page aliases throughout our system), we also need a Before Header Application Computation that fills the Application Item APP_PAGE_ALIAS (see http://oraclequirks.blogspot.com/2008/02/build-and-use-apex-page-alias.html).
The actual insert into the log table is a simple SQL statement that could be executed in the APEX process, but we prefer to keep code that writes to tables in PL/SQL packages, separate from APEX: it’s more maintainable there, error handling can be more rigorously enforced, and we end up with a single PL/SQL procedure to perform each action, rather than several APEX PL/SQL regions doing more-or-less the same thing.
Underscore acts as wildcard in PL/SQL INSTR function for CLOB
Oracle 10.2.0.4.0, Windows 32-bit, NLS_CHARACTERSET AL32UTF8.
We have a PL/SQL utility function that retrieves the nth item from a delimited string. The string parameter is a CLOB so that it can be used with input of any length. Normally the delimiters are commas, and the function has worked fine for years.
An APEX user’s sort order preference for a static report is stored in column WWV_FLOW_PREFERENCES$.ATTRIBUTE_VALUE as a string that looks something like fsp_sort_1_desc. We used our function to retrieve the various parts of this string, with ‘_’ as the delimiter, but it always returned a null. To our surprise it turned out that the underscore acted as a wildcard and matched any character in the string. That is expected behaviour in a SQL LIKE clause, but should not happen in PL/SQL. There is in fact a bug report in Oracle Metalink which describes this behaviour. There is no fix for Oracle version 10g – the bug is said to be fixed in Oracle 11.2. Our workaround was to replace all the underscores with another delimiter before calling the function.
APEX Deployment Script
We are used to working in more formal coding environments than our in-house Oracle/Apex development setup, for which we have a dedicated Oracle server and no full-time DBA. When we had to start making periodic application releases to a remote live server, we expected to be able to write a script that would do the release automatically. It proved a little more difficult than we thought it would – we needed a PL/SQL package, a PERL script, and an APEX command line utility. If you’re faced with the same problem, here’s how we did it.
We used Windows servers for this project, so the final product was two Windows batch files – however it wouldn’t be difficult to write PERL or shell scripts using the same methods.
Of course you will need to customise the batch files and possibly the PL/SQL package for your own environment, and you should test the deployment process to destruction before you let it loose on a live server.
Why did we need an Apex deployment script?
We were coming to the end of a large Apex project developed and to be run on Microsoft Windows Servers. The data was in a single dedicated Oracle workspace but there were three separate Apex applications and 17 PL/SQL packages amounting to nearly 30,000 lines of code.
There were three dedicated Windows Oracle servers – a development server at our office and staging and live servers at a remote co-location facility. We were making formal code and Apex applications releases from the development server to the staging server for user acceptance testing, then releasing from the staging to the live server when the system was approved. Each version of the project had to be retained so that releases could be rolled back if there was any problem.
The manual process
We started off doing both deployments (development->stage and stage->live) manually.
For development->stage:
- Export each of the 3 Apex applications from the development server to a text file with a name that represents the new version number, then import it into the staging server.
- Copy each PL/SQL package from the development to the staging server, taking care not to overwrite sections of code that are site-specific; that includes SMTP server names and addresses, sender email addresses, and database links. We did this using cut and paste (carefully) between PL/SQL developer windows.
- Save the current state of the PL/SQL packages under the new version name – we used Subversion for this.
- Copy the style sheet and images from a directory on the development server to one on the staging server. It was easy to forget this step.
This process was complicated, time-consuming, and inherently risky – after several Apex versions had been generated and saved on disk it was easy to pick the wrong one for import. It was also essential to know how to edit PL/SQL code so that the site-specific code sections were retained – which meant that only a programmer could make a deployment.
The automated solution
Our goal was a single script (ie a Windows batch file) to run on the originating server that would export a project version in the form of SQL scripts to the target server, and a single script to run on the target server to import the new version.
A sample export batch file can be found here. It relies on a PL/SQL package, a PERL script, and the APEX utility APEXExport.class, which is in the standard APEX 3.1 distribution under apex\utilities\oracle\apex.
To run this, you’d save it as a Windows batch file (.bat or .cmd extension) on the source Oracle server then invoke it from the Windows command line with the version number as the only parameter. The source and destination file locations, and the numbers of the Apex applications to be deployed are hard-coded in the batch file. The names of the packages, functions, and procedures to be deployed are package-level constants in the PL/SQL package. The name of the Oracle directory for output PL/SQL code is also a package-level constant. The directory name and its physical path on disk needs to be set up (once) from Oracle – see below.
Here’s a corresponding sample import batch file. You’d save it as a Windows batch file on the target Oracle server then run it with the same version number as was given to the export batch file.
And here’s some more detail about how it works:
1. Exporting PL/SQL packages, functions and procedures (your code)
Where to write the export file containing your code
Oracle can write to external files by means of the UTL_FILE package. In order to use this, we needed to define an external directory on the Oracle server:
CREATE OR REPLACE DIRECTORY CODE_SAVE AS ‘C:\MYDIRECTORY\ETC’
Where to get your code from
The standard Oracle view USER_SOURCE contains all the package, function, and procedure code visible to the current user. To turn the code into executable SQL scripts we just needed to surround it with “CREATE OR REPLACE” and “/”.
2. Site-specific code
We used a simple mechanism for identifying conditional code sections. Each such section was given a start and finish label on a separate Oracle commented line. We arbitrarily chose ‘!~’ and ‘~!’ as begin and end delimiters for the labels because those strings didn’t occur anywhere else in our code.
The labels consisted of two parts separated by an underscore(_): an identifier that specifies the server for which the code section should be live, and the word BEGIN or END. In the exported code the relevant section for the target server is uncommented and all other sections remain commented out.
Thus a package specification could contain the following on a development server: --!~DEVELOPMENT_BEGIN~! smtp_host CONSTANT VARCHAR2(256) := 'smtp.developmentdomain.com'; --!~DEVELOPMENT_END~! --!~LIVE_BEGIN~! /* smtp_host CONSTANT VARCHAR2(256) := 'smtp.livedomain.com'; */ --!~LIVE_END~! Which would be rendered on a live server as: --!~DEVELOPMENT_BEGIN~! /* smtp_host CONSTANT VARCHAR2(256) := 'smtp.developmentdomain.com'; */ --!~DEVELOPMENT_END~! --!~LIVE_BEGIN~! smtp_host CONSTANT VARCHAR2(256) := 'smtp.livedomain.com'; --!~LIVE_END~!
3. Exporting Apex Applications
Apex comes with a command-line export tool written in Java. This produces a SQL script that can be directly invoked from SQL*Plus to reimport the application. It was not easy to make the export tool run, but after some trial and error, we found that (at least in our setup) the CLASSPATH environment variable needed to point to both the Oracle Java library and the location of the Apex export script.
To import an exported application into a different server from the one that exported it, the security group id needs to be changed to that of the target server. That involves rewriting the security group id in the following line of code:
wwv_flow_api.set_security_group_id(p_security_group_id=>SomeBigNumber);
We did this by processing the output Apex script with a simple PERL script. Instead of using a literal security group ID to replace SomeBigNumber, we rewrote this line as
wwv_flow_api.set_security_group_id(p_security_group_id=> apex_util.find_security_group_id(‘MY_WORKSPACE’));
That allowed the import to work on any server with the correct workspace name.
Of course you can use any other text substitution tool to do the replacement instead of a PERL script, as long as it can be invoked from the command line.
4. Versions
This should work on any recent version of Oracle and Apex, but it has been tested only on:
Apex 3.1.2
Oracle 10.2.0.4
Additional Software
Java is likely to be already installed on your Windows server, but PERL may not be.
PERL
We used ActiveState ActivePerl 5.10.0.1004.
32-bit Windws download:
http://www.activestate.com/store/download_file.aspx?binGUID=c8a0cfb5-29dc-484c-9b12-227ed449dacb
64-bit Windws download:
http://www.activestate.com/store/download_file.aspx?binGUID=01ab9ee2-63be-4f69-8b27-fafda0563882
JAVA
Java for Windows (currently Version 6 Update 11) download is available from:
http://javadl.sun.com/webapps/download/AutoDL?BundleId=26223
APEXtras 