Translate into your own language

Showing posts with label ORA Errors. Show all posts
Showing posts with label ORA Errors. Show all posts

Thursday, February 14, 2019

Step by step - How to resolve ORA-01555 from the EXPDP


Scenario - we have around 2 TB of database and daily export job is scheduled to run at 12 AM EST. But it is failing with the below error. There are multiple occurrence of this error. I am just putting out one to show the exact error.

ORA-31693: Table data object "RDR"."REGW_EXTRACT" failed to load/unload and is being skipped due to error:
ORA-02354: error in exporting/importing data
ORA-01555: snapshot too old: rollback segment number 10 with nam"_SYSSMU10_3196945471$" too small 

So we need to heck the few things:

1.      Check the total duration of the export job.
2.      Find the undo retention of the database.
3.      Find the undo table space size
4.      Find the job running timing

Solution:

1.      Change the undo_retention to the total time the export is taking. But remember to add atlease 30 mins more. Suppose export is taking 6 hours then keep the undo_retention to atleast 6:30 hours. Mostly it will resolve the issue.

2.      Also make sure we have enough space on the undo table space.

3.      Also make sure to run the export in non-business hours.



Saturday, September 15, 2018

ORA-01722: invalid number while running catupgrd.sql during database upgrade

Recently we encountered this issue while upgrading from 11.2.0.3 to 11.2.0.4.

ISSUE

catupgrd.sql script was failing with the below error:

DOC> run in the old oracle home prior to upgrading a pre-11.2 database: 
DOC> 
DOC> SELECT TO_NUMBER('MUST_BE_SAME_TIMEZONE_FILE_VERSION') 
DOC> * 
DOC> ERROR at line 1: 
DOC> ORA-01722: invalid number 
DOC> 
DOC> 
DOC> o Action: 
DOC> Shutdown database ("alter system checkpoint" and then "shutdown abort"). 
DOC> Revert to the original ORACLE_HOME and start the database. 
DOC> Run pre-upgrade tool against the database. 
DOC> Review and take appropriate actions based on the pre-upgrade 
DOC> output before opening the datatabase in the new software version. 
-- 
The following error is generated if (1) the old release uses a time 
DOC> zone file version newer than the one shipped with the new oracle 
DOC> release and (2) the new oracle home has not been patched yet: 
DOC> 
DOC> SELECT TO_NUMBER('MUST_PATCH_TIMEZONE_FILE_VERSION_ON_NEW_ORACLE_HOME') 
DOC> * 
DOC> ERROR at line 1: 
DOC> ORA-01722: invalid number 
DOC> 
DOC> o Action: 
DOC> Shutdown database ("alter system checkpoint" and then "shutdown abort"). 
DOC> Patch new ORACLE_HOME to the same time zone file version as used 
DOC> in the old ORACLE_HOME. 


SOLUTION

Get the output of the below queries from 11.2.0.3 and 11.2.0.4 homes:

1) SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value 
FROM DATABASE_PROPERTIES 
WHERE PROPERTY_NAME LIKE 'DST_%' 
ORDER BY PROPERTY_NAME; 

2) SELECT version FROM v$timezone_file; 

3) ls -ltr $ORACLE_HOME/ORACORE/ZONEINFO 


OUTPUT:

on 11.2.0.3 
========

1. SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value 
FROM DATABASE_PROPERTIES 
WHERE PROPERTY_NAME LIKE 'DST_%' 
ORDER BY PROPERTY_NAME; 

PROPERTY_NAME VALUE 
------------------------------ ---------- 
DST_PRIMARY_TT_VERSION 18 
DST_SECONDARY_TT_VERSION 0 
DST_UPGRADE_STATE NONE 

2. SELECT version FROM v$timezone_file; 

SYS@host1> SELECT version FROM v$timezone_file; 

VERSION 
---------- 
18 

3.   @host1> cd /u01/sq/ora_1/db/11.2.0.3/oracore/zoneinfo/ 
@host1> ls -tlr 
total 14568 
-rw-r--r-- 1 oracle oinstall 6068 Jun 7 2010 readme.txt 
-rw-r--r-- 1 oracle oinstall 31525 Jun 7 2010 timezdif.csv 
-rw-r--r-- 1 oracle oinstall 351525 Aug 18 2011 timezone_9.dat 
-rw-r--r-- 1 oracle oinstall 302100 Aug 18 2011 timezone_8.dat 
-rw-r--r-- 1 oracle oinstall 286815 Aug 18 2011 timezone_7.dat 
-rw-r--r-- 1 oracle oinstall 286217 Aug 18 2011 timezone_6.dat 
-rw-r--r-- 1 oracle oinstall 286310 Aug 18 2011 timezone_5.dat 
-rw-r--r-- 1 oracle oinstall 286264 Aug 18 2011 timezone_4.dat 
-rw-r--r-- 1 oracle oinstall 286651 Aug 18 2011 timezone_3.dat 
-rw-r--r-- 1 oracle oinstall 274900 Aug 18 2011 timezone_2.dat 
-rw-r--r-- 1 oracle oinstall 274427 Aug 18 2011 timezone_1.dat 
-rw-r--r-- 1 oracle oinstall 344448 Aug 18 2011 timezone_14.dat 
-rw-r--r-- 1 oracle oinstall 344425 Aug 18 2011 timezone_13.dat 
-rw-r--r-- 1 oracle oinstall 345024 Aug 18 2011 timezone_12.dat 
-rw-r--r-- 1 oracle oinstall 345356 Aug 18 2011 timezone_11.dat 
-rw-r--r-- 1 oracle oinstall 345637 Aug 18 2011 timezone_10.dat 
-rw-r--r-- 1 oracle oinstall 801410 Aug 18 2011 timezlrg_9.dat 
-rw-r--r-- 1 oracle oinstall 616723 Aug 18 2011 timezlrg_8.dat 
-rw-r--r-- 1 oracle oinstall 601242 Aug 18 2011 timezlrg_7.dat 
-rw-r--r-- 1 oracle oinstall 586750 Aug 18 2011 timezlrg_6.dat 
-rw-r--r-- 1 oracle oinstall 587487 Aug 18 2011 timezlrg_5.dat 
-rw-r--r-- 1 oracle oinstall 531137 Aug 18 2011 timezlrg_4.dat 
-rw-r--r-- 1 oracle oinstall 527717 Aug 18 2011 timezlrg_3.dat 
-rw-r--r-- 1 oracle oinstall 507957 Aug 18 2011 timezlrg_2.dat 
-rw-r--r-- 1 oracle oinstall 493675 Aug 18 2011 timezlrg_1.dat 
-rw-r--r-- 1 oracle oinstall 791430 Aug 18 2011 timezlrg_14.dat 
-rw-r--r-- 1 oracle oinstall 782475 Aug 18 2011 timezlrg_13.dat 
-rw-r--r-- 1 oracle oinstall 785621 Aug 18 2011 timezlrg_12.dat 
-rw-r--r-- 1 oracle oinstall 787272 Aug 18 2011 timezlrg_11.dat 
-rw-r--r-- 1 oracle oinstall 792894 Aug 18 2011 timezlrg_10.dat 
-rwxr-xr-x 1 oracle oinstall 781669 Jun 11 2012 timezlrg_18.dat >>>>>>>>>>>>>>>>> 
-rwxr-xr-x 1 oracle oinstall 341718 Jun 11 2012 timezone_18.dat >>>>>>>>>>>>> 
-rwxr-xr-x 1 oracle oinstall 7150 Jun 11 2012 readme_18.txt 
drwxr-xr-x 2 oracle oinstall 4096 May 6 2013 big 
drwxr-xr-x 2 oracle oinstall 4096 May 6 2013 little 

on 11.2.0.4 
========

1. SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value 
FROM DATABASE_PROPERTIES 
WHERE PROPERTY_NAME LIKE 'DST_%' 
ORDER BY PROPERTY_NAME; 

PROPERTY_NAME VALUE 
------------------------------ ---------- 
DST_PRIMARY_TT_VERSION 18 
DST_SECONDARY_TT_VERSION 0 
DST_UPGRADE_STATE NONE 

2. SQL> SELECT version FROM v$timezone_file; 

no rows selected 

3. @host1> cd /u01/sq/ora_1/db/11.2.0.4/oracore/zoneinfo/ 
@host1> ls -tlr 
total 14568 
-rw-r--r-- 1 oracle oinstall 6068 Jun 7 2010 readme.txt 
-rw-r--r-- 1 oracle oinstall 31525 Jun 7 2010 timezdif.csv 
-rw-r--r-- 1 oracle oinstall 351525 Jul 16 2013 timezone_9.dat 
-rw-r--r-- 1 oracle oinstall 302100 Jul 16 2013 timezone_8.dat 
-rw-r--r-- 1 oracle oinstall 286815 Jul 16 2013 timezone_7.dat 
-rw-r--r-- 1 oracle oinstall 286217 Jul 16 2013 timezone_6.dat 
-rw-r--r-- 1 oracle oinstall 286310 Jul 16 2013 timezone_5.dat 
-rw-r--r-- 1 oracle oinstall 286264 Jul 16 2013 timezone_4.dat 
-rw-r--r-- 1 oracle oinstall 286651 Jul 16 2013 timezone_3.dat 
-rw-r--r-- 1 oracle oinstall 274900 Jul 16 2013 timezone_2.dat 
-rw-r--r-- 1 oracle oinstall 274427 Jul 16 2013 timezone_1.dat 
-rw-r--r-- 1 oracle oinstall 344448 Jul 16 2013 timezone_14.dat 
-rw-r--r-- 1 oracle oinstall 344425 Jul 16 2013 timezone_13.dat 
-rw-r--r-- 1 oracle oinstall 345024 Jul 16 2013 timezone_12.dat 
-rw-r--r-- 1 oracle oinstall 345356 Jul 16 2013 timezone_11.dat 
-rw-r--r-- 1 oracle oinstall 345637 Jul 16 2013 timezone_10.dat 
-rw-r--r-- 1 oracle oinstall 801410 Jul 16 2013 timezlrg_9.dat 
-rw-r--r-- 1 oracle oinstall 616723 Jul 16 2013 timezlrg_8.dat 
-rw-r--r-- 1 oracle oinstall 601242 Jul 16 2013 timezlrg_7.dat 
-rw-r--r-- 1 oracle oinstall 586750 Jul 16 2013 timezlrg_6.dat 
-rw-r--r-- 1 oracle oinstall 587487 Jul 16 2013 timezlrg_5.dat 
-rw-r--r-- 1 oracle oinstall 531137 Jul 16 2013 timezlrg_4.dat 
-rw-r--r-- 1 oracle oinstall 527717 Jul 16 2013 timezlrg_3.dat 
-rw-r--r-- 1 oracle oinstall 507957 Jul 16 2013 timezlrg_2.dat 
-rw-r--r-- 1 oracle oinstall 493675 Jul 16 2013 timezlrg_1.dat 
-rw-r--r-- 1 oracle oinstall 791430 Jul 16 2013 timezlrg_14.dat 
-rw-r--r-- 1 oracle oinstall 782475 Jul 16 2013 timezlrg_13.dat 
-rw-r--r-- 1 oracle oinstall 785621 Jul 16 2013 timezlrg_12.dat 
-rw-r--r-- 1 oracle oinstall 787272 Jul 16 2013 timezlrg_11.dat 
-rw-r--r-- 1 oracle oinstall 792894 Jul 16 2013 timezlrg_10.dat 
-rw-r--r-- 1 oracle oinstall 341718 Sep 18 2013 timezone_17.dat 
-rw-r--r-- 1 oracle oinstall 779003 Sep 18 2013 timezlrg_17.dat 
-rw-r--r-- 1 oracle oinstall 6905 Aug 31 22:38 readme_17.txt 
drwxr-xr-x 2 oracle oinstall 4096 Aug 31 23:04 big 
drwxr-xr-x 2 oracle oinstall 4096 Aug 31 23:04 little 
@host1> 

from the above output we can see that the result for 2nd query on 11.2.0.4 is empty(NO ROW SELECED) because of 18 version TZ files does not exist at new 11.2.0.4 home.

Hence to resolve this issue we copied the TZ files from 11.2.0.3 home to new 11.2.0.4 home.
Make sure to copy the 18 version of files from 11.2.0.3 to 11.2.0.4 homes on below locations:

/u01/sq/ora_1/db/11.2.0.4/oracore/zoneinfo/
/u01/sq/ora_1/db/11.2.0.4/oracore/zoneinfo/big
/u01/sq/ora_1/db/11.2.0.4/oracore/zoneinfo/little

And we ran the catupgrd.sql script again and it was successful.

==========================*****========================================

Tuesday, May 24, 2016

How to Troubleshoot ORA-00600 Internal error in Oracle

ORA-00600 and ORA-07445 Troubleshooting are an integral part of Oracle Database Administrator's life. In some cases these error could crash database instance's or in others does very minor impact.

So, I would suggest to start working on ORA-00600 and ORA-07445 immediately, you notice them. My agenda for this post is to explain the approach for solving ORA-00600 and ORA-07445 Internal error, which would be more helpful rather than solving one or two issues.

What is ORA-00600 and ORA-07445 Internal error ?

As we know, basic code of database is written in C and C++ Language. When a function in C call another function but got some unexpected results and which is not handled by Oracle Code, then DBA encounter ORA-00600 or ORA-07445 Internal error. On further discussion in this article, we will also see which function is failed and case these errors.

Troubleshooting ORA-00600 

1. Getting Error Detail from Alter log and Trace file:

Whenever, this error arise, it is written in to database alert log files with trace file having details about this error. So, first task is to find trace file from alter log holding about this error.
Oracle 11g Alter log entry with error ORA-00600 and trace files path

Wed May 25 15:41:47 2016
Errors in file /u01/app/oracle/diag/rdbms/remotedba/remotedba1/trace/remotedba1_pr00_19608.trc (incident=222458):
ORA-00600: internal error code, arguments: [1433], [60], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/remotedba/remotedba1/incident/incdir_222458/remotedba1_pr00_19608_i222458.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

In case of Oracle 10g, Just A small difference, Instead of Incident file detail you will see trace file path. Open the trace/incident files and you will find description about this error. Though, it's not very human readable.


2. Login to https://support.oracle.com to Troubleshoot:

If, you need an accurate solution of this error, then you can only find at https://support.oracle.com. So, first of all, you must have an Oracle Support user ID and Password.

Login to support.oracle.com

Search for Article "153788.1" in knowledge base. This will open an article "ORA-600/ORA-7445/ORA-700 Error Look-up Tool [Article ID 153788.1]" This looks like this.



3. Search Error in this tool:

In the above screen choose error code DBA is facing in alter log files (ORA-600 or ORA-7445). Next is filling "Error Code First Argument". Whatever you are getting in fist [] after ORA-600 or ORA 7445 is called as first argument.

E.g. ORA-00600: internal error code, arguments: [1433]

In this error 1433 is the first argument.

You just need to copy first argument in "Error Code First Argument" box. Here, I am taking ORA-00600: internal error code, arguments: [1433] as an test case. Choose Oracle version database is running on from "Oracle RDBMS Version" drop box. Here, I am choosing 11.2.0.3. After filling all this information. Click "Look-up Error". This will search Oracle knowledge base for matching documents.



Document 138413.1 is the matching document with given error in Alert log file. This document has all details about bug, affected versions, workaround or fix for it. Click on the link for the document and see the details.

In this case, below are the details.

    a. Bug number: 12899768

    b. Fixed in versions: 11.2.0.2.8, 11.2.0.2.BP18, 11.2.0.3.BP11, 12.1.0.0

    c. Workaround: Increase _messages init.ora parameter to a large value.

Apply the suggested solution and your problem is solved.

Wednesday, May 4, 2016

Archivelog Backup/Delete using RMAN

How to change archive log destination to diffent disk group:

1. SQL> alter system set log_archive_dest_1='location=+SPARE_DISKS' scope=both;


alter system set log_archive_dest_1='location=/ora_backup/arc_bkp_2may' scope=both;
System altered.

SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            +SPARE_DISKS
Oldest online log sequence     288
Next log sequence to archive   290
Current log sequence           290
SQL>

alter system set log_archive_dest_1='location=+RMAN_DUPACTIVE' scope=both;


Script to mvoe archive log to diffent location on disk:

connect target sys/${SYSPASSWORD}@${SERVICE_NAME}
connect catalog rcat/rcat@TESTRAC


RMAN>spool log to rman_log

RMAN> run {
allocate channel ch001 device type disk;
allocate channel ch002 device type disk;

backup as compressed backupset archivelog all not backed up 1 times delete input
TAG 'archbkp_Dec28_arc' FORMAT '/orabackup/tmp_bkp/%d_%s_%p_%c_%t.arc.rman';
backup current controlfile
TAG 'archbkp_Dec28_ctl' FORMAT '/orabackup/tmp_bkp/%d_%s_%p_%c_%t.ctl.rman';

release channel ch001;
release channel ch002;

}


Restoring Archive logs

RESTORE ARCHIVELOG FROM SEQUENCE 210004 UNTIL SEQUENCE 210064;

RESTORE ARCHIVELOG FROM SEQUENCE 203 UNTIL SEQUENCE 264;


Checkinng the archive location size and its spaces 

sqlplus "/as sysdba"

archive log list;

!df -h or !df -kgt

show parameter db_recovery;

NAME                      TYPE         VALUE
----                      ----         -----
db_recovery_file_dest     string       +FLASHDG
db_recovery_file_dest_size big integer 150G

if +FRA,
select * from V$FLASH_RECOVERY_AREA_USAGE;
(see what kind of files are available in the Flash Recovery Area)

set pages 9999 lines 300
col name format a40
select name,
to_char(space_limit, '999,999,999,999') as space_limit,
to_char(space_limit - space_used + space_reclaimable,
'999,999,999,999') as space_available,
round((space_used - space_reclaimable)/space_limit * 100, 1) as pct_full
from v$recovery_file_dest;

NAME SPACE_LIMIT SPACE_USED SPACE_RECLAIMABLE NUMBER_OF_FILES
---- ----------- ---------- ----------------- ---------------

if SPACE_USED is full, then increase the db_recovery_file_dest_size

rman target /
crosscheck archivelog all;
delete expired archivelog all;
crosscheck backup;

SQL>alter system set db_recovery_file_dest_size=200G scope=both;
System altered.

rman target /

show all;
LIST BACKUP OF ARCHIVELOG FROM TIME 'sysdate-1'; ------List all archivelog backups for the past 24 hours
list backup of archivelog all completed before 'sysdate -1';

list backup of archivelog all;
list backup of archivelog all tag 'TAGXXX';
list archivelog all backed up 1 times to DEVICE TYPE DISK completed before 'sysdate-7';

crosscheck archivelog all;
list expired archivelog ALL;
delete expired archivelog all;

delete obsolete device type disk;
crosscheck backup;
delete expired backup device type disk;
delete expired backupset device type disk;

ls -ltr archivebackup.cmd

nohup rman target / cmdfile=archivebackup.cmd log=archivebackup_dbname_DDMONYY.log &
nohup: appending output to `nohup.out'

tail -f archivebackup_dbname_DDMONYY.log

+FRA/+RECO01 filled with archivelogs
====================================

Backup archivelogs to disk
--------------------------
run
{
allocate channel ch1 type DISK;
allocate channel ch2 type DISK;
allocate channel ch3 type DISK;
allocate channel ch4 type DISK;
BACKUP ARCHIVELOG ALL FILESPERSET 10 DELETE INPUT;
}

Backup archivelogs to disk (specified location,if there is no space to take bakup at archive default location)
--------------------------------------------------------------------------------------------------------------
RUN
{
allocate channel ch1 type DISK;
allocate channel ch2 type DISK;
allocate channel ch3 type DISK;
allocate channel ch4 type DISK;
BACKUP ARCHIVELOG ALL format '/location/arch_%d_%p_%s.rman' FILESPERSET 10 DELETE INPUT;
}

Backup archivelogs to tape (PROD Environment)
--------------------------
run
{
allocate channel ch1 type 'sbt_tape';
allocate channel ch2 type 'sbt_tape';
allocate channel ch3 type 'sbt_tape';
allocate channel ch4 type 'sbt_tape';
BACKUP ARCHIVELOG ALL FILESPERSET 10 DELETE INPUT;
}

+FRA filled with Bacupsets (PROD Environment)
==========================
Move older backupset to tape (to keep more recent backups on disk and older backups only on tape)
-------------------------------------------------------------------------------------------------
run
{
allocate channel ch1 type 'sbt_tape';
allocate channel ch2 type 'sbt_tape';
allocate channel ch3 type 'sbt_tape';
allocate channel ch4 type 'sbt_tape';
BACKUP BACKUPSET COMPLETED BEFORE 'SYSDATE-7' DELETE INPUT;
}

if required,
BACKUP BACKUPSET ALL FILESPERSET 10 DELETE INPUT;

Delete BACKUPSET backed up to tape/disk (PROD Environment)
-----------------------------------------
DELETE NOPROMPT BACKUPSET COMPLETED BEFORE 'SYSDATE-14' DEVICE TYPE DISK;

+FRA filled with Bacupsets (NON-PROD Environment)
==========================
Move older backupset to tape (to keep more recent backups on disk and older backups only on tape)
-------------------------------------------------------------------------------------------------
run
{
allocate channel ch1 type 'sbt_tape';
allocate channel ch2 type 'sbt_tape';
allocate channel ch3 type 'sbt_tape';
allocate channel ch4 type 'sbt_tape';
DELETE BACKUPSET COMPLETED BEFORE 'SYSDATE-4';
}

Delete archivelogs backed up to tape/disk (PROD Environment)
-----------------------------------------
DELETE archivelog UNTIL TIME 'SYSDATE-7' BACKED UP 1 TIMES TO DEVICE TYPE DISK;
DELETE archivelog UNTIL TIME 'SYSDATE-7' BACKED UP 1 TIMES TO DEVICE TYPE SBT_TAPE;


Delete archivelogs backed up to tape/disk (NON-PROD Environment)
-----------------------------------------
DELETE archivelog all BACKED UP 1 TIMES TO DEVICE TYPE DISK;

list archivelog all backed up 1 times to DEVICE TYPE DISK completed before 'sysdate-7';
delete archivelog all backed up 1 times to DEVICE TYPE DISK completed before 'sysdate-7';

backup archivelog [all] [until time 'sysdate']| [sequence between 100 to 110 thread 1] as filesperset 5 delete

input;

backup archivelog from sequence 100 until sequence 110 thread 1 delete input;

backup archivelog from logseq 100 until logseq 110 thread 1 delete input;

report obsolete;
DELETE OBSOLETE REDUNDANCY = 3;
DELETE OBSOLETE RECOVERY WINDOW OF 7 DAYS;

Tuesday, May 3, 2016

OPatch failed with error code 73 " Following executables are active :"

The process of applying patch is to stop all the services from the existing home where we are going to apply the patch. We do our best but sometime when we issue the opatch apply command it throws the below error:

OPatch failed with error code 73 " Following executables are active :". See below figure:



Solution: The solution of this error is very simple. We have to use /sbin/fuser command to find the executable process id. It gives extra m with the process id like 123m. Now we have to grep the process id and kill it.  Please find the below figure:


After killing the executable when we run the opatch apply, it completes successfully.
Please find the below figure.




Monday, April 18, 2016

How to resolve network - database connectivity issue

DBA generally face this issue many times that a user has reported that he or she can’t connect to a database. You know there are many components involved with network connectivity and want to figure out the root cause of the problem.

Use the below steps as guidelines when diagnosing Oracle database network connectivity issues:

1. Use the operating system ping utility to determine whether the remote box is accessible—for example:
$ ping dwdb
dwdb is alive

If ping doesn’t work, work with your system or network administrator to ensure you have server-to-server connectivity in place.

2. Use telnet to see if you can connect to the remote server and port (that the listener is listening on)—for example:
$ telnet ora03 1521
Trying 127.0.0.1...
Connected to ora03.
Escape character is '^]'.

The prior output indicates that connectivity to a server and port is okay. If the
prior command hangs, then contact your SA or network administrator for
further assistance.

3. Use tnsping to determine whether Oracle Net is working. This utility will verify that an Oracle Net connection can be made to a database via the network—for example:

$ tnsping dwrep
..........
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)
(HOST = dwdb1.us.farm.com)(PORT = 1521))
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = DWREP)))
OK (500 msec)

If tnsping can’t contact the remote database, verify that the remote listener and database are both up and running. On the remote box, use the lsnrctl status command to verify that the listener is up. Verify that the remote database is available by establishing a local connection as a non-SYS account (SYS can often connect to a troubled database when other schemas will not work).


4. Verify that the TNS information is correct. If the remote listener and database are working, then ensure that the mechanism for determining TNS information (like the tnsnames.ora file) contains the correct information.

Sometimes the client machine will have multiple TNS_ADMIN locations and
tnsnames.ora files. One way to verify whether a particular tnsnames.ora file is
being used is to rename it and see whether you get a different error when
attempting to connect to the remote database.


Network connectivity issues can be troublesome to diagnose because there are several architectural components that have to be in place for it to work correctly. You need to have the following in place:

  • A functional network
  • Open ports from point to point
  • Oracle Net correctly installed and configured
  • Target database and listener up and running
  • Correct navigational information from the client to the target database

If you’re still having issues, examine the client sqlnet.log file and the remote server listener.log file. Sometimes these log files will show additional information that will pinpoint the issue.

Sunday, April 17, 2016

ORA-09817: Write to audit file failed Linux Error: 28: No space left on device

Users are reporting that they can’t connect to a database. You log on to the database server, attempt to connect to SQL*Plus, and receive this error:
ORA-09817: Write to audit file failed.
Linux Error: 28: No space left on device
Additional information: 12

You want to quickly determine if a mount point is full and where the largest files are within this mount point.

To resove this, use the df command to identify disk space issues. This example uses the -
h to format the output so that space is reported in megabytes or gigabytes:

$ df –h

Here is some sample output:

The prior output indicates that the root (/) file system is full on this server. In this situation, once a full mount point is identified, then use the find command to locate the largest files contained in a directory structure. This example navigates to the ORACLE_HOME directory and then connects the find, ls, sort, and head commands to identify the largest largest top 10 files beneath that directory:

$ cd $ORACLE_HOME

$ find . -ls | sort -nrk7 | head -10

If you have a full mount point, also consider looking for the following types of files that can be moved or removed:


  • Deleting database trace files
  • Removing large Oracle Net log files
  • Moving, compressing, or deleting old archive redo log files
  • Removing old installation files or binaries
  • If you have datafiles with ample free space, consider resizing them to smaller sizes


Another way to identify where the disk space is being used is to find the largest space-consuming directories beneath a given directory. This example combines the du, sort, and head commands to show the ten largest directories beneath the current working directory:

$ du -S . | sort -nr | head -10

The prior command is particularly useful for identifying a directory that might not necessarily have large files in it, but lots of small files consuming space (like trace files).

Friday, April 15, 2016

ORA-15001: diskgroup "'+DATA_DISK" does not exist or is not mounted after OS Patching

We regularly encounter this type of error when Linux admin apply the patch on database server.
But when the Linux team confirms and you try to start the database you can see the below error:

Note- It has nothing to do with the database

ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA_DISK/DBNAME/spfileDBNAME.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA_DISK/DBNAME/spfileDBNAME.ora
ORA-15056: additional error message
ORA-15001: diskgroup "'+DATA_DISK" does not exist or is not mounted

No need to worry here. We can simply match the "oracleasm" and "kernal" version.
if it does not match then immediatly contact Linux team who performed the patch.
Below are the simple steps for matching:

[oracle@host1~]$ uname -a
Linux tryprorarac2c.intra.searshc.com 2.6.18-408.el5xen #1 SMP Fri Dec 11 14:07:27 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

[oracle@host1 ~]$ rpm -qa |grep oracleasm
oracleasmlib-2.0.4-1.el5
oracleasm-support-2.1.8-1.el5
oracleasm-2.6.18-408.el5-2.0.5-1.el5
[oracle@host1~]$ uname -r
2.6.18-408.el5xen
[oracle@host1~]$

Linux team will update the oracleasm version to match with the kernal version and then we can start the database successfully.

Now match the "oracleasm" and "kernal" version again and verify again. We can see it matches perfectly.

[oracle@host1~]$ rpm -qa |grep oracleasm
oracleasm-2.6.18-408.el5xen-2.0.5-1.el5
oracleasmlib-2.0.4-1.el5
oracleasm-support-2.1.8-1.el5
[oracle@host1~]$ uname -r
2.6.18-408.el5xen

Monday, March 21, 2016

EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet.

I strongly believe if you’ve deployed agents a coupled of times, you’ve encountered the following error after the installation:
[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

The first thing that should be done is checking the log file. Switch to the following directory and tail the log file:
cd /u01/oracle/product/10.2.0/agent10g/sysman/log
tail -f emagent.trc

Here’s the output from the log file:
2012-07-26 03:46:33 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository
2012-07-26 03:47:08 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository
2012-07-26 03:47:08 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository

I checked the status of the agent and saw that it’s running :
[oracle@dg2 bin]$ ./emctl status agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.3.0
OMS Version : 10.2.0.3.0
Protocol Version : 10.2.0.2.0
Agent Home : /u01/oracle/product/10.2.0/agent10g
Agent binaries : /u01/oracle/product/10.2.0/agent10g
Agent Process ID : 27772
Parent Process ID : 27755
Agent URL : http://dg2:3872/emd/main/
Repository URL : http://dg1:4889/em/upload/
Started at : 2012-07-26 03:42:04
Started by user : oracle
Last Reload : 2012-07-26 03:42:04
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 21
Size of XML files pending upload(MB) : 3.25
Available disk space on upload filesystem : 58.86%
Last attempted heartbeat to OMS : 2012-07-26 03:45:22
Last successful heartbeat to OMS : unknown
—————————————————————
Agent is Running and Ready
[oracle@dg2 bin]$
[oracle@dg2 bin]$

So I decided to clear unnecessary files that can’t be uploaded to the Grid Control under the following directories:
$AGENT_HOME/sysman/emd/collection
$AGENT_HOME/sysman/emd/state
$AGENT_HOME/sysman/emd/recv
$AGENT_HOME/sysman/emd/upload
$AGENT_HOME/sysman/emd/lastupld.xml

Then stopped the agent, secured it, cleared unnecessary files again with clearstate parameter, started the agent and uploaded.

[oracle@dg2 bin]$ ./emctl stop agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Stopping agent …. stopped.

[oracle@dg2 bin]$ ./emctl secure agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Enter Agent Registration password :
Agent is already stopped… Done.
Securing agent… Started.
Requesting an HTTPS Upload URL from the OMS… Done.
Requesting an Oracle Wallet and Agent Key from the OMS…
Done.
Check if HTTPS Upload URL is accessible from the agent… Done.
Configuring Agent for HTTPS in CENTRAL_AGENT mode… Done.
EMD_URL set in /u01/oracle/product/10.2.0/agent10g/sysman/config/emd.properties
Securing agent… Successful.
[oracle@dg2 bin]$

[oracle@dg2 bin]$ ./emctl clearstate agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
EMD clearstate completed successfully
[oracle@dg2 bin]$ ./emctl start agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ….. started.

[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload completed successfully

[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload completed successfully
[oracle@dg2 bin]$

Sunday, March 20, 2016

Resolving ORA-01000 open cursor error

We are frequently getting the Maximum Open Cursors exceeded error, and we want to resolve the error.
Solution:

One of the first things we need to do when we receive the ORA-01000: “maximum open cursors exceeded” error is to check the value of the initialization parameter open_cursors.

We can view the current limit for open cursors by issuing the following command:

SQL> sho parameter open_cursors
NAME TYPE VALUE
------------------------------------ ----------- ---------
open_cursors integer 300

The parameter OPEN_CURSORS sets the maximum number of cursors a session can have open at once. We specify this parameter to control the number of open cursors. Keeping the parameter’s value too low will result in a session receiving the ORA-01000 error. There’s no harm in specifying a very large value for the OPEN_CURSORS parameter (unless we expect all sessions to simultaneously max out their cursors, which is unlikely), so we can usually resolve cursor-related errors simply by raising the parameter value
to a large number. However, we may sometimes find that raising the value of the open_cursors parameter doesn’t “fix” the problem. In such cases, investigate which processes are using the open cursors by issuing the following query:

SQL> select a.value, s.username,s.sid,s.serial#,s.program,s.inst_id
from gv$sesstat a,gv$statname b,gv$session s
where a.statistic# = b.statistic# and s.sid=a.sid
and b.name='opened cursors current'

The GV$OPEN_CURSOR (or the V$OPEN_CURSOR) view shows all the cursors that each user session has currently opened and parsed, or cached. We can issue the following query to identify the sessions with a high number of opened and parsed or cached cursors.

SQL> select saddr, sid, user_name, address,hash_value,sql_id, sql_text
from gv$open_cursor
where sid in
(select sid from v$open_cursor
group by sid having count(*) > &threshold);

The query lists all sessions with an open cursor count greater than the threshold we specify. This way, we can limit the query’s output and focus just on the sessions that have opened, parsed, or cached a large number of cursors.

we can get the actual SQL code and the open cursor count for a specific session by issuing the following query:

SQl> select sql_id,substr(sql_text,1,50) sql_text, count(*)
from gv$open_cursor where sid=81
group by sql_id,substr(sql_text,1,50)
order by sql_id;

The output shows the SQL code for all open cursors in the session with the SID 81. we can examine all SQL statements with a high open cursor count, to see why the session was keeping a large number of cursors open.

How It Works:

If wer application is not closing open cursors, then setting the OPEN_CURSORS parameter to a higher value won’t really help we. we may momentarily resolve the issue, but we’re likely to run into the same issue a little later. If the application layer never closes the ref cursors created by the PL/SQL code, the database will simply hang on to the server resources for the used cursors. we must fix the application logic so it closes the cursors—the problem isn’t really in the database. If we’re using a Java application deployed on an application server such as the Oracle WebLogic Server, the WebLogic Server’s JDBC connection pools provide open database connections for applications. Any prepared statements in each of these connections will use a cursor. Multiple application server instances and multiple JDBC connection pools will mean that the database needs to
support all the cursors. If multiple requests share the same session ID, the open cursor problem may be due to implicit cursors. The only solution then is to close the connection after each request. A cursor leak is when the database opens cursors but doesn’t close them. we can run a 10046 trace for a session to find out if it’s closing its cursors:

SQL> alter session set events '10046 trace name context forever, level 12';

If we notice that the same SQL statement is associated with different cursors, it means that the application isn’t closing its cursors. If the application doesn’t close its cursors after opening them, Oracle assigns different cursor numbers for the next SQL statement it executes. If the cursor is closed, instead, Oracle will re-use the same cursor number for the next cursor it assigns. Thus, if we see the item PARSING IN CURSOR #nnnn progressively increase in the output for the 10046 trace, it means that the application is not closing the cursors. Note that while leaving cursors open may be due to a faulty application design, developers may also intentionally leave cursors open to reduce soft parsing, or when
they use the session cursor cache.

we can use the SESSION_CACHED_CURSORS initialization parameter to set the maximum number of cached closed cursors for each session. The default setting is 50. we can use this parameter to prevent a session from opening an excessive number of cursors, thereby filling the library cache or forcing excessive hard parses. Repeated parse calls for a SQL statement leads Oracle to move the session cursor for that statement into the session cursor cache. The database satisfies subsequent parse calls by using the cached cursor instead of re-opening the cursor.

 When we re-execute a SQL statement, Oracle will first try to find a parsed version of that statement in the shared pool—if it finds the parsed version in the shared pool, a soft parse occurs. Oracle is forced to perform the much more expensive hard parse if it doesn’t find the parsed version of the statement in the shared pool. While a soft parse is much less expensive than a hard parse, a large number of soft parses can affect performance, because they do require CPU usage and library cache latches. To reduce the number of soft parses, Oracle caches the recent closed cursors of each session in a local session cache for that session—Oracle stores any cursor for which a minimum of three parse calls were made,
thus avoiding having to cache every single session cursor, which will fill up the cursor cache.
The default value of 50 for the SESSION_CACHED_CURSORS initialization parameter may be too low for many databases. we can check if the database is bumping against the maximum limit for sessioncached cursors by issuing the following statement:

SQL> select max(value) from v$sesstat
2 where statistic# in (select statistic# from v$statname
3* where name = 'session cursor cache count');
MAX(VALUE)
----------
49

The query shows the maximum number of session cursors that have been cached in the past. Since this number (49) is virtually the same as the default value (or the value we’ve set) for the SESSION_CACHED_CURSORS parameter, we must set the parameter's value to a larger number. Session cursor caches use the shared pool. If we’re using automatic memory management, there’s nothing for we to do after we reset the SESSION_CACHED_CURSORS parameter—the database will bump up the shared pool size if necessary. we can find out how many cursors each session has in its session cursor cache by issuing the following query:

SQL> select a.value,s.username,s.sid,s.serial#
2 from v$sesstat a, v$statname b,v$session s
3 where a.statistic#=b.statistic# and s.sid=a.sid
4* and b.name='session cursor cache count';

Resolving ORA-04031 (stream pool) error while exporting

When I executed datapump export (expdp) job on a 11.2.0.2 database, it got failed with below error message
ORA-31626: job does not exist
ORA-31637: cannot create job SYS_EXPORT_FULL_01 for user SYS
ORA-06512: at “SYS.DBMS_SYS_ERROR”, line 95
ORA-06512: at “SYS.KUPV$FT_INT”, line 600
ORA-39080: failed to create queues “KUPC$C_1_20070823095248” and “KUPC$S_1_20070823095248” for Data Pump job ORA-06512: at “SYS.DBMS_SYS_ERROR”, line 95
ORA-06512: at “SYS.KUPC$QUE_INT”, line 1580
ORA-04031: unable to allocate 4194344 bytes of shared memory (“streams pool”,”unknown object”,”streams pool”,”fixed allocation callback
when we look at last line, like me, many people wonder why it is showing stream pool here and what is the relation between stream pool and expdp?
Here is the background…
Datapump jobs (either expdp/impdp), when initiated, will use buffer queues in order to transfer the data directly from the master table to the dump file. Instead of creating new buffer queue, datapump operations will try to use the existing queues in stream pool memory area.
Normally, Stream pool will have buffer queues in order to store messages in memory that supports capture processes, apply processes, XStream outbound servers, and XStream inbound servers.
Ideally, Stream pool size will be managed (increase/decrease) automatically when ASMM or AMM is enabled. But, sometimes with this automatic memory management, it might happen that STREAM_POOL_SIZE has not been set and pointing to a size of 0. This is the cause for the error mentioned above.
Solution:
Set STREAM_POOL_SIZE parameter to a value greater than 0. Usually 48m should be the min size and this value will be more if there is a large database or a database with high work load.

Resolving ORA-01031 while configuring EM 10g

EMCTL is one service which we never know whether it will run fine or not…. :-)
Just kidding, here is one such problem with Enterprise Manager configuration and its solution
Problem:
When trying to configure EM for 10g or trying to drop repository or trying to recreate repository you may get below error
[oracle@issgascop218 ~]$ emca -deconfig dbcontrol db -repos drop
STARTED EMCA at Aug 24, 2010 7:51:44 AM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.
Enter the following information:
Database SID: PASQ
Listener port number: 1521
Password for SYS user:
Password for SYSMAN user:
Do you wish to continue? [yes(Y)/no(N)]: y
Aug 24, 2010 7:51:54 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /opt/oracle/product/10.2.0/cfgtoollogs/emca/PASQ/emca_2010-08-24_07-51-44-AM.log.
Aug 24, 2010 7:51:55 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) …
Aug 24, 2010 7:51:57 AM oracle.sysman.emcp.EMReposConfig stopDBMSJobs
WARNING: Error initializing SQL connection. SQL operations cannot be performed
Aug 24, 2010 7:51:57 AM oracle.sysman.emcp.EMReposConfig invoke
WARNING: Unable to remove DBMS jobs.
Aug 24, 2010 7:51:57 AM oracle.sysman.emcp.EMReposConfig dropRepository
INFO: Dropping the EM repository (this may take a while) …
Aug 24, 2010 7:51:57 AM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /opt/oracle/product/10.2.0/sysman/admin/emdrep/bin/RepManager -connect (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=issgascop218)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=PASQ.issgascop218.global.iss.biz))) -repos_user SYSMAN -action drop -verbose -output_file /opt/oracle/product/10.2.0/cfgtoollogs/emca/PASQ/emca_repos_drop_2010-08-24_07-51-57-AM.log
Aug 24, 2010 7:51:57 AM oracle.sysman.emcp.EMReposConfig invoke
SEVERE: Error dropping the repository
Aug 24, 2010 7:51:57 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Refer to the log file at /opt/oracle/product/10.2.0/cfgtoollogs/emca/PASQ/emca_repos_drop_<date>.log for more details.
Aug 24, 2010 7:51:57 AM oracle.sysman.emcp.EMConfig perform
SEVERE: Error dropping the repository
Refer to the log file at /opt/oracle/product/10.2.0/cfgtoollogs/emca/PASQ/emca_2010-08-24_07-51-44-AM.log for more details.
Could not complete the configuration. Refer to the log file at /opt/oracle/product/10.2.0/cfgtoollogs/emca/PASQ/emca_2010-08-24_07-51-44-AM.log for more details.
When i opened log file, i found below information…
[24-08-2010 07:51:57] Enter SYS user’s password :
[24-08-2010 07:51:57]
[24-08-2010 07:51:57] Enter repository user password :
[24-08-2010 07:51:57]
[24-08-2010 07:51:57] Getting temporary tablespace from database…
[24-08-2010 07:51:57] Could not connect to SYS/(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=issgascop218)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=PASQ.issgascop218.global.iss.biz))): ORA-01031: insufficient privileges (DBD ERROR: OCISessionBegin)
One of possible reasons for ORA-01031 is OS user is not added to either oinstall or dba group. But in my case, its there.
After a search, i found that the reason is lack of PASSWORD FILE. Then i created password file which solved the issue
[oracle@issgascop218 ~]$ cd $ORACLE_HOME/dbs
[oracle@issgascop218 dbs]$ orapwd file=orapw$ORACLE_SID password=oracle entries=1 force=y
[oracle@issgascop218 dbs]$ ls -ltr
total 16096
-rw-r—–  1 oracle dba    8385 Sep 11  1998 init.ora
-rw-r—–  1 oracle dba   12920 May  3  2001 initdw.ora
-rw-rw—-  1 oracle dba      24 Mar 22  2007 lkPAS
-rw-rw—-  1 oracle dba    1552 Mar 22  2007 hc_PAS.dat
-rw-rw—-  1 oracle dba    1552 May  9  2007 hc_PASQ.dat
-rw-rw—-  1 oracle dba      24 May  9  2007 lkPASQ
-rw-r—–  1 oracle dba    2430 May  9  2007 initPASQ.ora
-rw-r—–  1 oracle dba    2560 Dec 28  2007 spfilePAS.oraold
-rw-r—–  1 oracle dba    2393 Dec 28  2007 initPAS.ora
-rw-r—–  1 oracle dba    1536 Apr 30 11:54 orapwPAS
-rw-r—–  1 oracle dba    2560 Jul  1 22:00 spfilePAS.ora
-rw-r—–  1 oracle dba    3584 Jul 12 22:00 spfilePASQ.ora
-rw-r—–  1 oracle dba 7716864 Aug 23 18:30 snapcf_PASQ.f
-rw-r—–  1 oracle dba 8601600 Aug 23 18:57 snapcf_PAS.f
-rw-r—–  1 oracle dba    1536 Aug 24 07:54 orapwPASQ
[oracle@issgascop218 dbs]$ emca -config dbcontrol db -repos recreate
Last few lines of above command
INFO: Repository successfully created
Aug 24, 2010 8:01:56 AM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) …
Aug 24, 2010 8:03:32 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Aug 24, 2010 8:03:33 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://issgascop218:5500/em <<<<<<<<<<<
Enterprise Manager configuration completed successfully
So, whenever you get ora-01031 while performing some action on EM, plz do check if this solution works out