How to Troubleshoot ORA-00600 and ORA-07445 Internal error in Oracle ?

ORA-00600 and ORA-07445 Troubleshooting are an integral part of Oracle Database Administrator's life. In some cases these error could crash database instance's or in others does very minor impact.

So, I would suggest to start working on ORA-00600 and ORA-07445 immediately, you notice them. My agenda for this post is to explain the approach for solving ORA-00600 and ORA-07445 Internal error, which would be more helpful rather than solving one or two issues.

What is ORA-00600 and ORA-07445 Internal error ?

As we know, basic code of database is written in C and C++ Language. When a function in C call another function but got some unexpected results and which is not handled by Oracle Code, then DBA encounter ORA-00600 or ORA-07445 Internal error. On further discussion in this article, we will also see which function is failed and case these errors.

Troubleshooting ORA-00600 and ORA-07445

You will find hundreds of post's on internet solving ORA-00600 and ORA-07445, but most of them are dealing with only one or two error messages.

Few from my own blog:

http://www.dbas-oracle.com/2011/07/ora-600-kwqbdrcp101-after-drop-or.html
http://www.dbas-oracle.com/2011/06/ora-00600-internal-error-code-arguments.html
http://www.dbas-oracle.com/2011/05/ora-00600-internal-error-code-arguments.html
http://www.dbas-oracle.com/2011/04/ora-07445-exception-encountered-core.html

 Here, We are going to discussion, general approach to investigate and solve these issues.

1. Getting Error Detail from Alter log and Trace file:

Whenever, this error arise, it is written in to database alert log files with trace file having details about this error. So, first task is to find trace file from alter log holding about this error.

Oracle 11g Alter log entry with error ORA-00600 and trace files path

Wed Feb 13 15:41:47 2013
Errors in file /u01/app/oracle/diag/rdbms/remotedba/remotedba1/trace/remotedba1_pr00_19608.trc (incident=222458):
ORA-00600: internal error code, arguments: [1433], [60], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/remotedba/remotedba1/incident/incdir_222458/remotedba1_pr00_19608_i222458.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

Oracle 11g Alter log entry with error ORA-07445 and trace files path

Sun May 05 00:30:42 2013
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x4255535D] [PC:0x90FD91A, kglic0()+1100] [flags: 0x0, count: 1]
Errors in file /home/oracle/admin/remotedb/diag/rdbms/remotedb5/remotedb/trace/remotedb_m000_20640.trc  (incident=480694):
ORA-07445: exception encountered: core dump [kglic0()+1100] [SIGSEGV] [ADDR:0x4255535D] [PC:0x90FD91A] [Address not mapped to object] []
Incident details in: /home/oracle/admin/remotedb/diag/rdbms/remotedb5/remotedb/incident/incdir_480694/remotedb_m000_20640_i480694.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun May 05 00:31:31 2013

In case of Oracle 10g, Just A small difference, Instead of Incident file detail you will see trace file path. Open the trace/incident files and you will find description about this error. Though, it's not very human readable.

2. Login to https://support.oracle.com to Troubleshoot:

If, you need an accurate solution of this error, then you can only find at https://support.oracle.com. So, first of all, you must have an Oracle Support user ID and Password.

Login to support.oracle.com

Search for Article "153788.1" in knowledge base. This will open an article "ORA-600/ORA-7445/ORA-700 Error Look-up Tool [Article ID 153788.1]" This looks like this.


3. Search Error in this tool:

In the above screen choose error code DBA is facing in alter log files (ORA-600 or ORA-7445). Next is filling "Error Code First Argument". Whatever you are getting in fist [] after ORA-600 or ORA-7445 is called as first argument.

E.g. ORA-00600: internal error code, arguments: [1433]

In this error 1433 is the first argument.

ORA-07445: exception encountered: core dump [kglic0()+1100]

In this, kglic0 is the first argument. You just need to copy first argument in "Error Code First Argument" box. Here, I am taking ORA-00600: internal error code, arguments: [1433] as an test case.

Choose Oracle version database is running on from "Oracle RDBMS Version" drop box. Here, I am choosing 11.2.0.3. After filling all this information. Click "Look-up Error". This will search Oracle knowledge base for matching documents.


Document 138413.1 is the matching document with given error in Alert log file. This document has all details about bug, affected versions, workaround or fix for it. Click on the link for the document and see the details.

In this case, below are the details.

    a. Bug number: 12899768  
    b. Fixed in versions: 11.2.0.2.8, 11.2.0.2.BP18, 11.2.0.3.BP11, 12.1.0.0
    c. Workaround: Increase _messages init.ora parameter to a large value.

Apply the suggested solution and your problem is solved.

Now, let's take another case ORA-07445: exception encountered: core dump [kglic0()+1100]. I tried to match this Look-up tool, below is the screen shot.



It matches with document ID 1321840.1. To further make sure, database is hitting this bug DBA can match call stack (It is flow of C or C++ functions called to complete the task)of the trace file with Document given for both type of errors. You can find this in error trace file by searching "call stack"

Below is call stack for ORA-07445.

skdstdst()+36        call     kgdsdst()        
ksedst1()+98         call     skdstdst()       
ksedst()+34          call     ksedst1()        
dbkedDefDump()+2741  call     ksedst()         
ksedmp()+36          call     dbkedDefDump()   
ssexhd()+2366        call     ksedmp()         
__sighandler()       call     ssexhd()         
kglic0()+1100        signal   __sighandler()
kksIterCursorStat()  call     kglic0()+295           
kewrrtsq_rank_topsq  call     kksIterCursorStat()
l()+235                
kewrbtsq_build_tops  call     kewrrtsq_rank_topsq
ql()+123                      l()              
kewrftsq_flush_tops  call     kewrbtsq_build_tops
ql()+674                      ql()             
kewrft_flush_table(  call     kewrftsq_flush_tops
)+397                         ql()             
kewrftec_flush_tabl  call     kewrft_flush_table(
e_ehdlcx()+747                )                
kewrfat_flush_all_t  call     kewrftec_flush_tabl
ables()+1401                  e_ehdlcx()       
kewrfos_flush_onesn  call     kewrfat_flush_all_t
ap()+165                      ables()          
kewrfsc_flush_snaps  call     kewrfos_flush_onesn
hot_c()+607                   ap()             
kewrafs_auto_flush_  call     kewrfsc_flush_snaps
slave()+752                   hot_c()          
kebm_slave_main()+5  call     kewrafs_auto_flush_
86                            slave()          
ksvrdp()+1737        call     kebm_slave_main()
opirip()+669         call     ksvrdp()         
opidrv()+598         call     opirip()         
sou2o()+98           call     opidrv()         
opimai_real()+261    call     sou2o()          
ssthrdmain()+252     call     opimai_real()    
main()+196           call     ssthrdmain()     
__libc_start_main()  call     main()           
+244 _start()+36          call     __libc_start_main()

You will find below function at starting of all calls in call stack, this is how functions are called in C and C++ to start work.

ksvrdp()+1737        call     kebm_slave_main() 
opirip()+669         call     ksvrdp()          
opidrv()+598         call     opirip()          
sou2o()+98           call     opidrv()          
opimai_real()+261    call     sou2o()           
ssthrdmain()+252     call     opimai_real()     
main()+196           call     ssthrdmain()      
__libc_start_main()  call     main()            
+244                                            
_start()+36          call     __libc_start_main()

You will find below function at ending of all calls in call stack, this is how functions are called in C and C++ in case of exception occurs.

skdstdst()+36        call     kgdsdst()    
ksedst1()+98         call     skdstdst()                                                                                  
ksedst()+34          call     ksedst1()    
dbkedDefDump()+2741  call     ksedst()     
ksedmp()+36          call     dbkedDefDump()
ssexhd()+2366        call     ksedmp()     
__sighandler()       call     ssexhd()     
kglic0()+1100        signal   __sighandler()

So, start matching function sequences after ksvrdp (default function ends) from error trace file with call stack given in Document and check if match happens.

In this case, document ID 1321840.1 says:

ERROR:
  ORA-7445 failing function "kglic0"

If we see before exception handling call stack last calling function in our call stack was "kglic0". So this is a perfect match with document ID 1321840.1 error message. So, You can apply suggested solution given in the solution document.

There could be situations when, Search by DBA in look-up tool doesn't show any result like given below. In this case, either you can  click on "Do a general Search for Knowledge" to find some related documents. This might help you but final solution would be to open as SR with Oracle Support.



Summary of the document.

1. For troubleshooting ORA-00600 and ORA-07455, Support.oracle.com is the best tool to work.
2. User ORA-600/ORA-7445/ORA-700 Error Look-up Tool [Article ID 153788.1] for troubleshooting these errors (Search in knowledge base for this article).
3. Provide input in error look up tool and find relevant document. (if founed)
4. Match call stack in trace file with call stack in document (if given) to confirm exact match.
5. If you could not find any matching document, click on Do a general search for Knowledge.
6. Before apply any solution in document, Make sure error code, code first argument and RDBMS version must match with your error environment.
7. In all errors and both the cases (document match and mismatch) DBA has to open a Service Request to take confirmation about bug and it's fix.
8.  DBA should not apply any fix directly on production environment. He should test first of any test env and then apply on prod env.

Please share if this post helps you to solve your ORA-00600 and ORA-074555  Internal error.

2 comments:

  1. So the resolution of the error ORA-00600: internal error code, arguments: [1433] is to increase _messages init.ora parameter to a large value.

    ReplyDelete