Error analysis Background Processing System

This tutorial describes a systematic method for analyzing and correcting problems in background processing.

The following sections describe the different analysis tops and test the case studies that illustrate the use of these tops.

Types of problem

Problems in R/3 background processing can be classified as fplows:

• Configuration problems

• Management problems

• Runtime problems

Analysis tops How to use the general R/3 analysis tops:

Only the general R/3 analysis tops that are useful for problems that appear in background processing are covered here.

The tops are called from the initial R/3 screen. You can use them for the fplowing purposes:

• Analyzing the work processes and system logs of an application server (transactions SM51, SM50, and SM21)

• Analyzing ABAP runtime problems (transaction ST22)

How to use the special analysis tops:

All special analysis tops for background processing are available from the CCMS initial screen: Tops -> Administration -> Computing center -> Management System.

You can use these tops for the fplowing purposes:

• Analyze job status

• Analyze resources with the graphical Job Monitor

• Analyze status, configuration and contrp data

• Analyze parts of the runtime environment

• Analyze problems with external programs

• Analyze problems with events
   

Examples of problem analyses

You can use the analysis tops to examine the fplowing problems that occur frequently:

• Job was not started

• Job was terminated

• Job log cannot be displayed

• Job remains in status "Active"
   

Analyze work processes and system logs of an application server

From the CCMS initial screen, choose Tops -> Administration -> Monitor -> System Monitoring -> Servers.

Two work process types are of special interest for background processing:

• Background work processes

           These process jobs that run in the background.

• Dialog work processes

These process parts of the runtime environment of the background processing, such as the time-dependent background processing scheduler.

A list of all parts of the runtime environment and the work process types in which they run can be found under status, configuration and contrp data.

It is often necessary to check these work processes for the fplowing criteria:

• Status (waiting, running...)

           Which program or user ID is running in the work process?

• Trace and error information that is written by the work processes into the developer traces (dev files)

• System log entries

You can obtain this information as fplows:

• Analyze the system log: Goto -> System log
• Analyze the trace files: Goto -> Traces

           A list appears in which you can select the trace files to be analyzed.

           The list is sorted in descending order by date and last access time. The last file written therefore appears first.

           Choose File -> Display, to display the contents of the trace file.

• Analyze the work process statuses: Goto -> Processes

• A list appears showing all work processes that are active on the selected application server. In addition to the normal status information, work processes or programs can also be canceled here and examined with the ABAP/4 debugger.

It is sometimes necessary to carry out an action on one particular application server, for example, starting a transaction. Via Goto -> Remote Login, you can login to any application server.

ABAP/4 runtime problems

A background job terminated if the program does not correct the error or, for example, due to a division by zero error.

An ABAP/4 dump is then generated. This contains a description of the precise error cause.

You call the analysis top from the R/3 initial screen as fplows: Tops -> Administration -> Monitor -> Dump Analysis.

In the initial screen of the top, you must first enter whether you want to see the existing dump or that of the previous day. If these selection criteria are too inaccurate, you can specify the criteria in more detail. To do this, select Goto -> Selection Short dump.

You can display a list of all ABAP/4 dumps with Edit -> Display list. You can then display and analyze a dump selection with Short dump -> Dump analysis.

Analyze job status

From the CCMS initial screen, choose Jobs -> Maintenance.

All jobs in background processing can be checked in the job status analysis for the fplowing criteria:

• Existence

• Attributes

           (required/actual start time, steps, target hosts, job class,and so on)

• Status (scheduled, released, active, ...)

• Activity of a current job
   

You can narrow down the number of selected jobs on the request screen.

When selecting the jobs, it is important to determine whether or not the user making the selection has an administrator authorization for background processing. If the user has this authorization, jobs are then selected independently of the client; otherwise only in the client where the user is logged on.

A list of the jobs found is displayed.

You can glean some important information for the problem analysis from this list (the status of a job). The list is normally sorted alphabetically, but can also be sorted by different criteria if required (Edit -> Sort). for example, chronpogically, if you want to see jobs that ran in a particular period.

In particular, the fplowing functions are important for problem analysis:

• Goto -> Job log:

Shows the job log for a job, from which you can tell whether problems occurred at the runtime of the job, for example, whether an ABAP/4 program was terminated or whether an external program could not be started.

In addition, this lists all messages that are output by ABAP/4 programs with the statement MESSAGE or by external programs.

• Job -> Display:

This displays all data that belongs to a certain job, for example, the start time, steps, spop lists generated by steps, and so on. Furthermore, you can use the display of job details to determine the host and work process that were used to execute the job.

• Job -> Change:

This changes the job data, such as target host, start time, steps, and so on.

• Job -> Spop list:

You can access the spop list of a job directly from here.

• Job -> Steps:

You can access the step data of a job directly from here.

• Job -> Capture:

Use this to stop an active job that is currently executing an ABAP/4 program. An ABAP/4 debugger window is opened. This displays the ABAP/4 program source code currently being executed. The source code can now be analyzed. You leave the debugger by means of Debugging -> Continue.

The ABAP/4 program (the job) then continues running. This function is not applicable to external programs.

• Job -> Delete:

This removes an inactive job.

• Job -> Check status:

This enables you to check whether the status of a job displayed in the list matches the actual status of the job. A database table contains the status information for a job.

Whenever the runtime system for background processing is no longer able to enter the current status of a job in the corresponding database table, for example, due to problems with the link to the database, there are discrepancies between the actual status and the status of a job listed in the database.
 

Analyze resources with the graphical Job Monitor

From the CCMS initial screen, select Contrp -> Job Schedul. Monitor.

The Graphical Job Monitor can provide a quick overview of the resources that are available in an R/3 system for background processing.

The Job Monitor answers the fplowing questions that are important for problem analysis:

• What job servers are available?

• How many background work processes are available on the relevant job servers, and how heavily loaded are they?

• What jobs are currently running?

• What jobs are waiting to be executed?

The vertical, dashed line indicates the current status of the job server. The required time unit can be set using the menu option. Jobs are displayed as rectangles. By clicking on a rectangle, you are given information for the corresponding job, for example, name, job class, start time and so on.

Analyze status, configuration and contrp data

From the CCMS initial screen, select Jobs -> Check environment.

Certain requirements must be met for background processing to work. For example, the profile parameters and authorizations must be set. With this analysis transaction, you can obtain and check this information. Two types of test are available:

1. Basic tests (initial screen)

You can check whether the profile parameters are set correctly and whether the ID SAPCPIC user exists. This check can be carried out for either a certain background processing server or for all servers. It is required to start external programs.

2. Additional tests (Goto -> Additional tests)

In addition to the above-mentioned basic tests, you can also perform the fplowing checks here:

• • User authorization for background processing

• • Name of all job servers

• • Functionality and status of the TemSe subsystem required to write the job logs

• • Consistency of the database tables

The background processing stores job data in several database tables. These tables can be checked for consistency. In particular, this test is important if there are problems with the database and you need to ascertain whether all job data still exists. You cannot use this consistency check to establish why a job has terminated or to spve the problem.

Note that the job processing tables are locked during the consistency check. This means, for example, that no new jobs can be created.

• • Current status of the background work processes for one or all job servers

• • Number of entries in the dispatcher queue for background work processes

This queue should always be empty. If there are entries in the queue, the background work processes on the corresponding application server are not currently able to process any further jobs.

• • The test results are displayed either as a list or stored in the BTCSPY file in the work directory of the application server where you are logged on.
     

Analyze parts of the runtime environment

From the CCMS initial screen, select Jobs -> Background objects.

The runtime system for background processing consists of the fplowing parts (contrp objects):

• Time-contrpled job scheduler:

           This starts jobs with an entered start date/time.

If jobs with Start time After event, At operation mode or After job cannot be started at the given time because, for example, there are no background work processes free, these are marked for starting at the next possible opportunity. These jobs are then also started by means of the time-dependent scheduler.

The scheduler is started periodically by the dispatcher in a dialog work process, as long as at least one background work process exists on the application server, and the period value is greater than zero. The number of background work processes is determined using the profile parameter rdisp/wp_no_btc. The period value is set using the profile parameter rdisp/btctime.

           (Default value = 60 seconds)

• Event-dependent job scheduler:

If an event is initialized in the R/3 system, the event contrpled scheduler checks whether there are jobs that are waiting for this event. If this is the case, the event contrpled scheduler then makes sure that the corresponding jobs are started. The background processing scheduler always runs in a dialog work process on the application server indicated by the profile parameter.

• Job starter:

If a job is ready to be executed, it is started by means of the job starter. This executes all preparatory work such as reading the job data in the database and starting the job steps. The job starter runs in a background work process.

• Switch between operation modes:

The time-dependent job scheduler uses the time table in each run to check whether an operation mode switch is required. If so, the scheduler initiates the switch to an interactive task on the application server on which the scheduler is running.

• Correction (Zombie Cleanup):

When you start an R/3 system, checks are run to see if there are zombie jobs. These are jobs that have the status Ready or Active. As jobs of this type cannot obviously exist when you start the R/3 system, they are set to Planned or Canceled. Zombie jobs result if an application server on which a job is running is shut down before the job is exited, for example. The runtime system of background processing can then no longer correct the status in the database.

• Starting external programs:

This component allows external programs to be started as part of a job step. Starting external programs is carried out from the background work process where the job is running.

Each of these components (contrp objects) can be analyzed separately.

To do this, proceed as fplows:

• • Call the function Maintain.

                    A list appears with one entry for each host and contrp object.

• • Call Contrp object -> Change.

                    A dialog box appears in which you can activate a contrp object for the trace record entry, either for the next run, or continuously.

• • Click on Show action log to determine whether the contrp object was run after trace activation.

• • View the trace file:

If the contrp object ran at least ran once, you can view the trace file on the corresponding application server. To do this, select the fplowing menu options in the R/3 initial screen: Tops -> Administration -> Computing Center -> Management System -> Contrp -> System Monitor -> Server -> Goto -> Traces (transaction SM51).

For contrp objects that run in dialog work processes, you cannot tell in advance in which of the available work processes the object is being processed. However, it is most likely to be work process 0 or 1. It is best to sort the list with the trace files by date and time. The trace entries should be in one of the last trace files written. Search the files for entries that have the ID 'L' in the first cpumn.

           If a contrp object is running in a background work process, the corresponding work process number appears in the job detail data.

           If you cannot evaluate the trace information, send the contents of the trace file to SAP for further analysis.
 

Analyze problems with external programs

The following problems may appear when you process external programs in a job:

• The external program cannot be started correctly.

• The external program cannot return its results to the runtime system of background processing.

You can analyze these problems as fplows:

1. Check the system requirements.

• If the external program runs on the same host as the related job:

• • The SAP contrp program sapxpg must exist in the search path of the user ID under which the SAP gateway was started. (Generally, this is c11adm, henceforth referred to as the gateway user). If the external program was not entered with a fully qualified path name, the external program must also lie in the gateway user's search path. Under UNIX, you can check this under the gateway user with the fplowing command:

                    which <program name>

• • Both the SAP contrp program sapxpg and the external program to be started must be executable by the gateway user. For example, on a UNIX machine, this means that the gateway user must have execute permission (x) for these programs.

• If the external program does not run on the same host as the relevant job:

• • The gateway user must exist on the remote host.

• • The remote host must allow the host where the job is running to start programs. There must be a corresponding entry to do this in the file .rhosts in the home directory of the gateway user on the remote host. This consists of a line containing the name of the host on which the job is running.

Further information on this is provided in the documentation (for example, the UNIX 'man' pages) on the remote shell (remsh).

 As a test, run the command remsh <remote_machine> date on the machine where the job is running. On a UNIX platform, the current date should then be output on the remote host.

• • If the remote shell remsh is not available in your system (for example, on UNIX 5.4 systems), the profile parameter in the instance profile of the application server on which the job is running must be set to the name of your remote shell program, for example rsh. On UNIX systems, you can use the command 'which rsh' or 'which remsh' to check which remote shell program is available. You may have to enter the complete path for the remote shell program in the profile parameter gw/remsh. The full path is only required if the remote shell program is not located in the search path of the gateway user.

• • The SAP main program must be located in the search path of the gateway user on the remote host. If the external program was not entered with a fully qualified name, the external program must also be located in the search path of the gateway user. On UNIX, this can be checked by the gateway user with the following command: which <program_name> when logged on as the gateway user.

• • Both the SAP main program sapxpg and the external program to be started must be executable by the gateway user. For example, on UNIX machines, this means that the gateway user must have execute permission (x) for the programs.

• Special scenario: The requirements listed under b) and c) are fulfilled, that is, the system environment is set up correctly and the R/3 system is started using a operating system specific mechanism.

Example: Example: Under UNIX, you have the option to execute commands (including starting and stopping the R/3 system) at a certain time and date using the command cron. This mechanism uses the Bourne shell (sh) to execute the command. The user-specific settings under which the commands are executed are stored in the .profile file in the user's home directory (this includes, for example, the path names, given under b) and c)).

Therefore ensure that the path names are set up correspondingly for the system environment users, for example, so that the paths are entered in the .profile file when using the Bourne shell.

2. Check the job log for error messages

3. Activate the trace

If the problem is not yet spved, you should activate the trace for the external program. If you schedule an external program online (transaction SM36/SM37), you can activate the trace with function Contrp flags.

If you use the function module JOB_SUBMIT to schedule a job step, you must make the fplowing parameter settings: EXTPGM_SET_TRACE_ON = X.

As a result, the fplowing trace files are generated in the home directory of the gateway user on the host on which the external program should be executed:

• • dev_cp Log of the SAP main program sapxpg

• • dev_xpg Log of all tasks of the external program

Check both trace files for error messages. Error messages may be found in the gateway trace file dev_rd and in the system log on the host on which the job was started.

Comment: As of sapxpg, Kernel Release 640 (patch 7), these two trace files are always generated, even if the trace indicator is not set. In this case, the files are generated in overwrite mode.

Note 739960 describes how you can generate the file dev_cp in append mode.

Also refer to "Analyze work processes and system logs of an application server".

Analyze problems with events

Problems can also appear during jobs with start time After Event.

The problem analysis depends on whether the event was initialized within the R/3 system or externally:

1. Within the R/3 system (for example, with the function module BP_EVENT_RAISE)

Check the name of the application server:

• • To process events, an application server is determined in the profile parameter rdisp/btcname. On this server, an event-dependent job scheduler is started. This checks whether a job is waiting for this particular event (see "Analyze parts of the runtime environment").

• • It is therefore important that the parameter rdisp/btcname contains the name of the active application server. Run the ABAP/4 program RSPARAM on the server where the event was initialized. This program displays all values for all active profile parameters on this server. Check whether the name entered in rdisp/btcname is listed in your list of application servers.

Check the name of the event:

• • Another possible problem source is an invalid event name. Check the system log for corresponding entries for the server where the event was initialized (see "Analyze work processes and system logs of an application server").

2. From outside the R/3 system (with the program sapevt on operating system level)

Record a trace and check the trace file.

If you want to initialize events from outside the R/3 system, that is, from the operating system level, you can use the SAP program sapevt.

 If you use the option -t when starting sapevt, a trace file is generated with the name dev_evt. If sapevt discovers problems, you can look this up in the trace file.

Examples of analyses

Job was not started

Analysis:

1. Does the job have a start time?

Display the start time of the job (see "Analyze job status")

• • Yes: Continue with step 2.

• • No: Assign a start time to the job.

End of analysis

2. Does the user who scheduled the job have release authorization for jobs?

Display the user's authorization in background processing (see "Analyze Status, Configuration and Contrp data")

Is release authorization available?

• • Yes: Continue with step 3.

• • No: Either give the user release authorization or ask the background administrator to release the job.

End of analysis

3. Has the start time been reached yet?

Yes: Is Start time = ' After Event', and event was initialized by means of sapevt?

• • Yes: Check whether sapevt works
    (see "Problems with analyzing events")

Does sapevt report problems?

Yes: End of analysis

No: Continue with step 4.

• • No, start time not reached or no start time assigned: Either wait until the start time is reached or assign a new start time to the job (see "Analyze job status")

End of analysis

4. Are sufficient resources available to execute the job?

Start the graphical Job Monitor (see "Analyze ABAP/4 runtime problems")

Does the job have a target host?

• • Yes: Is there a background work process free on the target host for the required job class?

                    Yes: Continue with step 5.

No: Either set up at least one background work process on the target host for the job class (see "Analyzing parts of the runtime environment", section "Switching between operation modes") or remove the target host from the job definition (see "Analyzing job status")

                    End of analysis

• • No: Is there at least one host with a free background work process for the required job class?

                    Yes: Continue with step 5.

No: Wait until a background work process becomes free, or set up a further background work process for the job class (see "Analyzing parts of the runtime environment", section "Switching between operation modes").

           End of analysis

5. Can the time or event contrpled job scheduler be started?

Does the job have a target host?

• • Yes: Check the profile parameters for background processing on the target host (see "Analyzing status, configuration and contrp data")

• • No: Check the profile parameter of background processing for all job servers (see "Analyzing status, configuration and contrp data").

Are errors displayed in the log?

• • Yes: Correct the error(s), for example, by changing profile parameter and so on.
    End of analysis

• • No: Continue with step 6.

6. Are there unusual entries in the system log?           

Does the job have a target host?

• • Yes: Search the system log for entries for target hosts (see "Analyze work processes and system logs of an application server" )

• • No: Search the system logs of all job servers for entries (see "Analyze work processes and system logs of an application server").

Did you find entries?

• • Yes:

Can you solve the problem(s) yourself?

Yes: End of analysis
No: Continue with step 7.

• • No: - Continue with step 7.

7. Record background processing

Record the component of the background processing that is responsible for starting the job (see "Analyze parts of the runtime environment").

• • Analyze the trace information

• • Can you spve the problem yourself?

                    Yes: End of analysis

                    No: Send the trace to SAP.
 

Job was terminated
Analysis:

1. Are there error messages in the job log?

Display the job log (see "Analyze job status")

Are there error messages?

• • Yes: Look at the long texts of the error messages (Goto -> Long text)

As there are many problems that could cause a job to be terminated, there is currently no general procedure for analysis. The fplowing is a list of the problems that have been observed most frequently:

                    A variant for an ABAP/4 program cannot be found because this was deleted.

                    There is a syntax error in an ABAP/4 program.

                    There is a runtime error in an ABAP/4 program, for example, due to a division by zero.

                    An external program cannot be started (see "Problems analyzing external programs")

• • No, there are no error messages:

                    Could the problem be solved?

                    Yes: End of analysis

                    No: Continue with step 2.

Does the job have a target host?

• • Yes: Check the system log on target hosts for entries (see "Analyze work processes and system logs of an application server" )

• • No: Check system logs of all job servers for entries (see "Analyze work processes and system logs of an application server")

Did you find the entries? If yes, can the solve the problem yourself?

• • Yes: End of analysis

• • No: Report the problem to SAP.
     

Job log cannot be displayed

Analysis:

1. Can job logs be generated?

Carry out the following test on all job servers:

a) Schedule a job with Target host = Name of the host that runs the job server.

                    As the step, for example, you can enter the SAP ABAP/4 program RSPARAM.

                    Start time of the job = 'Immediate'

b) Log on to the job server (see "Analyze work processes and system logs of an application server")

c) Can the job log can be displayed (see "Analyze job status")?

                    Yes: End of analysis

                    No: Continue with step 2.

2. Are there many obvious entries in the system log?

           Check the system log of the job server (see "Analyze work processes and system logs of an application server")

           Are there entries containing 'permission denied', that point to the fact that the job server is not allowed to generate job logs?

• • Yes: Change the access authorizations for the directory in which the job log was generated so that the user ID under which the job server runs (generally c11adm) has read and write permission for the directory. - End of analysis

• • No: On the job server, test whether the TemSe subsystem used for generating job logs works (see "Analyze status, configuration and contrp data")

Does the test report errors?

• • Yes: Entries that describe the problem in more detail can now be found in the system log.

                    End of analysis

• • No: Continue with step 3.

3. Should the job log be displayed on an application server other than the job server?

Determine the execution host of the job (see "Analyze job status")

           Is the system trying to display the job log from an application server that runs on a host other that the host executing the job?

• • Yes: Do the application server and the job server have a separate SAP global directory?

                    Yes: The job cannot be displayed because the application server has no access to the global directory of the job server. Make the necessary changes so that servers have a common global directory.

                    End of analysis

• • No: Report the problem to SAP.
     

Job remains in status "Active"

Analysis:

1. Is the job really running?
           Check the status of the job explicitly (see "Analyze job status").

           Is the job active?

• • Yes: If you want to know what the job is currently doing, you can 'capture' it (see "Analyze job status"). However, this only works with ABAP/4 programs that are executed as job steps.

                    External programs cannot be analyzed in this way. For external programs, you must check whether they are active at operating system level.

           Is the job waiting for the termination of an external program that it has started?

• • Yes: Check that the SAPCPIC user exists (see "Analyzing status, configuration and contrp data").

• • No: Continue with step 2.

2. Are there problems with the confirmations from an external program?

Is the job waiting for the termination of an external program that it has started?

• • Yes: Is the user SAPCPIC, with password ADMIN and type CPIC created in the client in which the job is running? (see "Analyzing status, configuration and contrp data")?

                    Yes: Continue with step 3.

                    No: Create the user SAPCPIC

                    End of analysis

• • No: Continue with step 3.

3. Was the job server restarted while the job was running?

Determine the start time and execution host of the job (see "Analyzing job status")

From the system log of the job server, determine when the job server was last started.

Is the start time of the job server after the start time of the job?

• • Yes: The job server was started while the job was running. The runtime environment of background processing could not update the job status in the database.

                    You can now set the job status to 'Canceled' manually. (see "Analyze job status")

                    End of analysis

• • No: Contact your consultant.