In this section, we cover the steps needed to run and interpret the PVT system. Like other conformance test systems, the PVT is not, nor can it be, a totally automated process. The PVT code and documentation are best seen as components of an integrated and interdependent system, which includes the operator as its active, directing component. We include in the notion of operation the process of interpreting the behavior of the programs, particularly with regard to conformance.
Let us refer to the execution of a set of PVT programs as a session. For each session, a primary workstation must be specified, together with an optional set of secondary workstations. The primary workstation is the one whose features are thoroughly tested by the PVT programs. Secondary workstations are tested only by certain special-purpose programs such as those dealing with multiple workstations. Thus, for each workstation to be tested in depth, the operator must run a distinct session.
In order to conform, a PHIGS implementation must support at least one workstation of category OUTIN. No other workstations need be supported. If other workstations are accessible, however, they must have the capabilities associated with their categories.
Thus, a typical scenario for conformance testing is first, a session in which an OUTIN workstation is designated as primary and in which all the PVT programs are executed. Such a session checks the minimum requirements for conformance.
Subsequent sessions deal with the capabilities of other workstations of whatever category. Each of these workstations is designated in turn as primary and an appropriate subset of the PVT is executed for it. Since there is no need to re-run tests which are not relevant to the capabilities of these other workstations, not all PVT programs need be executed in these subsequent sessions. Use the information in the CHARACTERISTICS entry of the program design documentation (summarized in Appendix: Test Program Characteristics and in the pgm-char.prt file) to decide which programs are pertinent, according to the table below.
Workstation
Category CHARACTERISTICS
-------- ---------------
OUTIN yn-- or ny-- or yy--
OUTPUT yn--
INPUT ny--
Programs with a CHARACTERISTIC entry of "--nn" require no operator interaction and may be run in batch mode; otherwise there must be an operator for the test. By convention, within a module, all the passive programs precede the interactive ones (e.g. P01-P05 are passive, P06-P09 are interactive).
Figure 4 summarizes the way in which several sessions are performed
so as to constitute a complete test of conformance.
for each PHIGS implementation (hardware platform + software)
for each language binding (Fortran, C)
INITPH for primary OUTIN workstation
execute all PVT tests for this language
use COMPLT to check completeness of session
for each additional workstation type
INITPH for this workstation type
execute workstation PVT tests of appropriate category
(OUTIN,OUTPUT,INPUT) for this language
(see PGM_CHAR.PRT file for guidance)
use COMPLT to check completeness of session
next workstation type
next language binding
next implementation
Figure 4: Use of PVT sessions to test conformance
The PVT Configuration file contains the information which is specific to the PHIGS implementation being tested, but the same for all test programs within the PVT session. Its purpose is to allow an operator to specify such information only once at the beginning of each session, rather than repeating it for each program. The INITPH program creates this file, based on the operator's responses. The program is stored as initph.f or initph.c in the PVT root. It uses some subroutines from the global subroutine library, and also from PHIGS itself, so these libraries must be available as it is compiled and linked.
Execute INITPH, and respond as prompted. Most responses are in the form of an integer. When parameter values are requested, they should be supplied as needed by the Fortran binding. Conversion to C will be done automatically by the system. E.g. the error file parameter to <open phigs> is supplied by you as an integer, not a character string. You should be prepared to supply the following information to INITPH:
The first two questions concern the input parameters to be passed to the <open phigs> function, whenever that function is needed in a PVT program. Simply supply the numeric values your implementation requires.
The next questions concern the workstations accessible to this implementation. Tell INITPH the total number of accessible workstations (primary and secondary). Then, for each of these, supply the three parameters by which <open workstation> can open it: workstation identifier, connection identifier, and workstation type. Be sure that the first set of parameters refers to the primary workstation.
Next, INITPH will ask you whether you want a message to be generated whenever the implementation successfully passes a TC in a test program. You can specify either that such messages are always suppressed, always generated, or that each program asks the operator what to do, so that he or she can selectively suppress pass-messages at run-time. No other type of message can be suppressed.
The next questions deal with the destination of messages as they are generated. First, indicate whether or not messages are to be sent to the operator (typically on the screen). Next, indicate to which files messages should be written. Individual message files are created once per execution of a test program. By default they are given the same name as the program, but with a "msg" suffix, rather than "f" or "c". The global message file is a cumulative file to which messages are appended whenever a test program is run.
These are independent choices; messages can be sent to any combination of the three destinations: operator, individual file, or global file. Each enabled destination receives exactly the same set of messages.
If you specify a global file, you must then provide an absolute name for this file, so that all programs can write to it. This response is, of course, not in the form of an integer, as are the others. You may want to specify a distinct name for the global message file of each PVT session. It may be useful for the name to reflect the language binding and the type of the primary workstation of the session which it is recording. E.g. "init-vt340-c.msg" might be used to indicate the validation of a VT-340 terminal using the C binding.
Since some operating systems have reserved logical unit numbers in Fortran, you are also asked to provide these for the individual and/or global file, if they have been designated as destinations.
You must specify the maximum number of characters per line which should be generated when the PVT system formats a message. Some messages may be quite long and would not fit on a reasonably-sized single line. Message text is never truncated; rather, it is simply broken into lines of the specified size.
Many of the PVT programs that test error handling do so by reading the error file created by the implementation. If possible, the ESETUP subroutine in 09.01 will assign the name "pxx.erf" to the error file generated by the program pxx, by explicitly opening the file and then passing its logical unit number to <open phigs>. Otherwise, the system default name is used. Tell INITPH whether your implementation allows opening of the error file prior to <open phigs>.
When the PVT attempts to read and examine the error file, it does not assume that there is exactly one record per error message. Therefore, if each message contains a special character string (such as "PHIGS_ERROR:"), supply this value so that multi-record error messages can be distinguished automatically.
Finally, INITPH will attempt to automatically create an error file with a single error message; the text of this message will be used as a separator when later tests analyze the contents of error files generated by the system. Therefore, this single-message error file must be readable by those tests, under the name "errsep.erf". If the error file cannot be explicitly named by the program, you must re-name it after execution of INITPH from its system name to "errsep.erf". INITPH will display a message if this is necessary.
Many of the interactive tests randomize the choices presented to the operator so that the correct responses are not predictable (see section below on operator interaction). For some purposes, however, it is desirable that the tests execute with repeatable displays and operator prompts. If you want to get repeatable behavior, enter a real number between 0.1 and 0.9. This value will be used as the seed for a random number sequence. Thus, re-initializing to a distinct value between 0.1 and 0.9 will cause repeatable behavior within the new session, but distinct from that of the previous session.
Entering any value outside the range of 0.1-0.9 causes the system to use a time function to set the seed for the random sequence, and thus generates truly random operator choices.
If the primary workstation is capable of visual output, answer yes, otherwise no. The normal answer is "yes". The question is here to allow for testing of INPUT-only workstations and metafile workstations in later versions of the PVT.
When running the interactive tests, the system poses questions to the operator. This option lets you choose the mechanism for transmitting those questions: 1-Fortran print (or its C equivalent), 2-PHIGS <message>, or 3-PHIGS <text>. In many systems, Fortran PRINT will write to another window. PHIGS <text> will use some of the PHIGS display space for questions, possibly leaving less room for the picture under examination.
If using Fortran print or PHIGS <message>, you must specify the maximum number of characters per line in interactive prompts.
You may specify the means by which the operator is to respond to prompts: either 1-Fortran read (or its C equivalent), or 2-PHIGS <request string>; (and the device number for <request string>).
If you are using PHIGS for operator input or output, you must specify where the dialogue area (containing operator prompts and responses) should be. The choices are: 1-dialogue at bottom of screen or PHIGS window, 2-dialogue at right, or 3-toggle picture and dialogue so that the screen can alternate betweem the two. Since the picture area will use the largest square remaining on the screen after the dialogue area has been reserved, it is recommended that the dialogue area be on the right for wide screens, and on the bottom for tall screens.
You must then specify what percentage of the screen should be reserved for the dialogue area. Something in the range of 15-30 percent is usually a reasonable choice.
Finally, if you are using PHIGS for operator input, specify what percentage of the dialogue area is to be reserved for the echo of operator responses (the remaining area is used for prompts). Since prompts are usually larger than the responses, something like 10-20 percent is a good choice.
You must tell the system the ratio of meters to DC units for this workstation. You may either enter the number directly, or physically measure a diagonal line on the screen, and INITPH will compute the result.
At the successful conclusion of INITPH, the operator receives a report on the names of the files to which the PVT configuration file and PVT configuration report file (the human-readable version) have been written.
In order to understand the significance of the programs' output for conformance, you should read over the module documentation and those sections of the standard referred to in the #S entries of the SRs. You will then have the appropriate background information to allow you to focus on the details of each test result.
Invoke the system-dependent command(s) needed to compile, link and execute each program in the session. There are a number of possible outcomes, each of which are dealt with below. We assume throughout the following sections that the PVT programs are themselves valid. If you encounter some aspect of a program which you believe is incorrect, please submit a comment to that effect as described in section 1.
Normally, test programs are executed with the error handler defined by the PVT (residing in errlib in the PVT root) in effect. Those programs, however, that test the behavior of the system error handler may have to be invoked in a different way, so as to exclude errlib. These programs include all those in modules numbered 09.01.xx.02, and 09.01/p03-p05.
Note that some nodes pertain only to a specific language binding. For version 2.1 of the PVT these are:
Fortran only: C only:
09.01.10 09.01.11
09.01.10.01 09.01.11.01
09.01.10.02 09.01.11.02
09.02.10 09.02.11
11.01 11.02
If your compiler does not support some feature of ANSI Fortran [FORT78] or ANSI C [C1989] used by a PVT program, you must find a different way to perform the equivalent operation. Note, however, that such lack of support constitutes non-conformance, since the Fortran binding [PHFOR90] incorporates the Fortran standard [FORT78] and the C binding [PHC91] incorporates the C standard [C1989]. Similarly, problems in the system software which performs linking of subroutines must be resolved by you. Any problem with system software which prevents a valid invocation of a PHIGS function by a program renders the implementation non-conforming.
Uncorrected failure to begin execution of a program should be noted by running the OPRCMT program. This will record operator comments as messages in the specified destination files. Therefore the results of all tests (even non-executing tests) are saved in a uniform way. Even though the OPRCMT program resides in the PVT root, you should be in the sub-directory of the appropriate module when executing OPRCMT so that the individual message file will be written in the correct location. For example, suppose program 06.01.02/01 cannot be executed. Then, while still in that sub-directory, execute OPRCMT, enter the identifier of the module and program ("06.01.02/01"), and then enter your comment(s) as prompted by the program.
The PVT programs are written conservatively so as to fall well within the operational limits of any reasonable PHIGS implementation. Furthermore, the PERHND (or perr_hand, for C) subroutine supplied in errlib.f (or errlib.c) should handle any errors signalled by PHIGS functions. Therefore, it is to be expected that every PVT program, once started, will proceed until completion under the control of the appropriate PVT subroutine, such as ENDIT for successful completion, or UNMSG for aborted completion.
If the PHIGS system itself aborts execution, it means that the implementation fails the test and does not conform, since no unhandled PHIGS error conditions should arise within the PVT code. If not otherwise evident, this outcome is indicated by the absence of two SY-type messages (the first containing the error count, the second to identify PVT completion) at the end of the message file(s).
You may wish to use the OPRCMT program (see preceding section) to enter an operator comment on the cause of failure, but be aware that the individual message file generated by the original program may be over-written. The cumulative message file will contain the output of both the aborted program and OPRCMT.
The interactive tests all present a graphical display to the operator in the picture area of the screen and then ask him/her a question about it. See the OPERATOR SCRIPT entries in the DOC.TXT files for detailed instructions if the screen is not self-explanatory.
In general, the format of the response should be obvious from the question. Most questions are single integer choices, such as "Which fill area is different?" referring to a display of perhaps 8 numbered fill areas. Answer such questions with a single integer, in the example given, 1 through 8. If there is no unique correct answer (e.g. all fill areas are the same, or several differ from each other, or the screen is blank) respond with "0" (see below).
When a list is called for (e.g. "Which primitives are highlighted?"), enter a list of integers separated by commas, e.g. "2,3,5,6". If the order of the list is significant, the prompt will so indicate. The single character "n" denotes the empty list (e.g. your response should be "n" if none of the displayed primitives were highlighted).
Other response types are: real numbers, "y" or "n" for yes/no questions, and character strings. If a response has an incorrect format, PVT issues a warning and lets you re-enter the response.
Except for yes/no questions, which require "y" or "n", a response of "0" is always valid and means that the question is unanswerable. After a "0" response, the operator is given an opportunity to enter a free-form comment explaining the nature of the problem. The comment is then recorded in the output message file. An operator comment is entered as several text lines, and terminated by a line containing a single period. If the first line of the comment is the termination line, no message is generated. Usually a "0" response causes the associated test case to fail.
Also, the operator may respond to any prompt with "@" (at-sign). This allows him/her to enter a comment pertaining to the current picture and prompt, but without causing failure as would the "0" response. The comment is made before actually answering the question. As with "0", the comment is recorded as a message and terminated by a line with a single period. After entering the comment, the original question is put to the operator again and must still be answered.
When the operator has indicated to INITPH that the dialogue and picture area are to be toggled (see section 4.2.7), the toggling is controlled by entering "t" or "T". The effect is to switch between displaying the dialogue area and the picture area. The actual answer may be entered when either area is visible.
The PVT programs sometimes examine whether they can proceed to execute, or whether anomalous conditions have arisen such that further execution is futile. In the latter case, before stopping, the program will issue a message of type UN or NC, indicating the reason for aborting further testing. Normally, all the TCs within the program that are appropriate for the implementation are executed.
In either case, the output of a completed PVT program consists of a sequence of messages. These messages are sent to the destinations specified in the PVT configuration file. They are followed by three records each consisting of a single period. This serves to separate visually the output of each test in the global message file.
The messages are categorized into six types in order to help the operator assess their significance. The output is formatted so that these types appear in the leftmost columns, with the text of the messages indented. The following table describes the circumstances associated with each of the message types.
Message Types
SY: System message, generated by INITGL and ENDIT. These
messages herald the beginning and completion of execution
of a PVT program. The next-to-last message contains a
summary of the number of TCs executed and failures
detected.
OK: The standard message resulting from a passed test case,
generated by PASS or IFPF. INITPH allows the operator to
suppress messages of this type.
FA: The standard message resulting from a failed test case,
generated by FAIL or IFPF.
IN: Informational message, generated by INMSG or SIGTST.
Some situation was encountered which, while not
erroneous, is worth noting, for example, the
implementation is such that a certain condition cannot be
tested. Also, this type of message can be used for the
purpose of conveying useful but not conformance-related
information about the implementation, such as support of
optional PHIGS features.
OP: Message as entered by the operator, either as a result of
entering "0", or "@" in response to a prompt. Typically,
this message is used to record some unusual or unexpected
visual feature of the display.
UN: Unanticipated error message, generated by UNMSG. This
message is generated when the program detects some
anomalous condition that prevents further processing. No
inference is to be drawn concerning implementation
conformance.
NC: Unanticipated non-conformance error message, generated by
NCMSG or CHKINQ. The program must abort because a feature
of PHIGS which is not under test, but needed for the test,
has failed. Since the incidental feature is mandated by
the standard, its failure indicates non-conformance. Most
commonly, this message type is generated by CHKINQ when a
required inquire function fails.
The operator may inspect the output at several levels of detail. In the simplest case, he or she may simply look at the error count at the end, to see whether or not all tests were passed. By scanning the left column for FA or NC type messages, the operator can focus on problems of non-conformance and ignore the other output. Finally, one can note the content of informational messages, which tests were passed or failed, which SRs are implicated in the result, and what the precise conditions of the test were.
Obviously, these is no foolproof procedure for analyzing unexpected results, but we recommend the following steps:
When you believe you have completed a session you may use the COMPLT utility to examine the global message file produced by the session so as to check that no programs have been skipped. COMPLT does not work in the absence of a global file. This utility is a simple Fortran or C program; it does not call PHIGS.
As prompted, enter the name of the global message file to be checked, whether the file was produced by the Fortran or C binding, and the expected subset of the PVT, based on the category of the primary workstation. COMPLT will report any discrepancies between the expected subset of the PVT and the actual subset executed, as reflected by the presence of "End execution" messages in the global message file. The OPRCMT utility generates the "End execution" message expected by COMPLT; therefore invoking OPRCMT for any programs in the session that have failed to run to completion is required in order to preclude error reports from COMPLT.
Users who are interested in debugging as well as conformance checking may wish to augment the tests so as to produce more detailed information. We recommend that the extra output from such enhancements follow the usual conventions of the PVT system. In particular, we recommend that the information be formatted into a character variable (e.g. using Fortran's WRITE statement) and then written out as an IN type message with the INMSG subroutine. There are many examples of this in the normal code. See these if you need further guidance.
Previous section is Section 3: Installation.