Code pages and Unicode support in StreamServe

The StreamServer and the StreamServe tools support Unicode version 2.1, in the following encoding schemes:

•	UTF-8 (with and without BOM)

•

UTF-16

•

UTF-16BE

•

UTF-16LE

In Unicode version 2.1, the support is restricted to characters in the Basic Multilingual Plane (BMP) for all Unicode encoding schemes. UTF-16 in the BMP is referred to as UCS-2 in the StreamServe tools.

How the StreamServer handles character encoding

Internally, the StreamServer handles all data in the UCS-2 encoding form. This means that the input data must be converted to UCS-2 before the StreamServer starts processing the data. When the StreamServer has finished processing the data, the output is encoded using the appropriate code page.

In order to convert the input to UCS-2, without corrupting the input data, the StreamServer must know which code page is used to encode the input data. You must specify this when you configure your Project. You must also specify which code page to use to encode the output data.

Example 1

ISO 8859-15 encoded input and output.

In this example, input data is ISO 8859-15 encoded. The StreamServer converts the input data to UCS-2, processes the data, and uses ISO 8859-15 to encode the output data before sending it to the printer.

 

Preparing the workstation environment

Before you configure your Projects in the Design Center you may have to prepare your workstation environment with respect to the fonts and code pages to be used, and to the language version of your operating system.

Whenever possible you should use the appropriate language version of the operating system. For example, always try to configure a Greek Project on a Greek operating system.

You must make sure all required fonts are available, as well as complex script support (e.g. for Arabic or Hebrew) if needed. Consult the Microsoft® Windows® documentation or your System Administrator for information on how to do this.

Specifying code pages for input and output data

Code pages for the input can be specified as filters in filter chains that you add to the input connector. A code page can also be specified for the Event, either by using the scripting function ConvCurrMsgToUC in a retrieved script, or by using lookup tables or script aliases. Code pages for the output are specified in the output connector configuration (Platform).

See Specifying code pages for input data and Specifying code pages for output data.

If you do not specify a code page for the input data, the StreamServer may fail to process the data correctly. However, if input data conforms to ISO 8859-1 (Latin 1) you do not have to specify a code page for the input. Similarly, if both the input and output data conforms to ISO 8859-1 you do not have to specify a code page for the output.

Bidirectional text

Plain text data that contains Arabic or Hebrew text in logical order is treated the same way as data that contains unidirectional left-to-right text. Arabic/Hebrew text in visual order must be reordered to logical order before the StreamServer processes the text. Output from the StreamServer can also be reordered from logical to visual order if required (e.g. Arabic text in PDF output). See Bidirectional text.

Export files from the Design Center

All configuration files included in the export from the Design Center are UTF-8 encoded.

Table files and function files

All table files and function files must be UTF-8 encoded. See Specifying code pages for table files and Specifying code pages for function files.

Log files

If you want the StreamServer log to display “non-Latin 1” characters you can use the startup arguments -logfilecp and -logcp. See Specifying code pages for the log.

ODBC scripting functions

The following ODBC scripting functions can be used to specify code pages when retrieving input from an ODBC data source:

•	OdbcSetCodepage

•	OdbcConnectW

See the Scripting reference for information.