Section 4 - Adding Custom File Formats

In RAYZ, all file format reader/writers are implemented as plugins. This makes adding new ones simple.

As with other aspects of RAYZ plugins, Adding a custom file format is built around a function table. In this case, the table is defined in CPI_ImageProvider.h. Here is a sample table from the RAYZ file reader for SGI's file format:

static RPI_ImageInfo dinfo = {
    {
        "sgi",              // internal name
        "SGI Image",        // string that appears in format menu
        "Silicon Grail",    // author
        "1.0",              // version
        NULL,               // URL for user help, if any
        NULL                // icon name, if any
    },
    SgiInit,                // init plugin
    SgiShutDown,            // shut down plugin
    NULL,                   // ParameterChanged function
    SgiOpenForRead,         // Open file for reading
    SgiReadImage,           // ReadImage function
    SgiReadImage,           // ReadSubImages function - in this case, the
                            // ReadImage func can also handle sub-images...
    SgiOpenForWrite,        // Open file for writing
    SgiWriteImage,          // WriteImage function
    NULL,                   // Not Used
    SgiCloseImage,          // close image 
    NULL,                   // FrameSeek function
                            // Flags
    CPI_IMAGE_RGB | CPI_IMAGE_RGBA | 
    CPI_IMAGE_UINT8 | CPI_IMAGE_UINT16 | CPI_IMAGE_FLOAT32,
    "sgi;rgb"               // file name extensions for this format
};

Let's look at each of the fields in this structure:

Init - This is called once when RAYZ starts up. It initializes the plugin and defines the user interface elements, if any. It should return CPI_TRUE if there is no error. This routine is not optional - if there is no work to do, you must still define a routine that returns CPI_TRUE.

ShutDown - This is called once when RAYZ exits. It should return CPI_TRUE if there was no error. This routine is not optional - if there is no work to do, you must still define a routine that returns CPI_TRUE.

ParameterChanged - This is called if there is a parameter defined for this file format, and if the user changes the value of that parameter. Note that most formats do not have user parameters - choices such as which bit depth to read a file into are handled by RAYZ outside of the plugin mechanism. See the later section on Parameters.

OpenForRead - This is called before a file can be read. In particular, this is called by the file browser in order to fill in information about the file X and Y size, bit depth, etc. The function's arguments are metadata which contain information about the file so that it can be opened.

The other thing that happens here is that the user-defined state variables, stored in a structure, is returned to RAYZ to be passed to other file I/O routines as required.

ReadImage - This actually reads the image. Things to note about this:

ReadSubImage - This is the function definition for reading in sub-images. Unless your format requires some special processing for sub-images, we recommend that you handle sub-images in the normal ReadImage function.

RAYZ will request sub-images in many situations, including thumbnails on the nodes, in the image browser, and in the clip editor. If you do not provide a ReadSubImage routine, RAYZ will create the images it needs from the full size image which is available, but of course this is less efficient than only reading what is necessary.

WriteImage - This is the function that writes an image to disk.

Not Used - This function entry point is left for future expansion.

CloseImage - Close the file and return status.

FrameSeek - For formats which contain multiple images, this defines a mechanism for getting to the correct image.

Flags - These bit flags define aspects of the file format. These can be OR'ed together to help define the supported features of a particular image format. Defined constants which may be used by the programmer are
CPI_IMAGE_RGB supports RGB channels
CPI_IMAGE_RGBA supports RGBA channels
CPI_IMAGE_UINT8 supports 8 bit images
CPI_IMAGE_UINT16 supports 16 bit images
CPI_IMAGE_FLOAT32 suports float images
CPI_IMAGE_PLANAR channels are stored separately, ie all red then all green then all blue, instead of RGB interleaved
CPI_IMAGE_REVERSE channels are reversed, eg ABGR
CPI_IMAGE_MOTION supports multiple frames
CPI_IMAGE_READFULL must read image all at once
CPI_IMAGE_WRITEFULL must write image all at once
CPI_IMAGE_LOG supports log format
CPI_IMAGE_LOG_ONLY requires log format
CPI_IMAGE_EASY_TO_DETERMINE the format can be known just from the header (default)
CPI_IMAGE_HARD_TO_DETERMINE the format can be guessed from the header
CPI_IMAGE_CANNOT_DETERMINE the header is no use in determining the format, or there is no header in this format

The DETERMINE flags are used as follows: the list of formats is sorted by that flag when an image is to be opened. RAYZ will try all the formats which is is sure about, followed by the difficult ones. The ones listed as CANNOT_DETERMINE are only opened if their filename matches the extension, or if the user specifically states that an image is in that format.

File Name Extensions - This is a list of the file name extensions that represent this format. Strings are separated by semi-colons, as in "tif;tiff" or as in this example, where SGI files may end with either .sgi or .rgb

An Example: the .com Format

To illustrate how some of this works, we are going to invent and implement our own image format. The complete source for this is given in source/Com.C. There are also jpeg and tiff source examples provided; these illustrate other aspects of the API, and can be used as reference.

The definition of the .com (compact) format is as follows:
Header (128 bytes): magic number (4 bytes) | xsize (4 bytes) | ysize (4 bytes) | unused (116 bytes)
Image Data: 1 byte per pixel, each is 3 bits red, 3 bits green, 3 bits blue

The header has a magic number. This is so that files of this type can be identified exactly by the software that reads them. When RAYZ attempts to open an image file for reading, it first looks at the suffix for a guide to what format it is. If there is no suffix, it then goes through the list of known formats, looking for a match. If there were no magic number for this format, it could "volunteer" to open and read any images which had no suffix, even though they wouldn't be understood. Having a magic number prevents this.

This format requires no user parameters - by definition, all files contain only RGB data, and when a .com file is read into RAYZ, it comes in as an 8 bit image. Therefore, the Init and ShutDown functions are empty (note: although they are empty, they must exist and return CPI_TRUE), and there does not need to be a ParameterChanged function. So far, that looks like this:

static RPI_ImageInfo dinfo = {
    {
        "com",
        "Com",
        "Silicon Grail",
        "1.0", 
        NULL, 
        NULL
    },
    ComInit,
    ComShutDown,
    NULL,               //ComParmChanged
    ComOpenForRead,
    ComReadImage,
    ComReadImage,        // note same routine for both types of read
    ComOpenForWrite,
    ComWriteImage,
    NULL,               //Not used
    ComCloseImage,
    NULL,               //FrameSeek
    CPI_IMAGE_RGB | CPI_IMAGE_UINT8,
    "com"
};

//////////////////////////////////////

static CPI_Bool
ComInit( CPI_Bool /* initforread */ )
{
    return CPI_TRUE;
}

//////////////////////////////////////

static CPI_Bool
ComShutDown( void )
{
    return CPI_TRUE;
}
We also need to define the usual register and unregister functions - as usual, these should be declared with the CPIDSOEXPORT macro to enable Windows versions to link properly:
CPIDSOEXPORT CPI_Bool
rpiPluginInit( void )
{
    return cpiRegisterImageReader( &dinfo );
}

//////////////////////////////////////

CPIDSOEXPORT void
upiPluginCleanup( void )
{
    cpiUnregisterImageReader( "com" );
}
Again, note that we register the structure's address, but we unregister the internal name of the format (in this case, "com").

Next, we'll define the OpenForWrite function. In our example, this is very simple - it opens the file and saves the file pointer in a structure which is returned for later use. We get the incoming image size with the call to cpiGetImageFullSize(); there are equivalent functions to get the number of channels and the bit depth, but we don't need those here. The header is created and written out. And that's it. Notice that the header byte order is swapped if the software is executing on a little-endian machine. For portability, your plugin should check and use that information (see the section on Portability for more information).

Now let's look at the actual WriteImage function. In our case, the only real complexity is that we want to deal with images coming from RAYZ in all of 8bit, 16bit and float formats. (It is possible to specify that your format will only accept certain kinds of images; for example, writing out Cineon format images requires that the incoming image be in 16bit format. Refer to the section on Error Handling for more information.)

To do this, we use the incoming image context to determine which kind of image we have, and then pack it into a series of 8bit bytes as per the format. Note here that we use RAYZ-provided routines for allocating and freeing memory; these provide efficient and portable mechanisms for doing that.

The resulting two routines are shown here, without the specifics of the packing routines - those can be seen in the supplied source example, but they just clutter up this explanation:

static CPI_PrivateData
ComOpenForWrite( const CPI_Metadata     meta,
                 CPI_Metadata           imageinfo )
{
    ComHeader    header;
    CPI_Int32    width, height;
    imageState  *retval = (imageState *)cpiAlloc( sizeof( imageState ) );

    retval->myDevice = cpiOpen( meta );

    if ( retval->myDevice == NULL )
    {
        cpiError( "Couldn't open file for write!" );
        cpiFree( retval );
        return NULL;
    }

    /* set up header and write it */
    cpiGetImageFullSize( imageinfo, &width, &height );
    header.xsize = (CPI_Uint32)width;
    header.ysize = (CPI_Uint32)height;

    if ( cpiIsLittleEndian() )
        SwapComHeader( &header );

    cpiWrite( retval->myDevice, (char *)&header, sizeof( ComHeader ) );

    return (CPI_PrivateData *)retval;
}

/* Output size is just sizeX * sizeY bytes, regardless of number of
   channels or bit depth */

static CPI_Bool
ComWriteImage( CPI_PrivateData handle, CPI_Image *image )
{
    imageState  *state  = (imageState *)handle;
    CPI_Bool     retval = CPI_TRUE;
    CPI_Uint8   *myPackedPtr;
    CPI_Uint64   writeSize;

    if ( image != NULL )
    {
        // set up output size
        writeSize = image->myContext.sizeX * image->myContext.sizeY *
                    sizeof( CPI_Uint8 );

        // allocate buffer for output image
        myPackedPtr = (CPI_Uint8 *)cpiAlloc( writeSize );

        // convert to .com format
        if ( image->myContext.bitsPerPel == 8 )
            PackCom8( myPackedPtr, (CPI_Uint8 *)image->data,
                      writeSize, image->myContext.channels );
        else if ( image->myContext.bitsPerPel == 16 )
            PackCom16( myPackedPtr, (CPI_Uint16 *)image->data,
                       writeSize, image->myContext.channels );
        else
            PackComF( myPackedPtr, (CPI_Float32 *)image->data,
                      writeSize, image->myContext.channels );

        if ( writeSize != cpiWrite( state->myDevice,
                                    (char *)myPackedPtr,
                                    writeSize ) )
        {
            cpiError ( "Couldn't write all image data" );
            retval = CPI_FALSE;
        }

        cpiFree( myPackedPtr );
    }
    else
    {
        cpiError( "Can't write NULL .com image!" );
        retval = CPI_FALSE;
    }

    return retval;
}

The next thing to do is to define the functions for reading the .com format. The first, OpenForRead, is simple - it opens the file using the provided cpiOpen() call, which takes metadata to get the file name and returns a file pointer. Then it reads the header, swaps it if necessary, and then sets up the image info in the passed metadata, using the cpi commands. In our case, the bit depth and the number of channels are defined by the format, so we just hardwire their values. We also pass back the X and Y size that we got from the header. That looks like this:

static CPI_PrivateData
ComOpenForRead( const CPI_Metadata    meta,
                CPI_Metadata          imageinfo )
{
    imageState  *retval     = (imageState *)cpiAlloc( sizeof( imageState ) );

    if ( NULL == retval )
        return NULL;

    retval->myDevice = cpiOpen( meta );
    if ( NULL == retval->myDevice )
    {
        cpiError( "Unable to open file for read" );
        cpiFree( retval );
        return NULL;
    }

    if ( sizeof( ComHeader ) != cpiRead( retval->myDevice,
                                         (char *)&retval->myHeader,
                                         HEADERSIZE ) )
    {
        cpiError( "Couldn't read com image header!" );
        cpiClose( retval->myDevice );
        cpiFree( retval );
        return NULL;
    }
    else
    {
        if ( cpiIsLittleEndian() )
            SwapComHeader( &retval->myHeader );

        /* the .com format has a magic number associated with it,
           so that these files can be accurately recognized even
           if the suffix is missing. This also prevents errors 
           involving attempting to read the wrong format.
        */
        if ( retval->myHeader.magic == MAGICNUM )
        {
            cpiSetImageFullSize( imageinfo, retval->myHeader.xsize,
                                 retval->myHeader.ysize );
            /* these params are fixed by the format definition */
            cpiSetImageBitDepth( imageinfo, 8 );
            cpiSetImageChannels( imageinfo, 3 );
        }
        else
        {
            cpiError( "Magic number doesn't match!" );
            cpiClose( retval->myDevice );
            cpiFree( retval );

            retval = NULL;
        }
    }

    return retval;
}

Finally, we need to define the Reading function. Ideally, we would construct this so that it is able to read both full images and sub-images. For most formats, it is more efficient if the sub-images can be handled in the main Read function. However, if reading and delivering a sub-image is difficult or complex and requires its own handling, you can define a separate function for that.

We could define our own reader to simply grab the whole image at once, and then unpack it - however, it is more memory efficient to read images as scanlines, so that's how this is implemented.

The result is not very complex - we read in a scanline at a time, and unpack into a set of RGB 8bit pixels. Note that the OpenRead defined to RAYZ that the result was going to be 8 bits, RGB. So that is the kind of image memory we are passed to put the result into. In RAYZ, the user can ask for an input image to be in 8 bit, 16 bit or float format, but that choice is handled after the read, by RAYZ - not by the application programmer. You only need to be concerned with delivering the type of image which is "generic" to your format - RAYZ will handle the conversion to other formats.

(An exception to this would be a situation where the user was choosing between, for example, RGB and Z depth. In this case, you have to query the user for what the output is, and then deliver what is requested)

Here is the ReadImage function:


static CPI_Bool
ComReadImage( CPI_PrivateData handle, CPI_Image *image )
{
    CPI_Uint8   *packedPtr;
    CPI_Uint32   offset, y;
    imageState  *state      = (imageState *)handle;
    CPI_Uint8   *dataPtr    = (CPI_Uint8 *)image->data;
    CPI_Uint32   rowSize    = image->myContext.sizeX;
    CPI_Uint32   end        = image->myContext.offsetY +
                              image->myContext.sizeY;

    /* sanity check */
    if ( image == NULL )
    {
        cpiError( "ComReadImage: destination image is NULL" );
        return CPI_FALSE;
    }

    /* allocate temporary line buffer */
    packedPtr = (CPI_Uint8 *)cpiAlloc( rowSize );

    /* compute offset into file and seek to start of data */
    offset = HEADERSIZE + image->myContext.offsetX +
             (image->myContext.offsetY * rowSize);
    cpiSetPosition( state->myDevice, offset, CPI_SEEK_START );

    /* read in each row, and then unpack - the call to cpiGetLine
       is a safe way to position the data pointer for each row */
    for (y = image->myContext.offsetY; y < end; y++)
    {
        cpiRead( state->myDevice, (char *)packedPtr, rowSize );
        dataPtr = (CPI_Uint8 *)cpiGetLine( image, y);
        UnPackCom( packedPtr, dataPtr, rowSize );
    }

    cpiFree( packedPtr );

    return CPI_TRUE;
}
And that makes up the complete .com file I/O package. The source and the header file are included in the source directory - this example compiles and works.

Other source examples for tiff and jpeg are also included in the API kit. These are the actual file handlers used in RAYZ, and they illustrate other aspects of writing a file handler.

User Parameters

Many or most image formats do not require input from the user to be written or read successfully, since the header contains all the information which is required. However, if your format does require (or allow) input from the user, there is a mechanism for handling that. This looks just like the node or LUT parameter definition and value return.

Note: The RAYZ Image In node contains many user-settable parameters for things like bit depth, pixel ratio, etc. These are all handled after the image is read in, by RAYZ. There is no access to these parameters in the application interface.

To add a parameter section for your format, define the parameters in the Init function. These are the same calls that the node panels use, and they can be referenced in the section on adding Widgets . Here is an example from the .tiff reader/writer:

/* define the menu arrays */

static const char *compmenu[] =
{
    "Zip Compression",
    "LZW Compression",
    "Jpeg Compression",
    "No Compression",
    NULL
};

static const char *ziplevelmenu[] =
{
    "1 - Fast",
    "2",
    "3",
    "4",
    "5",
    "6 - Default",
    "7",
    "8",
    "9 - Max compression - slow",
    NULL
};

static CPI_Bool
TiffInit( CPI_Bool initforread )
{
    /* initforread is CPI_TRUE if this call is being used to set
       up an Image In node - in that case, you would define the
       parameters which are relevant (if any) for reading your
       format.
       initforread is CPI_FALSE for format information in the
       Render Panel - in this case, you set up parameters for 
       writing out the format, which is what is done here */

    if ( CPI_FALSE == initforread )
    {
        cpiAddMenu( "compression",
                    "Compression",
                    1,
                    compmenu,
                    "Set type of compression to use",
                    NULL );
        cpiAddMenu( "zip_level",
                    "Zip Level",
                    5,
                    ziplevelmenu,
                    "Level of zip compression to use",
                    NULL );
        cpiAddInteger( "jpeg_quality",
                       "Jpeg Quality Level",
                       90,
                       1,
                       100,
                       CPI_PARM_ANIMATABLE,
                       "Adjust the quality of the jpeg image to be written",
                       NULL );
    }

    return CPI_TRUE;
}
Then to retrieve and use these values, the calls are the same as for the node parameters - for example, to get the jpeg quality level from the above slider, you do this
cpiGetInteger( &compParm, "compression", time );
The ParmChanged function can be used to handle state changes based on user input. For example, the .tiff writer enables/disables parameters based on the kind of compression chosen. That function looks like this:
static void
TiffParmChanged( const char * /*parmname*/ )
{
    CPI_Bool     enablezip, enablejpeg;
    CPI_Int32    ival;

    enablezip   = CPI_FALSE;
    enablejpeg  = CPI_FALSE;

    if ( cpiGetInteger( &ival, "compression", 0 ) )
    {
        switch ( ival )
        {
            case 0:
                enablezip = CPI_TRUE;
                break;

            case 2:
                enablejpeg = CPI_TRUE;
                break;

            default:
                break;
        }
    }

    cpiEnableParameter( "zip_level", enablezip );
    cpiEnableParameter( "jpeg_quality", enablejpeg );
}
The ParmChanged function takes an argument, 'parmname', which is the name of the parameter which generated the call. This name is based on the internal name you used when you defined the parameters. For example, in the .tiff reader/writer that is defined above, the three parameter names are
compression
zip_level
jpeg_quality
You should always test the parmname string for a NULL value before using it, as there are times when this routine can be called with a NULL value (as in initial setup).

Error Handling

You can use the same cpiError functions that a node uses, in order to signal a user or software error to the user. For example, the .tiff reader does not support 1 bit tiff files, so it contains the test
        if ( 1 == retval->bitspersample )
        {
            cpiError( "1 Bit Tiff Files Not Supported" );
            TIFFFlushData( retval->tif );
            TIFFClose( retval->tif ); // This will close the file
            cpiFree( retval );
            return NULL;
        }
Similarly, cpiWarning() can be used to draw the user's attention to something which is not fatal, but which they might want to be aware of. An example might be an unsupported file extension or header flag. The function is used with a warning string, as in
cpiWarning( "Interpretation of Targa file extension area not supported " );
This will turn the Image In node yellow and place the text in the status line area of the interface.

Warnings are currently only valid during file reading - warnings issued during file writing are not displayed to the user.

Portability Issues

RAYZ provides several cpi calls that you should use in order to ensure that your plugin works across the range of supported systems, which is currently
SGI IRIX
Linux (Intel)
Linux (Alpha)
Linux (Power PC)
Windows NT/2000

For memory allocation and deallocation, use cpiAlloc() and cpiFree().

You should always use the test if ( cpiIsLittleEndian() ) to determine if the current environment is little-endian or big-endian, and you can use the routines cpiSwapShort(), cpiSwapLong() and cpiSwapFloat() to convert the values between environments.


[Previous Page] [Next Page]
[Table of Contents] [Index]

Copyright © 2002 Silicon Grail Inc.
736 Seward Street, Hollywood, CA 90038