Section 7.1 - Adding Custom File Managers

The RAYZ API allows you to build an interface to a custom file manager, such as an Accom or other DDR, or an in-house asset management system. The documentation contains an example of an alternate file manager, namely ftp. This section details the construction of the ftp file manager.

The FTP Example

To see an alternate file manager in action, start RAYZ and drop an Image In node. Click on the file browser, and then choose "FTP File Manager" from the File Manager menu in the lower right corner of the browser window. Then in the Directory text entry field at the top of the browser, enter the location of a known ftp server, for example "ftp://ftp.sgrail.com" (no quotes).

You should see the single directory "pub" shown in the directory list. You can then browse the Silicon Grail ftp site as if it were a local file system. There are no images there, so you will not be able to load any, but this should give you an idea of how this works.

Examining the FTP File Manager Example

As with the rest of the RAYZ API, a new file manager is defined by filling out the fields in a structure. This is defined in CPI_FileManagerProvider.h. For the FTP example, the structure is defined as follows (required entries are shown in bold; optional entries can be filled in with NULL):

static RPI_FMInfo dinfo = {
    {
        "ftp",                  // internal name
        "FTP File Manager",     // menu name
        "Silicon Grail",        // author
        "1.0",                  // version
         NULL, NULL             // help strings
    },

    FTPInit,                    // initialize
    FTPShutDown,                // shut down
    FTPGetFileIDList,           // GetFileIDList
    FTPGetMetadataToOpen,       // GetMetadataToOpen
    NULL,                       // FillSeqInfo
    FTPConvertFileID,           // ConvertFileID
    FTPConcatFileID,            // ConcatFileID
    FTPSplitFileID,             // SplitFileID
    NULL,                       // GetFileType
    NULL,                       // ChangeFileType
    FTPGetFileIDInfo,           // GetFileIDInfo
    FTPGetDeviceName,           // GetDeviceName
    FTPCreateDirID,             // CreateDirID
    NULL,                       // RemoveDirID
    NULL,                       // RemoveFileID
    NULL,                       // RenameFileID
    NULL,                       // TemporaryFileID
    FTPGetDefaultDirID          // GetDefaultDirID
};
The signatures and uses of each of the struct values is given below.

Internal Name -

Menu Name - This is the string which will appear in the browser's File Manager menu

Author - Any text string

Version - This is not used by RAYZ, but is available for the programmer

Help Strings -

Initialize - Apart from initializing any global variables which my be necessary, the main purpose of this routine is to allocate a block of private data which will be passed to subsequent functions via the 'handle' parameter. For example, the FTP manager is initialized like this (simplified for example purposes):

typedef struct _managerState
{
    netbuf      *nControl;
    char         servername[256];
    char         username[256];
} managerState;

...

static CPI_PrivateData
FTPInit( void )
{
    managerState *retval = (managerState *)cpiAlloc( sizeof( managerState ) );

    return (CPI_PrivateData)retval;
}
If this function is not defined, the value of the 'handle' parameter in subsequent calls will be NULL. See the ShutDown explanation (next) for an example of that. If for some reason you want to define this function, but not allocate any private data, then just return NULL from the Init function.

Shut Down - This function is called just before exit. The incoming parameter is a pointer to any storage allocated in the Init routine, or NULL if none was allocated. If you allocated memory in the Init, it should be freed here. For example

static void
FTPShutDown( CPI_PrivateData myHandle )
{
    managerState *state = (managerState *)myHandle;

    cpiFree( state );
}

Get File ID List - (required) This function returns two lists, one of the sub-directories relative to the current location, and the other of files in the current directory. These lists should be sorted before exit - there is a helper function for sorting (cpiSortFileIDList) which can be used. Look in CPI_FileID.h

Directories should always be added, but files can be added or not, depending on the application. For directories, the "." (current directory) entry shouldn't be added, as it is expected. But the ".." (parent directory) can be added or not, as you choose.

The signature of this function is as follows:

CPI_Bool  getFileIDList( CPI_PrivateData handle,
                         CPI_FileIDList  dirIDList,
                         CPI_FileIDList  fileIDList,
                         const char     *dirID,
                         CPI_Uint32      flags,
                         CPI_ExtListFn   fileExtFn );
where
handle is a pointer to the private data allocated in the Init function.
dirIDList is the returned list of directories contained within the current directory.
fileIDList is the returned list of files in the current directory.
dirID is the current directory.
flags are bit flags which may be OR'ed together. Currently there is only one flag which may be set - CPI_NO_HIDDEN_IDS. If set, this means that no hidden files should be added. In Unix, this means that files which start with a '.' (period) will be ignored.
fileExtFn is a pointer to a function which should be called with the file extension to see if it matches. The function should return CPI_TRUE if the file should be added. This argument can be NULL, in which case all files are considered to be OK. Extensions are passed without any leading '.' (period), eg "jpg", not ".jpg"

As an illustration of the core of the getFileIDList function, this is (part of) what happens within the FTP sample code:

    if ( isDir )
    {
        cpiAddFileIDToList( dirIDList, fileName );
    }
    else
    {
        extension = strrchr( fileName, '.' );
        if ( NULL != extension )
            ++extension;

        if ( NULL == fileExtFn ||
             CPI_FALSE != fileExtFn( fileName, extension ) )
        {
            cpiAddFileIDToList( fileIDList, fileName );
        }
    }

Get Metadata - (required) This function returns a reference to some metadata, which will be passed to the device's open routine, where it can be used to open the requested file. This is the device which is defined in the next section.

Other values can be passed, via the metadata mechanism, at this point. The mechanism is very similar to the metadata that the Nodes use to get values down to the Image Operations. The signature of this function is

CPI_Metadata  getMetadataToOpen( CPI_PrivateData    myHandle,
                                 const char        *fileID,
                                 CPI_Float32        curSample )

where
myHandle is the pointer to the private data (if any) allocated in the Init routine
fileID is the fileID which is going to be packed into metadata
curSample is the current frame number (as a floating point value) which is used to expand any time-related variables in the fileID, before converting them to metadata

As an example, the FTP sample code implements this routine as follows:

static CPI_Metadata
FTPGetMetadataToOpen( CPI_PrivateData    /*myHandle*/,
                      const char        *fileID,
                      CPI_Float32        curSample )
{
    CPI_Metadata     retval     = NULL;
    const char      *expandedFID    = FTPExpandVars( fileID, curSample );

    if ( NULL != expandedFID )
    {
        retval = cpiCreateMetadata();
        if ( NULL != retval )
        {
            char    username[256];
            char    password[256];
            char    server[256];
            char    fileID[256];

            FTPParseDirectoryInfo( expandedFID,
                                   username,
                                   password,
                                   server,
                                   fileID );

            cpiSetFilename( retval, fileID );
            cpiSetDeviceName( retval, FTPGetDeviceName() );
            cpiSetMetaString( retval, "SERVERNAME", server );
            cpiSetMetaString( retval, "USERNAME", username );
            cpiSetMetaString( retval, "PASSWORD", password );
        }
    }

    return retval;
}

Fill Sequence Info - This function takes a fileIDList, obtained from a previous call to getFileIDList(), and returns a FileIDSeqList, which is a list of sequence pairs. A pair is made up of a string that represents the sequence (for example, foobar.$F.jpeg) and a metadata object that contains metadata info about the start, end and increment for that sequence.

The sequence format string is completely arbitrary - if you wanted to implement an alternative sequence representation, say foobar.####.jpeg, you could do that. It would be up to you to recognize this sequence format in the getMetadataToOpen function defined above.

There are several calls available to manipulate the sequence ID list, which are defined in CPI_FileID.h. These are:

CPI_FileIDSeqList  cpiCreateFileIDSeqList( void );
void               cpiDeleteFileIDSeqList( CPI_FileIDSeqList seq);

CPI_Uint32         cpiGetFileIDSeqListSize( CPI_FileIDSeqList seq );

void               cpiAddFileIDToSeqList( CPI_FileIDSeqList idlist,
                                          const char       *fileid,
                                          CPI_Metadata      metadata );
There are also routines to get at entries of the file ID list and iterate through it. These are defined as follows:
CPI_Uint32  cpiGetFileIDListSize( CPI_FileIDList  idlist );
void        cpiAddFileIDToList( CPI_FileIDList    idlist,
                                const char       *fileid );
const char *cpiGetFileIDFromList( CPI_FileIDList  idlist,
                                  CPI_Uint32      idx );
As an example, here is a psuedo-code fragment from the default Grail version of this function.
CPI_Bool
CustomFillSeqInfo( CPI_PrivateData       handle,
                   CPI_FileIDSeqList     seqlist,
                   const CPI_FileIDList  filelist )
{
    CPI_Uint32  numents = cpiGetFileIDListSize( filelist );
    CPI_Uint32  i;
    char       *fileID;

    for ( i = 0; i < numents; i++ )
    {
        fileID = cpiGetFileIDFromList( filelist, i );

        if ( fileID is the last file in a sequence )
        {
            CPI_Metadata    meta = piCreateMetadata();

            cpiSetImageRangeStart( meta, curstart );
            cpiSetImageRangeEnd( meta, curend );
            cpiSetImageRangeIncr( meta, curincr );
        }
    }

    return CPI_TRUE;
}
In addition, you can retrieve the metadata values with the following calls:
CPI_Bool   cpiGetImageRangeEnd( const CPI_Metadata    meta,
                                CPI_Int32            *result );
CPI_Bool   cpiGetImageRangeStart( const CPI_Metadata  meta,
                                  CPI_Int32          *result );
CPI_Bool   cpiGetImageRangeIncr( const CPI_Metadata   meta,
                                 CPI_Int32           *result );

If the FillSeqInfo function is not defined, then your file manager will use the Grail default sequence behavior, namely collecting together numbered file names into foobar.$F.suffix (in fact, it will be a variant of $F, such as ${F}, $F2, $F3, etc).

There may be situations where assembling individual files into sequences is not desired - for example, a browser which looks for job control files or parameter channel files. In that case, you would need to define this function, but its only job would be to copy each file from the fileID list to the sequenceID list; the corresponding metadata value would be NULL.

You can also call the default routine explicitly yourself from within this function. The value of this would be that, for example, you might want to massage the file ID list first, then have it parsed into sequences in the normal manner, or you might want to get the sequence list first, and then manipulate it. Either way, you can call

CPI_Bool cpiDefaultSeqSearch( CPI_FileIDSeqList     seqlist,
                              const CPI_FileIDList  fids);

Convert File ID -

char *ConvertFileID( CPI_PrivateData  handle,
                       const char      *oldFileID );

This function converts a fileID to the native file system. For example, the default (Grail) file manager will try to convert fileIDs between Unix and Windows NT naming conventions. It does this by looking for a set of preferences (accessed through the cpiPref* calls - see Section 6) and finding drive letter-to-mount point mappings, and by converting backslashes to forward slashes, or vice versa.

Also, if there is any cleanup to the filename, it should be done here. For example, if the user types /usr/////foo/bar/ and that is not valid, or at least unaesthetic, this function should clean it up and return something like /usr/foo/bar

If this function is not provided, then the default filesystem file manager will be used to do the conversion. If you are constructing file IDs that are consistent with the normal native file system, you do not need to provide this function.

If the function returns NULL, then it is assumed that no translation was necessary. The returned string should be allocated with cpiAlloc, and freed with cpiFree. It should also not be referenced after the function exits.

Concatenate File ID -

char *ConcatFileID( CPI_PrivateData  handle,
                    const char      *dirID,
                    const char      *fileID );

This function constructs a fully qualified file identifier from the passed in directory and file IDs. For example, for the Unix filesystem file manager, a dirID of "/usr/tmp" and a fileID of "foobar" would result in "/usr/tmp/foobar".

The returned string should be allocated with cpiAlloc, and freed with cpiFree. It should also not be referenced after the function exits. If the return value is NULL, then it is assumed that there was some sort of error.

Split File ID -

CPI_Bool SplitFileID)( CPI_PrivateData    handle,
                       char             **dirID,
                       char             **fileID,
                       const char        *fileAndDirID );

This function splits a full file ID into a directory and a file name, if possible. For example, if /usr/tmp/foobar is passed in, this should split it into "/usr/tmp" and "foobar" returning CPI_TRUE. If "/" is passed in (or something like "D:\" under NT), it should fail, returning CPI_FALSE. If "/usr" (or "/usr/") is passed in, should return "/" and "usr" (a directory can be considered a file ID).

The returned strings should be allocated with cpiAlloc and freed with cpiFree. If this function is not provided, then the default filesystem file manager function will be used.

Get File Type -

CPI_Bool GetFileType( CPI_PrivateData     handle,
                      char              **fileType,
                      const char         *fileID );
Returns in fileType the type of the file. The file type is most commonly known as the file extension, i.e., "foobar.tiff" would return "tiff" as the fileType. Some filemanagers may not have file names with actual extensions and will insert those as appropriate - e.g. older Accom devices just had frame numbers as the file names, but there is still a file type associated with that file.

The returned string should be allocated with cpiAlloc and freed with cpiFree. The function should return CPI_TRUE if it was successful, otherwise CPI_FALSE. If this function is not provided, then the default filesystem file manager function will be used.

Change File Type -

CPI_Bool ChangeFileType( CPI_PrivateData   handle,
                         char             **newID,
                         const char        *oldID,
                         const char        *newType,
                         const char        *oldType );

The purpose of this function is just to manipulate strings to allow a file extension to be changed. For example, you might be passed in an oldID of "foobar.tiff" and a newType of "jpeg". The result, contained in newID, would be "foobar.jpeg" This comes up when, say, the user changes the filetype that she wants to write out, so we want to make the fileID match the user's selection.

The oldType will be passed in if Rayz was able to determine what that type is - it does this by querying your plugin. If it determines that the type is one it knows about, it will be passed in here. Otherwise, the oldType will be NULL. In that case, just append the new type to the oldID.

The returned string should be allocated with cpiAlloc and freed with cpiFree. The function should return CPI_TRUE if it was successful, otherwise CPI_FALSE. If this function is not provided, then the default filesystem file manager function will be used.

Get File ID Info -

CPI_Metadata GetFileIDInfo( CPI_PrivateData  handle,
                            const char      *fileID );
Returns a metadata item which contains information about the passed in fileID. This is not necessarily info for opening a file, but is more intended to be user information about things like file size, modification date, etc. If there is a problem, the function should return NULL. The programmer can create their own metadata by using string data, and/or he can use the specific metadata calls listed below. These should be self-explanatory:
CPI_Bool  cpiSetFilename( CPI_Metadata          meta,
                          const char           *filename );
CPI_Bool  cpiGetFilename( const CPI_Metadata    meta,
                          char                 *result,
                          size_t                buffersize );
CPI_Bool  cpiSetDeviceName( CPI_Metadata        meta,
                            const char         *devname );
CPI_Bool  cpiGetDeviceName( const CPI_Metadata   meta,
                            char                *result,
                            size_t              buffersize );
CPI_Bool  cpiSetRead( CPI_Metadata            meta,
                      CPI_Bool                onoff );
CPI_Bool  cpiGetRead( const CPI_Metadata      meta,
                      CPI_Bool               *result );
CPI_Bool  cpiSetWrite( CPI_Metadata           meta,
                       CPI_Bool               onoff );
CPI_Bool  cpiGetWrite( const CPI_Metadata     meta,
                       CPI_Bool              *result );
CPI_Bool  cpiSetTrunc( CPI_Metadata           meta,
                       CPI_Bool               onoff );
CPI_Bool  cpiGetTrunc( const CPI_Metadata     meta,
                       CPI_Bool              *result );
CPI_Bool  cpiSetFileSize( CPI_Metadata        meta,
                          CPI_Uint64          filesize );
CPI_Bool  cpiGetFileSize( const CPI_Metadata  meta,
                          CPI_Uint64         *result );
CPI_Bool  cpiSetFileLastMod( CPI_Metadata     meta,
                             const char      *time );
CPI_Bool  cpiGetFileLastMod( const CPI_Metadata   meta,
                             char                *result,
                             size_t               buffersize );
CPI_Bool  cpiSetIsDir( CPI_Metadata         meta,
                       CPI_Bool             onoff );
CPI_Bool  cpiGetIsDir( const CPI_Metadata   meta,
                       CPI_Bool            *result );
CPI_Bool  cpiSetIsFile( CPI_Metadata        meta,
                        CPI_Bool            onoff );
CPI_Bool  cpiGetIsFile( const CPI_Metadata  meta,
                        CPI_Bool           *result );

Get Device Name - Returns a string which will not be freed, representing the default device name for this file manager plugin. If not supplied, the default device "filesystem" will be used. Here is an example from the FTP plugin.

static const char *
FTPGetDeviceName( void )
{
    return "ftp";
}

Create Directory ID -

CPI_Bool CreateDirID( CPI_PrivateData   handle,
                      const char       *newdirID,
                      CPI_Bool          createParent);
This function is called to create a directory before a write operation. newdirID is the name of the directory to be created. If createParent is CPI_TRUE, then you should also create all directories leading up to the named directory.

CPI_TRUE should be returned for a successful operation.

Remove Directory ID -

CPI_Bool RemoveDirID( CPI_PrivateData   handle,
                      const char       *dirID );
This function should remove the given directory, and all its contents. Return CPI_TRUE if the operation was successful.

Remove File ID -

CPI_Bool RemoveFileID( CPI_PrivateData  handle,
                       const char      *fileID );
This function should remove the passed fileID. However, remember that the fileID can represent a sequence, so the function could be passed an argument like "foobar.$F.tiff" or even "foobar.[1-9]?[1-0]*.tiff". If this function is not defined, then the default (Grail) filesystem remove function will be used, with Grail wildcard expansion.

Rename File ID -

CPI_Bool RenameFileID( CPI_PrivateData  handle,
                       const char      *oldID,
                       const char      *newID );
This function should change the name of the file from oldID to newID. It returns CPI_TRUE if the operation was successful. This function is optional.

Temporary File ID -

char *TemporaryFileID)( CPI_PrivateData   handle,
                        const char       *fileID );
Returns the name of a temporary file to render into. When Rayz renders an output file, it opens a temporary file and puts the results there - when complete, the temporary file is renamed to the chosen output file. In this way, rendering errors do not destroy existing files.

The resulting string should be created using cpiAlloc here; it will be freed by Rayz.

Note: this function will not be called if you have not defined a renameFileID function (previous function).

Get Default Directory ID -

char *GetDefaultDirID( void );
This function returns the name of the default (starting) directory. For example, the Unix filesystem will return the results from getcwd() (get current directory).

The string should be allocated with cpiAlloc - it will be freed by Rayz. The string should not be referenced after the function exits. If this function is not provided, the default filesystem file manager will be used.

Custom Devices

Many file managers will require a custom device to go with it; the FTP example provided is one such case. If your application requires a custom device driver, then you should refer to the next section, Creating a Custom Device.

In some cases, it won't be necessary to create a device; for example, it would be possible to define a file manager that simply overlayed a different kind of file organization onto a unix-style system. You might find images in this system by referring to Shows and Shots (and perhaps image resolutions), rather than directories - the file manager would make the connection between Show and Shot name and the underlying paths and file names.


[Previous Page] [Next Page]
[Table of Contents] [Index]

Copyright © 2002 Silicon Grail Inc.
736 Seward Street, Hollywood, CA 90038