About Caching |
Shake can use caching to speed up reprocessing tremendously. What it does is saves a snapshot of certain nodes, with all of the parameters that make up that node, away in a directory called the cache directory. When Shake processes any tree, it very quickly scans the directory to see if it has done this exact tree before. If it has, it calls up the snapshot rather than re-calculate it.
The cached images are labeled with input files, date of file creation, the node tree, and all of the parameters in the nodes, including resolution/proxy settings, to guarantee identical results.
The Shake cache is persistent. If you quit out of the program, and then re-launch the program the next day, it will still call up the files cached the previous day.
You can have four different settings for the cache:
Type | Setting | Function |
None | 0 | No images are read from or written to the cache. |
Read Only | 1 | Images are only read from the cache. No images are written to it. |
Regular | 2 | Images are read from and written to the cache in a conservative algorithm. Only non-animated branches are cached. |
Aggressive | 3 | As Regular, but animated branches are also cached. This can result in a lot of cache activity, so make sure you have a large cache disk if you are going to use this mode. |
In addition, Shake 2.2 now has transient caching, which is a temporary cache of viewed images. This means you can usually toggle between two images quickly. As you work on nodes that aren't connected to the cached image, the cache will be disposed of.
The cache has a default maximum size of 512 Mb. When files are written to the cache that will push its size above the maximum limit, the oldest cached files will be removed, allowing for new space. Because of this, if you are using the Aggressive caching, you should have a lot of space, since otherwise you will constantly be deleting old file and writing new ones, which defeats the intended purpose of the cache.
By default, the cache is written to either:
NT: [SystemDisk]:/TEMP/Shake/cache
or
SGI: /var/tmp/Shake/cache
If you want to clear disk space, you can remove these directories, but of course all the caching information is lost. This is, however, not vital to the script rendering. It simply forces Shake to completely re-render the compositing tree. Also note that Shake will automatically create the directories if they do not exist.
You can change the caching settings in the nreal.h file stored in the include directory where your software is installed, usually
NT: [SystemDisk]:/Program Files/Shake/
or
SGI: /usr/nreal/shake2/
At the bottom of the file you will find several settings. Look for the string 'cache':
// this plug sets the amount of memory available to nodes when processing cache.cacheMemory = 64; // these plugs control the size and location of the disk cache // diskCache.cacheMemory controls how much memory is reserved // for disk buffering // diskCache.cacheLocation = "/var/tmp/Shake/cache"; // diskCache.cacheMaxFile = 2048; diskCache.cacheSize = 512; diskCache.cacheMemory = 64;
There are two main memory settings in a startup .h file,
cache.cacheMemory = 64; diskCache.cacheMemory = 64;
The first, cache.cacheMemory, controls how much memory is available to nodes when processing. The second, diskCache.cacheMemory, is for the caching of frames into memory.
The cache.cacheMemory setting works on a per-proc basis, and 64 Mb is usually enough. For particularly complex scripts, you might want to bump this up to 96 Mb. It is not necessary to increase it to a very large number to get better performance, since it controls the amount of memory Shake can use for processing, but not for caching of frames (which is controlled by diskCache.cacheMemory). Processing memory is needed only by layer nodes and some operators that can't work 'in-place' (i.e require a temp buffer), but even then, most of these nodes grab only a small amount of memory. You can override this setting on the command-line with the -mem setting.
On the other end, scaling diskCache.cacheMemory proportionally to the amount of memory available on your machine is going to improve performance significantly, as more images will be kept around. Set to approximately 25% to 30% of your total memory, or enough to hold about 4 or 5 images in memory. A 2k, 4-channel float image takes about 51 MB of memory.
So, here is an example for memory settings. Let's say you have a 1 GB , 2-proc machine and you want to leave at least 150 MB for OS overhead. 1024-150 = 874 MB. A 2k image at float is 2048 across * 1556 high * 4 channels * 4 bytes-per-channel (float) = 50,987,008. About 51 MB for float - 16 bit is half of that. Assuming you will want to boost your memory for the script up to 96 MB per proc, you would set cache.cacheMemory = 96; On a two processor machine this takes 192 MB. 874-192 = 682 MB. You want to have enough free RAM for 2 viewer buffers at float 2k, so that's another 102 MB. 682-102 = 580 MB. You want to be able to keep at least 5 full 2k float images in the cache, thats 5*51 = 255 MB. Set diskcache.cacheMemory = 256. 580-256 = 324 MB. This leaves 324 MB RAM left for flipbooks and whatever else you may be doing. At 1/3 proxy (still at float, though) you will be about 6 MB per frame so you could get about 80 frames into a flipbook.
Here are some additional functions for the startup .h files:
Function | What it does |
diskCache.cacheLocation="/var/tmp/Shake/cache"; diskCache.cacheLocation="C:/TEMP/Shake/cache"; |
Set this to a location with gobs of unused space (and expect to use it all). |
diskCache.cacheMaxFile=2048; | The maximum number of files in the cache before they start getting replaced. Least used, first out. You should leave this setting unchanged. |
diskCache.cacheMaxFileSize = 2048*2048*4*2; |
These values represent width*height*channels*bytes This is probably the most difficult to figure. You want to make sure that you can fit AT THE VERY LEAST 4 to 5 images in memory cache (the more images you can fit in memory, the better your interactivity). So say you want to cache 2048x1556 8-bit images, then you are looking at about 12.7 Mb per image (derived from the formula above). For this to work well, you'll need to designate about 64 Mb of cache memory (hence the default setting for cache.cacheMemory). This means you need about 256 Mb total memory. Of course some kind of balance needs to be found here with your working resolution, work habits, proxy settings, etc. |
diskCache.cacheSize | The maximum size in Mb of your cache directory. |