right the file goes in your HOME/nreal/include/startup/
shake scans this directory on startup and will load any custom settings from it.
its a plain text file, use text edit or VI, plain text files only mind, no rich text. Call it whatever you want as long as it has a .h extension, startup.h or memory.h, you can have multiple files if you want.
the three lines you'll be adding are:
cache.cacheMemory = 96;
diskCache.cacheMaxFileSize = 2048*2048*4*2;
diskCache.cacheSize = 512;
diskCache.cacheMemory = 500;
you want to up the disk cache size and memory parameters to suit, and yes they are case sensitive.
you can also edit these values in the nreal.h file in your Applications/Shake/Shake.app/Contents/Resources but be very carefull, you don't want to screw up this file. If you do, it's a re-install if you don't have a backup.
if you do get stuck, take a look under the "tunning the amount of RAM shake uses" section in the manual, that should sort you out.