Using Shake To Format 10bit 422 Yuv Into Floating Point Linear

[user_deleted]

cineon is not a linear file format, and doesnt support float afaik, so you might wanna check if digital fusion can read .iff files. otherwise use .exr (or check the manual for other file formats that support float).
but i'm not sure why 16bit isnt enough for you.. the original HDCAM was 8bit afterall (but hey, if you have the storage and processing power

anyway, just make a FileIn node with your QT file, add a Bytes with your desired bit depth, and a FileOut with your desired file format (make sure to add a # or some @@@@ signs to make it a image sequence). and check for gamma differences. you might want to read up on a few things before you tackle stuff like that though.
++ christoph ++

[user_deleted]

Thank you, The end product is going out to film, the effects house is used to receiving 2k scans as DPX log files from their clients like me, their work is very high quality and I'd rather promote my footage to their standards rather than demote them to mine, they'll use their 3d work and composit into my scans using Fusion, I can't send them all my work and trust their upconversion to RGB. Thay rarely do that so I'd rather talk with you all first to determin the best workflow for the entire show and do it myself on my Shake system, I might as well up convert the entire project and then reconvert the show back to quicktimes as 10 bit RGB linear files where they'll stay in that format through the rest of the workflow to film inter positive.

Yes, they will accept .iff files, I'd imagine that they use Maya and these would be the files they'll composit my footage with anyway, I should ask them what they're going to use and convert my stuff to that format?

Can I batch all three thousand clips at once using Shake or would I need to do them one at a time?

Do you think I'll be able to figure it out following your suggestions above? Will I need to include any custom LUTs to compensate going from YUV to RGB? Are there any provisions for using a custom or preset gamma in that conversion node?

Thanks, Keith

[user_deleted]

keith,

this is fairly complex and if you do things wrong you'll deeply regret it.. so you might want to ask somebody to help you since this is not something we can explain on a few internet posts..
that said, a few pointers:

QUOTE(keicol @ 03/12/06, 08:11 AM) [snapback]230776[/snapback]
[...] I can't send them all my work and trust their upconversion to RGB.

why not? it seems to me that they should have a better understanding of the matter than you..
of course, i can imagine that they refuse to work with quicktime files or charge you extra.

QUOTE
I might as well up convert the entire project and then reconvert the show back to quicktimes as 10 bit RGB linear files where they'll stay in that format through the rest of the workflow to film inter positive.

10bit linear RGB is usually a cineon log file, not linear....
and if you're thinking about going to log files, you really should get some help with that if you never have done that before.

QUOTE
Yes, they will accept .iff files, I'd imagine that they use Maya and these would be the files they'll composit my footage with anyway, I should ask them what they're going to use and convert my stuff to that format?

of course you should talk to them! that's the first thing you usually do.
(and while you're at it, ask them if they really want it in float)

QUOTE
Can I batch all three thousand clips at once using Shake or would I need to do them one at a time?

you can batch them, but you'll need to read up on shake terminal commands and scripting. this has been discussed here recently, so search the archives.

QUOTE
Do you think I'll be able to figure it out following your suggestions above?

i dont know, depends on how much time you spend on it...
the basics are easy, just make a test

QUOTE
Will I need to include any custom LUTs to compensate going from YUV to RGB?

no, the quicktime FileIn converter will do all the work for you..

QUOTE
Are there any provisions for using a custom or preset gamma in that conversion node?

as said, you dont need the Colorspace conversion node. to check for the gamma, you could export a few test frames out of FCP using the quicktime conversion.. load that into shake and compare to the YUV qt clip.

++ christoph ++

[deke-kincaid]

Also a quick note. Going to float is a waste of space(unless your using exr). A 10 bit YUV file will easily fit in a 16 bit file. Anything more is just extra file size for nothing.

[user_deleted]

I'm working with an off shore third world different language effects house who have been trained in a film to film work flow, and that's what they know.

Quicktime files are an anomaly for them and in addition, quicktime files aren't able to be run through the Fusion render farm preventing their fifty machines from converting my quicktime files into image files which they will use in their composites and render farms.

I worry that they will simply do something crazy just to get my files into a format that they can work with and am researching this myself in hopes of preserving the integrity of my work thus far.

10 bit linear RGB 444 is a quicktime codec (Kona2) that will maintain the linear workflow from what was originally captured on HDcam 8 bit YUV (linear) and I expect that the effects house will work in a linear floating point space and as I put it all back on my Final Cut Pro System, add my credits colorize with Final Touch and export back to tape at HDcam SR 10bit 444 RGB Linear which will be delivered to the tape to film printing house.

I would like to hire someone to help me through this process but since it's somewhat of an unusual workflow I'd expect that they'll have to do a bit of research themselves and there's noting better than being on the same page to know when your hired help is going off in the wrong direction. Then there's the old saying that if someone's gonna screw it up, it might as well be me, it is after all, my stuff.

Here's the questions I need to ask:

1. (for my effects house) What kind of files will they import from their 3d machines to composite into my footage. They've requested that I provide them with DPX log files, they don't explain why, I had to call Fusion tech support to figure all this out about their usual film workflow and the render farm thing.

2. What kinds of files will the film recorders be able to use for printing the inter-positive. I know that DPX log is their norm and that they can import HDcam SR which is becoming a standard for a digital intermediate tape delivery format. I'd actually prefer to deliver in a file format thereby saving from ever having to go back to tape and keeping files on hard disks for hand delivery (cheaper that way).

Every one out of six shots on my project will include CGI work, it'll be easier to convert the whole project to a file format that will be compatible with the 3d files being made for the show. Then keep the whole show in the RGB space either log or linear through color-correction, titles and credits, and finally the film print out. The main question is, how do I take my 10bit YUV 422 quicktime files and get them into the format best suited for my effects work and staying in that colorspace through the rest of the processes. I was wanting to use Shakes superior abilities to make these files as I trust Apple, always will.

Thanks for understanding my dilemma.
Keith C

[deke-kincaid]

Finaltouch comes with a commandline utility that converts quicktime movies to dpx files (qt2dpx I think). Look at the manuals for ref on how to use it.

[user_deleted]

It's actually the AJA that has that utility, it's a rather primitive tool that's found free elswhere on the internet, it does a simple gamma adjustment and header rewrite to any 10 bit RGB quicktime file, problem is that I have a 10 bit YUV 422 quicktime file which needs more attention and finese, hence my posts here.

[deke-kincaid]

I know about the kona one, that is something different and not what I am refering to. Look at the commandline one that comes with finaltouch. It can write out dpx log files, cineon, and you can set the white/black point.

http://www.siliconcolor.com/support/docs/Toolset/index.html

[user_deleted]

Thanks for the link, yea, I don't own that program yet but will after NAB, I do own Shake 4 and think it wise to bring someone in to make this happen for me, any suggestions on a local hire, LA area?

Thanks again, Keith

[deke-kincaid]

Finaltouch install is on their site in the download section. Only the main application looks for a license, not the commandline tools.

[user_deleted]

Okay, sorry for all the whining, I just got back from the office and am glad to say that I followed the steps that Christopher laid out above and I added a few more steps that I had to figure out myself and was able to pull it off very easily, I'm rather embarrassed at how simple it turned out to be, it took a bit to figure out the naming process and navigating around the directories, figuring out how to render more than just the first frame in the sequence took me about an hour of reading manuals but once I figured out to hit the auto button that was a snap, all the images rendered out finally, so far I've made floating point log and linear files, Dpx, Cineon, Sgi, Targa files and have uploaded them all for the CGI folks on the other side of the globe to test then return back with test composites for me to convert back to QT keeping it all in 10 bit RGB, exactly what I was wanting, Thank you all, the internet is a very powerful democratic place, I hope to help others with the process of shooting HDcam and making a film out all on a simple Mac computer, a feat previously reserved only for the privileged! THANK YOU!!!

[user_deleted]

QUOTE(keicol @ 03/14/06, 06:53 AM) [snapback]230985[/snapback]
[...] all the images rendered out finally, so far I've made floating point log and linear files, Dpx, Cineon, Sgi, Targa files

glad it all worked out, although i'm still wondering why you're insisting in using float images..
in fact, all the formats you listed above dont support float, so i'm not sure it you really understand what float is and why and when you need it.
you probably also want to avoid log files.
++ christoph ++

[user_deleted]

Well, too much research can be a bad thing, I've learned that working in log which is what the company wants is not good, from what I understand when working in log irregularities can get amplified along the path which is why I want to work in linear and so that rules out Dpx and Cineon files which is their preferred delivery format. They don't convert to lin from log and work in lin then export log, they stay log all the way through. Since I'm starting with HDcam footage (8 bit 4:2:2 YUV) which was put to HDcam tape as 3:1:1 seven times compressed 8 bit YUV then uncompressed off tape to 8 bit YUV 422 to HD SDI into my Kona 2 and brought into my computer as 10 bit YUV 422, I'd like to stop any further degradation of my image and from my research (and maybe I'm being too anal) working with Dpx Log may not be best for my footage on down the composite pipeline. With Shake I can deliver any format to the Fusion guys that I want and they'll deliver back to me what I ask them for, however they won't change to linear workspace if I deliver to them Log file types.
I was thinking of testing .exr and .iff and .flx files today. Maybe I should just give them what they want, Dpx Log and be done with it, I can't really see any difference beyond some contrast issues that I can't fix later in color correction. I've sent them the files I've converted, they've done sample composites and sent the files back for me to evaluate, I look at the overall image for picture quality, I look at it's waveform and the punch it up a thousand times and compare pixels of the different color channels. It's a long drawn out process. I'm in search of the perfect way to turn shit into gold, a futile exercise that foolish men have gotten addicted too throughout history.

[user_deleted]

converting to log, if done properly, wont degrade your footage...
all i'm saying is that you want to make sure it's done properly.

anyway, i'd be suspicious of any effect house that cant accept anything but log files, and cant tell you which parameters they prefer you to use to create them.
++ christoph ++

[deke-kincaid]

QUOTE(keicol @ 03/14/06, 08:32 AM) [snapback]231042[/snapback]
which is why I want to work in linear and so that rules out Dpx and Cineon files which is their preferred delivery format.
No, dpx and cineon is just a container. It can store linear color files just fine. Actually the Sony Cinealta and any hard drive based HD Cameras all write out dpx linear files.

QUOTE(keicol @ 03/14/06, 08:32 AM) [snapback]231042[/snapback]
They don't convert to lin from log and work in lin then export log, they stay log all the way through.
I doubt that. Most compositing software converts to linear when working and then convert back to log at the end of the tree on fileout. I know fusion does this. If they are working all in log throughout their pipeline, I would have serious doubts about working with this company. Most likely something is getting lost in the translation or a person that doesn't know what they are talking about is doing the communicating to you(ie producer, PA or it guy).

[user_deleted]

I called fusion tech support and asked them if they convert log to lin if importing Dpx log files and work in lin when compositing with files from lightwave and the tech support guy said that fusion works in whatever file format you decide to work in, if you import and export dpx log files than that's what you work in, there's no log to lin conversion that's obligitory or expected, I spoke with the guys at the effects house and they verified that log is good, they do it all the time, never had a problem.....

I, on the other hand, have found in all my research that log is not good and refuse to give them log files, I am wondering if maybe Shake can output Lin Dpx or cineon files, I don't think I saw a button for that as an option assuming that all their exports were therefore Log.

I'm getting a favor from this company who actually do fantastic work, they're just not multi format trained, each person at this company was trained to do one thing, it's off shore, third world and run by a friend from here who's not an expert at this format thing either, they work film in film out, i.e. Dpx log in Dpx log out.

Their artists are so great and this project of ours will put them on the map, I hope.....

Meanwhile it's basically up to me to determin the least degrading workflow and their guys will follow it though, I just have to propose it, what I give them file wise is what they'll work in and deliver back to me.

Keith

[user_deleted]

QUOTE(keicol @ 03/14/06, 11:15 PM) [snapback]231089[/snapback]
I, on the other hand, have found in all my research that log is not good and refuse to give them log files,

log is not good? it's more tricky to handle, but it's actually better than linear in some respects.. however, some stuff just cant be done in log space and that's why most compositing is done in linear anyway. maybe you do some more research.

QUOTE
I am wondering if maybe Shake can output Lin Dpx or cineon files, I don't think I saw a button for that as an option assuming that all their exports were therefore Log.

you use a LogLin node for converting, not the fileout node...
however, if they refuse to work with linear .tga, .sgi or .iff files, i dont see how they would be happy with linear .cin files.

++ christoph ++

[user_deleted]

10 bit Cineon files(Log files?), from my meager research notes, were originally developed back in the days when computers were slower and storage and ram cost more, it was determined that the highlights could be compressed saving more data space for blacks and midtones verses highlights, the result was a smaller 10 bit file.

the problems I was referring to with log is that according to one book I read is that when when working in log anomalies may become exaggerated on down the line, operations that involve multiplying and dividing (mult, Gamma, Over, and so on) may return unpredictable results because values are being unevenly pushed in the higher or lower ranges.

That's why I said, log is bad,

I would prefer to do as you said, deliver log, convert to lin, at the end convert back to log, My visual effects people said that they would work in Linear if I delivered in linear, seems fair.

[user_deleted]

QUOTE(keicol @ 03/15/06, 02:04 AM) [snapback]231118[/snapback]
10 bit Cineon files(Log files?), from my meager research notes, were originally developed back in the days when computers were slower and storage and ram cost more, it was determined that the highlights could be compressed saving more data space for blacks and midtones verses highlights, the result was a smaller 10 bit file.

so you think that these days, 30TB of fast, fail safe storage for feature film project in 4K float is cheap? and processing times a snap?
++ christoph ++

[user_deleted]

Basically you can comp certain types of shots quite happily in log with no loss if you are not doing anything which requires a linear computational colour space.

if you are using CGI then you need to bring it and your film plate together into ONE colour space to composite them. Most people prefer to do this in Linear space. This will actually require you to apply a lookup (in the case of shake a logLin) to BOTH the CGI and the scan. Doing this at floating point bit depth is the safest way to ensure no data is clipped.

As you are capturing data which is already compressed and colour managed to a broadcast spec (4:2:2 YUV) none of this should really be an issue as you are alrady locked to a video colour space paradigm !

It seems that lot of the confusion on this thread is because people are unable to find good solid information about film compositing practises and that there are a lot of assumptions going on based on terms tha thave not been correctly defined.

I can see that some folks are on top of all this stuff to. for all those who aren't

Log : is a colour encoding format which is based on Kodak's method of compressing the data on a film negative into 10bits of digital information designed to represent a MUCH larger range of Colour values. This is typified by the logarithmic graph used in the encoding of the values. Colors in the toe and head of this graph contain lots of dat but are not easily viewable on conventional display devices. Hence the use of refernce black and white points !

Lin : a colour space in which colour value of a pixel is assumed to have a linear relationship with its brightness. Tape and digital imagery (CGI) are not linear as they also have a gamma correction embeded by default. Linear footage is obtained by negating BOTH logarithmic and video gamma encoding

Flaoting Point /Bit Depth : Log Lin and video have no baring on bit depth whatsoever. Bit depth is a measure of the ammount of data stored perpixel in each colour channel and can be any of the conventional values 8, 10, 12 (for those of you with discreet systems) 16 or Float (32). but can also be other values.