Color Subsampling Notation

It's time to return to this blog and I'm getting started by taking some of the most popular posts and updating them and featuring content that still sparks conversations like one I had today with a friend and colleague that inspired me to revisit this post (I originally posted this topic in 2008).

Color Subsampling gets confused with color precision (8 bits per channel, 10 bits per channel, etc.) and color channels (Does 4:2:0 mean there isn't ANY Cr channel samples?).

In actuality, the number is really best characterized as a ratio. (click on the chart for a large visual)

“4” in the first slot is easiest to think of as representing the baseline of four pixels (and these ratios only apply to digital video signals).  The first number represents the first channel of the RGB or Y'CbCr group.

The second number and third number are frequently thought to represent the remaining two channels, but actually the second number refers to the sampling frequency of both the second and third channels horizontally and the third number was originally intended to indicate the sampling frequency of both vertically, though the system was developed without really considering vertical subsampling systems like 4:2:0. 

In the current system, the third number is either the same as the second number as in 4:2:2 and 4:1:1 indicating no vertical subsampling…all the vertical color difference samples are there in each column that has a horizontal color difference sample. In ratios where the third number is zero, the “0” indicates that there is a 2:1 vertical subsample in addition to the horizontal color difference subsample.

4:4:4

A designation of 4:4:4 would mean that there is a discreet sample for each of three color channels making up the signal for every pixel. While this could apply to either RGB or Y'CbCr used for video, 4:4:4 would most often be seen with an RGB signal, but 4:4:4 could refer to a Y'CbCr color sampling scheme 

RGB does not subsample one color channel in relation to another, so 4:2:2 (or 4:1:1, etc...) would never refer to RGB.

4:2:2

This number is most prevalent in high-end video formats and refers to a discrete sample for Y’ (luma) on every pixel and samples for each color difference signal is sampled at one value for every two pixels. While in theory this sounds like the elimination of a lot of information (a third actually) compared to 4:4:4, the human eye prioritizes the detail in the luma portion of the image and most humans would be hard pressed to see the difference between a color Y’ CB CR image in 4:4:4 and one in 4:2:2. In fact, 4:2:2 is good enough that most video types that are designated as “uncompressed” are actually color subsampled at 4:2:2.

4:1:1

NTSC DV introduced us to this aggressive, lossy color subsampling scheme. For every four Y’ samples horizontally, there is only one sample for Cb and Cr.  This creates a 4x1 four pixel horizontal “block” with common color difference values, though each pixel has a discreet Y’ value so the pixels aren’t identical. 

While DV footage was used extensively, even in broadcasting, it can be a challenge for special effects and compositing as chroma keying and green and blue screen work requires a lot of subtle tonal variations to recognize irregular vertical edges. Canopus and Matrox each created custom methods of decode for DV to attempt to improve (effectively interpolating to 4:2:2) the four pixel horizontal spread for better keying, and many software keyers have similar measures in place. 

It's interesting to note that even though 4:2:0 subsampling is thought by many to be somewhat inferior to 4:1:1, 4:2:0 (compression set aside from color subsample for a moment) can actually be slightly easier to composite or key as there is only one pixel of interpolated value in either the vertical or horizontal direction, while 4:1:1 interpolates 3 pixel values horizontally.

4:2:0

PAL DV users and anyone who outputs to MPEG has seen this number. Many people find it confusing at first as it appears the notation as a Y’ sample for each pixel, a Cb sample for every two pixels, and no samples whatsoever for Cr.  In reality, there are the same number of color difference samples as NTSC DV with the pixels arranged differently. 

Also confusing: all the color difference sample sites for the various approaches to 4:2:0 are not standard. (see chart) JPEG/MPEG-1 structures the samples so that they’re sited in the center of the four pixel block. MPEG-2 sites the samples between pixels vertically, and PAL DV sites the difference samples on alternating lines. Even with the color difference samples sited differently for different applications of 4:2:0, you could say there are still four pixel blocks that net out to the same amount of color difference samples as 4:1:1 and simply picture these 4 pixel “blocks” as square (2x2) instead of a horizontal line (4x1) like NTSC DV’s 4:1:1.

4:2:2:4, 4:4:4:4

As if all this isn’t complicated enough…you could add a number. 4:2:2:4 or 4:4:4:4 refer to 4:2:2 or 4:4:4 color sampling with the addition of an alpha channel for keying purposes. The fourth channel would carry an 8 bit or 10 bit (depending on the image format) grayscale map indicating relative transparency of each pixel in the image. The alpha number is always the same as the Y' sample.

3:1:1

This ratio appeared during the period of HDCAM's introduction.   Playback is 1920x1080, but actually record 1440x1080 to tape. In my opinion the most confusing aspect is not so much that there is a different baseline number, but whether or not that number is a proportion of “4” in itself as 1440/1920 is 3 of 4. 

The interpolation to 1920x1080 4:2:2 (this is how the manufacturer presents the specs on the playout picture) and the color difference subsampling ratio of 3:1:1 are separate issues and their mathematical scale to full raster 1920x1080 is most likely coincidental. 3 equates to 1440 horizontal Y’ samples and 1 is a ratio to 3 designating 480 horizontal color difference samples. This notation is NOT on the chart as it does not exist anywhere but in file storage, and the end user can only access HDCAM footage as 4:2:2 SDI output without a proprietary post solution.

As we continue to see new formats and frame sizes, we'll continue to see new approaches to storing and encoding images, but the color subsampling notations will likely stay in place for the foreseeable future.

TimK

A Brisk Start for Adobe's Production Premium CS6

Well, Adobe has now had CS6 out for several months and at least Premiere Pro already seems to be generating more buzz than even the CS5 release...and frankly that's saying something.

Only time will tell if Premiere Pro may have reached the feature and function set that will really gain traction in the higher end entertainment sector of the industry, but I suspect you'll be surprised at the amount of entertainment and feature production that will be attributed to Premiere Pro in the next year.

The new release of Premiere Pro has been keeping me busy as well...I've spent some time on the phone with production facilities around the USA who are changing their workflow to switch to Premiere Pro, which has been an interesting study in having to change how you even think about what it is you do, much less just how you do it.

I'm scheduled to train U.S. military personnel who are migrating to Adobe for their work in combat camera in just a couple weeks, I've been in the studio to do my Class On Demand training course for Premiere Pro...and my fxphd.com class - "PRM203, Intermediate Premiere Pro CS6" is under way.  I've also wedged in some time to do the technical editing on two books...one being Adobe's Classroom in a Book for Premiere Pro CS6, and another for Focal that isn't released yet.

I've also had a spike in production lately and I have to say that even as a long time Premiere Pro user, I like the changes that Adobe is making...The new keyboard shortcuts may be different, some features have changed, but the changes feel good and I have to say that this software feels pretty nimble relative to CS5...which was no slouch.

More to come on Premiere Pro in coming days, but I will point out that the blog archives are still available by keyword over to the right and down the page.  I still get regular requests for the color sub-sampling and vectorscope articles particularly.

TimK

Continuity or Innovation? FCPX makes the trade.

Well, all the buzz about the coming out party for FCPX is starting to appear to actually be a surprise for Apple as FCP users have now triggered an unprecedented refund-due-to-disappointment operation at the app store...even the recent "I need to hold my iPhone with a BBQ tongs in order to get decent call clarity" debacle now seems tame in comparison.

When well-respected FCP advocates like Larry Jordan need to publish statements to their user-brethren to clarify the limitations on their involvement in a software release, and Walter Biscardi immediately starts to share his journey to an alternate application, I think it's clear that the development philosophy, if not the majority of the process wasn't really opened to the installed user base for discussion, even though apparently Apple did solicit input.  As Michael Wohl, an FCP user who had pre-release access to FCPX put it in a recent LAFCPUG meeting demo, he submitted "pages of notes...", then indicating none seemed to have been used.

While I have to admit that as a bit of a software advocate myself for a competitive product, I can relate to the feeling you're not being taken seriously when you're asked for this type of input...however, depending on the company, a user asked for input may not be completely informed of larger company plans or initiatives, either because of corporate confidentiality, or even just because the company may be interested in a completely unbiased perspective.

In any case, the reaction of shock on the part of users seems odd as Apple is known for decisive innovation, as well as pretty effective feature-set intuition when it comes to creating products that are popular and profitable.  Apple's genius has never been so much in knowing what the market wants as much as it has been in persuading the market that what Apple has is what the market wants.  Steve Jobs himself has been quoted multiple times uttering statements that are various iterations of "Customers usually know what they want when you show it to them..."  In my opinion, Apple's track record indicates this cannot possibly be a significantly flawed philosophy.

In the case of FCPX, Apple has made a move to get into position for the "everybody will edit video" marketplace and I think that they'll likely see revenues in a couple years that will validate their fundamental revamp in FCP's operating philosophy from a stockholder benefit perspective.

However, when a company decides to change their market target, that effort can be especially clumsy when the existing customer segment they've been highly publicizing as an indication of their product's legitimacy is actually part of the casualties.   Apple has been standing on the shoulders of their vast installed base of professional Final Cut Pro users who edit high profile entertainment projects in network television and film to show proof of their established legitimacy for the last couple years as their users have been waiting for some sort of significant upgrade.  Apple's customers (who have a brand loyalty that is likely the envy of any business in any market) have been some of the most effective promoters of the Final Cut Pro application, even as it has been left behind in features and versatility by its competitors in the years since its last significant release.

Final Cut Pro's installed base bought it some time, as Avid's installed base did 5 years before.  Even as it became obvious during the "DV years" that multiple players in the industry had passed Avid in feature set and cost-effectiveness, Avid wasn't completely out of the race until a significant portion of their users got frustrated enough to undergo considerable stress to change platforms...not something a professional facility can do spontaneously.

However, after spending their "customer base inertia" capital buying a couple years of customer patience, Apple created a new piece of software.  No upgrades.  No legacy version project loading.  Existing  FXplug-ins (a completely new plugin standard for Apple alone that the industry was forced to comply with, introduced a few short years and one version ago) aren't compatible.

To top it all off, the new "professional" application not only looks like Apple's consumer editing application, it's actually project-compatible with it.  While this iMovie compatibility isn't a cardinal sin, it certainly is an embarrassing feature to have front-and-center for a customer base and a company that has traditionally been so professional image-conscious up to now.

Ryan from FilmRiot in this clip sees the iMovie compatibility and legacy FCP version incompatibility as a "...big middle finger to professional editors signed: Apple..." and it gets more entertaining from there:




Is what Apple did wrong?

Apple is a business and businesses make money, and Apple has shown that they know how to do that.

Is what Apple did foolish?

I think if even professional editors are honest with themselves, the combination of more varied and shallow markets for video content  developing simultaneously with a significant cultural shift toward media literacy can't possibly be ignored as a force in moving video creation "down-market."  Odds are that in a short time, Apple will see revenue figures that endorse their re-targeting efforts.

Does FCPX change the professional post market?

I think so.

Adobe Premiere loads projects from versions that are ancient, even in cases where not all the features translate, you can still load the basic project and revise or update it.  I get calls from clients to do this with regularity, and I don't think my business is that unique.  Avid makes efforts to maintain backward compatibility.  This is how you keep a massive user base massive in the professional world.  Does it restrict how far you can jump from version to version?  I would think it would have to, but if you are creating a product for professional use, that product's ability to keep productive continuity flowing and therefore keep your customer's groceries paid for, is the most critical feature that product has.

Apple allows a user to run their legacy version of FCP along with FCPX on the same machine (since it's not an "upgrade" I don't see how it could work any other way...), but previous versions of FCP are no longer for sale (Edit: 7/7/11 it now appears that Apple will offer FCP7 for sale again...for those who wish to buy additional seats before it's permanently EOL'd).  Of course the way Apple upgrades hardware and their OS, it's not likely that FCP legacy will operate on Apple systems all that far into the future.

FCPX is enough of a philosophical and operational departure that FCP users will have to take some time to adapt...but the clock is ticking as the next Apple OS revision in a year will sneak up fast, not to mention big changes in the media architecture (QuickTime) are moving forward...

While all this might be on-plan for Apple's ultimate goals, it will have to change its marketing pitch for its new editing system as the user names and faces it's been putting out there to be the image and reputation of its professional editing product appear to be left standing outside the new target area for FCPX.

...and it doesn't seem like the fact is lost on anyone that iMovie projects load in FCPX and the features and television programs edited in FCP projects previous...load in Adobe Premiere Pro.

I thought it interesting that Michael Horton of the Los Angeles Final Cut Pro User's Group was quoted as concluding the recent meeting with some at least partial humor: "...LAFCPUG has never been about just FCP; we're about the craft of editing, the art of editing — no matter what tool you use. We just draw the line at PCs."


Um...OK...perhaps the name of the user group may need revision?   


MMSG perhaps?  (The Mac Myopia Support Group) 




TimK

Apple's 'Sneak Peek' offers a few answers, but lots of questions.

Well, we finally know what Apple's been working on during the last couple years while Final Cut Pro users have been waiting and wondering...  They offered a glimpse of the new "Final Cut Pro X" on the Supermeet stage they appropriated from their various industry partners one week prior to the event.

Emmanuel Pampuri has the presentation, captured from the audience, on Vimeo... 

I will say that Apple is certainly capable of creating more marketing momentum from small amounts of precisely applied information than any company in any business that I've ever seen.  I don't think that it is overstating the situation to say that this is their genius.

In this case the momentum seems divided between two camps.  There are those who look at what little they saw and imply a "prosumer direction" due to the streamlined interface and its similarity to iMovie...

Scott Simmons compares the iMovie and FCPX "magnetic timelines..."

There are those who have anointed FCPX as "the future of editing..."

Videomaker magazine declares FCPX "Best Editing Software of NAB 2011"

...I guess there wasn't a category for "Best Keynote Presentation Showing Screen Grabs of Software that Doesn't Exist Yet.".

I'd say that the jury will be out until we see something we can at least test drive.  QuickTime has been left to seed as the first computer company with a 64 bit OS now needs to completely gut its 32 bit media architecture 8 years later, so the changes coming from Apple will undoubtedly be significant.

I for one don't believe that a streamlined interface is a blatant cue that the application will lack the sophistication a professional needs...but then I also believe that presenting "background rendering" as if it's cutting edge technology when it was a feature in Pinnacle editing software nearly a decade ago (before Avid acquired them), is pure Apple, playing catch up without ever losing the swagger.

One thing is certain, it's nearly impossible to find a neutral opinion on what transpired during the presentation, which keeps the industry buzzing...which is what illustrates the genius in not revealing too much, but tossing a few hints out to seed the prognostication of Apple's motivations, intentions, and direction.

Hopefully the buzz level stays high enough that they get the bounce they'd like at release as no matter what Apple's charts say about their user base growth and market share, Avid and Adobe are gaining ground every day that there isn't a freshly minted Final Cut Pro version to evaluate.

Of course, once Apple's media architecture starts experiencing version 1 bumps in the road, look for Apple's computer platform to get some scrutiny as Microsoft's Windows 7 builds on an already fairly solid start...

That's all I've got as my crystal ball is starting to fog up...

TimK

NAB 2011...let the insanity begin.

Visit me in the AJA Video Systems booth SL4420

OK...one week and counting until the National Association of Broadcasters tradeshow in Las Vegas and things are already starting to look interesting...

GoPro buys CineForm.

Press Release

I own 5 GoPro POV cameras which I originally bought for automotive work, then started to use them for various other things...I really like them and they are an incredible piece of equipment for the money.

I use CineForm Intermediate codecs for many purposes within various HD post production workflows and have for years.  It's an excellent, efficient, and very high quality video compression system that has in the last few years, also become one of the industry's truly useful metadata management systems.  One benefit of using CineForm's workflow is the ability to use their primary color correction system along with the rest of the metadata management tools in their application called "FirstLight", including the ability to do alteration of visual stereo (3D) settings on CineForm's industry-leading file structure which puts the right and left eye video clips inside the same video file.

(Disclaimer: I wrote the current manual for FirstLight as well as use the application regularly...I will not claim complete objectivity...though there are almost no competing tools to compare it to...)

GoPro has been selling their cameras to almost every conceivable user at every level of video production from a teenage BMX racer or skateboard performer to professional video production.  CineForm has been focused on several professional production markets as well as video enthusiasts to some extent, but with the CineForm RAW format being used in the Silicon Imaging SI2K camera (and therefore figuring prominently in the production of the movies "Slumdog Millionaire" and more recently "127 Hours") and also providing a convenient post path for the RED camera, CineForm's sweetest sweet spot is probably higher end more than anything.

So who saw GoPro's purchase of CineForm coming?  Not me.  I've been working withn CineForm products and even working with the company for a number of years and with 5 GoPro cameras and additional GoPro mounting widgets for nearly any circumstance, I can at least characterize myself as a loyal customer on that end of the spectrum, and I would not have guessed this one.

GoPro's upcoming stereo (3D) offering was shown in prototype form at NAB 2010 and I am sure that CineForm's elegant handling of visual stereo figures prominently into the general landscape of the situation.

A couple things are certain.  GoPro's business track record speaks for itself.  A product line that has only recently really had any marketing push beyond us users pointing other users toward the product has built an impressive enterprise which seems to have a very clear view of their market's needs and expectations.  CineForm has been cranking out impressive technical innovations for a number of years now and their video codec and workflow plays a quiet role in many little known but high-stakes workflows.

The possibilities for this unlikely marriage boggle the imagination...hopefully the realities coming are just as tantalizing...

TimK

3D Viewing "Not Necessarily Bad" -American Optometric Association

The American Optemetric Association has apparently pointed out that viewing entertainment content (television, movies, games) is not necessarily a bad thing for young children.

This was apparently all triggered by Nintendo's warning label on their new 3DS hand held gaming system (google translated version here ).

Michael Grotticelli's January 21st entry in his Broadcast Engineering.com blog brought this to my attention...and it's an important topic for me as a parent...and as a content creator.

The gamer site 1up.com also has an article taking it even farther, quoting a New York Times article that describes studies using Rhesus monkeys, subjecting one group to constant 3D content for 3 months versus a control group that didn't showed no difference in visual development. It leaves out any findings in the way of the ability of the 3D viewing monkeys to ever get a life, a job or to move out of their parent's basement...but I digress.

I'll go on record as having had some concerns about the long term effects of viewing 3D content in marathon gaming sessions, or even several feature films back-to-back (Star Wars 1-6 if it were all in stereo on a rainy weekend for instance...) on young children. My 11 year old son certainly seems to become hearing-challenged when he's in a particularly prickly HALO scenario and my wife is calling the family to report for dinner...

With the nature of gaming and "forced" or synthetic 3D, it seems that 3D animation for gaming is probably the most cost-effective area to add stereo imaging in all of entertainment. Adding a render camera to a 3D animation to provide the other stereo "eye" is far easier and faster than adding, syncing, and managing a second camera on a feature shoot...at least for the moment. Kids (and some adults of course, though with less concern for visual "development") tend to play games for hours on end. What effect will this have long-term? Particularly in cases where a child may not have an athletic or otherwise physically "active" life in addition to their interests in things competitive, but virtual?

I couldn't find the particular findings that Grotticelli was referring to on the AOA site (not that I'm challenging it, just that "trust but verify" thing is the only Reagan-era strategy that really grabbed me...), but there are several other articles I could find.

First, an article on how many people have vision imperfections that at the least, prevent them from perceiving the difference between visual stereo and standard imaging, and at worst give them the headaches, nausea, etc that are mentioned by some people who view stereo (3D) content:

AOA.org : "...Not Everyone Can See What All the Hype is About"

And second, an article that reminds us of the inability of children to judge when to stop doing something enjoyable, often past the point of apparent exhaustion.

AOA.org : Impact of Computer Use on Children's Vision

Bottom line? I'm probably not nearly as informed as I should be if my goal was to lead you toward some specific conclusion.

The fact remains that human stereo vision and the way a human brain processes input has to be affected at some level by viewing stereo visual material as when that edit comes where we change from a close up to an establishing shot of the next sequence, our brain sees that we are changing from focusing on something closeup to something far away, but our eyes don't re-converge at all as the actual image plane we focus on is stationary. Does this help eye-brain cooperative function, or cause it to "disconnect" to maintain immersion in the experience? We move from place to place, and change our focus from point to point, but it's all done for us as the focal point remains the reflected, emitted, or transmitted image plane we are viewing.

Am I saying I'm opposed to 3D movies and entertainment? Absolutely not.

I'm saying we all need to be aware of the long term impact of any activity that we repeat often enough whether it's carpal tunnel issues from typing, health or safety issues from drinking (I'm a Single Malt Scotch guy myself), or the effects on our visual acuity and development from viewing our entertainment content in 3D.

...and criticism of Nintendo for issuing a common-sense warning on their products in a world where most of us can't even eat in a way that keeps us healthy, seems absurd.

TimK

2010...Moving, teaching, but not blogging

Hi all,

I just wanted to reach back to the blog and get going with writing again...

I know the entire year has past and things have been busy with Adobe releasing CS5 to the world and all the training commitments that came with that...

I have two CS5 training DVDs available from Class On Demand, one is an introduction to the CS5 Production Premium Suite product, and the other, a deeper look at Premiere Pro CS5 itself.

I've also been teaching Premiere Pro CS5 at FXPHD.com, we're on the second term now, and I think that there are a lot of people seeing what Adobe Premiere Pro is really capable of for the first time.

In a very short couple of weeks, I'll be doing a free webinar for BorisFX, so look for that...

AND... I'll be picking up the blogging function again.

2011 here we come...

TimK