Selecting and masking hair is very difficult even with all the tools Photoshop has at its disposal and different subjects need different techniques, or combination of techniques.
It is very difficult to create one mask for both hair and body together, and a much more accurate result can be obtained by masking them separately.
Even with your best effort the selection you make your mask will often have a colour fringe. This can be removed.
You will often need to make a selection look more realistic by painting in extra hair at the edges of a mask.
Using Channels
Channels can be used to make very accurate selections where there is good contrast between light and dark, and is probably the most accurate way to mask hair when the subject and background make this suitable to use. See Using a Channel as a Layer Mask for fuller details.
Select Subject can do a really good job of selecting a whole subject, including most of the hair, as it uses artificial intelligence to analyse the whole scene.
Colour Range can be used to either select the hair or the background where there is a large colour difference between hair and background.
Focus Area allows you to select the area that are in focus in an image.
These selections are then further refined for hair using the Select and Mask Tool.
Mask Hair and Body Separately
It is very difficult to create one mask for both hair and body together, and a much more accurate result can be obtained by masking them separately.
When Selecting and Masking a person for a Composite it may be necessary to select and mask the hair separately from the body to give the best selection for both.
How to Cut Out Anything in Photoshop (Select and mask Crash Course) - Video by Photoshop Training Channel. This is a portrait selection which treats the hair and body separately and explains all the tools and exactly how to use them. It also shows (Ref) a better way of decontaminating the edge colour by creating a blank layer above your saved selection, adding a Clipping Mask to link it to the selection, and then painting over the hairs that need decontaminating with a selected hair colour to match that area. You can further refine this technique(Ref) by selecting only the edges of the hair by contracting the hair mask by 2 pixels and then inverting the mask, which, with the clipping mask now selects the outer 2 pixels of fine hair.
Superimpose a Selection from Another Head of Hair
Where the hair outline is very characteristic e.g. very curly hair, you can add a hair outline from another well cut out head of hair and match it to this head.
Use the Liquify Filter to roughly change the hair outline to match your subject then match one section at a time.
Refine the Edges of Your Selection
Refining the Edges of hair is a problem as it consists of such fine detail that is easily lost, and this fine detail seems to hang on to the background colour so that decontaminating the edge colour can be a real problem. Light hair also seems to let enough background colour show through it that it takes on some of the background colour itself.
The Select and Mask Tool is the best tool for refining the edge of a selection for hair. However, the settings needed will be different from those required for the rest of the body, which is often better masked separately. Sometimes manipulating the image to increase the contrast and separation of the hair with the background can help this tool work better.
Refining the Edges Using a Clipping Mask is usually the best way to overcome this problem as it allows you to use your existing mask to paint in hair colour only in the masked areas and so hides colour fringing very accurately.
Where the edges of a selection still look bad or have gone granular, they can be blurred slightly with the Blur Tool.
A hair brush can also be used to hide the edge problems.
Use a Hair Brush to Further Refine the Edges
Painting in extra hair with the Brush Tool to hide the edges can be useful where a selection does not look right and this can be done with a fine brush by hand, or even better to use a special Custom Hair Brush shape which matches the hair edge. Select a colour from the actual hair to paint with and lay down darker areas first and then lighter hairs over the top.
Even a few strands of hair, painted in, soften the edge of the hair mask and make it much more realistic, as the fine detail of individual hairs and flyaways at the edge of the hair are what gives it realism.
Match the sharpness and granularity of the painted hairs to the image by adding blur and noise to blend it in.
Using a Grey background and shoot everything sharp as blurred edges are hard to mask properly.- See Compositing > Cutting Out a Figure.
Use Channels, chose the channel with the highest contrast between hair and background, usually blue. Duplicate the channel and work on the duplicate. Invert the channel. Edit the channel to be pure white subject on a pure black background using Levels top slider, brush, lasso, and dodge and burn on the shadows and highlights to emphasis fine hair detail globally or with a selection. ---To Use Calculations. (See Menu > Image > ) ---Emphasise Fine Hairs by selecting the hair mask (CTRL + CMD), creating a blank layer, which will have the mask applied and filling this with a dark selected hair color (ALT + BACKSPACE). Add a black layer mask and paint in the fine hairs in white to emphasise them. Cloning is better with the next layer in of hair which has more texture. ---Use a Hair Brush to refine an edge e.g. Story Art > Alice Hair. Can paint on the outside in black or on the inside in white. See Brushes.
Background Eraser Tool. Brush hardness 50%. Set the eyedropper to Sampling Once, Limits to Discontiguous, Tolerance by trial and error, increasing until it goes too far and then backing off. Protect Foreground Colour is on and sample the hair colour to set this as a brush colour. Remove colour fringing by selecting the mask and cloning etc. as above. . ,
When You Can't Mask Accurately
Select the hair and extend it using Liquify. Then paint back hair on the mask with a pressure sensitive brush so that the underlying hair shows through. This is very time consuming but can mask hair where other methods fail. (Ref)
Using Blending Modes
Using a darken or lighten blend mode can be useful for bringing outline fine hairs back into an edit. (Ref 10:54)
Videos
How to Cut Out Hair in Photoshop by Phlearn. Using channels to select dark hair from a plain white background and clipping masks to remove fringing.
by Tutvid. Editing an image from a difficult background using Calculations, selecting out of focus hair, and selecting very frizzy hair from a similarly coloured background.
How to Cut Out Hair in Photoshop by Phlearn (Phlearn Pro tutorial series only available by subscription) was used as the inspiration for this page. It is my recommended resource for understanding how to cut out hair from any subject and really opened my eyes to all the different ways of doing this.
