Very often when you make a mask for a subject the edges are not perfect and you need a toolkit of techniques to refine the edge of the selection or mask.
Sometimes you may need to actually alter the way the selection or mask was made and go back and adjust something to make it work better.
Often this is not due to a problem with the mask or selection, but is just a natural colour fringe or colour contamination, which can be edited away.
The Select and Mask Tool is the best place in Photoshop to refine the edge of a selection and remove colour fringing.
Hair is particularly difficult to create clean edges with and the article on Selecting and Masking Hair deals with this specifically.
The Dodge Tool and the Burn Tool can be used to refine the edge of a mask by darkening or lightening the grey areas at the edge of a mask to make a more precise edge. The Smudge Tool can be used to push in the outline where an edge does not need to be sharp, such as a slightly out of focus area.
To remove fringing add a blank layer above the selection and apply a Clipping Mask to clip this layer to the selection so that it only shows where there is an image on that layer.
Paint on the fringing with the Brush Tool to remove it. Changing the Blending Mode to Hue, Darken or Overlay may help. (Ref)
Use the Clone Stamp Tool to paint over the fringe from a similar coloured area.
You can use a plain Brush Tool to paint in edges of a selection or a mask to improve it.
Custom Brushes can be used or created for specific purposes, such as a Hair Brush or one for fur or foliage.
Smoothing an Edge
Try to blur any selected edge to the same amount as it was sharp in the original photo.
Feathering.
Blur and Increase Contrast. Select an edge e.g. with a Lasso. Blur with a Gaussian Blur which blurs and smoothes the edge, then increase the contrast with Image > Adjustments > Brightness/Contrast. The Legacy setting may be stronger. (Piximperfect).
Sometimes blending works better if a little of the background colour is painted over the edges.
Green Screen Colour Fringe Removal
Select > Colour > Use the eyedropper to select the green and the fuzziness to refine the edge. (Using Composite Photography to Create a Fantasy World by Karen Alsop)
