A garbage matte is a rough, simple matte that isolates unwanted elements from the primary element in an image. Back in the phto-chemical days of special effects (that means pre-digital), in order to make models move, it was mounted onto a blue pylon and the camera moved on motion control tracks, giving the model the apearance of movement. Alot of times the camera would have to move 30 feet away from a model to get the apearance of a model doing a flyby and then flying into the distance. Since the lighting of the bluewscreen wa paramount to pull a decent matte, the bluescreen wouldn't be that big. So when the camera pulled away you would see the outside of the bluescreen, and the motion control tracks, lighting rigs to light the model, etc. in order to easily remove those, animators in the rotoscope department would draw onto animation cells a border within the bluescreen, but around the model, and then color that area outside with pure black ink. Most cases they wouldn't do one for each frame, just as many as needed during the move to hide anything outside the bluescreen area. When the optical department would pull a matte from the bluescreen film, they would put a red filter into the optical printer and composite that onto high contrast black and white film. On black and white film, purple registers as black, so the bluescreen would black, but the model would still register on the film. From this they would pull the different holdout mattes (the different plugs that they burn into the different elements to make all black aroudn the model, and a black hole the shape of the model over the background so that when they compsoite them in the optical printer, the images go together like a puzzle. However, the red filter would only take out the blue screen itself, and anything outside of the bluescreen area described above would still be in the frame. This is what the garbage matte is for. It is for covering them up in the optical printer during the final composites. In the final stages of optical compositing, they would take the original bluescreen image, along with the different light passes (usually lighting a model takes 2 to 3 different passes. One for the sun light, one for the shaddow area, and one to try to cover the areas that have bluescreen spill on the model, as well as any internal lights the model has. Each pass is exposed on the optical printer in successive order with the holdout matte and garbage matter over that. Now, the black that is created from the red filter over the blue onto B/W film is a different shade of black to the one made by the black ink from the garbage matter. Therefore, when you composite the final shot, you will see what lookes like ghostly splotches around the models over the background. Look at the TIE fighters in the Death Start Trench run and you will see a ghostly sort of form around the ships. And sice each ships had to be shot seperatately, you will see a different one for each ship. This is the film registering the difference between the garbage matte black and the matte pulled from the bluescreen. I know i probably confused you more but that is the most layman of terms to explain it.
Now imagine the time if takes to do this for each ship. Ken Ralston, who is a special effects supervisor, but was compositing supervisior on all three OT Star Wars films went a little cuckoo working on the final space battle scene on Return of the Jedi. The one shot of the Tie fighters flying up to the Mellenium Falcon and the shot of the falcon flying through the swarm of Tie fighters. That had probably 100 models in that one shot alone. Each model was a seperate element. Then you had the mattes, holdout mattes for each element, and the garbage mattes. You also had the laser animation, the moving starfileds, moving Death Star, etc. There must have been a few hundred pieces of film composited together just to make that one shot. Ken Ralston was working literally 20 to 22 hours a day. He went so cukoo at one point that he took of his white Nike shoe, shot it and composited it into one of the shots as a Star Destroyer. If you look real close it is there. It is a shot looking at the shot from the front.
In the old days if it was a bluescreen shot with a person they would have to print those frames onto a 4 pin aimation frame and rotoscope (draw a black outline and paint black) around the people. This was another form of garbage matte, just in the shape of the person for every frame. Sometimes they would have to do it for bodyparts that had bluescreen bleed, like the nose, hair, etc. For tron there was one guy who was responsible for rotoing the character RAM. The actor was walking down the street and a guy he had never met before walked up to him and said "You're Dan Shor" The actor said "Yes I am. Do I know you?" And the guy said "I hate your nose." And Danny Shor said "What?" And the guy said "I am the rotoscope animator on tron that is working on your mattes and I have been working on you for 6 months and I hate your nose." This one guy had been working for 6 months on just his character and the film wasn't done yet.
For Superman, there was a rumor that animators went blind painting out the wires for each flying shot. In those days they couldnt just do wire removal like they can now, and each part of teh frame the wire went over had to be painted the exact color of as the colors around the wires. If they didn't match, they had to repaint the whole frame again. and they had to do this for every single frame that wires showed up in. That is 24 frames a second. And some of the fling rigs had from 8 to ten cables holding Reeve up.
With digital technology, the carbage matte is just painting blue over all of those things around the bluescreen with the same shade of blue that the bluescreen is in the shot. Therefore no halos around the objects.