As per this study, as we have normally only seen a yellow banana and that color association is quite strong in our minds, hence when we perceive a 'different' colored banana, we are bound to see it more yellowish than is the actual hue in which the different color banana is presented.
Basically, they used 2 extremely good experiments that show that when viewing a banana (which is generally yellow), the yellow color perception is automatically activated in our brains: thus a gray matched banana would appear yellowish; while the task that requires matching a pink banana to a gray background would result in a bluish-gray banana, as blue is the opponent color for yellow and blue is added to the background gray to compensate for the memory-activated yellow color perception.
It is interesting to draw parallels here with the stroop test. In this test, color words like 'red', 'yellow' etc also appear to invoke automatic activation of the corresponding color in the brain and thus interferes with the correct naming of the actual color in which the color word is presented. Developing Intelligence has a very interesting and promising post, in which he explores the current research and computation models, that seem to suggest that the mechanism underlying stroop interference is not directed inhibition of prepotent responses, but lateral excitation among color and linguistic perception modules, with color perception area of the brain being always activated when a color linguistic term is presented and in the incongruent trials more activation seen in this to-be-ignored module as the conflicting activations of color - one due to the actual color of the word and the other due to the color perception activated by the linguistic color word ('red' ) both competing against each other lead to more activation. This is in contrast to the view that the more activation is due to directed inhibition . The new explanation advocated seems also to fit with the brain anatomy, with there being only local inhibition processes and is reconcilable with a lack of long range inhibiting pathways in the neocortex.
Thus to me, it seems more and more possible that stroop effect may be due to actual 'yellowish' hue perception in brain on watching the linguistic term 'yellow'. I know that the two examples are not the same-- a yellow banana actually has yellow color and thus its memory may affect the perception of a strange colored banana; but maybe the 'yellow' linguistic term is also somehow related in our mind very strongly with actual yellow hue perception and maybe we are all synaesthetic to the extent that all of us literally see the linguistic color terms in color rather than in black-and-white (or whatever the text color).