The readings of the poem which see it as focusing on sexual desire point to the following elements in the poem:

It is maids who hear the goblins cry. In the nineteenth century maids meant virgins. Does this suggest that their sexual inexperience makes them sensitive to the cries of the fruit-merchant men?
In English writing food and particularly fruit is used in descriptions of sexual pleasure. Or, food is associated with various erogenous parts of the body. So 'Taste them and try' becomes an invitation to sexual excess.
It's also traditional to talk of loss of virginity in terms of cutting off a lock of hair. This links to Alexander Pope's poem, Rape of the Lock (1712). This makes the negotiation between the goblin men and Laura as about a loss of sexual innocence. For Laura the incident has huge significance 'She dropped a tear more rare than pearl' (l.127)
The most obvious link between sexual desire and eating fruit is the description of Laura gorging on the fruit in an excessive way. Pick out examples of her excessive indulgence and the sense of intoxicating (i.e. leaving her drunk) pleasure in the lines below:

[she] sucked their fruit globes fair or red:

Sweeter than honey from the rock,

Stronger than man-rejoicing wine,

Clearer than water flowed that juice;

She never tasted such before,

How should it cloy with length of use?

She sucked and sucked and sucked the more

Fruits which that unknown orchard bore;

She sucked until her lips were sore;

Then flung the emptied rinds away

But gathered up one kernel stone,

And knew not was it night or day

As she turned home alone.

If the interpretation that this is about desire is correct, as a Victorian woman, she would know the consequences of giving into desire before marriage hence the heart-breaking description of her sense of longing for this pleasure but being denied it as a fallen woman (crucially she can't hear the goblin men, she is no longer a 'maid')

She said not one word in her heart's sore ache;

gnashed her teeth for baulked desire, and wept / As if her heart would break

She thought of Jeanie in her grave,

Who should have been a bride;

But who for joys brides hope to have

Fell sick and died

Might the point of the story of Jeanie be that the loss of sexual innocence is a kind of death?

Richard Dadd Dadd Richard The Fairy Fellers Master Stroke1