A central problem for short-story writers in the nineteenth century was how to use newly developed eighteenth-century conventions of realism to communicate spiritual meaning formerly explored allegorically in the mythic romance. In the older tale or romance form, characters functioned as psychic projections of basic human fears and desires; in the new fiction, the characters had to be presented as if they were real. Early nineteenth-century writers solved the problem by combining conventions of allegorical romance and realistic fiction. In "Young Goodman Brown," for example, Hawthorne joined allegorical and realistic elements in an ambiguous mix of dream and reality in which his protagonist sometimes seems motivated by the demands of the allegory and sometimes by the demands of his own psyche.
In "The Fall of the House of Usher," Edgar Allan Poe presented an “as if” real character entering the hermetically-sealed world of the artwork dominated by an obsessed and ultimately metaphoric character. And in "Bartleby, the Scrivener," Herman Melville moved closer to realistic conventions by beginning with what appears to be the "real world,” only to have that world invaded by an emblematic character who transforms reality into metaphor. The problem for all three of these early narrative experimentalists was how to bridge the gap between romance conventions, in which characters embody psychic states, and realistic conventions, in which characters possess individual psyches. By rejecting the supernatural and suggesting the strangeness of life as a function of individual psychology, the nineteenth-century short story no longer presented the drama of the clash between the sacred and the profane as existing in realm of the spiritual, but rather in the minds of the individual.
However, it is not until the end of the nineteenth century that Chekhov, the great master of the short story, perfected the form’s ability to present spiritual reality in realistic terms by focusing on the essentially mysterious and hidden nature of the basic human desire to transcend the everyday and live in the realm of spiritual reality. Critic Peter Bitsilli has suggested that the complexity of Chekhov's characters leads us to feel there is something about them we do not understand, a something hidden from us, a something that is part of Chekhov's appeal. Although the theme of the basic desire of the secret self could be illustrated in any number of Chekhov’s short fictions, the paradigmatic statement can be found in one of his most famous stories, “Lady with the Dog.” Near the end of what seems to be merely an anecdotal tale of adultery, the central male character agonizes over the division he senses in himself.
He had two lives: one, open, seen and known by all who cared to know, full of relative truth and of relative falsehood, exactly like the lives of his friends and acquaintances; and another life, running its course in secret. And through some strange, perhaps accidental, conjunction of circumstances, everything that was essential, of interest, and of value to him, everything in which he was sincere and did not deceive himself, everything that made the kernel of his life, was hidden from other people; and all that was false in him, the sheath in which he hid himself to conceal the truth—such, for instance, as his work in the bank, his discussions at the club, his “lower race,” his presence with his wife at anniversary festivities—all that was open.
In Chekhov’s great story, the secrecy of Gurov’s idealized desire constitutes true reality for him, just as the sacred constituted true reality for primitive man and woman. Indeed, in the modern short story, idealized human desire--unsayable, unrealizable, always hovering, like religious experience in the realm of the "not yet"--replaces the sacred revelation embodied in primal short-fiction forms. When Anna leaves, Gurov thinks it has been just another episode or adventure in his life, nothing left but a memory that would visit him only from time to time. But she haunts him, and he imagines her to be lovelier and himself to be finer than they actually were in Yalta. The story ends with the couple agonizing about how to avoid the secrecy and to be free of their intolerable bondage. “How? How?” Gurov asks. But, of course there is no answer, no way that the romantic, spiritual ideal they store up in their ghostly hearts can ever be actualized, except, of course, as it is manifested in the short story—as the immanent, the “not yet.”
James Joyce confronted the quintessential problem of the modern short story in "The Dead": How is it possible for a realistic narrative to convey meaning and significance? It is the same problem that Chekhov had to deal with--how to arrange concrete details in such a way that they develop into a pattern equivalent to theme. Joyce's achievement in this story, its contribution to the development of the short story as a genre, can be best understood if we see its most basic theme as the difference between the kind of reality that realistic prose imitates and the kind of reality that romantic prose reveals.
Thematically, the conflict in "The Dead" that reflects its realistic/lyrical split is the difference revealed to both Gabriel and the reader between public life and private life, between life as it is in everyday experience and life perceived as the objectification of desire. The party portion of "The Dead" is the story of Gabriel's public life, and his only psychic interest is what kind of figure he is going to cut publicly. However, throughout the party period of the story, there are moments--particularly those moments that focus on the past, on music, and on marital union--when reality is not presented as here and now, but as a mixture of memory and desire. Thematically, the basic issue the story poses is: In which one of these realms does true reality reside? Gabriel's discovery at the end of the story is not only that his wife has an inner life inaccessible to him but that his own life has been an outer life only. Filled with desire and the memory of intimacy, wishing Gretta to at one with him, Gabriel is annoyed that she seems so distracted. When he discovers that she has a secret life that has nothing to do with him, he tries to use his typical public devices of irony, but the very simplicity of her story undercuts the effort, and he sees the inadequacy of his public self.
In the much-discussed lyrical ending of "The Dead," Gabriel confronts the irony that the dead Michael is more alive than he is. "Generous tears" fill his eyes because he knows that he has never lived the life of desire, only the untransformed life of the everyday. The ending, in which Gabriel, awake and alone while his wife sleeps beside him, allows himself to lose self and imaginatively merge into a mythic lyrical sense of oneness, makes it possible for the reader to begin the story over again with this end in mind. "The Dead" is not a story that can be understood the way most novels are read--one thing after another--but the way the modern short story must be read--aesthetically patterned in such a way that only the end makes the rest of the story meaningful.
The theme of the inaccessibility of the private life and the inadequacy of public life is most emphatically explored in Sherwood Anderson's Winesburg, Ohio The most frequent remark made about the characters in Winesburg, Ohio is that they are psychic deformities, cut off from society, adrift in their own consciousness. The story that most centrally focuses on Anderson’s effort to explore the secret life of his Winesburg grotesques is “Hands.” Anderson’s suggestion that the secret of Biddlebaum's hands is a job for a poet is part of the basic change in the short story signaled by Chekhov. Anderson struggles with the problem of the prose writer trying to communicate something subtle and delicate, feeling the words are clumsy, for all he has are the events and explanation. What he needs is a way to use language the way the poet does, to transcend language.
The hidden, secret self is a persistent characteristic of the modern short story as a genre, understood by writers who know the form well and have mastered it. This could easily be illustrated in any number of well-known stories by Andre Dubus, Raymond Carver, Alice Munro, and other modern masters of the form. For example, in Munro’s best work, the hidden story of emotion and secret life, communicated by atmosphere and tone, is always about something more enigmatic and unspeakable than the story generated by characters and what happens next.