It’s late, maybe 1:30 a.m. I’m sitting on the floor of our two-bedroom suite at the Patterson Inn, peering out an east-facing window. Even at this hour, the streets below are alive with the after-midnight crowd for which Denver’s Capitol Hill neighborhood is known. Every now and again, I glance over at the king-size bed where my husband lies asleep, sure the revelers will have disturbed him. But he does not stir, leaving me alone to consider the remarkable place in which I find myself tonight.

The Patterson Inn, located at the corner of 11th Avenue and Pennsylvania Street, opened as a nine-suite bed-and-breakfast in October 2012, but the mansion is not what you might call new-build. In fact, the nearly 14,000-square-foot red sandstone home with French Chateau and Richardsonian Romanesque features was built in 1891 by Thomas B. Croke, a Denver-area businessman and later a state senator.

Croke bought the land—three lots on what was then known as Block 77 of Brown’s Bluff—and commissioned Isaac Hodgson Jr. to design the main structure and a carriage house. In those days—a few decades after the Pikes Peak gold rush, just 15 years after Colorado became a state, and 17 years before the Capitol building was completed—the elevated east side of Denver was quickly becoming the place to live for the city’s elite. Croke and his family moved into the lavish new home, but, according to historical documents, they didn’t stay long. Although the reason Croke left his recently completed home may never be known, a flailing economy and the silver market crash of 1893 could have been factors that led him to make a real-estate deal with Thomas Patterson, the subsequent and best-known owner of the home.

Although the nighttime air is cool, I open the window and try to imagine what the area might have been like 120 years ago. At that time, Denver bustled with a population of roughly 100,000 inhabitants who worked in wholesale trade, manufacturing, agriculture and ranching, and mining, or for the railroads. Ladies decked out in high-necked, puffy-shouldered blouses and bell-shaped skirts and men in narrow trousers and tailored jackets strolled along these neighborhood streets. Horse-drawn carriages plied the roads. Thomas Patterson—who was a majority owner of the Rocky Mountain News—his wife, Kate, and their two daughters would’ve been seen coming and going from what became known as the Croke-Patterson Mansion, a structure listed on the National Register of Historic Places.

The green glow from the digital clock flashes 2:10 a.m., and I decide it’s finally time to close the window, slide under the down comforter, and shut my eyes. As I tiptoe toward the bed, though, a chill catches my bare legs and a feeling of unease stops me where I stand. I look around the Cheshire suite—a whimsical accommodation with an Alice in Wonderland theme, bright colors, quirky furnishings, and a beautifully renovated bathroom—to see and hear nothing that could’ve caused my angst. I shrug it off, but as I pull up the covers I can’t help but ask myself a question: Could the rumor that the Croke-Patterson Mansion is haunted be true?

Although the light of day helps, there’s little argument that the Croke-Patterson Mansion—now the Patterson Inn—looks haunted from the outside. There’s the vaulted, multilevel dark gray roof. The turrets and towers. The small windows in odd locations. It’s got that Beauty and the Beast, creepy-old-castle vibe. But Brian Higgins—the Denver architect who fell in love with the old house, bought it with his family, and fully renovated it—isn’t keen on talking about the eerie folklore that surrounds his property. Instead, he wants people to think of it as an upscale and charming inn nestled into one of Denver’s hippest urban communities.

And, of course, the Patterson Inn is both of those things. Rooms start at $169 per night, a rate that includes a wine and hors d’oeuvres hour as well as a breakfast of quiches, meats, cheeses, pastries, fruit, juices, coffee, and teas in the dining room. The refinished hardwoods, the original stained glass windows, the detailed plasterwork, the 1890s-era weight-and-pulley windows, the restored chandeliers—the house is simultaneously old and new, and decidedly enchanting. Each bedroom’s decor is loosely based on a movie—Prague takes its cue from Harry Potter, the Library from Clue—and has a smattering of one-of-a-kind pieces of furniture and complementary artwork. The beds are unquestionably comfortable. The bathrooms have amazing showers. The flat-screens are what you wish you had in your own home. And the location is within walking distance of some of the city’s trendiest restaurants, coolest bars, and most popular tourist attractions.

Still, it’s hard to ignore the Patterson’s past and the mysterious anecdotes—some real, some unconfirmed—that the mansion can’t seem to shake. Tales abound of a frail female spirit who roams the home’s grand staircase; of eerie cries of an infant from a bygone era; of dogs living in the house that jumped to their deaths from the third-level tower room window; of the cold spot just in front of the large mirror in the main foyer; and of the suicide of a doctor’s wife from cyanide gas poisoning. Higgins knows the stories. In fact, he was so acutely aware of the house’s history and associated legends that he decided to film the demolition and renovation process, which began in October 2011 and lasted through September 2012.

When Higgins bought the mansion for $565,000—a bargain for the area—he knew he would need to spend at least that much to rehabilitate the dilapidated home. The red sandstone on the exterior was in need of repair. The plumbing and most of the electrical would need to be replaced. And he’d have to install an elevator. Although there were hiccups along the way—like knocking down a ceiling only to find another ceiling above it, and an unexplained fire on the second floor just days before the inn was supposed to open—Higgins says the overhaul didn’t give him more than he could handle as an architect and developer. In other ways, though, the house did divulge more than he could’ve imagined.

“We pulled items out of the walls that really took us back to the early days of the house,” Higgins says. “Things like old children’s clothing, a newspaper from 1891, eyeglasses from the turn of the century, clips from a garter belt, and a large candle from the ’40s.” Higgins even found small, old glass vials about the size of AA batteries that had been thrown into a fireplace. There is, of course, no way to determine to whom the items belonged; however, Higgins knows these details—as well as his forthcoming documentary film—will only add to the rich lore of the Croke-Patterson Mansion.

Hours before my late-night windowside musings, my husband and I stroll the blocks surrounding the Patterson Inn. Although we live less than five miles from Capitol Hill, we’ve never wandered on foot through the area’s leafy streets. Tucked amongst apartment buildings and high-rises are remnants of the neighborhood the Pattersons once knew. Every few blocks a historic home—the Capitol Hill, Boettcher, and Grant-Humphreys mansions—stands resolute in the face of time and reminds us why this part of Denver, and more specifically a long section of nearby Grant Street, was once called Millionaire’s Row.

After our twilight amble, we head to the intersection of Grant Street and Seventh Avenue, where we duck into the seven-month-old Vesper Lounge for a cocktail—the on-tap old fashioned seems appropriate, although the Silver Gin Fizz is just as tasty. We chat with the barkeep until it’s time to walk next door to Mizuna, one of Denver’s finest fine-dining establishments, for our dinner reservation. Over seared veal tenderloin, Skuna Bay salmon, and a bottle of Syrah—we know we’re walking “home,” after all—we talk about our plans for the next morning.

From the inn, we have our choice of easy-to-walk-to attractions—a handful of places that, even though we live in Denver, we sometimes forget to make time to enjoy: the Molly Brown House Museum, the Denver Art Museum, the Clyfford Still Museum, Cheesman Park, and the History Colorado Center. As we linger over a dessert of warm chocolate cake, we decide to wait until the morning to settle on a destination. After all, in the backs of our minds we know we could choose a different option altogether: sleeping in late, relishing in a breakfast we didn’t cook, and spending some time in a home that is an extraordinary experience in its own right.