Shale oil/gas reserves are a major focus for current and future oil/gas exploration. The nanopores in shale vary in size and shape, greatly affecting the phase behavior and multiphase flow of reservoir fluids. Understanding the fluid phase behavior in nanopores is crucial for enhancing the shale oil/gas recovery efficiency. Numerous studies have explored the impact of pore sizes on phase behavior, while relatively few studies have been conducted on the influence of pore shapes on phase behavior. This paper develops a novel two-phase flash algorithm for confined fluids considering the effect of pore size and shape at specified volume and temperature. The proposed algorithm performs well under varying overall molar densities and temperatures. The results confirm that confined fluids in nanopores exhibit distinct phase behaviors compared to bulk fluids. Specifically, at a specified molar density, the dew and cricondentherm point temperatures rise in nanopores, the bubble point temperature decreases. Liquid-phase equilibrium pressure, vapor-phase saturation, and heavy component concentrations in both phases are lower in nanopores, while liquid-phase saturation and light component concentrations in both phases are higher. As pore size decreases and polygon sides of the pore cross-section increase, the phase behavior diverges further from bulk fluids. Polygons with more than 10 sides behave similarly to circular pores. The effects of nanopore cross-sectional shape on the phase behavior of fluids are different near the dew point and bubble point. This effect of the dew point temperature peaks at around 2 nm, while its influence gradually diminishes for the bubble point. As pore radius increases, the shape effect on capillary pressure, equilibrium pressure, and vapor composition peaks at 2 nm, but diminishes for saturation of vapor and liquid phases and liquid composition.