Personality detection is an emerging task benefiting numerous fields. The existing studies on text-based personality detection rarely concern silent users who never publish social texts due to the lack of their posts. Simultaneously, the mainstream methods lack an effective pathway for silent user representation to detect the personality. To solve the silent user problems, we propose a psycholinguistic knowledge-guided graph network, PKGN. Our method is composed of neighbor post metric, graph initialization & learning, and classification. Under the guidance of psychological knowledge, our model first selects high-quality posts from neighbors as the posts of silent users through the neighbor post metric. In the graph initialization & learning, psychologically relevant categories are introduced to build the bipartite graph for each silent user and obtain the user representation via GATv2. Then, we utilize linear classifiers for personality classification. We conducted extensive experiments on a new real-world dataset, including 1581 samples. To conduct a baseline benchmark for the silent user personality detection task, we apply the neighbor post metric to combine with the existing work. From the experimental results, our model achieves 64.11% average accuracy and 63.21% average macro-F1, outperforming mainstream methods in most individual personality traits and comprehensive comparisons. Furthermore, the introduction of psycholinguistic knowledge benefits the model performance. In neighbor post metric comparison, the psycholinguistic knowledge from LIWC reduces the standard variances of psychological category count and improves the detection results (3.01% for average accuracy and 2.34% for average macro-F1). In the ablation study, all the psychologically relevant categories contribute to the model performance (ranging from 0.02% to 3.02% for average macro-F1).