In modern greenhouses, complicated tasks and unstructured environments generate the imperious demand for advanced semantic information about each object at work scenes. A significant problem that mainstream methods intend to resolve is that the refinement and understanding of environmental information cannot efficiently cover the entire task in real time. Therefore, this paper proposes a panoptic semantic mapping method to identify each object that is supposed to be concerned in greenhouses. This method builds grid maps with advanced semantic information based on RGB and depth images. For the agricultural task with tomato as the working object, the categories of various objects in the grid map are divided into four groups: fruits, pedicels, stems and obstacles. This method consists of three steps: semantic segmentation from RGB images with K-Net, reconstruction of point cloud data based on depth images and semantic masks and transformation of the point cloud data into OctoMap. Experimental results show the semantic segmentation algorithm reaches a mean precision of semantic segmentation of 93.83%, a mean IoU of 88.39% and an average accuracy of 98.28%. Meanwhile, the refresh frequency of publishing point cloud data with advanced semantic information holds steady at 2 Hz with the resolution of 8 mm.