Video summarization can present a large video content in a concise and precise form by extraction of the most relevant and salient parts from the video and can also keep the essence of the video intact. Point of interest refers to the particular perspective or viewpoint of a user at which the user focuses on while viewing a video. The paper presents a non-interactive video summarization technique, where the summarizer creates summary of a video based on the point of interest of the user. Given a group of videos, the summarizer first calculates the point of interest of the user and then based on that point of interest generates summaries for the videos in the group. To find out the point of interest of a user, a collection of many different videos grouped in different categories is taken into consideration and the similarity in the intragroup of the videos is considered as the point of interest. Though existing viewpoint-based video summarization techniques address the importance of diverse perspectives, many rely on user preferences or explicit viewpoint annotations. Our proposed framework distinguishes itself by automatically identifying and summarizing multiple viewpoints present within a video without requiring such prior knowledge. The experiments are carried out on a publicly available dataset, and the results are fairly evaluated and compared to prove the efficiency of the technique.