Recently, the importance of generating accurate question and answer summaries has increased because the number of documents has increased, but it is difficult to summarize them from unstructured documents where the answer is not associated with the question. To address this problem, first we make a group of answer sentences, which corresponds to one question based on the heuristic rules, and we find the corresponding answer from the summaries and subtopics of the question by using the BM25-based similarity calculation. Second, we use the large language model (LLM) to generate a summary of the answers from the answers found. Experimental results showed that our methods significantly outperformed LLM-based answer generation that inputs whole answer sentences including irrelevant parts, which correspond to another question, to LLM and that our methods were practical compared to human-generated gold summaries.