Buildings are the most frequently changing and valuable features in geographic databases, and building vectors are the basis for their spatial analysis and editing. They play important roles in various industries and sectors such as basic surveying and mapping, urban planning, emergency management, etc. Traditional building extraction methods based on images using manually designed features or convolutional neural networks have difficulties in ensuring boundary accuracy and generalization, and the raster format cannot be directly applied. The current building vector extraction still relies on manual work, which is costly, inefficient, and cannot meet the automation needs of surveying and mapping production. Therefore, achieving efficient and regular building vector extraction is a key problem that needs to be solved urgently in the field of intelligent surveying and mapping. Low-altitude unmanned aerial vehicles (UAVs) have advantages such as flexibility, speed, efficiency, and low cost, and their produced DOMs have rapidly developed into a new data carrier and acquisition tool in the surveying and mapping field, which is an important data basis for building extraction. Segment Anything Model is a representative of the large promptable segmentation models, which can achieve zero-shot generalization while ensuring accuracy, and has a broad application prospect in industries such as medical, agriculture, remote sensing, etc., providing new technical support for building semantic segmentation. This paper firstly designs a building vector extraction process for oblique aerial DOMs based on Segment Anything Model, and extracts the building vector contours. Then, aiming at the problems of redundant nodes, over-smoothed contour points and non-linear edge lines of the initial building vectors, a building contour fitting algorithm based on length and angle double-weighted least squares linear fitting is proposed to recover the building contours. It is based on the basic principle that longer contour lines always have higher confidence. Taking the long edge as the reference edge, the node length and angle confidence are calculated by associating the nodes with the contour lines. Finally, using the Douglas-Peuker algorithm to roughly determine the corner positions, and then using the double-weighted least squares linear fitting to recover the precise building corners, the identification and regular contour fitting of building vectors are realized. In rural building areas with complex structure, close distribution and complex situation, this method effectively recovers high-precision building corners, avoids sharp noise to some extent, and achieves the best results compared with other contour fitting algorithms in terms of intersection over union, segment quantity difference, Hausdorff distance and other indicators, and extracts high-precision and regularized building vectors. The contour fitting algorithm designed in this paper is not constrained by the building shape, owns good generalization ability, and has high application promotion value.