如何根据从 JSON 请求中获取的数据在模型中创建行，这些请求来自使用 "best practice" 进行的第三方爬网

Question

描述问题：

我想从 dataforseo 爬取数据，并通过 model.create() 方法将它们直接保存在模型数据库中，具有多模型与模型的多关系所以例如在模型 A 中有

与模型的多对多关系 B
与模型的多对多关系C
与模型的多对多关系 D

and model B 与model C
有关系

所以我的问题是如何通过模型A创建方法

顺利保存对上述所有模型的JSON响应

响应来自请求：

[
   {
      "title":"title",
      "url":"url",
      "description":"description",
      "pre_snippet":"pre_snippet",
      "extended_snippet":"extended_snippet",
      "images":"images",
      "amp_version":"amp_version",
      "rating":{
         "rating_type":"rating_type",
         "value":"value",
         "votes_count":"votes_count",
         "rating_max":"rating_max"
      },
      "price":{
         "current":"current",
         "regular":"regular",
         "max_value":"max_value"
      },
      "highlighted":{
         "name_highlighted":"name_highlighted"
      }
   },
   {
      "title":"title",
      "url":"url",
      "description":"description",
      "pre_snippet":"pre_snippet",
      "extended_snippet":"extended_snippet",
      "images":"images",
      "amp_version":"amp_version",
      "rating":{
         "rating_type":"rating_type",
         "value":"value",
         "votes_count":"votes_count",
         "rating_max":"rating_max"
      },
      "price":{
         "current":"current",
         "regular":"regular",
         "max_value":"max_value"
      },
      "highlighted":{
         "name_highlighted":"name_highlighted"
      }
   }
]

代码：

view.py file

@api_view(['POST'])
@parser_classes((JSONParser,))
def crawl_data(request):
    """
    A view that can accept POST requests with JSON content.
    """
    Product.create(
         title=request.data[title]
         url=request.data[url]
         describtion=request.data[describtion]
         ...
         )
    return Response({'received data': request.data})

models.py



class Highlighted(models.Model):
    name_highlighted = models.CharField(max_length=100)

    def __str__(self):
        return str(self.name_highlighted)


class Rating(models.Model):
    rating_type = models.CharField(max_length=500, null=True, blank=True)  # make unique
    value = models.CharField(max_length=500, null=True, blank=True)
    votes_count = models.CharField(max_length=500, null=True, blank=True)
    rating_max = models.CharField(max_length=500, null=True, blank=True)

    def __str__(self):
        return str(self.value)


class Price(models.Model):
    current = models.CharField(max_length=500, null=True, blank=True, default="none")
    regular = models.CharField(max_length=500, null=True, blank=True)
    max_value = models.CharField(max_length=500, null=True, blank=True)

    def __str__(self):
        return str(self.current)


class Product(models.Model):
    title = models.CharField(max_length=500, null=True, blank=True)
    url = models.CharField(max_length=500, null=True, blank=True)
    description = models.CharField(max_length=500, null=True, blank=True)
    pre_snippet = models.CharField(max_length=500, null=True, blank=True)
    extended_snippet = models.CharField(max_length=500, null=True, blank=True)
    images = models.CharField(max_length=500, null=True, blank=True)
    amp_version = models.CharField(max_length=500, null=True, blank=True)
    rating = models.ManyToManyField(Rating, null=True, blank=True)
    price = models.ManyToManyField(Price, null=True, blank=True)
    highlighted = models.ManyToManyField(Highlighted)

    def __str__(self):
        return str(self.url)

任何帮助感激不尽

Answer 1

您可以通过在 models.py 文件中执行以下操作来覆盖创建方法：

试试这个：

在产品模型中，您可以像这样覆盖创建模型


class Product(models.Model):
    title = models.CharField(max_length=500, null=True, blank=True)
    url = models.CharField(max_length=500, null=True, blank=True)
    description = models.CharField(max_length=500, null=True, blank=True)
    pre_snippet = models.CharField(max_length=500, null=True, blank=True)
    extended_snippet = models.CharField(max_length=500, null=True, blank=True)
    images = models.CharField(max_length=500, null=True, blank=True)
    amp_version = models.CharField(max_length=500, null=True, blank=True)
    rating = models.ManyToManyField(Rating, null=True, blank=True)
    price = models.ManyToManyField(Price, null=True, blank=True)
    highlighted = models.ManyToManyField(Highlighted)
    extended_people_also_search = models.ManyToManyField(ExtendedPeopleAlsoSearch)

    def __str__(self):
        return str(self.url)

    @classmethod
    def create(cls, **kwargs):
        product = cls.objects.create(
            title=kwargs['title'],
            url=kwargs['url'],
            description=kwargs['description'],
            pre_snippet=kwargs['pre_snippet'],
            extended_snippet=kwargs['extended_snippet'],
            images=kwargs['images'],
            amp_version=kwargs['amp_version'],

        )

        # add price dict

        price, created = Price.objects.get_or_create(current=kwargs['price']["current"],
                                                     regular=kwargs['price']["regular"],
                                                     max_value=kwargs['price']["max_value"],
                                                     )
        product.price.add(price)

        # add rating dict
        rating, created = Rating.objects.get_or_create(rating_type=kwargs['rating']["rating_type"],
                                                       value=kwargs['rating']["value"],
                                                       votes_count=kwargs['rating']["votes_count"],
                                                       rating_max=kwargs['rating']["rating_max"],
                                                       )
        product.rating.add(rating)

        return product

    def __str__(self):
        return str(self.url)

并且在您的 views.py 函数爬网中，您可以遍历 JSON 响应并将它们添加到您的模型中，如下所示：

@api_view(['POST'])
@parser_classes((JSONParser,))
def crawl_data(request):
    """
    A view that can accept POST requests with JSON content.
    """
    for product_data in request.data:
        product = Product.create(**product_data)
    # DO SOMETHING WITH product
    return Response({'received data': request.data})

如何根据从 JSON 请求中获取的数据在模型中创建行，这些请求来自使用 "best practice" 进行的第三方爬网

how to create row in model based on data fetched from JSON request from third-party crawling with "best practice"

python

django

json

django-models

django-rest-framework