我是 webscraping/scrapy 和 python 的新手

Scrapy 版本：Scrapy 2.5.1 作业系统：windows IDE：pycharm

我正在尝试使用scrapy中的FEEDS选项自动从网站汇出报废资料以下载到excel

尝试了以下解决方案，但没有奏效

根据档案中提供的示例在我的蜘蛛类中评论 custom_settings 后，我还尝试在我的settings.py档案中添加相同的内容：https ://docs.scrapy.org/en/latest/topics/feed-exports.html?highlight =饲料#饲料

现在我使用spider_closed（信号）通过将所有抓取的项目资料存盘在一个名为result的阵列中来将资料写入CSV来实作我的要求

class SpiderFC(scrapy.Spider):
name = "FC"

start_urls = [
     url,
]
custom_setting = {"FEEDS": {r"C:\Users\rreddy\PycharmProjects\fcdc\webscrp\outputfinal.csv": {"format": "csv", "overwrite": True}}}

@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
    spider = super(SpiderFC, cls).from_crawler(crawler, *args, **kwargs)
    crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
    return spider

def __init__(self, name=None):
    super().__init__(name)
    self.count = None

def parse(self, response, **kwargs):
    # each item scrapped from parent page has links where the actual data need to be scrapped so i follow each link and scrape data
      yield response.follow(notice_href_follow, callback=self.parse_item,
                                      meta={'item': item, 'index': index, 'next_page': next_page})
       
def parse_item(self, response):
    # logic for items to scrape goes here
    # they are saved to temp list and appended to result array and then temp list is cleared
        result.append(it) # result data is used at the end to write to csv
        item.clear()
        if next_page:
            yield next(self.follow_next(response, next_page))
        
            
def follow_next(self, response, next_page):
    next_page_url = urljoin(url, next_page[0])
    yield response.follow(next_page_url, callback=self.parse)

蜘蛛关闭信号

def spider_closed(self, spider):
    with open(output_path, mode="a", newline='') as f:
        writer = csv.writer(f)
        for v in result:
            writer.writerow([v["city"]])

当所有资料都被抓取并且所有请求都完成后，spider_closed信号会将资料写入 csv 但我试图避免这种逻辑或代码并使用来自 scrapy 的内置汇出器，但我在汇出资料时遇到了麻烦

uj5u.com热心网友回复：

检查你的路径。如果您在 Windows 上，则在custom_settings例如下面提供完整路径

custom_settings = {
    "FEEDS":{r"C:\Users\Name\Path\To\outputfinal.csv" : {"format" : "csv", "overwrite":True}}
}

如果您使用的是 linux 或 MAC，则提供如下路径：

custom_settings = {
    "FEEDS":{r"/Path/to/folder/fcdc/webscrp/outputfinal.csv" : {"format" : "csv", "overwrite":True}}
}

或者提供如下的相对路径，这将fcdc>>webscrp>>outputfinal.csv在运行蜘蛛的目录中创建一个档案夹结构。

custom_settings = {
    "FEEDS":{r"./fcdc/webscrp/outputfinal.csv" : {"format" : "csv", "overwrite":True}}
}