本教學的最終目標是完成以下架構。
- 在 Ubuntu 24 上安裝 FireCrowl（API 伺服器 & 工作程式）
- 將 Node.js 安裝於系統全域，並全域使用
pnpm指令 - 導入 Rust 工具鏈（rustup 等），編譯 FireCrawl 以 Rust 撰寫的 HTML 轉換函式庫（html-transformer）
- 安裝 Playwright 的相依套件，並建立自訂的 Playwright 微服務指令碼
- 建立兩個用於啟動 FireCrowl 的 systemd 服務（伺服器用 & 工作程式用），
- OS 啟動時自動啟動
- 可透過手動執行
sudo systemctl restart firecrowl-server/sudo systemctl restart firecrowl-workers等指令控制
(Optional) +-----------------+ | Dify | +-----------------+ | | HTTP / REST API v +------------------------------------------------------------------+ | FireCrawl (Node.js, TypeScript, pnpm) | | Directory: /home/firecrawl/apps/api | | | | +----------------------+ (pnpm run start) | | | API Server |-----------------------------------------+ | | - Express | | | | - BullMQ Dashboard | | | +----------------------+ | | ^ | | | (Queue tasks) | | +----------------------+ (pnpm run workers) | | | Workers |-----------------------------------------+ | | - Scraping, parsing, | | | | indexing | | | +----------------------+ | | | | - .env 中設定的各項選項（API 金鑰、PORT、HOST、 | | PLAYWRIGHT_MICROSERVICE_URL等） | | | | - Rust HTML Transformer: | | -> Built via Cargo in: | | /home/firecrawl/apps/api/sharedLibs/html-transformer | | -> Generates: libhtml_transformer.so | | -> Used by FireCrawl for fast HTML parsing (or falls back to | | Cheerio) | +------------------------------------------------------------------+ | | Task Queue / Rate Limit v +----------------------------------+ | Redis (localhost) | | - Used by BullMQ for job queuing| +----------------------------------+ +---------------------------------------------+ | systemd (Ubuntu 24) | | | | +--------------------------+ | | | firecrowl-server | | | | - ExecStart=pnpm run start| | | +--------------------------+ | | +--------------------------+ | | | firecrowl-workers | | | | - ExecStart=pnpm run | | | | workers | | | +--------------------------+ | | +--------------------------+ | | | firecrowl-playwright | | | | - ExecStart=pnpm run | | | | playwright-service | | | +--------------------------+ | | (Automatic start at boot, process monitoring,| | logs managed via systemd) | +---------------------------------------------+前提條件：
- 在 Ubuntu 24（代號 “noble”）上不使用 Docker 執行
- OS 使用者可以管理員權限操作
- 預計將 FireCrowl 儲存庫複製（clone）至
/home/firecrawl目錄 - Redis 已安裝（
sudo apt install -y redis-server）並正在執行
STEP1. 系統準備與安裝 Node.js / pnpm #
STEP1.1 系統更新與安裝開發工具 #
sudo apt update sudo apt install -y build-essential pkg-config curl git libssl-devSTEP1.2 從 NodeSource 安裝 Node.js #
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - sudo apt-get install -y nodejsSTEP1.3 全域安裝 pnpm #
sudo npm install -g pnpm※請以
which pnpm確認執行路徑（例如：/usr/local/bin/pnpm）。
STEP2. 複製 FireCrawl 儲存庫並安裝相依套件 #
STEP2.1 複製 FireCrawl #
cd /home/firecrawl git clone https://github.com/mendableai/firecrawl.gitSTEP2.2 安裝相依套件 #
cd /home/firecrawl/apps/api pnpm installSTEP2.3 設定 .env 檔案 #
建立或編輯
/home/firecrawl/apps/api/.env，設定所需的環境變數。範例：# ===== Required ENVS ====== NUM_WORKERS_PER_QUEUE=8 PORT=3002 HOST=0.0.0.0 REDIS_URL=redis://localhost:6379 REDIS_RATE_LIMIT_URL=redis://localhost:6379 PLAYWRIGHT_MICROSERVICE_URL=http://localhost:3000/html USE_DB_AUTHENTICATION=false # ===== Optional ENVS ====== TEST_API_KEY=fc-bestnet BULL_AUTH_KEY=fc-bestnet ...（其他選項視需求添加）
STEP3. 導入 Rust 工具鏈並編譯 HTML Transformer #
STEP3.1 安裝 Rust 工具鏈 #
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env rustc --version cargo --versionSTEP3.2 確認 FireCrawl 內的 Rust 函式庫目錄 #
在儲存庫根目錄執行以下指令，尋找
Cargo.toml所在的目錄。cd /home/firecrawl find . -type f -name Cargo.toml | grep -i html-transformer例如：找到
./sharedLibs/html-transformer/Cargo.toml後，切換至該目錄。STEP3.3 編譯 Rust 函式庫 #
cd /home/firecrawl/sharedLibs/html-transformer cargo build --release編譯成功後會產生
target/release/libhtml_transformer.so。確認方式：ls target/release/libhtml_transformer.soSTEP3.4 配置函式庫或設定環境變數 #
- 方法 A：若 FireCrawl 的 Node.js 程式碼以相對路徑讀取，則不需額外作業
- 方法 B：視需要將其加入
LD_LIBRARY_PATH中
export LD_LIBRARY_PATH=/home/firecrawl/sharedLibs/html-transformer/target/release:$LD_LIBRARY_PATH若要在 systemd 中自動設定，請在服務檔案中加入
Environment=LD_LIBRARY_PATH=...。
STEP4. 設定 Playwright 並建立微服務指令碼 #
STEP4.1 安裝 Playwright 相依套件 #
Ubuntu 24.04 適用的相依函式庫範例：
sudo apt-get install -y libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 \ libdrm2 libxkbcommon0 libgtk-3-0 libpango-1.0-0 libcairo2 libgdk-pixbuf2.0-0 \ libgbm1 libatspi2.0-0 libx11-xcb1 libxcomposite1 libxdamage1 libxfixes3 \ libxrandr2 libxrender1 libxtst6 libxcb1 libxi6 libxcursor1 ca-certificates \ fonts-liberation xdg-utilsSTEP4.2 安裝 Playwright 瀏覽器 #
pnpm exec playwright installSTEP4.3 建立 Playwright 微服務指令碼 #
在
/home/firecrawl/apps/api建立playwright-service.js，貼上以下範例程式碼。// playwright-service.js const http = require('http'); const { chromium } = require('playwright'); const PORT = 3000; const server = http.createServer(async (req, res) => { if (req.method === 'GET' && req.url.startsWith('/html')) { try { const urlParam = new URL(req.url, `http://localhost:${PORT}`).searchParams.get('url'); if (!urlParam) { res.writeHead(400, { 'Content-Type': 'text/plain' }); return res.end('Missing ?url parameter'); } const browser = await chromium.launch({ headless: true }); const page = await browser.newPage(); await page.goto(urlParam, { waitUntil: 'networkidle' }); const content = await page.content(); res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' }); res.end(content); await browser.close(); } catch (err) { console.error('Playwright microservice error:', err); res.writeHead(500, { 'Content-Type': 'text/plain' }); res.end('Playwright error occurred'); } } else { res.writeHead(404, { 'Content-Type': 'text/plain' }); res.end('Not found'); } }); server.listen(PORT, '0.0.0.0', () => { console.log(`Playwright microservice listening on http://0.0.0.0:${PORT}/html`); });STEP4.4 在 package.json 中新增指令碼 #
請在
/home/firecrawl/apps/api/package.json的"scripts"中新增下列內容（與現有項目合併）。"playwright-service": "node playwright-service.js"
STEP5. 設定 Systemd 服務 #
為了讓 FireCrawl 伺服器、工作程式、Playwright 微服務能自動啟動，需要建立 systemd 單元檔案。
STEP5.1 FireCrawl 伺服器服務 (/etc/systemd/system/firecrowl-server.service) #
[Unit] Description=FireCrowl Server After=network.target [Service] User=firecrawl Group=firecrawl WorkingDirectory=/home/firecrawl/apps/api Environment=PATH=/usr/local/bin:/usr/bin:/bin ExecStart=/bin/bash -c 'pnpm run start' Restart=always RestartSec=5 Type=simple [Install] WantedBy=multi-user.targetSTEP5.2 FireCrawl 工作程式服務 (/etc/systemd/system/firecrowl-workers.service) #
[Unit] Description=FireCrowl Workers After=network.target [Service] User=firecrawl Group=firecrawl WorkingDirectory=/home/firecrawl/apps/api Environment=PATH=/usr/local/bin:/usr/bin:/bin ExecStart=/bin/bash -c 'pnpm run workers' Restart=always RestartSec=5 Type=simple [Install] WantedBy=multi-user.targetSTEP5.3 Playwright 微服務服務 (/etc/systemd/system/firecrowl-playwright.service) #
[Unit] Description=FireCrowl Playwright Microservice After=network.target [Service] User=firecrawl Group=firecrawl WorkingDirectory=/home/firecrawl/apps/api Environment=PATH=/home/firecrawl/.nvm/versions/node/v20.18.3/bin:/usr/local/bin:/usr/bin:/bin ExecStart=/bin/bash -c 'pnpm run playwright-service' Restart=always RestartSec=5 Type=simple [Install] WantedBy=multi-user.target注意：
- 若使用
User=firecrawl，需事先建立firecrawl使用者，並以sudo chown -R firecrawl:firecrawl /home/firecrawl等指令設定擁有權。 Environment=PATH=...中設定的是透過 nvm 安裝的 Node.js 路徑，請依實際環境修改。- 若未以相對路徑讀取 Rust 函式庫，則也需要設定
LD_LIBRARY_PATH。
STEP5.4 啟用並啟動 Systemd 服務 #
sudo systemctl daemon-reload sudo systemctl enable firecrowl-server sudo systemctl enable firecrowl-workers sudo systemctl enable firecrowl-playwright sudo systemctl start firecrowl-server sudo systemctl start firecrowl-workers sudo systemctl start firecrowl-playwright請以
sudo systemctl status <service>確認各服務是否為active (running)狀態。
STEP6. 重新啟動 FireCrawl 並驗證 #
- 確認 .env 設定：
請特別確認PLAYWRIGHT_MICROSERVICE_URL是否為http://localhost:3000/html設定 - FireCrawl 的伺服器與工作程式已啟動：
若日誌中出現「Scrape via fetch…」等訊息，表示 HTTP 請求運作正常 - 確認 Playwright 服務運作狀況：
在瀏覽器中存取http://<伺服器IP>:3000/html?url=https://example.com，若能取得回傳的 HTML 即為正常
STEP7. 最終確認 #
- Rust 函式庫：
確認target/release/libhtml_transformer.so存在，且日誌中沒有錯誤訊息 - Playwright 微服務：
測試是否正確啟動並執行 JavaScript 轉譯 - 整體連動：
最終確認 FireCrawl 伺服器、工作程式、Playwright 是否協同運作，並能正常回應來自 Dify 等的請求
