> ## Documentation Index
> Fetch the complete documentation index at: https://larkup.de/documentation/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawl at Scale

> Crawl websites, upload files, pull integrations, and track every ingestion job.

Larkup gives you one **Data** page for every source. Add content once, and Larkup indexes it in the background.

<Frame caption="The Data page, with a tab for every source">
  <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=0d40442a5860437f92fee81984e024bd" alt="Data page with Files, Text, Website, Media, and Integrations tabs" width="3012" height="1442" data-path="documentation/images/add/website-tab.png" />
</Frame>

<Tabs>
  <Tab title="Website">
    Search by keyword or paste a direct URL. The built in local crawler works immediately; switch to the Cloud web crawler in Settings for JavaScript heavy or protected sites.

    <Frame caption="Starting a crawl by keyword">
      <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=fea3373c6a2980a5197463115915b757" alt="Website search results ready to add" width="3012" height="1442" data-path="documentation/images/crawl/crawl-start.png" />
    </Frame>

    Jobs run in the background. Track pages processed and status right in the tab.

    <Frame caption="A running crawl job">
      <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=970c7086938333dc7558ed0a24136142" alt="Running crawl job with page count and status" width="3012" height="1442" data-path="documentation/images/crawl/crawl-jobs-view.png" />
    </Frame>
  </Tab>

  <Tab title="Files">
    Drop in PDF, Word, CSV, JSON, Excel, or plain text. Structured files get a column and schema preview before you confirm.

    <Frame caption="Files staged with a schema preview">
      <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=83e1a4c7d29d00c44d76ea7137c3acb7" alt="Files tab with a PDF and an Excel schema preview" width="3012" height="1442" data-path="documentation/images/add/files-tab2.png" />
    </Frame>
  </Tab>

  <Tab title="Text">
    Paste notes, transcripts, or any text you want to search, with a title.

    <Frame caption="Adding raw text directly">
      <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=a248b348f08b4d827a9d48877a36a635" alt="Add Text tab with a title and content field" width="3012" height="1442" data-path="documentation/images/add/text-tab.png" />
    </Frame>
  </Tab>

  <Tab title="Media">
    Upload images, video, or audio, or paste a YouTube URL. Install the Video Intelligence marketplace tool for transcription and frame analysis. See [Chat with Video](/documentation/documentation/guide/video-chat).

    <Frame caption="Importing a video from a URL">
      <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=870aee2a1ab5cf00d4386e6f35662ccd" alt="Add Media tab with a YouTube URL import" width="3012" height="1442" data-path="documentation/images/add/media-tab.png" />
    </Frame>
  </Tab>

  <Tab title="Integrations">
    Connect Notion, Google Drive, Slack, Jira, and more. Choose **Configure** or **Connect**, authorize through OAuth or an API key, and Larkup pulls the content in and indexes it.

    <Frame caption="Available integrations">
      <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=6294d368e5e1e6c33371ae4ae04d1d15" alt="Integrations tab with Notion, GitHub, Jira, Slack, and more" width="3012" height="1442" data-path="documentation/images/add/integrations-tab.png" />
    </Frame>

    | Service      | Category           | What gets indexed                     |
    | ------------ | ------------------ | ------------------------------------- |
    | Notion       | Documentation      | Pages, databases, and knowledge bases |
    | Google Drive | Documentation      | Files and shared drives               |
    | Slack        | Communication      | Channels, messages, and DMs           |
    | Confluence   | Documentation      | Pages, spaces, and knowledge bases    |
    | Jira         | Project management | Issues, sprints, and projects         |
    | Linear       | Project management | Issues, projects, and roadmaps        |
    | GitHub       | Engineering        | Repositories and README knowledge     |
    | Salesforce   | CRM                | Accounts, contacts, and opportunities |

    Don't see what you need? Select **Ask for integration**. Sync again anytime to pull fresh content.
  </Tab>
</Tabs>

## Your corpus

Every loaded document lands in **Data → All Data**, with its source, status, and character count.

<Frame caption="All Data, every indexed document in one table">
  <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=d485f3a248f354e3c171dfa412132db7" alt="All Data table with indexed documents" width="1811" height="868" data-path="documentation/images/add/data-table-view.png" />
</Frame>

## Scraping settings

Configure the web scraper and web search provider under **Settings → Search & Scraping**. No `.env` editing required.

<Frame caption="Search & Scraping settings">
  <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=b73545a55f5826c4423c732d14270f08" alt="Web search and web scraper provider settings" width="3012" height="1442" data-path="documentation/images/crawl/search-scraping-settings.png" />
</Frame>

Prefer hosted crawling? Add a Firecrawl or Serper.dev API key here instead of running the local crawler. Organizations that route traffic through a proxy can set it here too.

<Frame caption="Proxy settings for secure scraping">
  <img src="https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?fit=max&auto=format&n=13l_fOT1PJRw-f4o&q=85&s=1d2ea5f899fe70209e782bfdee23d6ee" alt="Proxy server, username, and password fields" width="3012" height="1442" data-path="documentation/images/crawl/proxy-settings.png" />
</Frame>