Frequently Asked Questions

Site Search Service

What is Site Search?

Site Search is an on-demand SaaS Site Search offering from loxal to enable website operators to provide search functionality for their websites. Site Search is run and operated by loxal, whereas you simply integrate the provided search bar into your website to use our service.

What is the advantage of using Site Search vs using search backends like Elasticsearch directly?

Site Search removes Elasticsearch as your direct dependency when introducing search to a website. In fact Site Search does not only rely on Elasticsearch but on additional components from loxal that improve the quality of search results. Site Search is also a managed service that removes the burden of operations from your organization. It provides a simple API that is tailored for the purpose of providing website search with all its specifics.

What is the difference between sites and pages?

A site is a website that can contain thousands of pages. You may have two sites, en.example.com and de.example.com, and each may contain thousands of pages in the respective language.

What is a tenant?

A tenant is an organizational entity that can manage dozens of independent sites. You may have a company that is assigned to a tenant.

How can I index a website and make it searchable?

Currently a website needs to expose an RSS / Atom feed to make it easily searchable. This is the fastest & easiest way to get up and running. Use this curl command:

curl -X POST https://search.loxal.net/sites/rss?feedUrl=https://www.nasa.gov/rss/dyn/breaking_news.rss

…to call our API accordingly. You can still make a website searchable that does not expose any feeds by using our API and indexing it page by page — see “How can I index a page and make it searchable?” below.

How can I create a new site and get started with Site Search?

After executing this curl command, you will also obtain the siteId and siteSecret of the newly created site:

curl -v -X POST https://search.loxal.net/sites

How can I index a page within an already existing site and make it searchable?

When a page with the provided URL already exists, it is updated with the given title & body. Execute this curl command with a siteSecret that corresponds to the provided siteId:

curl -v -X PUT \
  'https://search.loxal.net/sites/5e307324-4161-11ea-9866-1fdc7723fa74/pages?siteSecret=6f10e912-4161-11ea-bb83-7b2c6510169b' \
    -H 'content-type: application/json' \
    -d '{
      "title": "Test Page",
      "body": "Test Content",
      "url": "https://example.com/page"
  }'

How can I update an already indexed page?

When updating a page, the URL in the payload assures that only the page with the provided URL is updated with a new title and content. If the page with the provided URL does not exist in a site's index, the page is automatically added to the site index. Execute this curl command with a siteSecret that corresponds to the provided siteId:

curl -v -X PUT \
  'https://search.loxal.net/sites/5e307324-4161-11ea-9866-1fdc7723fa74/pages?siteSecret=6f10e912-4161-11ea-bb83-7b2c6510169b' \
    -H 'content-type: application/json' \
    -d '{
      "title": "Test Page",
      "body": "Test Content",
      "url": "https://example.com/page"
  }'

How can I delete an indexed page?

You need to provide a siteSecret that corresponds to the provided siteId, as well as a page's primary ID:

curl -v -X DELETE \
  https://search.loxal.net/sites/d87fdcef-a84d-462d-ba84-61df9805536a/pages/0c42f3ef01536bd29510d7b4d178fc7e6cbc1d26095ac3a759bf638f80bfa3c9?siteSecret=b554a1e7-3e87-44ab-b353-f1fd8a423bbe

What is a site secret?

A site secret is a token that you should keep confidential and not publish to any public repository. The site secret enables you to add new pages and update existing pages within a site. You need to provide a siteSecret as a query parameter to authorize operations that modify the index of a site.

How can I make my website searchable?

The easiest & quickest way is to use the Site Search Gadget to create an index from an RSS / Atom feed and make it instantly searchable without even embedding it into a website. If your website does not expose any feeds, use our open API to index your website page by page.

Where can I find API documentation and specification for Site Search?

Site Search' API is technically documented according to the OpenAPI 3.0 specification, using Swagger. You can look up the required schemas and also use the API right from your browser.

What does Site Search cost?

During the pilot phase, Site Search is offered at a single flat pilot plan of €49 / month for one domain (up to 2,000 pages). A 14-day free evaluation is available up-front. Full pricing tiers will return after the pilot phase — email info@loxal.net for custom needs.

All prices are total prices; loxal operates under the small-business scheme of § 19 UStG (Kleinunternehmerregelung) and therefore does not charge or separately show VAT.

I do not want to use Site Search anymore. Do I have to delete my data?

You do not have to do anything. We automatically delete your indexed data after a certain time of inactivity. We would appreciate to learn the reasons behind your decision to stop using our service.

After using the Site Search Gadget I get search results that do not correspond to the specified siteId.

Delete your browser's cookies for search.loxal.net, specifically the “override-site” cookie.

Does Site Search support categories / labels / tags?

Categories as supported by Site Search are a flat labels list: a page can belong to many categories and a category may contain many pages, but no subcategories. A category is assigned to a page at indexation time. Currently we support assigning categories to a page via robots.txt declarations. Please reach out to describe the way you want to assign categories to pages.

Who is using Site Search?

The following website's search is powered by Site Search:

• loxal Company
• …and many more. Just look at the bottom of this page.

Crawler

Sitemaps support

The crawler can either crawl a site using its sitemaps or by following its links. Nested sitemaps are supported as well.

What is the impact on my web server when Site Search crawls my site to index its pages?

The Getting Started Gadget crawls a site at a speed of around two pages per second. Even very moderate web servers are able to handle this load. The regular recrawls for subscribed customers happen at around the same speed, except for very large websites with more than 10k pages, where we can increase the recrawling speed to 100 pages per second or more. In that case we notify the service subscriber and provide a speed their web servers can handle without any disruption to their customers.

Content Extraction Selector

This feature gives you the opportunity to provide a CSS selector that is used to extract a page's content. Instead of indexing everything in a page's body, you can limit content extraction to a subsection of a page. If the selector cannot match anything, the crawling mechanism falls back to page body extraction.

Does the crawler support dynamic, JavaScript-generated content?

Like most (all?) other crawlers, the Site Search crawler cannot extract runtime-generated content. A common practice is to provide the crawler with HTML snapshots of the page content you want to be crawled.

Can the crawler access sites or pages that require authentication, e.g. Basic Auth protected sites?

Not yet, but if this feature is frequently requested, we might consider implementing this functionality.

Search Bar

Do I need the search bar to use Site Search?

No, not necessarily; the search bar is a sample implementation to quickly show you how a search bar may look and interact with the Site Search service. Still, the search bar is a production-grade implementation and ready for production deployments.

Can I use my own search bar with Site Search?

Yes, you can use any search bar, written in any language that also runs on native platforms, as long as it follows Site Search' API specification.

Can I customize the layout and/or design of the search bar?

Yes, just use data-search-style to customize the search bar box and data-results-style to customize the results list.

Can I customize the highlighted words within the search snippet previews?

To customize the highlighted words, override the pf-highlight CSS class with your desired definitions.

How do I integrate the Site Search search bar into my website?

Embed the following HTML fragment into your website and adjust the siteId provided in the snippet:

<div id="searchbar"></div>
<finder-config
    data-siteId="a2e8d60b-0696-47ea-bc48-982598ee35bd"
    data-append-as-child-to="#searchbar"
    data-search-style="width: 500px; font-size: 2em; text-indent: .5em;"
></finder-config>
<script type="module">
    import init, { init_finder } from "https://www.loxal.net/app/finder/finder.js";
    (async () => { await init(); init_finder(); })();
</script>

Can I use another search bar language?

The search bar uses just a few labels that are not translated yet. Just reach out to receive a translation in your desired language on short notice.

How can I see the search queries that users are performing, or include them in my Google Analytics reports?

Place the code snippet below at the bottom of the body of the page where you have integrated Site Search' search bar. Replace the value of data-analyticsTrackingId with your Google Analytics Tracking ID:

<div id="searchbar"></div>
<finder-config
    data-siteId="a2e8d60b-0696-47ea-bc48-982598ee35bd"
    data-analyticsTrackingId="UA-12345678-9"
    data-append-as-child-to="#searchbar"
    data-search-style="width: 500px; font-size: 2em; text-indent: .5em;"
></finder-config>
<script type="module">
    import init, { init_finder } from "https://www.loxal.net/app/finder/finder.js";
    (async () => { await init(); init_finder(); })();
</script>

Afterwards enable “Site Search Tracking” in your Google Analytics admin dashboard and specify the search query parameter query. Please consider a Google Analytics-specific delay of up to 30 minutes before you can see the search queries.