Search module overview

The search system allows Guests, Members, and Admins to find resources on interaction-design.org quickly and easily. The search system is currently powered by Laravel Scout with Algolia driver (in production).

• Search indexing
  • Configuration
  • Updating the indexes
  • Filtering
• Search engine differences per environment
  • Testing changes to Algolia locally or on staging
  • Common errors on Bugsnag
• Running searches
  • Server-side search
  • Multi-index searches

Search indexing

A search index is a collection of data that is used to create a search experience. In our context, each item in the index is related to an individual model. The search index item is generally similar to the model properties, but some properties might be removed or renamed. There may even be some custom 'computed' properties that are used to enhance the search experience.

All index PHP classes are stored in the App/Modules/Search/Indexes namespace. Each index is a trait that uses Laravel scout's searchable trait. The naming convention for the index is the name of the model it is going to index, followed by SearchIndex:

• ArticleSearchIndex.php
• CourseSearchIndex.php

The related model will need to use the trait in order for it to work:

...

final class Course
{
    use CourseSearchIndex;

    ...
}

There are a few methods that are commonly used within the search index traits:

• searchableAs() - By default, each index will have a name which matches the models' table name. This method allows you to override the name.
• shouldSearchable() - Determines whether the model should be searchable or not. As an example of usage, it can be used to hide unpublished courses from search. It must return a bool value.
• toSearchableArray() - This method allows you to customise the properties that are indexed. You can change the names of properties, add custom properties or remove properties altogether using this method.

Configuration

Each search index has a configuration file that allows for control over options such as searchable attributes, available custom rankings, and filters, typo tolerance, etc. These configuration files are specific to Algolia and the Scout Extended package. You can read more about the options available algolia docs.

Updating the indexes

Laravel Scout automatically hooks into the model's update event and fires a job that updates the details of the model on the index whenever anything changes. However, if you have changed some configuration or made changes to the properties available on the index, you might need to re-create the entire index. The command for flushing all indexes and re-importing is php artisan scout:import. You can also specify a specific index by passing the model as an argument, e.g.: php artisan scout:import "App\Modules\Course\Models\Course".

DANGER

Once the frontend search UI is live, it is dangerous to run php artisan scout:import as searching will not work until it is finished. Instead, Algolia allows for zero downtime reimports using the php artisan scout:reimport command.

DANGER

Refreshing the search index, especially the "Learning Resource" one which aggregates multiple search indexes (Article, Masterclass, Course, etc.), is a resource-intensive task that consumes a significant amount of memory. That's why it might be necessary to reimport the search index on your local environment rather than on the production server.

docker compose exec app php -d memory_limit=4000M artisan scout:reimport "App\Modules\Search\Indexes\LearningResourceSearchIndex"

Filtering

The search feature supports filtering by model type. To add a new option to the list of filters remember to define model_type as one of the attributes returned from the toSearchableArray method.

Search engine differences per environment

Algolia is generally only run in production. The local and staging environments use the collection engine to handle searches. We don't run Algolia on local or staging for the following reasons:

• Making changes to searchable models on local and staging while using Algolia would affect the search experience on production
• Running scout:import locally would break searching on production
• It would skew search analytics

The collection engine will still allow for searching to work, but it won't be able to do more complex searches, such as searches across relationships. It also won't return results according to the custom ranking rules set up for the index.

Testing changes to Algolia locally or on staging

If you need to make a change to the search configuration files, or you need to add some more complex search functionality that is only supported on Algolia, you may need to temporarily run Algolia locally or on staging. The safest way to do so is to use the "[TEST] ixdf_web" Algolia application instead of the production one - "[PROD] ixdf_web". You'd need the following .env variables:

ALGOLIA_APP_ID= #"Application ID" on Algolia API key page
ALGOLIA_SECRET= #"Admin API Key" on Algolia API key page

SCOUT_DRIVER=algolia

You can get the Algolia API keys for the test application here.

Once you've made your changes and synced or imported them, you can verify that they have work by logging into the Algolia dashboard and testing out the search experience and the configuration for the affected indexes.

WARNING

Once your changes ship to production, you might need to SSH into the server and sync the Scout configuration files and/or re-import the search indexes in order for the changes to take effect.

DANGER

Make sure you swap the SCOUT_DRIVER .env variable back to collection once done. Even though the test Algolia account won't impact changes on production, we are running a free version of Algolia there, so you might burn through the usage credits.

Common errors on Bugsnag

Since we are dealing with an external API to handle search, there are a few common errors that might pop up on Bugsnag.

Unreachable hosts is an error that happens when the Algolia API request fails. This could be due:

• Temporary downtime of Algolia - If you see this error, you can check the Algolia status page here.
• Search timeout - A heavy enough search might have caused a timeout in the connection. Please investigate the url of the api request to find the search query that caused the timeout.
• Throttled request - Too many requests being made at the same time, could cause this issue, specially considering this functionality is available for Guests. Please investigate the status code of the request, if it's 429, it's a throttled request.
• Network failure - If the request fails due to network issues, it might be a problem with the client's network connection. If this happens we can expect a status code in the range of 4xx

These are known issues, that we can't do much about, but we can monitor them and take action if necessary. If you see a lot of these errors, it might be worth investigating if there is a problem with the search functionality.

Running searches

Server-side search

Any model which has a search index can be searched using the static search() method:

$courses = Course::search('Search term')->get();

The query will be fuzzy-searched across the searchable attributes defined for that index and relevant results will be returned. In a similar way to building Eloquent queries, you can chain additional methods onto the search query, such as where(), whereBetween(), whereIn(), etc. You can read more here.

Multi-index searches

There are some places on the site where we want members and guests to be able to search across multiple indexes at once. Using Algolia's instantSearch library, you can run a multi-index search manually, but it would be quite a lot of work to set this up for 4 or 5 search indexes.

An alternative is to use an aggregator. This allows you to group a bunch of indexes together and allow them to be searched as if they were one index.