Custom Web Search with Aigents

7 min readJun 17, 2020

Since the inception of the Aigents® project, the scope of the Webspace available for the Aigents user was narrowed down to the list of sites the user configures for their account themselves or have extracted by the Aigents engine from histories of user interactions. Now it changes and the full-blown large-scale Web search (meta-search, in fact) and latent monitoring gets available.

Aigents® “meta-search report with category detection (clustering), sentiment analysis and graph visualization available at Aigents Web Demo site https://aigents.com

What’s new?

The new features of the Aigents Core Platform Open Source release 2.9.0 are news monitoring with global Web search and integration of the custom Web search with sentiment analysis. These features are relying on commercial API-s, such as Google Custom Search, SerpApi (the two already supported), and Gigablast (to be supported in the future).

Aigents custom search with sentiment analysis based on internal Aigents database and Google search integration using Google Search API and SerpAPI interfaces — presented by Anton Kolonin

Global Web Search Monitoring means that once you have any “Topics” set up, you do not necessarily need to set up the “Sites” so they are crawled by the Aigents. The Aigents will start monitoring the search results given the topics and create the news items right out of the search results, applying personal and social relevance assessment to them and measuring sentiment.
Custom Web Search can be run in chat sessions either on the Aigents Web Demo site in the “Chat” view or using Telegram and Facebook Messenger. You can run the queries with multiple options (described below) with your text input and get the results in the same text or as a well-formed HTML reports with optional graphs.
Sentiment Analysis has been implemented earlier for news monitoring, and now it applies to the results of the Custom Web Search as well, so the sentiment assessment can be done with respect to every finding returned by the search and with respect to categories discovered for these items.

Using Aigents bot in Telegram and Facebook Messenger to do the Custom Web Search interactively with results returned in either plain text or HTML report.

What are the magic spells?

Setting up the Global Web Search Monitoring is set up automatically once you have subscribed to the Aigents service with your support. At this point, even if no sites are configured under the “Sites” view, you will be getting the news feeds in the “News” view, having the news items extracted right from the search results checked periodically under the Aigents hood.

Providing support to Aigents project enabling Global Web Search Monitoring is done with the “Subscription” menu option under the Aigents icon on the Aigents Web Demo site.

The Custom Web Search and Sentiment Analysts are freely available for every user, however, you still need to be subscribed for commercial service (see above) in order to do the search via commercial APIs (Google Custom Search and SerpApi).

The minimum search command in an Aigents chat session is a “search <topic>” where the <topic> may be either a single word like pizza or a search query in single or double quotes like ’large pizza’ or ”smart pizza” (need plain quotes like on computer keyboard and not the fancy ones like used here in this Medium publication).

The search command may be extended with the following options. The options may be separated by commas or may not — see the video above for example.

search [topic] — topic pattern to search — as keyword or hierarchical pattern. The topic pattern may contain “variables” which may be used as properties in the other options such as “graph” or “sort”.
site [url] — URL to start the search from. If omitted, either the Aigents internal Web index is searched or commercial API is used (if the “engine” option is set and commercial subscription is present).
timeout [number] — limits the number of seconds to wait for results in chat before going into background searching mode (default — 10). When the timeout expires, the “Working” reply is returned and the search keeps running in the background. If the new search is requested during the previous search execution, the “Busy” reply is returned. When the search is completed running in the background, the results are returned to user asynchronously.
format [text|html|json|pdf|xlsx] — format of the results returned in a file, if not configured, then the results are returned in plain text (default — text, html — supported right now, json/pdf/xls/xlsx — not supported yet, will come in the next releases)
cluster [text|<property>] — specifies the property if the finding which has to be used for clustering and automatic category detection, either “text” or any variable name across names of the variables in a topic pattern (default — “text”).
sentiment — if set, requests automatic detection of either positive or negative sentiment of the texts found by search and representation of the sentiment with “emoji” like 😊 (“positive”)or 😞 (“negative”).
graph [category|sentiment|<property>] — identifies the name of one of the properties to be used to build graphs of relationships, at least two properties have to be specified or single “category” one. If the “category” is set, the property “category” can be used and the category labels of the clusters are used for the value of the property for building graphs. If the “sentiment” is set, ether “positive” 😊 or “negative” 😞 graph vertices are used to render relationships with others.
limit [number] — limits the maximum number of news items to return (default — 10).
time [today|yesterday|YYYY-MM-DD] — latest time to start the search back in the past (default — today).
novelty [new|all] — specifies if return only the new (“new”) findings or all (“all”) of them (default — “new”).
scope [site|web] — specifies whether to search following the links under only the specified web site domain (“site”) or under the entire web (“web”) (default — “site”).
range [number] — the “crawl range” limits the number of “hops” following links from page to page (default — 2).
minutes [number] — limits the maximum number of minutes to spend on search in background (default — 1). When this time expires, the search terminates and returns just what has been found do far.
period [days] — number of days to do the search back in the past in case if nothing is found for the day identified by time above (default — 3).
sort [text|category|<property>] — name of the field used for sorting, can be “text”, “category” (if the format is “html”), or any variable name (default — text).
order [asc|desc] — ascending or descending sorting order.
mode — [smart|track|find] — whether to use the existing path if present only (track) or always explore new paths (find) or track first and go to find if nothing is found (default — smart). See more details on “pathfinding” and “pathtracking” logic on our earlier publication.
engine [any|all|google|serpapi|gigablast] — if set, specifies which of the commercial search APIs to use — any one of them, all of them, or specific one (the “gigablast” option is not supported at the moment). If not set, the internal Aigents index is searched.

What’s next?

In one of the subsequent publications, we will tell how to use the Aigents Custom Web Search architecture to perform a targeted Deep Search with information extraction for specific topics of interest — this feature is still in active development and testing.

We are also considering to have a separate “Search” view in the Aigents Web Demo User Interface on https://aigents.com/, so the searches and analysis of the results could be done more interactively — instead of the HTML reports returned in the chat session. We need your feedback on this so try Aigents now and let us know what you think!

Well, what’s about the money?

Since both the global Web search the custom Web search are relying on commercial APIs (Google Custom Search and SerpApi), one needs to pay for a subscription for using these search API-s. Those who run their own instances of the Aigents platform can pay to the API providers directly. The users of the Aigents Web Demo site at https://aigents.com can just support the Aigents project via PayPal for this service.

However, supporting the Aigents project for using these features is only the one opportunity that one can take. You can earn money yourself hosting the Aigents service on your own hardware premises, setting up your own Google Custom Search and SerpApi accounts to provide the service and your own PayPal account to get paid for that. Moreover, you can add your own value to the service setting up your own sites and topics for your audience and rendering it under your own user interface and brand. To get started, just deploy the Aigents server following the instructions and deploy it on your hosting under your custom user interface referring to the original Aigents Web User Interface as a reference.

But there is more that you can do…

How to help?

Since the Aigents® project is being developed under open-source MIT license terms, anyone is welcome to contribute to the ecosystem with new developments and extensions that we are expecting to come next, as follows.

All that said, join Aigents now and let us know what you think!