Hi ,

The particular lua functions you mention, and related sections of the documentation, are specific to a certain set of connectors. The Facebook and Instagram Connectors, along with a few other social media-related ones, use APIs to obtain data that's structured as a feed (formatted either as JSON or XML), and these functions help in parsing that feed, to determine what constitutes a document, how to determine which links need following, etc.

Other connectors, such as the Web Connector, obtain data in different formats, such as files on a filesystem, individual web pages, or results from a database query. These don't necessarily require the same sort of fine-tuning for connector actions to run and for records to be passed through the rest of the ingest process. Where there are possibilities to alter this information, such as identifying patterns in a web page to treat as links, this may show up in configuration parameters such as LinkAttributes .

Most connectors, including the Web Connector, will have a set of common built-in lua functions that you'll find in the Reference, under Lua Functions and Methods . These are most commonly run as IngestActions to do things like adjust document fields before sending them on for ingestion. The "Manipulate Documents" section of the administration guide will usually provide some details under Run Lua Scripts .

Hello Geoff,

Thank you for the quick reply. I bring this up as there used to be a generic social media connector before it was split up into many different connectors. This form of social media connector supported these build-in lua functions I mentioned, but it is outdated. If there was to be an upgrade to a newer form of connector that use APIs to obtain data that's structured as a feed, what connector would be best? It seems that the web connector functions differently and could not be used as a replacement.

Hello Geoff,

Thank you for the quick reply. I bring this up as there used to be a generic social media connector before it was split up into many different connectors. This form of social media connector supported these build-in lua functions I mentioned, but it is outdated. If there was to be an upgrade to a newer form of connector that use APIs to obtain data that's structured as a feed, what connector would be best? It seems that the web connector functions differently and could not be used as a replacement.

Right, the Web Connector is more designed to deal with fully-rendered web pages, as a browser would. There isn't an out-of-the-box replacement for the old Social Media Feed Connector, so this may require more of a custom solution. (Though if the APIs in question belong to a commonly used product/site, it wouldn't hurt to put in an enhancement request. New connectors are added all the time.)