AI has a Chicken and Egg Problem
AI has a chicken and egg problem, and it goes something like this. In order to understand what the user is asking, you need to supply relevant grounding data in the prompt. But in order to supply relevant grounding data in the prompt, you need to understand what the user is asking. There are a number of attempted solutions to this problem, and none of them are entirely successful. One prominent solution is called a Vector Database. This white paper discusses some of the pros and cons of Vector Databases and alternate strategies that we have developed at Metazoa that are specifically designed for Salesforce.
Vector Databases
Some of the popular Vector Databases are called Chroma, Pinecone, Weaviate, Milvus and Faiss. Salesforce recently announced that there will be a Vector Database for Einstein as well. Vector Databases are designed to enhance AI prompts with relevant grounding data. You can’t simply include all possible data in the prompt because the token limits in Large Language Models (LLMs) are too restrictive.
Here is how a Vector Database works. First, you divide all potential grounding data into chunks and submit it to the vector database. User questions are turned into a vector that is used to retrieve the most relevant chunks of grounding data with a nearest neighbor search. This information is added invisibly to the prompt. Used in this manner, Vector Databases are similar to other Retrieval Augmented Generation (RAG) systems.
Vector Databases are best suited for freeform text environments. For example, if you have a book, you can vectorize each paragraph, and when the user asks questions about the book, you can pull some relevant paragraphs for grounding data. In Salesforce, you might vectorize technical support emails in order to enhance chats with an intelligent support assistant. There are some other examples where Vector Databases can be used to supply relevant grounding data for freeform text environments in Salesforce.
Problems and Limitations
But the most important information in Salesforce is contained in structured records, not freeform text. All of the data and metadata records will have technical names and labels that identify them. People rarely type a record name or label correctly in a question, and using a Vector Database to find “similar” names and labels will not be helpful. The core technology of a Vector Database depends on comparing the meaning of words, and that doesn’t work very well with structured data.
Token Limits
A related problem is that most Salesforce objects are orders of magnitude larger in size than even the largest token limit. There are a huge number of records to choose from, but only a tiny fraction of this information can be selected for grounding data. Selecting the wrong record will have catastrophic consequences for the prompt. When this happens, the LLM will answer the wrong question, reference incorrect information, or provide hallucinatory responses. The grounding data is usually invisible, so this can be really disconcerting for the end user. Selecting bad grounding data is usually worse than selecting nothing at all.
Implementation Difficulties
An additional issue is that Vector Databases are notoriously difficult to maintain. In Salesforce, records are constantly being created, updated, and deleted. This makes the most recent (and relevant) information impossible to include in the Vector Database. There is a huge security issue here as well. You now have a gigantic, fragmented database of vector encodings on some other platform with all the attendant difficulties. Lastly, some Vector Databases require multiple round trips to the LLM for vectorization. This strategy isn’t viable for chat scenarios until the performance of the available LLMs gets much better.
Beyond Vector Databases
At Metazoa we are currently using Vector Databases in some situations, but after extensive testing, we have come to the conclusion that they are not the best strategy for all scenarios. Metazoa has developed some special technologies to implement RAG on Salesforce that avoid the pitfalls of a Vector Database. The strategies below go a long way in providing effective and error free grounding data for prompts that must live within the token limit.
Context and Relationships
First, we make use of context where possible. If the user receives an error message from a deployment, then we include the details of the error and the metadata assets that caused the problem in the grounding data. If the user mentions an Account, we know that the associated Opportunities and Contacts are also important. Context is a seamless way to provide good grounding data. Our Intelligent Assistant only uses a Vector Database when the end user does not explicitly select the assets they are interested in.
Precision Grounding Data
Second, we streamline the selection of precision grounding data. This allows the prompt engineer or end user to select atomistic pieces of information to include in the prompt. For metadata, the prompt engineer can specify individual parent or child assets. For data, the prompt engineer can select related records and fields individually. This strategy gives us detailed focus on the required grounding data we need, which allows the reliable transformation of data and metadata for org management scenarios.
Dynamic Compression
Third, we have implemented dynamic compression of both data and metadata assets. This technology can rank the importance of individual XML elements and drop elements of lower importance to dynamically fit inside any given token limit. Most LLMs perform very well with even partial information about data or metadata. This strategy enables massive transformations of entire Salesforce orgs without errors when extra-large assets or records are encountered.
Prompt Automation
Lastly, sometimes the best way to work within the token limit is to divide a transaction into multiple parts. For example, our prompt engineering platform Metadata Studio can develop prompts that operate on individual data records or metadata assets. Then you can use Prompt Automation to apply that prompt to any number of selected data records or metadata assets. This provides an effective method of transforming an entire Salesforce org with AI through the reuse of a single prompt.
Conclusion
Vector Databases are a useful RAG technology for freeform text environments. For more sophisticated platforms like Salesforce that feature structured data, we recommend use of context and relationships, precision grounding data, dynamic compression, and prompt automation.
