agri1.ai: A Two-Sided Approach to LLMs, chatGPT in Agriculture - Frontend & Embedding and Domain-Specific Large Language Model for Agriculture

Welcome to the world of LLMS such as Claude, Llama and chatGPT in agriculture, welcome to agri1.ai, an initiative that aims to explore the potential of artificial intelligence (AI) in the agriculture industry. As the global population continues to grow, the demand for efficient and sustainable farming practices is more pressing than ever. AI, with its ability to analyze vast amounts of data and make accurate predictions, could be a game-changer in meeting this demand.

Introduction
Current state of agri1.ai
The future of agri1.ai & Domain-Specific Large Language Model for Agriculture
LLM Open Source Approaches and Models
Existing LLMs in Agriculture

Introduction

With agri1.ai, we’re taking a two-sided approach to harnessing the power of AI for agriculture. On one side, we’re developing a frontend interface that uses an existing Large Language Model (LLM), fine-tuning it, embedding it, and contextualizing it with public and internal data. On the other side, we’re exploring the possibility of creating our own domain-specific LLM for agriculture.

In rapidly changing environments, both in terms of climate and markets, the concept of agri1.ai becomes increasingly important. This is especially true for large agriculture-driven societies and regions such as the African continent, where a lack of knowledge can lead to significant challenges within agriculture. One of the missions of agri1.ai is to address these issues, supporting smallholder farmers in their struggle with rapidly changing climatic conditions, and providing better consultation for new agricultural cultures depending on climatic and soil conditions. The lack of education in certain parts of the world is also a motivation for us to support farmers through our initiative.

The Current State of agri1.ai: Bridging the Gap Between Humans and AI

At the heart of our initiative, agri1.ai serves as a dynamic platform, bridging the gap between humans in agriculture and the world of AI software and algorithms. Our primary goal is to facilitate a seamless interaction between these two entities, fostering a symbiotic relationship that enhances the efficiency and sustainability of agricultural practices.

Currently, a gri1.ai operates on the foundation of OpenAI’s GPT, a cutting-edge Large Language Model (LLM). We’ve partially adapted, fine-tuned this model to better comprehend and generate agriculture-centric text, enhancing its relevance and utility for our users. Furthermore, we’ve partly incorporated data embeddings, integrating both public and internal data, to augment the model’s contextual understanding of the agricultural domain.

In the realm of AI, simplicity is often the key to success. Building and deploying AI applications can be a complex process, and maintaining a sense of simplicity in our operations allows us to focus on delivering a high-quality, user-friendly service. By building upon an existing, hosted LLM, we’re able to leverage the power of advanced AI while maintaining a streamlined and efficient system.

One of the cornerstones of our operation is data governance. We recognize the critical importance of managing the availability, usability, integrity, and security of our users’ data. This comprehensive approach to data governance not only ensures the reliability and usefulness of the information provided by agri1.ai but also addresses key concerns such as regulatory compliance, privacy, quality, and security. We understand that agribusinesses have valid concerns about data leakage and the potential for LLMs to be trained on internal data, compromising data sovereignty. We want to assure our users that we take these concerns very seriously and are actively working on strategies to address these issues.

As we continue to refine and enhance agri1.ai , we’re also exploring the possibility of creating a new LLM by retraining, or fine-tuning an existing LLM. This approach could potentially allow us to create a more specialized and effective model for agriculture.

The Future of agri1.ai: Domain-Specific Large Language Model for Agriculture

While we’re proud of what we’ve achieved with agri1.ai so far, we’re not stopping there. We’re also exploring the possibility of creating our own domain-specific LLM for agriculture. This model, which we’re calling agriLLM (working title), would be trained on a large amount of agriculture-related text data, making it an expert in the language and nuances of the agriculture industry.

Creating agriLLM will be a complex process, involving data collection, data cleaning and preprocessing, model selection, model training, fine-tuning, evaluation and testing, and deployment. We’re also planning to involve experts in various fields of agriculture to help us build detailed training datasets andfine-tune the model.

Data Collection: The first step in building a domain-specific LLM for agriculture involves collecting a vast amount of data relevant to the field. This can include scientific articles, research papers, farming guides, weather reports, crop yield data, and more. The data should cover a wide range of topics within agriculture to ensure the model is well-rounded and knowledgeable in all aspects of the field. Tools like web scraping can be used to automate the data collection process from various online sources.
Data Preprocessing: Once the data is collected, it needs to be preprocessed to prepare it for training the LLM. This involves cleaning the data (removing duplicates, fixing missing or incorrect values), normalization (converting all text to lowercase, removing punctuation, and stop words), and tokenization (breaking down the text into individual words or phrases to create the vocabulary for the language model).
Model Selection and Configuration: The next step is to choose a suitable model architecture for the LLM. Transformer-based models like GPT-3 and BERT are popular choices due to their ability to handle long sequences of text and generate high-quality outputs. The model configuration, including the number of layers, attention heads, loss function, and hyperparameters, needs to be specified at this stage.
Model Training: The model is then trained on the preprocessed data. This involves presenting the model with sequences of words and training it to predict the next word in the sequence. The model adjusts its weights based on the difference between its prediction and the actual next word. This process is repeated millions of times until the model reaches a satisfactory level of performance.
Evaluation and Fine-tuning: After the initial training, the model is evaluated on a separate test dataset. Based on the evaluation results, the model may require some fine-tuning. This could involve adjusting its hyperparameters, changing the architecture, or training on additional data to improve its performance.
Domain-Specific Fine-tuning: To make the LLM specific to agriculture, it is fine-tuned on the domain-specific data collected in the first step. This helps the model to understand the unique terminology, context, and nuances of the agriculture domain.
Integration with agri1.ai: Once the domain-specific LLM is ready, it is integrated with the agri1.ai system. This involves setting up the necessary APIs and interfaces to allow agri1.ai to leverage the capabilities of the new LLM.
User Testing and Feedback: The updated agri1.ai system is then tested by end-users. Their feedback is collected and used to identify any issues or areas for improvement.
Continuous Improvement: Based on user feedback, the LLM is continuously updated and improved. This could involve further fine-tuning, adding more data to the training set, or tweaking the model architecture.
Monitoring and Maintenance: Finally, the performance of the LLM is continuously monitored to ensure it is providing accurate and useful output. Regular maintenance is also performed to keep the system running smoothly.

Building a domain-specific LLM for agriculture is a complex but achievable task. It involves a series of steps from data collection to continuous improvement. By following this process, we aim to develop an LLM that can provide accurate, relevant, and useful information to users in the agriculture industry.

Open Source Approaches and Models

We’re keeping a close eye on the developments in the wider AI community. One resource we’ve found particularly useful is the LMSYS leaderboard, which ranks various LLMs based on their performance. Some of the models on this leaderboard, such as OpenAI’s GPT-4 and Anthropic’s Claude-v1, could potentially be used as the foundation for agriLLM.

However, we’re also aware of the gap between proprietary and open-source models. While proprietary models like GPT-4 currently lead the pack, we’re optimistic about the potential of open-source models to catch up. One such open-source model is MosaicML, which provides a flexible and modular platform for machine learning models, and could potentially be used to train our own LLM.

MosaicML offers a range of features that could be beneficial for the development of agriLLM. It allows for the training of multi-billion-parameter models in hours, not days, and offers efficient scaling at large scales. It also provides automated performance enhancements, allowing users to stay on the bleeding edge of efficiency. MosaicML’s platform supports training large language models at scale with a single command, and it provides automatic resumption from node failures and loss spikes, which could be particularly useful for the long training times associated with large models like agriLLM.

Existing LLMs in Agriculture

In our research, we’ve come across a specific model to agriculture, named AgricultureBERT, a BERT-based language model that has been further pre-trained from the checkpoint of SciBERT. This model was trained on a balanced dataset of scientific and general works in the agriculture domain, encompassing knowledge from different areas of agriculture research and practical knowledge.

The corpus used to train AgricultureBERT contains 1.2 million paragraphs from the National Agricultural Library (NAL) from the US Government and 5.3 million paragraphs from books and common literature from the Agriculture Domain. The model was trained using the self-supervised learning approach of Masked Language Modeling (MLM), which involves masking 15% of the words in the input sentence and then having the model predict the masked words. This approach allows the model to learn a bidirectional representation of the sentence, which is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like GPT which internally masks the future tokens.

This existing model can provide valuable insights and serve as a useful starting point, our ultimate goal at agri1.ai is to develop our own domain-specific LLM for agriculture. We believe that by doing so, we can create a model that is even more tailored to the needs of the agriculture industry and that can provide even more accurate and relevant information to our users.

Keep It Agile: The Journey Continues

In the rapidly evolving field of AI, continuous learning and adaptation are key. This journey has been a profound learning experience, particularly for me, Max.

Understanding the unique ways in which users interact with AI within the agricultural context has been both enlightening and instructive. Each query we receive from farmers worldwide provides invaluable insights into the real-world challenges that agri1.ai can address. Our approach is iterative – we observe user interactions, engage in dialogue with users, develop solutions, ship them, and then reassess.

This cycle allows us to constantly refine and improve our product, ensuring it remains relevant and useful to our users. We’re excited about the potential of user interface (UI) and user experience (UX) enhancements to further improve the usability of agri1.ai . The pace of development in the AI scene is breathtaking, with new models and technologies emerging regularly. We’re committed to staying abreast of these developments, exploring how we can leverage them to enhance agri1.ai and better serve farmers and agribusinesses worldwide.

I do recognize that this is just the beginning. The journey of agri1.ai is an ongoing process, and I am committed to continuing to learn, adapt, and improve. I am excited about the potential of AI to transform agriculture, and I am grateful for the opportunity to be a part of this journey. Thank you for joining us on this adventure.