According to this post on X, LLaMa 3 used Intra-Document causal masking during pretraining to avoid cross-contamination when using sequence packing: Causal Masking vs. Intra-Document Causal Masking This seems to improve final performance of the model across wide range of tasks in contrast to simply putting a separator token - e.g. end of text token - between documents and hoping that the model will learn to not attend to unrelated documents during pre-training as stated in GPT-3 paper: During training we always train on sequences of the full nctx = 2048 token context window, packing multiple documents into a single sequence when documents are shorter than 2048, in order to increase computational efficiency. Sequences with multiple documents are not masked in any special way but instead documents within a sequence are delimited with a special end of text token, giving the language model the information necessary to infer that context separated by the end of text token is unrelated. This allows for efficient training without need for any special sequence-specific masking. I wanted to use the former - Intra-Document - approach in my experiments, but I stumbled upon one issue. In the standard transformer implementation the only interaction between tokens is via attention mechanism, so using the proper masking is enough. However, I use slightly modified version that also calculates a cumulative (prefix) sum on queries and keys to introduce positional bias that seems to work quite well when compared to RoPE or ALiBi. Without going into details here is relevant piece of code: # cumulative/prefix sum across sequence q_pos = q.cumsum(-3) k_pos = k.cumsum(-3) Now without Intra-Document causal-masking or during inference this code works fine, but when a sequence is packed it can contain a bunch of unrelated documents. I want to somehow prevent query and key contamination from past unrelated documents for "cumsum" operation and do it efficiently - without python loops. What would be the best approach here? submitted by /u/kiockete [link] [comments]

Researchers in Computational Aesthetics are exploring how computers can analyze and potentially create aesthetically pleasing visual experiences. A key component of this research is having access to large, diverse datasets of high-quality visual content. I'm curious to hear from the community: How valuable would a dataset of tens of thousands of diverse and high-quality images and videos be for research in Computational Aesthetics? What specific benefits might such a dataset offer (e.g., training AI models, evaluating aesthetic quality, exploring different artistic styles)? I'm interested in learning from researchers, artists, and anyone else interested in the intersection of AI and aesthetics. Additionally: A specific example of a research area in Computational Aesthetics that could benefit from such a dataset (e.g., style transfer, artistic quality assessment). A link to an article or resource that explains Computational Aesthetics for those unfamiliar with the field (optional). submitted by /u/Same_Half3758 [link] [comments]

Abstract: In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling. Link: xLSTM: Extended Long Short-Term Memory submitted by /u/Background_Thanks604 [link] [comments]

Hey everyone. For context, I’m a recent CS graduate and current entry level Data Engineer, and I’ve always loved learning about ML models and techniques and how to implement, deploy, and scale them. I’m looking for a good podcast to keep my knowledge of ML trends up to date, but the challenge is that I don’t really like listening to podcasts that are technical as I am still a newbie and generally understand complexities better if I read them. I’ve tried some podcasts but most of the time the stuff goes over my head and I get lost. Looking for something I can listen to without having to think too hard on my way to work. Would love any suggestions! submitted by /u/C-beenz [link] [comments]

This post presents a solution to automatically generate a meeting summary from a recorded virtual meeting (for example, using Amazon Chime) with several participants. The recording is transcribed to text using Amazon Transcribe and then processed using Amazon SageMaker Hugging Face containers to generate the meeting summary. The Hugging Face containers host a large language model (LLM) from the Hugging Face Hub.

This post is co-written with Tim Camara, Senior Product Manager at Veritone. Veritone is an artificial intelligence (AI) company based in Irvine, California. Founded in 2014, Veritone empowers people with AI-powered software and solutions for various applications, including media processing, analytics, advertising, and more. It offers solutions for media transcription, facial recognition, content summarization, object

A lot of works on parameter efficient fine tuning of transformers are coming out, but how much of them are actually being applied? Also I was curious what techniques do you normally use in the industry? submitted by /u/Inner_Programmer_329 [link] [comments]

Hi I'm doing weighted pruning, but I have one issuse here , so let's say I have a tensor so most of the tensors are nearly to zero so I changed that to zero , so nearly 40percent of the tensors zero now, does that mean my matrix is a sparse one or is it still dense , if it's not a sparse matrix , the computation will be same right , all row and column gets multipled , so then what is the purpose of weighted pruning then !! submitted by /u/IcyCockroach5501 [link] [comments]

Large language models (LLMs) have unlocked new possibilities for extracting information from unstructured text data. Although much of the current excitement is around LLMs for generative AI tasks, many of the key use cases that you might want to solve have not fundamentally changed. Tasks such as routing support tickets, recognizing customers intents from a

I'm curious to understand how much realistic potential there is that Apple can compete with Nvidia IF we make an assumption that they're starting with what we know about in the M series chips. Could they pull some of this IP to make purpose built "AI" chips that might compete? Context: Rumors that Apple might try to do this.. submitted by /u/playstation3d [link] [comments]

Github link Hey all, I'm Efe from Secondlaw AI. We are building physics-informed large AI models. Currently, we are focusing on weather modelling. To benchmark SOTA, we had to build a forecasting infra for all available large weather models and we could not find a solid tooling to do so, so we built Sykrim. Within <5 mins and <5 LOC you can run forecasts on par with global weather models that are run on 100K+ CPU HPCs! You can check out examples here. We are implementing more models & fine-tuning capabilities. Let us know if anything more we can add, happy to answer any questions! submitted by /u/0xe5e [link] [comments]

submitted by /u/the-amplituhedron [link] [comments]

I made YARI. It features a hybrid fusion search between BM25 and Cosine Similarity and is built on top of Redis. Uses: FastAPI, Celery and Redis. OpenAI’s API support for embedding generation and prompt completion. Please give me your feedback on it. Source: https://github.com/fighterbay/YARI submitted by /u/fighterbay [link] [comments]

Suppose I want to do a multi-class classification on text. One approach is to prompt engineer, however, this can output labels different to what I want. Here is an example: Extract the following labels from the text. Labels: Apples, Oranges. Text: I ate an apple and then a few oranges. Answer: Apples, Oranges The answer shown above being simply the expected answer. If we were to use prompts, some other possibilities would be [Apple, Orange], [Oranges, Apples] etc. In my case I do have an extensive set of labels that I can fine tune a model on. While I can train BERT to do this, I want to be able to add labels in the future, so want to try finetuning an LLM. Is there a way to train this so that we limit the words that can be output after Answer? One way I can think of is looking at the logits of the word, but this depends on the tokenization (eg. apple could be ap_, _ple). There is also the instructor library, but this doesn't work with transformer library models (eg. Llama-3) to my understanding, (at least not without hosting it separately). Would appreciate any hints/ thoughts about this. TIA submitted by /u/themathstudent [link] [comments]

Hey everyone, We're building Agent Cloud and we’ve been working in the RAG space since last couple of months and we’re open-source. Agent Cloud is an open-source platform enabling companies to build and deploy private LLM chat apps, empowering teams to securely interact with their data. AgentCloud internally uses Airbyte to build data pipelines allowing us to split, chunk, and embed data from over 300 data sources, including NoSQL databases like MongoDB. It simplifies the process of ingesting data into the vector store for the initial setup and subsequent scheduled updates, ensuring that the vector store information is always updated. AgentCloud uses Qdrant as the vector store to efficiently store and manage large sets of vector embeddings. For a given user query the RAG application fetches relevant documents from vector store by analyzing how similar their vector representation is compared to the query vector. You can find more info about how it works and how to use it in the project’s README and We're launching cloud version by end of this week. We’re also very open to contributions and added good first issues for beginners. ​ Sync strategies - we still need to implement ability to change to incremental append instead of full overwrite Chunking strategies - We have semantic chunking, we want to implement custom strategies that would work well with Airbyte connections - currently chunking message by message (Rust) Retrieval strategies - Currently we use agents to craft the query, we would either like more standard retrieval strategies that can be added out of the box in our RAG connector (TS, Python, Mongo) Conversation app ease of setup - we have a design pattern we would like to employ to make setup of conversation apps simpler. APIs - Publish our current Web App APIs as open API spec and more. Happy to answer any questions. [GitHub repo](https://github.com/rnadigital/agentcloud) submitted by /u/thewritingwallah [link] [comments]

Hello everyone I'm currently working on Whisper to specialize it in French railway language. I'm facing some issues with transcribing ambigous words, and recognizin station names. Initially, i tried training it with audio file totaling 2 hours, but the results didn't meet my expectations. I then turned to usings prompts, which solved the ambiguity problème, however since the context size is limited to 244 tokens, i can't include all station names. Could you please provide me with some tips? I'm new to this field. Thank you submitted by /u/Top-Set-1178 [link] [comments]

The EOS token used during pretraining marks "end of sequence", but it does not prevent information to flow across potentially unrelated documents. If so why to even include it during pretraining when we can add it later in SFT phase? submitted by /u/kiockete [link] [comments]

https://stackoverflow.co/company/press/archive/openai-partnership A couple of thoughts: - Pretty sure OPEN AI has already scraped Stack Overflow while training ChatGPT (if you don't believe it - please watch again the famous interview with Mira Murati) - so why do this? Maybe to have legal access to the content? - Since Chat GPT has been released, StackOverflow is declining in popularity (see chart below from Google trends) - so it makes sense for SO owners - Very interesting from the community perspective: developers created the entire content for free which will now be used to replace them, and they don't get the profit share ​ https://preview.redd.it/fudrujkniyyc1.png?width=968&format=png&auto=webp&s=e116159e61394557e03a6cad431aadc77f88807b submitted by /u/pg860 [link] [comments]

I’ve read that inference speed for models like Llama-2 70B is ~10 t/s at best. So that left me wondering how the extremely large models like GPT-4 (1T params?) do their fast 20 t/s inference. With 10x the params, they gotta have at least 3x the layers(?) So that should make its inference much slower. Am I missing anything? What kind of further improvements might these companies be doing to power their fast APIs? Edit: I must mention that you cannot parallelize across GPUs to help with latency of a single example when the data has to pass through model layers sequentially. And with the large model sizes, model parallelism, with its inter-GPU communication should make it even slower… submitted by /u/Fit-Flow-4180 [link] [comments]

Here are some options that are available in my region, I want to go with the 2011, because of how cost-effective the CPUs were for the amount of cores and threads, so there were 2 platform the X79 and the X99. DDR3 was significantly cheaper than DDR4 even though offering little to no performance drop, x99 boards were available with only DDR4 and didn't have any DDR3 boards. As for the GPU, I went with the mi50 16gb because it was available here for just around $130. So after some researching here is what I found: Concerns: I'm planning to do Video Generative Model Training, and I'm still relatively unsure whether or not Ram matters a lot, it seems like having a lot of ram you could do less streaming data on disk, and offload it to Ram for faster access from GPU. If you don't I assume it would just hinder data reading speed? As for storing Data, I don't know if I would actually need to build out a Storage Cluster for this? It seems like it's also possible to tream data to the nodes though it would be very slow? Or potentially just do data slicing so that the amount of data isn't too large for any node? Can I potentially train let say with 10TB of data first, then because my disk is full, delete the current batch data and get another 1OTB of data to then continue training, is that possible? As for MI50 as well, it seems like rocm has dropped support for this card, I was planning to use Zluda, basically a drop-in driver on top of CUDA for AMD, which uses the Rocm 5.7, is this going to affect the stability of the GPU at all if I'm training on Pytorch with Zluda? Option #1: Potentially Ram Restricted But less? Main: X79 5 slot 3.0 x8 Ram: 32gb DDR3 CPU: 2696v2 GPU: 5x MI50 16GB Option #2: - Ram Restricted? Main: X79 9 slot 3.0 x8 Ram: 32gb DDR3 CPU: Dual 2696v2 GPU: 9x MI50 16GB Option #3: Pcie Lanes Restricted? Main: X79 8 slot 2.0 * x1 Ram : 64gb DDR3 CPU: Dual 2696v2 GPU: 8x Mi50 16GB submitted by /u/Ok_Difference_4483 [link] [comments]

As a PhD student in Canada with an accepted paper at ICML, I'm curious about funding options for attending these costly conferences. While my supervisor covers some costs, the total can reach 3500-4000 CAD, including a 700 CAD registration fee. Are there other external funding sources available to cover the remaining expenses? submitted by /u/Personal_Click_6502 [link] [comments]

I just noticed some guy created a 120B Instruct variant of Llama 3 by merging it with itself (end result duplication of 60 / 80 layers). He seems to specialize in these Frankenstein models. For the life of me, I really don't understand this trend. These are easy breezy to create with mergekit, and I wonder about their commercial utility in the wild. Bud even concedes its not better than say, GPT-4. So what's the point? Oh wait, he gets to the end of his post and mentions he submitted it to Open LLM Leaderboard... there we go. The gamification of LLM leaderboard climbing is tiring. submitted by /u/Objective-Camel-3726 [link] [comments]

ResearchHub is rewarding peer reviews on various topics, including AI, for which I'm an editor. The paiement is ~150$ per peer review (paid in their cryptocurrency but easily exchangeable for dollars). Here are some papers for which a peer review bounty is currently available, but keep in mind new papers are often added (and you can also upload papers you'd find interesting to review): Physics Of Language Models: Part 3.3, Knowledge Capacity Scaling Laws Scaling Laws For Fine-Grained Mixture Of Experts Mixture-Of-Depths: Dynamically Allocating Compute In Transformer-Based Language Models Interpretability Analysis And Attention Mechanism Of Deep Learning-Based Microscopic Vision To get the bounty, simply post your review in the Peer Review tab of the paper, and you'll get the bounty if the quality is sufficient. I'd be happy to answer any question you have! submitted by /u/Troof_ [link] [comments]

Hello people I am currently working as a data scientist at startup. We have a requirement of extracting entities from the text of 10 billion tokens. I am not aware how to do it at this much scale. What should be the pipeline and so on. It would be helpful if you guys share your knowledge or good research paper/blog. Currently we are working on 18 entities and my boss wants me to get 93% accuracy. Thankyou submitted by /u/Boring_Astronaut_421 [link] [comments]

Meet LeRobot, a library hosting state-of-the-art deep learning for robotics. The next step of AI development is its application to our physical world. Thus, we are building a community-driven effort around AI for robotics, and it's open to everyone! Take a look at the code: https://github.com/huggingface/lerobot https://preview.redd.it/ugf4l8lfgryc1.png?width=3794&format=png&auto=webp&s=222825e897ba48eb07acedffb0662d5794af04e8 LeRobot is to robotics what the Transformers library is to NLP. It offers clean implementations of advanced AI models with pre-trained checkpoints. We also reimplemented 31 datasets from academia, and some simulation environments, allowing to get started without a physical robot. https://preview.redd.it/1d8qnrpggryc1.png?width=2563&format=png&auto=webp&s=3e0d6c10440c9d0970b867813d3a210dd9a7bab9 Additionally, the same models can be trained on real-world datasets. Here is a cool data visualization with rerun.io which is fully integrated with our video format optimized for training. The data originally comes from the Aloha project. [LINK TO VIDEO] https://preview.redd.it/86ihkcwhgryc1.png?width=2506&format=png&auto=webp&s=4f2ca7522a012d00d7327d90335d069dd099a321 Another visualization with LeRobot, this time on Mobile Aloha data, to learn navigation and manipulation totally end-to-end. Both datasets have been collected on trossenrobotics robot arms. [LINK TO VIDEO] https://preview.redd.it/qqtncqligryc1.png?width=1900&format=png&auto=webp&s=4f83c675b5c6f9dbded4b5b90a7a1c9f531c4086 LeRobot codebase has been validated by replicating state-of-the-art results in simulations. For example, here is the famous ACT policy which has been retrained and made available as a pretrained checkpoint: [LINK TO HF HUB] LeRobot also features the Diffusion Policy, a powerful imitation learning algorithm, and TDMPC, a reinforcement learning method that includes a world model, continuously learning from its interactions with the environment. https://preview.redd.it/br9ibrylgryc1.png?width=1684&format=png&auto=webp&s=8e5595f1dff5381e5f60c6776126f48187ec58d9 Come join our Discord channel. We are building a diverse community from various backgrounds, software and hardware, to develop the next generation of smart robots in the real-world! Thanks to the AI and robotics community without whom LeRobot won't have been possible. submitted by /u/Tamazy [link] [comments]

It turns out, that you can write Kolmogorov-Arnold Network as an MLP, with some repeats and shift before ReLU. https://colab.research.google.com/drive/1v3AHz5J3gk-vu4biESubJdOsUheycJNz submitted by /u/osamc [link] [comments]

submitted by /u/kiockete [link] [comments]

Hello folks! See post here: https://twitter.com/abacaj/status/1785147493728039111 I didn't understand what he meant by "with zero-training (actually just a simple 2 line config) you can get 32k context out of llama-3 models" Does someone know what this dynamic scaling trick is? Much appreciated! 🙂 submitted by /u/sunchipsster [link] [comments]

Please post your questions here instead of creating a new thread. Encourage others who create new posts for questions to post here instead! Thread will stay alive until next one so keep posting after the date in the title. Thanks to everyone for answering questions in the previous thread! submitted by /u/AutoModerator [link] [comments]

Today, we’re excited to announce the availability of Meta Llama 3 inference on AWS Trainium and AWS Inferentia based instances in Amazon SageMaker JumpStart. The Meta Llama 3 models are a collection of pre-trained and fine-tuned generative text models. Amazon Elastic Compute Cloud (Amazon EC2) Trn1 and Inf2 instances, powered by AWS Trainium and AWS

As more powerful large language models (LLMs) are used to perform a variety of tasks with greater accuracy, the number of applications and services that are being built with generative artificial intelligence (AI) is also growing. With great power comes responsibility, and organizations want to make sure that these LLMs produce responses that align with

We are excited to announce the general availability of two advanced recipes in Amazon Personalize, User-Personalization-v2 and Personalized-Ranking-v2 (v2 recipes), which are built on the cutting-edge Transformers architecture to support larger item catalogs with lower latency. In this post, we summarize the new enhancements, and guide you through the process of training a model and providing recommendations for your users.

Embeddings are integral to various natural language processing (NLP) applications, and their quality is crucial for optimal performance. They are commonly used in knowledge bases to represent textual data as dense vectors, enabling efficient similarity search and retrieval. In Retrieval Augmented Generation (RAG), embeddings are used to retrieve relevant passages from a corpus to provide

Large language models (LLMs) are making a significant impact in the realm of artificial intelligence (AI). Their impressive generative abilities have led to widespread adoption across various sectors and use cases, including content generation, sentiment analysis, chatbot development, and virtual assistant technology. Llama2 by Meta is an example of an LLM offered by AWS. Llama

Imagine harnessing the power of advanced language models to understand and respond to your customers’ inquiries. Amazon Bedrock, a fully managed service providing access to such models, makes this possible. Fine-tuning large language models (LLMs) on domain-specific data supercharges tasks like answering product questions or generating relevant content. In this post, we show how Amazon

Organizations often offer support in multiple languages, saying “contact us for translations.” However, customers who don’t speak the predominant language often don’t know that translations are available or how to request them. This can lead to poor customer experience and lost business. A better approach is proactively providing information in multiple languages so customers can

Numerous customers face challenges in managing diverse data sources and seek a chatbot solution capable of orchestrating these sources to offer comprehensive answers. This post presents a solution for developing a chatbot capable of answering queries from both documentation and databases, with straightforward deployment. Amazon Bedrock is a fully managed service that offers a choice

As of April 30, 2024 Amazon Q Business is generally available. Amazon Q Business is a conversational assistant powered by generative artificial intelligence (AI) that enhances workforce productivity by answering questions and completing tasks based on information in your enterprise systems. Your employees can access enterprise content securely and privately using web applications built with

In the fast-paced world of customer service, efficiency and accuracy are paramount. After each call, contact center agents often spend up to a third of the total call time summarizing the customer conversation. Additionally, manual summarization can lead to inconsistencies in the style and level of detail due to varying interpretations of note-taking guidelines. This

We believe generative artificial intelligence (AI) has the potential to transform virtually every customer experience. To make this possible, we’re rapidly innovating to provide the most comprehensive set of capabilities across the three layers of the generative AI stack. This includes the bottom layer with infrastructure to train Large Language Models (LLMs) and other Foundation

Today, we announced the General Availability of Amazon Q, the most capable generative AI powered assistant for accelerating software development and leveraging companies’ internal data. “During the preview, early indications signaled Amazon Q could help our customers’ employees become more than 80% more productive at their jobs; and with the new features we’re planning on

This is a guest post co-authored with Ville Tuulos (Co-founder and CEO) and Eddie Mattia (Data Scientist) of Outerbounds. To build a production-grade AI system today (for example, to do multilingual sentiment analysis of customer support conversations), what are the primary technical challenges? Historically, natural language processing (NLP) would be a primary research and development

This blog post is co-written with Pradeep Prabhakaran from Cohere.  Today, we are excited to announce that Cohere Command R and R+ foundation models are available through Amazon SageMaker JumpStart to deploy and run inference. Command R/R+ are the state-of-the-art retrieval augmented generation (RAG)-optimized models designed to tackle enterprise-grade workloads. In this post, we walk through how

This is a guest post by Mark McQuade, Malikeh Ehghaghi, and Shamane Siri from Arcee. In recent years, large language models (LLMs) have gained attention for their effectiveness, leading various industries to adapt general LLMs to their data for improved results, making efficient training and hardware availability crucial. At Arcee, we focus primarily on enhancing

Today, we are excited to announce that the DBRX model, an open, general-purpose large language model (LLM) developed by Databricks, is available for customers through Amazon SageMaker JumpStart to deploy with one click for running inference. The DBRX LLM employs a fine-grained mixture-of-experts (MoE) architecture, pre-trained on 12 trillion tokens of carefully curated data and

At AWS re:Invent 2023, we announced the general availability of Knowledge Bases for Amazon Bedrock. With Knowledge Bases for Amazon Bedrock, you can securely connect foundation models (FMs) in Amazon Bedrock to your company data for fully managed Retrieval Augmented Generation (RAG). In previous posts, we covered new capabilities like hybrid search support, metadata filtering