Portfolio item number 1
Short description of portfolio item number 1
Posts by Collection
portfolio
Portfolio item number 2
Short description of portfolio item number 2
publications
Observations on Building RAG Systems for Technical Documents
Published in ICLR 2024 Tiny Papers Track, 2024
Retrieval augmented generation (RAG) for technical documents creates challenges as embeddings do not often capture domain information. We review prior art for important factors affecting RAG and perform experiments to highlight best practices and potential challenges to build RAG systems for technical documents.
Recommended citation: Soman, Sumit, and Sujoy Roychowdhury. "Observations on Building RAG Systems for Technical Documents." ICLR 2024 Tiny Papers Track (2024).
Download Paper
Evaluation of RAG Metrics for Question Answering in the Telecom Domain
Published in ICML 2024 Workshop on Foundation Models in the Wild, 2024
Retrieval Augmented Generation (RAG) is widely used to enable Large Language Models (LLMs) perform Question Answering (QA) tasks in various domains. However, RAG based on open-source LLMs for specialized domains has challenges of evaluating generated responses. A popular framework in the literature is the RAG Assessment (RAGAS), a publicly available library which uses LLMs for evaluation. One disadvantage of RAGAS is the lack of details of derivation of numerical value of the evaluation metrics. One of the outcomes of this work is a modified version of this package for few metrics (faithfulness, context relevance, answer relevance, answer correctness, answer similarity and factual correctness) through which we provide the intermediate outputs of the prompts by using any LLMs. Next, we analyse the expert evaluations of the output of the modified RAGAS package and observe the challenges of using it in the telecom domain. We also study the effect of the metrics under correct vs. wrong retrieval and observe that few of the metrics have higher values for correct retrieval. We also study for differences in metrics between base embeddings and those domain adapted via pre-training and fine-tuning. Finally, we comment on the suitability and challenges of using these metrics for in-the-wild telecom QA task.
Recommended citation: Roychowdhury, S., Soman, S., Ranjani, H. G., Gunda, N., Chhabra, V., & Bala, S. K. Evaluation of RAG Metrics for Question Answering in the Telecom Domain. In ICML 2024 Workshop on Foundation Models in the Wild.
Download Paper
talks
First node, second node and methods performed thereby for handling data augmentation
Published:
Abstract: A method performed by a first node for handling data augmentation. The first node divides each epoch in an original dataset having an input space, into a set of batches. The first node generates a set of subsets of samples by selecting, within each batch from every set of batches, a respective plurality of subsets. The first node determines, using machine learning, a fourth set of clusters of data using the third set. The first node selects a fifth set of clusters from the fourth set based on a relevance criterion. The first node generates samples in each cluster of the fifth set, and refrains from generating samples in clusters of the fourth set excluded from the fifth set. The first node then generates a sixth set of augmented samples in the input space of the original dataset, by using the generated samples and applying a reverse projection approach.
Optimizing user equipment service level agreement violations for network slice allocation
Published:
Abstract: A method performed in a network node includes performing drift detection in a slice to identify a number of data points in drift in at least one of network specific performance parameters and/or user equipment, UE, specific performance parameters. The method includes obtaining weighting parameters of the network and UE specific performance parameters. The method includes combining a function of data points in drift of the network and UE specific performance parameters with each data point of the number of data points in drift weighed by the weighting parameters associated with data point. The method includes determining one or more service level agreement, SLA, violations as a weighted average of individual drift in one or more of the at least one of network specific performance parameters and UE specific performance parameters. The method includes performing an action based on determining the one or more SLA violations.
Node and methods performed thereby for handling drift in data
Published:
Abstract: A method performed by a node for handling drift in data. The node obtains a dataset comprising a plurality of datapoints corresponding to a plurality of values of one or more dependent variables for a plurality of first features over a time period. The node determines, using machine learning and explainability, in the absence of determining whether or not the plurality of datapoints has a drift, whether or not there has been a change in respective one or more characteristics of a subset of the plurality of first features having a largest contribution to a variability of the datapoints in the plurality of datapoints based on a threshold from a first time period to a second time period. The node then initiates application of a drift policy on the plurality of datapoints based on a result of the determination.
First node, second node, third node and methods performed thereby for handling predictive models
Published:
Abstract: A computer-implemented method, performed by a first node, for handling predictive models. The first node updates, using machine learning, a first predictive model of an indicator of performance of the communications system. The updating is based on respective explainability values respectively obtained from a first subset of a plurality of second nodes. The respective explainability values correspond to a first subset of respective second predictive models of the indicator of performance of the communications system (100), respectively determined by the first subset of the plurality of second nodes. The models in the first subset of respective second predictive models have a respective performance value above a threshold. The first node then provides an indication of the updated first predictive model to a third node comprised in the plurality of second nodes and excluded from the first subset, or to another node.
First node and methods performed thereby for handling location of a network node in a geographical area for operation in a communications system
Published:
Abstract: A computer-implemented method, performed by a first node, for handling location of a network node in a geographical area for operation in a communications system. The first node obtains first data indicating images of the geographical area over a first time period. The first node also obtains second data indicating data samples of performance indicators of radio communications, during the first time period, of devices in the geographical area. The first node determines, by performing a spatio-temporal correlation of the obtained first data and the obtained second data, one or more locations as candidates to place the network node for operation in the communications system. The determining is performed using machine learning or deep learning, and. The first node then outputs an indication of the determined one or more locations.
Identifying sequences of machine learning models
Published:
Abstract: A computer-implemented method of identifying sequences of machine learning (ML) models of a plurality of ML models during a training phase is provided. The method comprises generating (301) one or more sequences of the ML models The method further comprises determining (302) a performance of each sequence of the ML models. Furthermore, the method comprises identifying (303) a sequence with performance greater than a threshold performance.
First node, second node and methods performed thereby, for planning radio coverage in a space
Published:
Abstract: A computer-implemented method performed by a first node (111). The methods is for planning radio coverage in a space (170). The first node (111) operates in a communications system (100). The first node (111) determines (204), using machine learning (ML) and first radio coverage data from one or more first communications networks (120), an ML model. The ML model is to estimate a number of one or more radio antennas (142) necessary to provide radio coverage to the space (170). The estimate is to be performed in the absence of a floor plan corresponding to the space (170). The first node (111) also provides (208) an indication of the determined ML model to a second node (112) operating in the computer system (100).
teaching
Teaching experience 1
Undergraduate course, University 1, Department, 2014
This is a description of a teaching experience. You can use markdown like any other post.
Teaching experience 2
Workshop, University 1, Department, 2015
This is a description of a teaching experience. You can use markdown like any other post.