Interview for Data Scientist Role at Cognizant (Questions With Answers From Gemini - 2024 Apr 18)

Q1: Tell about yourself and your technical expertise.

Q2: How much would you rate yourself in Machine Learning?

Q3: What are the assumptions in Linear Regression?

Linear regression relies on several key assumptions to ensure the accuracy and validity of the model. Here are the main ones:


Linearity: This is the foundation of linear regression. It assumes there's a straight-line relationship between the independent variable (predictor) and the dependent variable (response).


Independence of Errors: The errors, or residuals, in your data should be independent of each other. This means the error for one observation shouldn't influence the error for another.


Homoscedasticity: This refers to constant variance of the errors. In simpler terms, the spread of the residuals should be consistent across all levels of the independent variable.


Normality of Errors: Ideally, the errors should follow a normal distribution (bell-shaped curve). This allows for reliable statistical tests based on the assumption.


No Multicollinearity: The independent variables shouldn't be highly correlated with each other. Multicollinearity can cause problems with interpreting the coefficients of the model.


No Endogeneity: There shouldn't be a causal relationship between the errors and the independent variables. This means the errors are due to random factors and not influenced by the independent variables themselves.

It's important to check these assumptions before interpreting the results of a linear regression analysis. Various diagnostic plots and tests can help you assess if these assumptions are met in your data. If not, there might be ways to address the issues through data transformation or alternative modeling techniques.

Hint: How about the acronym LINER?

The acronym LINER is a handy way to remember the key assumptions for making inferences in linear regression. Here's how each letter corresponds to an assumption:


L - Linearity:  This refers to the assumption that there's a straight-line relationship between the independent and dependent variables. You can check for this by visually inspecting a scatterplot of your data.


I - Independence:  This signifies that the errors (residuals) in your data are independent of each other. There shouldn't be any correlation between the error terms.


N - Normality:  This assumption refers to the normality of the errors. Ideally, the errors should follow a normal distribution (bell-shaped curve). This allows for reliable statistical tests based on this assumption.


E - Equal variance (Homoscedasticity):  This signifies that the variance of the errors is constant across all levels of the independent variable. The spread of the residuals should be consistent.


R - Random errors (No Endogeneity):  This refers to the absence of a causal relationship between the errors and the independent variables. The errors should be due to random factors and not influenced by the independent variables themselves.

By remembering LINER, you can quickly recall the key assumptions to consider when interpreting the results of a linear regression analysis.

Q4: What is the difference between decision tree and random forest?

Here's a breakdown of the key differences between decision trees and random forests:
Structure:


Decision Tree: A single tree-like model where each node represents a decision based on a feature, and the branches represent the outcome of those decisions. It follows a yes/no branching structure to arrive at a final prediction.


Random Forest: An ensemble of decision trees. Imagine a forest instead of a single tree. Each tree contributes to the final prediction, making it more robust.

Training Process:


Decision Tree: A single tree is grown based on the entire dataset. The tree splits the data at each node using the best split criteria (e.g., information gain) to minimize impurity (improve prediction accuracy) within its branches.


Random Forest: Trains multiple decision trees. Each tree uses a random subset of features and data points (bootstrapping) to grow, reducing the influence of any single feature or data point on the overall model.

Prediction:


Decision Tree: Makes a prediction by traversing the tree based on the input data, following the splits until reaching a leaf node (terminal decision).


Random Forest: Averages the predictions from all the trees in the forest for a more robust prediction.

General Performance:


Decision Tree: Easier to interpret due to its clear tree structure. Can be prone to overfitting, especially with complex datasets, as small changes in the data can significantly alter the tree structure.


Random Forest: Less interpretable due to the complex interaction of multiple trees. Generally more accurate and robust than a single decision tree due to averaging and reduced overfitting.

Choosing the Right Model:


Decision Tree: Preferable when interpretability is a priority, or when dealing with smaller datasets. Can be a good first model to get a basic understanding of the data.


Random Forest: Ideal for higher accuracy and robustness, especially for complex datasets. Often the go-to choice for various machine learning tasks due to its effectiveness.



Q5: What is Gini coefficient in context of modeling a decision tree?


In the context of decision trees, the Gini coefficient (or Gini impurity) is a measure of how well a specific split separates the data according to the target variable (what you're trying to predict). It essentially calculates the probability of a random draw from the data set landing in a different class than the prevailing class at that node.
Here's how it works:

Imagine you have a dataset with two classes (like yes/no or positive/negative).
At each node of the decision tree, you're considering splitting the data based on a particular feature.
The Gini coefficient calculates the probability of misclassifying a data point after the split.
A perfectly pure split would have a Gini coefficient of 0, meaning all data points belong to the same class after the split.
Conversely, a completely random split would have a Gini coefficient close to 1 (perfectly mixed).
The decision tree algorithm aims to find the split at each node that minimizes the Gini coefficient, essentially creating the most homogeneous groups (based on the target variable) at each step. This leads to a more accurate decision tree overall.
Here are some additional points to consider:

There are other impurity measures used in decision trees besides Gini impurity, such as information gain. The choice of measure can sometimes affect the structure of the resulting tree.
The Gini coefficient is relatively simple to calculate and interpret, making it a popular choice for decision tree algorithms.


Q6: What is residual?

In statistics, a residual refers to the difference between an observed value and a predicted value for a particular data point. It's essentially a measure of how far away the actual data point falls from the line of best fit (in regression analysis) or a predicted value from a model.
Here's a breakdown:

Observed Value: This is the actual data point you have for a specific variable you're measuring.
Predicted Value: This is the value estimated or forecasted by a model (e.g., linear regression line) for that data point.
Residual: This is the difference between the observed value and the predicted value. A positive residual indicates the observed value was higher than the predicted value, and a negative residual indicates the observed value was lower than the predicted value.
Residuals play a crucial role in assessing the quality of a model. Here's why they're important:

Evaluating Model Fit: By analyzing the distribution of residuals, we can see how well the model captures the underlying trend in the data. Ideally, the residuals should be randomly scattered around zero, indicating no consistent pattern of overestimation or underestimation by the model.
Identifying Outliers: Large residuals (positive or negative) can highlight potential outliers in the data. These outliers may require further investigation or might need to be handled appropriately depending on the analysis.
Improving the Model: Analyzing residuals can help identify areas where the model can be improved. If there are patterns in the residuals, it might suggest the model needs adjustments or a different model type might be more suitable for the data.
Here are some additional points to note:

Residuals are also sometimes referred to as errors.
They are calculated by subtracting the predicted value from the observed value for each data point.
Visualizing residuals through histograms or scatter plots is a common way to analyze their distribution and identify potential issues.


Q7: What is RMSE?

RMSE stands for Root Mean Squared Error. It's a common metric used to evaluate the performance of prediction models, especially in regression analysis. It tells you how far off, on average, the predictions of your model are from the actual values.
Here's how it works:

Calculate the residuals: For each data point, you find the difference between the predicted value and the actual value (observed value). This difference is the residual.
Square the residuals: Squaring the residuals ensures that positive and negative errors are treated equally.
Find the mean of the squared residuals: This essentially averages the squared errors, giving an idea of the average amount of error in your predictions.
Take the square root: The final step is to take the square root of the mean squared error. This transforms the units back to the original scale of your data, making the RMSE easier to interpret.
Lower RMSE indicates a better model:

A perfect score of 0 would indicate that your model perfectly predicts all the values, with no errors.
In practice, a RMSE of 0 is unlikely. However, lower RMSE values generally signify better model performance, meaning your predictions are, on average, closer to the actual values.
Here are some additional points to consider about RMSE:

RMSE is scale-dependent: Since it uses the same units as the data, it can be affected by the scale of your variables. For example, an RMSE of 10 might be significant for predicting housing prices, but less concerning for predicting daily temperatures.
It penalizes large errors more severely: Squaring the residuals gives more weight to larger errors in the final calculation. This can be helpful in some cases, but it's important to be aware of this property.
Not always the best choice: RMSE might not be the best metric for all situations. Depending on the context and the nature of your data, other metrics like mean absolute error (MAE) might be more suitable.
Overall, RMSE is a widely used and straightforward metric for evaluating prediction models, especially in regression analysis. It provides a good idea of how well your model's predictions match the actual values.

Q8: In which use cases, you should check RMSE and in which use case you should check MAE?

Here's a breakdown of when to use RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) for evaluating prediction models:
RMSE (Root Mean Squared Error):
Use Cases:

Focus on Larger Errors: When large errors are particularly undesirable in your predictions, RMSE is a good choice. Squaring the residuals in the calculation gives more weight to larger errors, making them stand out more significantly. This can be crucial in applications where big mistakes are more costly or impactful.
Continuous and Scale-Compatible Data: RMSE works well with continuous data (like temperature or sales figures) where the scale of the data is relevant. The units of RMSE are the same as the data, making it easier to interpret the error magnitude in that context.
Disadvantages:

Sensitive to Outliers: Similar to residuals, RMSE can be significantly affected by outliers since large errors are squared. If outliers are a major concern, MAE might be a better option.
Not Ideal for Large Error Ranges: When the range of your data's values is vast, RMSE might be misleading. Squaring errors can disproportionately amplify the impact of larger values on the final score.
MAE (Mean Absolute Error):
Use Cases:

Outliers are Present: When your data might have outliers that could skew the RMSE value, MAE is a more robust choice. It takes the absolute value of the differences, making it less influenced by extreme values.
Focus on Average Error Magnitude: If you simply want a general idea of how far off your predictions are on average, in the units of your data, MAE provides a clear and interpretable metric.
Cost-Sensitive Applications: In some scenarios, the cost of errors might not be symmetrical. For instance, underestimating sales by a small margin might be less concerning than overestimating production costs by a large amount. MAE can be a better choice in such cases where some errors are more impactful.
Disadvantages:

Doesn't Punish Large Errors:  Since MAE uses absolute values, all errors are weighted equally. This can be a drawback if large errors are particularly undesirable in your application.
Less Informative for Individual Errors: Unlike residuals, MAE doesn't provide detailed information about the distribution of errors for each data point.
Choosing Between RMSE and MAE:

If large errors are a major concern and the data is continuous with a relevant scale, RMSE is a good choice.
If outliers are a concern or you simply need a general idea of the average error in the data's units, use MAE.
For cost-sensitive applications where some errors have a greater impact, MAE might be more suitable.
In many cases, it's beneficial to use both RMSE and MAE together.  This gives you a more comprehensive picture of your model's performance. Analyze the residuals to understand the distribution of errors and identify potential issues, and then use both metrics (RMSE for larger error focus and MAE for overall error magnitude) to get a well-rounded assessment.


Q9: You have an outlier. How do you check for outliers? And how do you handle outliers in the dataset?

Q10: Let's say you're using a clustering algorithm. What all clustering algorithms do you know?

Q11: In kMeans, how do you choose the value for k that represents the number of clusters?


Choosing the optimal value of k (number of clusters) in k-means clustering can be a challenge, as there's no single definitive method. Here are some common approaches to help you decide:
Elbow Method:

This is a popular visual technique. You calculate the Within-Cluster Sum of Squares (WCSS) for a range of k values (e.g., 1 to 10). WCSS represents the total squared distance of each data point to its assigned cluster center.
Plot the WCSS on the y-axis and the number of clusters (k) on the x-axis.
As k increases, WCSS will continue to decrease (since you're essentially creating more clusters and fitting the data more closely). The elbow method suggests choosing the value of k at the "elbow" of the curve. This is the point where the rate of decrease in WCSS starts to slow down significantly. It's a good indication that adding more clusters isn't giving you much benefit in terms of reducing overall error.
Silhouette Analysis:

This method considers how well each data point is assigned to its cluster. It calculates a silhouette coefficient for each data point, ranging from -1 to 1.
A silhouette coefficient closer to 1 indicates a good assignment (the data point is far away from points in other clusters). A value closer to -1 suggests the data point might be better suited to a different cluster.
The average silhouette coefficient across all data points can be calculated for different k values. You can choose the k that leads to the highest average silhouette coefficient, indicating a better overall clustering structure.
Gap Statistic:

This is a more statistical approach that compares the WCSS of your data to the WCSS of randomly shuffled data. It essentially measures how well your clusters separate the data compared to a random distribution.
The gap statistic is calculated for a range of k values. You choose the k where the gap statistic is furthest above the expected gap under a null hypothesis of random data distribution.
Additional Considerations:

Domain Knowledge:  Incorporate your understanding of the data and the problem you're trying to solve. The optimal number of clusters might be based on some inherent structure or groupings within the data domain.
Evaluation Metrics: Depending on your specific application, you might have specific evaluation metrics (beyond WCSS or silhouette coefficients) that can help assess how well different k values perform.
In conclusion:
There's no guaranteed "best" method for choosing k in k-means. It's often a combination of techniques and some trial and error. The elbow method and silhouette analysis are good starting points due to their visual nature. The gap statistic offers a more formal approach. Consider your domain knowledge and the specific goals of your clustering task to make an informed decision.

Q12: What are the different metrics you know to measure the performance of a model on classification task?

Q13: What are the different metrics you know to measure the performance of a model on regression task?

Q14: What are the different metrics you know to measure the performance of a model on clustering task?

Q15: What is F1 score?

Q16: What is precision?

Q17: What is OCR?

Q18: What are the different types of files you can upload for OCR?

Q19: Which languages are supported by OCR?

Q20: How do you handle noise in images in OCR?

Q21: How do you deal with skew correction in computer vision?

Q22: Do you have any idea on character segmentation in OCR?

Q23: Can we detect hand written characters with OCR?

Q24: How much would you rate yourself in NLP?

Q25: What is the difference between lemmatization and stemming?

Q26: What is embedding?

Q27: Can you tell me how BERT model works?


BERT, which stands for Bidirectional Encoder Representations from Transformers, is a powerful pre-trained language model developed by Google. Here's a breakdown of its key functionalities:
Understanding the Building Block: Transformer:

BERT leverages the Transformer architecture, a deep learning model that excels at natural language processing tasks.
Unlike traditional models that process text sequentially (left-to-right or right-to-left), the Transformer can analyze the entire sentence at once, considering all the words and their relationships simultaneously. This is achieved through an attention mechanism that allows the model to focus on specific parts of the sentence relevant to the current word being processed.
Core Functions of BERT:


Pre-training on Unlabeled Data:

BERT is first pre-trained on a massive dataset of text and code, like books and Wikipedia articles. This pre-training doesn't involve any specific task or labeled data. The model learns general-purpose representations of words and how they're used in context.



Masked Language Modeling (MLM):

A key aspect of pre-training involves a technique called Masked Language Modeling (MLM). Here, some words in the training data are randomly masked (replaced with a special token). BERT then tries to predict the masked words based on the surrounding context. This helps the model understand the meaning and relationships between words.



Next Sentence Prediction (NSP):

Another pre-training strategy used is Next Sentence Prediction (NSP). BERT is given pairs of sentences and needs to predict if the second sentence is the actual following sentence in the original document. This helps the model understand the logical flow and coherence between sentences.


Fine-tuning for Specific Tasks:

Once pre-trained, BERT can be fine-tuned for various natural language processing tasks. This involves adding a small layer on top of the pre-trained BERT model and training it on a specific labeled dataset relevant to the desired task (e.g., sentiment analysis, question answering, text summarization).
Benefits of BERT:

BERT's pre-trained nature allows it to achieve state-of-the-art performance on many NLP tasks with minimal fine-tuning compared to training a model from scratch.
By considering the entire sentence at once and using the attention mechanism, BERT can capture complex relationships between words and context, leading to better understanding of language.
Overall, BERT is a powerful pre-trained language model that has revolutionized the field of NLP. Its ability to learn general-purpose language representations and then adapt to specific tasks makes it a valuable tool for various applications.


Q28: What is cosine similarity?

Q29: You want to read a PDF document, what are the ways of reading a PDF document?

Q30: You are given a sentence "Ashish Jain lives in Bengaluru."

You detect Ashish is a person and Bengaluru is a place. 
What kind of problem is this?

Q31: How does an NER model works? What algorithm are there for an NER model? 

Named Entity Recognition (NER) is a subfield of Natural Language Processing (NLP) tasked with identifying and classifying specific entities within text data. These entities can be people, organizations, locations, dates, monetary values, and other predefined categories. Here's a breakdown of how NER models typically work and the algorithms commonly used:
NER Workflow:


Text Preprocessing:

The raw text data undergoes preprocessing steps like tokenization (splitting text into words or characters) and normalization (handling lowercase/uppercase and special characters).



Feature Engineering:

Features are extracted from the text that might be helpful for identifying entities. These features can include word n-grams (sequences of words), part-of-speech tags, prefixes/suffixes, or word embeddings (numerical representations capturing semantic similarities).



Sequence Labeling:

The core of NER is the sequence labeling step. Here, the model assigns a label (e.g., "PER" for person, "LOC" for location) to each word or token in the sentence. This essentially predicts the entity type for each word in the sequence.


Common Algorithms for NER:


Rule-based NER:

This traditional approach relies on manually defined rules and patterns to identify entities. These rules consider linguistic features like capitalization, gazetteers (lists of known entities), and part-of-speech tags. While interpretable, rule-based systems can be labor-intensive to create and maintain, especially for complex scenarios.



Statistical NER:

This method leverages statistical models like Hidden Markov Models (HMMs) or Conditional Random Fields (CRFs) to predict entity labels. The model is trained on annotated data where each word is labeled with its corresponding entity type. During prediction, the model considers the sequence of words and their features to assign the most likely entity label to each word.



Neural Network-based NER:

Deep learning approaches using recurrent neural networks (RNNs) or convolutional neural networks (CNNs) are increasingly popular for NER. These models can learn complex patterns and relationships within the text data to identify entities. They are often pre-trained on large amounts of unlabeled text data and then fine-tuned on labeled NER datasets for specific tasks.


Choosing the Right Algorithm:
The choice of algorithm for NER depends on factors like:

Data availability:  Large datasets are often needed for training complex neural network models.
Task complexity:  For simpler NER tasks, rule-based or statistical methods might suffice. Deep learning approaches excel with intricate entity types or large amounts of data.
Interpretability:  Rule-based models offer clear explanations for entity recognition, while deep learning models can be more like black boxes.
Overall, NER models leverage various algorithms to tackle the task of identifying and classifying named entities within text data. By understanding the workflow and common approaches, you can select the most suitable method for your specific NER task.

Q32: How much would you rate yourself in Deep Learning?

Q33: What is an encoder and decoder as in a transformer?

Q34: What is the difference between Sigmoid and ReLU activation function?

Q35: In which scenario will you use Sigmoid and in which scenario would you use ReLU? 

Q36: What is k-fold cross validation?

Q37: Can you tell me the architecture of LSTM?

Q38: What all gates are there in LSTM?

Q39: What do Azure ML Studio do?

Q40: Have you deployed anything on Azure ML Studio?

Q41: How much you want rate yourself in Python?

Q42: De-duplicate this list of numbers [3, 5, 1, 5, 4, 1, 1, 2, 2, 3, 3] without using built-in functions.


# l = list(set(l))
# print(l)

l = [3, 5, 1, 5, 4, 1, 1, 2, 2, 3, 3]

l = sorted(l)
m = [-1]
# for i in l:
#     if i > m[-1]:
#         m.append(i)

_ = [m.append(i) for i in l if i > m[-1]]

m = m[1:]
print(m)

# --- --- --- --- ---

Q43: How would you replace all occurrences of a letter from a string.

name = 'Ashish Jain'

# replace A/a with e

import re

name = re.sub('[a]', 'e', name)
print(name)