Showing posts with label Data Analytics. Show all posts
Showing posts with label Data Analytics. Show all posts

Sunday, October 1, 2023

What are the Data Science Roles?

1. Data Strategist

Ideally, before a company collects any data, it hires a data strategist—a senior professional who understands how data can create value for businesses.

According to the famous data strategist Bernard Marr, there are four main ways how companies in different fields can use data science:

They can make data-driven decisions.

Data can help create smarter products and services.

Companies can use data to improve business processes.

They could create a new revenue stream via data monetization.

Companies often outsource such data roles. They hire external consultants to devise a plan that aligns with the organizational strategy.

Once a firm has a data strategy in place, it is time to ensure data availability. This is when a data architect comes into play.

2. Data Architect

A data architect (or data modeler) plans out high-level database structures. This involves the planning, organization, and management of information within a firm, ensuring its accuracy and accessibility. In addition, they must assess the needs of business stakeholders and optimize schemas to address them.

Such data roles are of crucial importance. Without proper data architecture, key business questions may remain answered due to the lack of coherence between different tables in the database.

A data architect is a senior professional and often a consultant. To become one, you'd need a solid resume and rigorous preparation for the interview process.

3. Data Engineer

The role of data engineers and data architects often overlaps—especially in smaller businesses. But there are key differences.

Data engineers build the infrastructure, organize tables, and set up the data to match the use cases defined by the architect. What's more, they handle the so-called ETL process, which stands for Extract, Transform, and Load. This involves retrieving data, processing it in a usable format, and moving it to a repository (the firm's database). Simply put, they pipe data into tables correctly.

Typically, they receive many ad-hoc ETL-related tasks throughout their work but rarely interact with business stakeholders directly. This is one of the best-paid data scientist roles, and for good reason. You need a plethora of skills to work in this position, including software engineering.

Okay, let's recap.

As you can see, the jobs in data science are interlinked and complement each other, but each position has slightly different requirements. First come data strategists who define how data can serve business goals. Next, the architect plans the database schemas necessary to achieve the objectives. Lastly, the engineers build the infrastructure and pipe the data into tables.

4. Data Analyst

Data analysts explore, clean, analyze, visualize, and present information, providing valuable insights for the business. They typically use SQL to access the database.

Next, they leverage an object-oriented programming language like Python or R to clean and analyze data and rely on visualization tools, such as Power BI or Tableau, to present the findings.

Side note: What is Data analysis?

Data analysis is the process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. Data analysis has multiple facets and approaches, encompassing diverse techniques under a variety of names, and is used in different business, science, and social science domains. In today's business world, data analysis plays a role in making decisions more scientific and helping businesses operate more effectively.

Data mining is a particular data analysis technique that focuses on statistical modeling and knowledge discovery for predictive rather than purely descriptive purposes, while business intelligence covers data analysis that relies heavily on aggregation, focusing mainly on business information. In statistical applications, data analysis can be divided into descriptive statistics, exploratory data analysis (EDA), and confirmatory data analysis (CDA). EDA focuses on discovering new features in the data while CDA focuses on confirming or falsifying existing hypotheses. Predictive analytics focuses on the application of statistical models for predictive forecasting or classification, while text analytics applies statistical, linguistic, and structural techniques to extract and classify information from textual sources, a species of unstructured data. All of the above are varieties of data analysis.

Data integration is a precursor to data analysis, and data analysis is closely linked to data visualization and data dissemination.

5. Business Intelligence Analyst

Data analyst's and BI analyst's duties overlap to a certain extent, but the latter has more of a reporting role. Their main focus is on building meaningful reports and dashboards and updating them frequently. More importantly, they have to satisfy stakeholders' informational needs at different levels of the organization.

6. Data Scientist

A data scientist has the skills of a data analyst but can leverage machine and deep learning to create models and make predictions based on past data.

We can distinguish three main types of data scientists:

# Traditional data scientists

# Research scientists

# Applied scientists

A traditional data scientist does all sorts of tasks, including data exploration, advanced statistical modeling, experimentation via A/B testing, and building and tuning machine learning models.

Research scientists primarily work on developing new machine learning models for large companies.

Applied scientists—frequently hired in big tech and larger companies—boast one of the highest-paid jobs in data science. These specialists combine data science and software engineering skills to productionize models.

More prominent companies prefer this combined skillset because it allows one person to oversee the entire ML implementation process—from the model building until productionization—which leads to quicker results. An applied scientist can work with data, model it for machine learning, select the correct algorithm, train the model, fine-tune hyperparameters, and then put the model in production.

As you can see, there's a significant overlap between data scientists, data analysts, and BI analysts. The image below is a simplified illustration of the similarities and differences between these data science roles.

7. ML Ops Engineer

Companies that don't have applied scientists hire ML Ops engineers. They are responsible for putting the ML models prepared by traditional data scientists into production.

In many instances, ML Ops engineers are former data scientists who have developed an engineering skillset. Their main responsibilities are to put the ML model in production and fix it if something breaks.

8. Data Product Manager

The last role we discuss in this article is that of а product manager. The person in this position is accountable for the success of a data product. They consider the bigger picture, identifying what product needs to be created, when to build it, and what resources are necessary.

A significant focus of such data science roles is data availability—determining whether to collect data internally or find ways to acquire it externally. Ultimately, product managers strategize the most effective ways to execute the production process.

Tags: Technology,Data Analytics,Interview Preparation

What is Data Science Lifecycle?

A data science lifecycle indicates the iterative steps taken to build, deliver and maintain any data science product. All data science projects are not built the same, so their life cycle varies as well. Still, we can picture a general lifecycle that includes some of the most common data science steps. A general data science lifecycle process includes the use of machine learning algorithms and statistical practices that result in better prediction models. Some of the most common data science steps involved in the entire process are data extraction, preparation, cleansing, modeling, evaluation etc. The world of data science refers to this general process as the “Cross Industry Standard Process for Data Mining”.

Who Are Involved in The Projects?

Domain Expert : The data science projects are applied in different domains or industries of real life like Banking, Healthcare, Petroleum industry etc. A domain expert is a person who has experience working in a particular domain and knows in and out about the domain.

Business analyst : A business analyst is required to understand the business needs in the domain identified. The person can guide in devising the right solution and timeline for the same.

Data Scientist : A data scientist is an expert in data science projects and has experience working with data and can work out the solution as to what data is needed to produce the required solution.

Machine Learning Engineer : A machine learning engineer can advise on which model to be applied to get the desired output and devise a solution to produce the correct and required output.

Data Engineer and Architect : Data architects and Data engineers are the experts in the modeling of data. Visualization of data for better understanding, as well as storage and efficient retrieval of data, are looked after by them.

The Lifecycle of Data Science

1. Problem identification

This is the crucial step in any Data Science project. The first thing is understanding in what way Data Science is useful in the domain under consideration and identification of appropriate tasks which are useful for the same. Domain experts and Data Scientists are the key persons in the problem identification of problem. Domain expert has in depth knowledge of the application domain and exactly what the problem is to be solved. Data Scientist understands the domain and help in identification of problem and possible solutions to the problems.

2. Business Understanding

Understanding what customer exactly wants from the business perspective is nothing but Business Understanding. Whether customer wish to do predictions or want to improve sales or minimize the loss or optimize any particular process etc forms the business goals. During business understanding, two important steps are followed:

KPI (Key Performance Indicator)

For any data science project, key performance indicators define the performance or success of the project. There is a need to be an agreement between the customer and data science project team on Business related indicators and related data science project goals. Depending on the business need the business indicators are devised and then accordingly the data science project team decides the goals and indicators. To better understand this let us see an example. Suppose the business need is to optimise the overall spendings of the company, then the data science goal will be to use the existing resources to manage double the clients. Defining the Key performance Indicators is very crucial for any data science projects as the cost of the solutions will be different for different goals.

SLA (Service Level Agreement)

Once the performance indicators are set then finalizing the service level agreement is important. As per the business goals the service level agreement terms are decided. For example, for any airline reservation system simultaneous processing of say 1000 users is required. Then the product must satisfy this service requirement is the part of service level agreement.

Once the performance indicators are agreed and service level agreement is completed then the project proceeds to the next important step.

3. Collecting Data

Data Collection is the important step as it forms the important base to achieve targeted business goals. There are various ways the data will flow into the system as listed below:

1. Surveys

2. Social media

3. Archives

4. Transactional data

5. Enterprise data

6. Statistical methods

4. Pre-processing data

Large data is collected from archives, daily transactions and intermediate records. The data is available in various formats and in various forms. Some data may be available in hard copy formats also. The data is scattered at various places on various servers. All these data are extracted and converted into single format and then processed. Typically, as data warehouse is constructed where the Extract, Transform and Loading (ETL) process or operations are carried out. In the data science project this ETL operation is vital and important. A data architect role is important in this stage who decides the structure of data warehouse and perform the steps of ETL operations.

5. Analyzing data

Now that the data is available and ready in the format required then next important step is to understand the data in depth. This understanding comes from analysis of data using various statistical tools available. A data engineer plays a vital role in analysis of data. This step is also called as Exploratory Data Analysis (EDA). Here the data is examined by formulating the various statistical functions and dependent and independent variables or features are identified. Careful analysis of data revels which data or features are important and what is the spread of data. Various plots are utilized to visualize the data for better understanding. The tools like Tableau, PowerBI etc are famous for performing Exploratory Data Analysis and Visualization. Knowledge of Data Science with Python and R is important for performing EDA on any type of data.

6. Data Modelling

Data modelling is the important next step once the data is analysed and visualized. The important components are retained in the dataset and thus data is further refined. Now the important is to decide how to model the data? What tasks are suitable for modelling? The tasks, like classification or regression, which is suitable is dependent upon what business value is required. In these tasks also many ways of modelling are available. The Machine Learning engineer applies various algorithms to the data and generates the output. While modelling the data many a times the models are first tested on dummy data similar to actual data.

7. Model Evaluation/ Monitoring

As there are various ways to model the data so it is important to decide which one is effective. For that model evaluation and monitoring phase is very crucial and important. The model is now tested with actual data. The data may be very few and in that case the output is monitored for improvement. There may be changes in data while model is being evaluated or tested and the output will drastically change depending on changes in data. So, while evaluating the model following two phases are important:

Data Drift Analysis

Changes in input data is called as data drift. Data drift is common phenomenon in data science as depending on the situation there will be changes in data. Analysis of this change is called Data Drift Analysis. The accuracy of the model depends on how well it handles this data drift. The changes in data are majorly because of change in statistical properties of data.

Model Drift Analysis

To discover the data drift machine learning techniques can be used. Also, more sophisticated methods like Adaptive Windowing, Page Hinkley etc. are available for use. Modelling Drift Analysis is important as we all know change is constant. Incremental learning also can be used effectively where the model is exposed to new data incrementally.

8. Model Training

Once the task and the model are finalised and data drift analysis modelling is finalized then the important step is to train the model. The training can be done is phases where the important parameters can be further fine tuned to get the required accurate output. The model is exposed to the actual data in production phase and output is monitored.

9. Model Deployment

Once the model is trained with the actual data and parameters are fine tuned then model is deployed. Now the model is exposed to real time data flowing into the system and output is generated. The model can be deployed as web service or as an embedded application in edge or mobile application. This is very important step as now model is exposed to real world.

10. Driving insights and generating BI reports

After model deployment in real world, the next step is to find out how the model is behaving in real-world scenario. The model is used to get insights that aid in strategic decisions related to business. The business goals are bound to these insights. Various reports are generated to see how business is driving. These reports help in finding out if key process indicators are achieved or not.

11. Taking a decision based on insight

For data science to do wonders, every step indicated above has to be done very carefully and accurately. When the steps are followed properly, then the reports generated in the above step help in making key decisions for the organization. The insights generated help in taking strategic decisions for example, the organization can predict that there will be a need for raw materials in advance. Data science can be of great help in making many important decisions related to business growth and better revenue generation.

Tags: Interview Preparation,Data Analytics,

Friday, August 4, 2023

Mapping the AI Finance Services Roadmap: Enhancing the Financial Landscape

Introduction

Artificial Intelligence (AI) has rapidly transformed the financial services industry, revolutionizing how we manage money, make investments, and access personalized financial advice. From robo-advisors to AI-driven risk management, the potential for AI in finance services is boundless. In this article, we'll navigate the AI Finance Services Roadmap, exploring the key milestones and opportunities that are reshaping the financial landscape and empowering consumers and businesses alike.



The Development of AI in the Financial Industry


Step 1: Personalized Financial Planning with Robo-Advisors

Robo-advisors have emerged as a revolutionary AI-powered tool that democratizes access to sophisticated financial planning. These platforms use AI algorithms to analyze an individual's financial situation, risk tolerance, and goals, enabling the creation of personalized investment portfolios. With lower fees and greater convenience, robo-advisors are transforming how we plan for our financial future.


Step 2: AI-Driven Credit Scoring and Lending

AI has revolutionized the lending process by introducing more efficient and accurate credit scoring models. By analyzing vast amounts of data, including transaction history, social media behavior, and online presence, AI algorithms can assess creditworthiness more effectively. This has opened up new avenues for individuals and businesses to access loans and credit facilities.


Step 3: Fraud Detection and Cybersecurity

The financial services industry faces persistent threats from cybercriminals. AI-based fraud detection systems can analyze vast data streams in real time, detecting suspicious activities and protecting against potential threats. By bolstering cybersecurity measures with AI, financial institutions can safeguard sensitive customer information and maintain trust in their services.


Step 4: AI-Powered Virtual Assistants

AI virtual assistants are reshaping customer interactions in the finance sector. These intelligent chatbots provide personalized support, answer inquiries, and perform routine tasks, enhancing the overall customer experience. By automating these processes, financial institutions can improve efficiency and focus on delivering high-value services to their clients.


Step 5: AI for Compliance and Regulatory Reporting

Compliance and regulatory reporting are critical aspects of the financial services industry. AI technologies can streamline these processes, ensuring adherence to complex regulations and reporting requirements. AI-driven solutions can identify potential compliance issues and proactively address them, reducing the risk of costly penalties and reputational damage.


Step 6: AI-Enhanced Risk Management

AI-powered risk management solutions provide more accurate and real-time risk assessment. These tools analyze historical data and market trends, enabling financial institutions to identify potential risks and make data-driven decisions. Enhanced risk management fosters stability and resilience, even in volatile market conditions.

Conclusion

The AI Finance Services Roadmap is shaping a future where financial services are more accessible, personalized, and secure than ever before. From robo-advisors offering tailored investment strategies to AI-driven fraud detection systems protecting against cyber threats, the transformative power of AI is revolutionizing the financial landscape. As we continue to innovate and embrace AI technologies, the potential for growth, efficiency, and customer satisfaction in the financial services industry is limitless. By navigating the AI Finance Services Roadmap, we can ensure a prosperous and inclusive financial future for individuals and businesses worldwide.

Overall, the AI finance services roadmap is promising. AI has the potential to improve efficiency, accuracy, and customer experience in the financial industry. However, there are also some challenges that need to be addressed before AI can be fully adopted in the financial sector.

I hope this article was helpful. If you have any questions, please feel free to leave a comment below.

Monday, July 24, 2023

Smoothing (Part of Data Analytics Course)

Highlighting Algorithm Followed For Smoothing The Data

1. Decide which kind of binning you want to use?

- Equal frequency

- Equal width

2. Once you have binned the data, you have to decide whether you are going to assign the bin with a value from:

- mean

- median

- boundary

3. Replace each bin value by the formula selected in Step 2.

Smoothing(noisy data)

Suppose a group of 12 sales price records has been sorted as follows:

5, 10, 11, 13, 15, 35, 50, 55, 72, 92, 204, 215

Partition them into three bins by each of the following methods.

Equal-frequency partitioning

What is Smoothing by bin mean/median/boundary?







How do we define the first bin?

We need a bin that encloses 5, 10 and 11.

(4.5, 11.5]: This is also correct but let’s look at Pandas.

What Pandas has created is:

(4.999, 12.5]: Range exlusive of 4.999 and starting from there. Also range inclusive of 12.5 and ending there.

Is it wrong? No.

Next bin:

(12.5, 42.5]: Is it wrapping the elements 13, 15 and 35?

Next bin would start at 42.5. Can we say this?

Smoothing(noisy data)

Suppose a group of 12 sales price records has been sorted as follows:

5, 10, 11, 13, 15, 35, 50, 55, 72, 92, 204, 215

Partition them into three bins by each of the following methods.

Equal-frequency partitioning

What is Smoothing by bin mean/median/boundary?

Replace each bin value is replaced by mean/median/nearest boundary

On smoothing by bin-boundary (bins follow equal-frequency partitioning):

Bin 1: 5, 13, 13, 13

As 5 is closer to boundary value ‘5’. And, 10, 11 are closer to boundary value ‘13’

Bin 2: 15, 15, 55, 55

Bin 3: 72, 72, 215, 215

Original:

Smoothing by equal-frequency binning using the mean of each bin

1. creation of bins 
In code: pd.qcut()
2. grouping the data according to bins In code: df.groupby()
3. find the mean of each group In code: df.groupby().mean()
4. create a map of bin labels and mean values In code: it is essentially a dictionary that looks like this: { '(4.999, 14.333]': 9.75, '(14.333, 60.667]': 38.75, '(60.667, 215.0]': 145.75 } A dictionary is simply key-value pairs. 5. Populate a new column containing the mean of each bin for each data point.

Smoothing (noisy data)

Suppose a group of 12 sales price records has been sorted as follows:

5, 10, 11, 13, 15, 35, 50, 55, 72, 92, 204, 215

Partition them into three bins by each of the following methods.

Equal-width partitioning

The width of each interval is (215 - 5)/3 = 70.

Perform Smoothing by bin mean/median/boundary.

Bins using equal width partitioning.

Elements: 5, 10, 11, 13, 15, 35, 50, 55, 72, 92, 204, 215

The width of each interval is (215 – 5)/3 = 70.

Domain for bin-1: 5 up to, but not, 75 (= 5 + 70)

Domain for bin-2: 75 to 144

Domain for bin-3: 145 Onwards (inc. 215 from the input data set)

Data Analytics Course Using Python (Index)

Chapters

Chapters Pending Publication

  • (6) decision tree
  • (7) measuring classification
  • (8) linear regression between two variables
  • (9) categorical encoding
  • (10) naive bayes classifier
  • (11) logistic regression
  • (12) support vector machines
  • (13) one-vs-rest and one-vs-one strategies for multiclass classification
  • (14) generalization and overfitting
  • (15) kNN classification
  • (16) clustering and kmeans
  • (17) clustering (kmeans)
  • (18) clustering (agglomerative hierarchical clustering)
  • (19) clustering - dbscan
  • (20) cluster evaluation
  • (21) random forest
  • (22) five regression algorithms
  • (23) Ensemble techniques
  • Exercises
Tags: Python,Data Analytics,Technology,

Equal Frequency Binning (Part of Data Analytics Course)

import pandas as pd
l = [5, 10, 11, 13, 15, 35, 50, 55, 72, 92, 204, 215]
s = pd.Series(l)
s.nunique()


12


bins = pd.qcut(s, 3) # Equal frequency binning

How do we make data smooth using mean?

df = pd.DataFrame( { 'data': l, 'bins': bins } ) df
t = df.groupby(['bins']).mean(['data']) t
map_of_mean_values = {} for i in t.iterrows(): # type(i[0]): Interval # type(i[1]): Series map_of_mean_values.update({str(i[0]): i[1][0]}) map_of_mean_values {'(4.999, 14.333]': 9.75, '(14.333, 60.667]': 38.75, '(60.667, 215.0]': 145.75} df['bins'] = df['bins'].astype(str) df['smoothed_values'] = df['bins'].apply(lambda x: map_of_mean_values[x]) df

Thursday, June 1, 2023

Ch 3 - Scaling and Normalization

Scaling vs. Normalization: What's the difference?

One of the reasons that it's easy to get confused between scaling and normalization is because the terms are sometimes used interchangeably and, to make it even more confusing, they are very similar! In both cases, you're transforming the values of numeric variables so that the transformed data points have specific helpful properties. The difference is that: % in scaling, you're changing the range of your data, while % in normalization, you're changing the shape of the distribution of your data. Ref: kaggle

Scaling

Why scale data? Example:
Marks1 Marks2 Marks3
Student 1 280 70 60
Student 2 200 60 55
Student 3 270 40 30
Euclidean distance(s1,s2) = 80.77 Euclidean Distance(s1,s3) = 43.58 Euclidean Distance(s2,s3) = 71.58 Euclidean distance between Student-1 and Student-2 is being dominated by the marks1. Same is the case for Euclidean Distance calculation between student-2 and student-3. There also the e. distance is being high because of difference in the marks-1 attribute. If we see the e. distance for student-1 and student-3, there the distance is not high because marks-1 are close to each each other (viz. 280 and 270). Unlike 200 & 270 or 200 & 280.

Why do we have to scale / normalize the input for an artificial neural network?

There are 2 Reasons why we have to Normalize Input Features before Feeding them to Neural Network: Reason 1: If a Feature in the Dataset is big in scale compared to others then this big scaled feature becomes dominating and as a result of that, Predictions of the Neural Network will not be Accurate. Example: In case of Employee Data, if we consider Age and Salary, Age will be a Two Digit Number while Salary can be 7 or 8 Digit (1 Million, etc..). In that Case, Salary will Dominate the Prediction of the Neural Network. But if we Normalize those Features, Values of both the Features will lie in the Range from (0 to 1). Reason 2: Front Propagation of Neural Networks involves the Dot Product of Weights with Input Features. So, if the Values are very high (for Image and Non-Image Data), Calculation of Output takes a lot of Computation Time as well as Memory. Same is the case during Back Propagation. Consequently, Model Converges slowly, if the Inputs are not Normalized. Example: If we perform Image Classification, Size of Image will be very huge, as the Value of each Pixel ranges from 0 to 255. Normalization in this case is very important. Mentioned below are the instances where Normalization is very important: % K-Means % K-Nearest-Neighbours % Principal Component Analysis (PCA) % Gradient Descent Ref: stackoverflow
z-score is 0.67 in both cases. Z-score is understood as how many standard deviations away the point is away from the mean.

What is there is an outlier in the data?

When there is outlier (on the higher side) in the data, MinMaxScaler sets the outlier data point to 1 and squashes all the other points to values near 0. So, if you doubt that there are outliers in the data, don't use MinMaxScaler. Use StandardScaler.

Now in code

Min Max Scaler

Standard Scaler

Q1: Which subject is easy?

Q2: Applying zscore normalization using Scikit Learn on Pandas DataFrame

Q3: Visualizing what minmax scaler and standard scaler do to the data.

Tags: Data Analytics,Technology,Python,