Difficult Conversations. How to Discuss What Matters Most (D. Stone, B. Patton, S. Heen, 2010)

Preface to Second Edition

The book 'Difficult Converstion' has been used to train oil-rig operators in the North Sea, Iñupiat negotiators in the oil-rich Northern Slope of Alaska, and business leaders at Saudi Aramco. It’s been used at the Boston Area Rape Crisis Center and the headquarters and field offices of UN-AIDS. Doctors, nurses, and administrators in hospitals across the United States have used it to deliver better patient care and develop more humane workplaces. Within the U.S. government, it’s distributed at the Department of Justice, the IRS, the Federal Reserve, and the Postal Service. During one administration, the White House made it required reading for its top sixteen hundred political appointees. Law schools, business schools, and colleges assign it, as do high school teachers, life coaches, therapists, and ministers.

The explanation for all of it is just this: people are people.
We have perceptions and thoughts and feelings, and we work and play with other human beings who have their own perceptions, thoughts, and feelings. 

In the present times, the space for difficult conversations has expanded because of the following reasons: 

• Your organization is flat, aligned, and right-sized, but you still can’t stand your boss.

• You fly three thousand miles and drive two hours to visit your elderly widowed father, and the first words out of his mouth are “You’re late!”

• You’ve got four e-mail addresses, two voice-mail accounts, and sit only feet away from your five closest colleagues, but not one of them has found a way to talk to you about what they apparently call your “confrontational style.”

• No matter how hard you try, you can’t seem to get your sales, manufacturing, and product development teams to see themselves as members of the same organization.

~ ~ ~

Relationships that deal productively with the inevitable stresses of life are more durable; people who are willing and able to “stick through the hard parts” emerge with a stronger sense of trust in each other and the relationship, because now they have a track record of having worked through something hard and seen that the relationship survived.

~ ~ ~

The ability to manage difficult conversations effectively is foundational to achieving almost any significant change. And in addition to supporting major change initiatives, these skills are increasingly needed simply to sustain business as usual. The relentless press of competition has forced most businesses to grow in scale to achieve efficiencies and competitive clout. Many industries are now global in reach. At the same time, the need to be responsive to the market – nimble, flexible, adaptive – has driven many organizations to be less hierarchical and to operate in a matrix that introduces more complexity to decision making and the ability to get things done.
This is a recipe for more conflict – and for more difficult conversations.

~ ~ ~

Introduction

A Difficult Conversation Is Anything You Find It Hard to Talk About 

Sexuality, race, gender, politics, and religion come quickly to mind as difficult topics to discuss, and for many of us they are.
Anytime we feel vulnerable or our self-esteem is implicated, when the issues at stake are important and the outcome uncertain, when we care deeply about what is being discussed or about the people with whom we are discussing it, there is potential for us to experience the conversation as difficult.
And, of course, there’s the stuff of everyday life, conversations that feel more ordinary but cause anxiety nonetheless: returning merchandise without a receipt, asking your secretary to do some photocopying, telling the painters not to smoke in the house.

~ ~ ~

The Dilemma: Avoid or Confront, It Seems There Is No Good Path 

Perhaps the neighbors’ dog keeps you up at night.
"Maybe I’ll get used to it," you might think. But then the dog barks again, and you resolve that tomorrow you are going to talk to the neighbors once and for all.
Because at some level we know the truth: If we try to avoid the problem, we’ll feel taken advantage of, our feelings will fester, we’ll wonder why we don’t stick up for ourselves, and we’ll rob the other person of the opportunity to improve things.
We may be rejected or attacked; we might hurt the other person in ways we didn’t intend; and the relationship might suffer.
If you follow the steps presented in this book, you will find difficult conversations becoming easier and causing less anxiety.
In fact, the people we’ve worked with, who have learned new approaches to dealing with their most challenging conversations, report less anxiety and greater effectiveness in all of their conversations.
The problems are complex, and the people involved are not easy to deal with.
But we have discovered that, regardless of context, the things that make difficult conversations difficult, and the errors in thinking and acting that compound those difficulties, are the same.
The people involved may be so emotionally troubled, the stakes so high, or the conflict so intense that a book – or even professional intervention – is unlikely to help.
But first and more important, it will help you understand better what you’re up against and why it makes sense to shift from a “message delivery stance” to a “learning stance.” Only then will you be able to understand and implement the steps of a learning conversation.

~ ~ ~

In studying hundreds of conversations of every kind we have discovered that there is an underlying structure to what’s going on, and understanding this structure, in itself, is a powerful first step in improving how we deal with these conversations.
Everything problematic that two people say, think, and feel in a difficult conversation falls into one of these three “conversations.” And everything in your difficult conversations does too.

1. The “What Happened?” Conversation. 
Most difficult conversations involve disagreement about what has happened or what should happen. Who said what and who did what? Who’s right, who meant what, and who’s to blame?

The “What Happened?” Conversation: What’s the Story Here? 

1.1 The Truth Assumption 

As we argue vociferously for our view, we often fail to question one crucial assumption upon which our whole stance in the conversation is built: I am right, you are wrong. This simple assumption causes endless grief. What am I right about? I am right that you drive too fast. I am right that you are unable to mentor younger colleagues. I am right that your comments at Thanksgiving were inappropriate. I am right that the patient should have received more medication after such a painful operation. I am right that the contractor overcharged me. I am right that I deserve a raise. I am right that the brochure is fine as it is. The number of things I am right about would fill a book.

There’s only one hitch: I am not right.

How could this be so? It seems impossible. Surely I must be right sometimes!

Well, no. The point is this: difficult conversations are almost never about getting the facts right. They are about conflicting perceptions, interpretations, and values. They are not about what a contract states, they are about what a contract means. They are not about which child-rearing book is most popular, they are about which child-rearing book we should follow.
They are not about what is true, they are about what is important.

1.2 The Intention Invention 

The second argument in the “What Happened?” Conversation is over intentions – yours and mine. Did you yell at me to hurt my feelings or merely to emphasize your point? Did you throw my cigarettes out because you’re trying to control my behavior or because you want to help me live up to my commitment to quit? What I think about your intentions will affect how I think about you and, ultimately, how our conversation goes.

The error we make in the realm of intentions is simple but profound : we assume we know the intentions of others when we don’t. Worse still, when we are unsure about someone’s intentions, we too often decide they are bad. The truth is, intentions are invisible. We assume them from other people’s behavior. In other words, we make them up, we invent them. But our invented stories about other people’s intentions are accurate much less often than we think. Why? Because people’s intentions, like so much else in difficult conversations, are complex. Sometimes people act with mixed intentions. Sometimes they act with no intention, or at least none related to us. And sometimes they act on good intentions that nonetheless hurt us. Because our view of others’ intentions (and their views of ours) are so important in difficult conversations, leaping to unfounded assumptions can be a disaster.

1.3 The Blame Frame 

The third error we make in the “What Happened?” Conversation has to do with blame. Most difficult conversations focus significant attention on who’s to blame for the mess we’re in. When the company loses its biggest client, for example, we know that there will shortly ensue a ruthless game of blame roulette. We don’t care where the ball lands, as long as it doesn’t land on us. Personal relationships are no different. Your relationship with your stepmother is strained? She’s to blame. She should stop bugging you about your messy room and the kids you hang out with.

Talking about fault is similar to talking about truth — it produces disagreement, denial, and little learning. It evokes fears of punishment and insists on an either/or answer. Nobody wants to be blamed, especially unfairly, so our energy goes into defending ourselves.

2. The Feelings Conversation.
Every difficult conversation also asks and answers questions about feelings.
We conduct an internal debate over whether this means we are competent or incompetent, a good person or bad, worthy of love or unlovable.

As we argue vociferously for our view, we often fail to question one crucial assumption upon which our whole stance in the conversation is built: I am right, you are wrong.
They are about conflicting perceptions, interpretations, and values.
These are not questions of right and wrong, but questions of interpretation and judgment.

The Feelings Conversation: What Should We Do with Our Emotions? 

Difficult conversations are not just about what happened; they also involve emotion. The question is not whether strong feelings will arise, but how to handle them when they do. Should you tell your boss how you really feel about his management style, or about the colleague who stole your idea? Should you share with your sister how hurt you feel that she stayed friends with your ex? And what should you do with the anger you are likely to experience if you decide to talk with that vendor about his sexist remarks?
In the presence of strong feelings, many of us work hard to stay rational.
Getting too deep into feelings is messy, clouds good judgment, and in some contexts — for example, at work — can seem just plain inappropriate. Bringing up feelings can also be scary or uncomfortable, and can make us feel vulnerable. After all, what if the other person dismisses our feelings or responds without real understanding? Or takes our feelings to heart in a way that wounds them or irrevocably damages the relationship? And once we’ve gotten our feelings off our chest, it’s their turn. Are we up to hearing all about their anger and pain?

An Opera Without Music (Part of 'Feelings Convesation')

The problem with this reasoning is that it fails to take account of one simple fact: difficult conversations do not just involve feelings, they are at their very core about feelings. Feelings are not some noisy byproduct of engaging in difficult talk, they are an integral part of the conflict. Engaging in a difficult conversation without talking about feelings is like staging an opera without the music. You’ll get the plot but miss the point.

Consider some of your own difficult conversations. What feelings are involved? Hurt or anger? Disappointment, shame, confusion? Do you feel treated unfairly or without respect? For some of us, even saying “I love you” or “I’m proud of you” can feel risky.

In the short term, engaging in a difficult conversation without talking about feelings may save you time and reduce your anxiety. It may also seem like a way to avoid certain serious risks – to you, to others, and to the relationship. But the question remains: if feelings are the issue, what have you accomplished if you don’t address them? Understanding feelings, talking about feelings, managing feelings – these are among the greatest challenges of being human. There is nothing that will make dealing with feelings easy and risk-free. Most of us, however, can do a better job in the Feelings Conversation than we are now. It may not seem like it, but talking about feelings is a skill that can be learned.

3. The Identity Conversation.

In the “What Happened?” Conversation, moving away from the truth assumption frees us to shift our purpose from proving we are right to understanding the perceptions, interpretations, and values of both sides.
And to offer our views as perceptions, interpretations, and values – not as “the truth.”

Did you throw my cigarettes out because you’re trying to control my behavior or because you want to help me live up to my commitment to quit? 
The error we make in the realm of intentions is simple but profound : we assume we know the intentions of others when we don’t.
Because people’s intentions, like so much else in difficult conversations, are complex.
Sometimes people act with mixed intentions.
Most difficult conversations focus significant attention on who’s to blame for the mess we’re in.

But in situations that give rise to difficult conversations, it is almost always true that what happened is the result of things both people did — or failed to do.
In the presence of strong feelings, many of us work hard to stay rational.
Getting too deep into feelings is messy, clouds good judgment, and in some contexts — for example, at work — can seem just plain inappropriate.
And once we’ve gotten our feelings off our chest, it’s their turn.
This line of reasoning suggests that we stay out of the Feelings Conversation altogether.

The Identity Conversation looks inward: it’s all about who we are and how we see ourselves.
In short: before, during, and after the difficult conversation, the Identity Conversation is about what I am saying to myself about me.
"Why does my sense of who I am in the world matter here?" Or Jack might be thinking, “This is about the brochure, not about me.” In fact, anytime a conversation feels difficult, it is in part precisely because it is about You, with a capital Y. In fact, what if your boss gives you good reasons for turning you down? Instead of wanting to persuade and get your way, you want to understand what has happened from the other person’s point of view, explain your point of view, share and understand feelings, and work together to figure out a way to manage the problem going forward.
We need to have a learning conversation.
This book will help you turn difficult conversations into learning conversations by helping you handle each of the Three Conversations more productively and improving your ability to handle all three at once.
This will help you shift to a learning stance when it’s your difficult conversation and you aren’t feeling very open.
Then we turn to the mechanics of how to talk productively about the issues that matter to you: finding the best ways to begin, inquiring and listening to learn, expressing yourself with power and clarity, and solving problems jointly, including how to get the conversation back on track when the going gets rough.
Michael’s version of the story is different from Jack’s: In the past couple of years I’ve really gone out of my way to try to help Jack out, and it seems one thing or another has always gone wrong.
But what really made me angry was the way Jack was making excuses about the chart instead of just fixing it.
He knew it wasn’t up to professional standards. 

~ ~ ~

When we argue, we tend to trade conclusions – the “bottom line” of what we think:

“Get a new mattress” versus “Stop trying to control me.”

“I’m going to New York to make it big” versus “You’re naive.”

“Couples counseling is helpful” versus “Couples counseling is a waste of time.”

But neither conclusion makes sense in the other person’s story.

Telling someone to change makes it less rather than more likely that they will.

Understand each other’s stories
Understanding each other’s stories from the inside won’t necessarily “solve” the problem, but as with Karen and Trevor, it’s an essential first step.
Second, we each have access to different information.
Sitting on his uncle’s shoulders, Andrew shouted with delight as football players, cheerleaders, and the school band rolled by on lavish floats.
His Uncle Doug, truck indifferent, hadn’t noticed a single truck.
Of course, neither Doug nor Andrew walked away from the parade thinking, “I enjoyed my particular perspective on the parade based on the information I paid attention to.” Each walked away thinking, “I enjoyed the parade.” Each assumes that what he paid attention to was what was significant about the experience.
Often we go through an entire conversation – or indeed an entire relationship – without ever realizing that each of us is paying attention to different things, that our views are based on different information.

~ ~ ~

Jack doesn’t know that Michael’s graphic designer has taken an unscheduled personal leave in the midst of their busiest season, affecting not just this project but others as well.
Jack doesn’t know that Michael has been dissatisfied with some of Jack’s work in the past.  But rather than assuming we already know everything we need to, we should assume that there is important information we don’t have access to.
Two especially important factors in how we interpret what we see are: 
(1) our past experiences and 
(2) the implicit rules we’ve learned about how things should and should not be done.

~ ~ ~

To celebrate the end of a long project, Bonnie and her coworkers scraped together the money to treat their supervisor, Caroline, to dinner at a nice restaurant.
This was not the first time Ollie had arrived late, and Thelma was so frustrated that she had trouble focusing for the first twenty minutes of their meeting.
Ollie was frustrated that Thelma was frustrated.

Thelma’s rule is “It is unprofessional and inconsiderate to be late.” Ollie’s rule is “It is unprofessional to obsess about small things so much that you can’t focus on what’s important.” Because Thelma and Ollie both interpret the situation through the lens of their own implicit rule, they each see the other person as acting inappropriately.

Our implicit rules often take the form of things people “should” or “shouldn’t” do:

“You should spend money on education, but not on clothes.”

“You should never criticize a colleague in front of others.”

"You should never leave the toilet seat up."

"Squeeze the toothpaste in the middle"

Or "Don't let the kids watch more than two hours of TV."

The list is endless.
There’s nothing wrong with having these rules.

~ ~ ~

Consider the disagreement between Tony and his wife, Keiko.

To her shock, Tony says he’s not going with her to visit his sister, but instead is going to watch the football game on TV.
When Keiko asks why, Tony mumbles something about this being a “big game,” and adds, “I’ll stop by the hospital tomorrow.” Keiko is deeply troubled by this.
That’s the most selfish, shallow, ridiculous thing I’ve ever heard!” But she catches herself in her own certainty, and instead of saying, “How could you do such a thing?” she negotiates herself to a place of curiosity.
Since Tony believes that his sister won’t care whether he comes today or tomorrow, he chooses in favor of his mental health. 

It can be awfully hard to stay curious about another person’s story when you have your own story to tell, especially if you’re thinking that only one story can really be right.
After all, your story is so different from theirs, and makes so much sense to you.
Part of the stress of staying curious can be relieved by adopting what we call the “And Stance.” We usually assume that we must either accept or reject the other person’s story, and that if we accept theirs, we must abandon our own.
The And Stance is based on the assumption that the world is complex, that you can feel hurt, angry, and wronged, and they can feel just as hurt, angry, and wronged.
Sometimes people have honest disagreements, but even so, the most useful question is not “Who’s right?” but “Now that we really understand each other, what’s a good way to manage this problem?”
You may be thinking that the advice to shift from certainty and arguing to curiosity and the And Stance generally makes sense, but that there must be exceptions.
Even if you understand another person’s story with genuine insight and empathy, you may still stumble on the next step, thinking that however much their story makes sense to them, you are still “right” and they are still “wrong.” For example, what about the conversation you have with your daughter about her smoking? What may help is to tell him about the impact his drinking has on you, and, further, to try to understand his story.

~ ~ ~

Later that evening things went from bad to worse:

LORI: I really resented it at the party, the way you treated me in front of our friends.

LEO: The way I treated you? What are you talking about?

LORI: About the ice cream. You act like you’re my father or something. You have this need to control me or put me down.

LEO: Lori, I wasn’t trying to hurt you. You said you were on a diet, and I’m just trying to help you stick to it. You’re so defensive. You hear everything as an attack on you, even when I’m trying to help.

LORI: Help!? Humiliating me in front of my friends is your idea of helping?

Exploring “Lori’s mistake” requires us to understand how our minds work when devising stories about what others intend, and to learn to recognize the set of questionable assumptions upon which these stories are built.
Here’s the problem: While we care deeply about other people’s intentions toward us, we don’t actually know what their intentions are.

We Assume Intentions from the Impact on Us  

Much of the first mistake can be traced to one basic error: we make an attribution about another person’s intentions based on the impact of their actions on us.
We feel hurt; therefore they intended to hurt us.
When Margaret finally saw the doctor a week later, she asked curtly how his vacation had been.
Yet knowing that he was not acting out of selfishness, but from an unrelated and generous motivation, left Margaret feeling substantially better about having to wait the extra week.
With business and even personal relationships increasingly conducted via e-mail, voice mail, faxes, and conference calls, we often have to read between the lines to figure out what people really mean.
What’s ironic — and all too human — about our tendency to attribute bad intentions to others is how differently we treat ourselves.

Are There Never Bad Intentions? 

The easiest and most common way of expressing these assumptions about the intentions of others is with an accusatory question: 
“How come you wanted to hurt me?” 
“Why do you ignore me like this?” 
“What have I done that makes you feel it’s okay to step all over me?” 

We might think we are sharing our hurt, frustration, anger, or confusion, but we are rather accusing.
For example, Leo is defensive throughout, and at the end, when he says that he sometimes wonders if Lori “starts these fights on purpose,” he actually accuses Lori of bad intentions.
If interviewed about their conversation afterward, both Lori and Leo would report that they were the victim of the other’s bad intentions.
You think your boss isn’t giving you enough responsibility.
When we think others have bad intentions toward us, it affects our behavior.
As we’ve seen, the mistake Lori makes of assuming she knows Leo’s intentions, though seemingly small, has big consequences.
Working to understand what the other person is really saying is particularly important because when someone says “You intended to hurt me” that isn’t quite what they mean.
The father who is too busy at work to attend his son’s basketball game doesn’t intend to hurt his son.
He would prefer not to hurt his son.
If the father responds to his son’s complaint by saying “I didn’t intend to hurt you,” he’s not addressing his son’s real concern: “You may not have intended to hurt me, but you knew you were hurting me, and you did it anyway.” It is useful to attempt to clarify your intentions.
Another problem with assuming that good intentions sanitize a negative impact is that intentions are often more complex than just “good” or “bad.” Are Leo’s intentions purely angelic? And he is sending a message to Lori that says, “I’m more interested in defending myself than I am in investigating the complexities of what might be going on for me in our relationship.” Interestingly, when people take on the job of thinking hard about their own intentions, it sends a profoundly positive message to the other person about the importance of the relationship.

~ ~ ~

Separating impact from intentions requires us to be aware of the automatic leap from “I was hurt” to “You intended to hurt me.” You can make this distinction by asking yourself three questions:

1. Actions: “What did the other person actually say or do?”

2. Impact: “What was the impact of this on me?”

3. Assumption: “Based on this impact, what assumption am I making about what the other person intended?”

Hold Your View as a Hypothesis.

Once you have clearly answered these three questions, the next step is to make absolutely certain that you recognize that your assumption about their intentions is just an assumption.
It is a guess, a hypothesis.

Your hypothesis is not based on nothing ; you know what was said or done. But as we’ve seen, this is not a lot of evidence to go on. Your guess might be right and it might be wrong. In fact, your reaction might even say as much about you as it does about what they did. Perhaps you’ve had a past experience that gives their action special meaning to you.

Share the Impact on You; Inquire About Their Intentions. 

You can use your answers to the three questions listed above to begin the difficult conversation itself: say what the other person did, tell them what its impact was on you, and explain your assumption about their intentions, taking care to label it as a hypothesis that you are checking rather than asserting to be true.

Consider how this would change the beginning of the conversation between Lori and Leo. Instead of beginning with an accusation, Lori can begin by identifying what Leo said, and what the impact was on her:

LORI: You know when you said, “Why don’t you lay off the ice cream”? Well, I felt hurt by that.

LEO: You did?

LORI: Yeah.

LEO: I was just trying to help you stay on your diet. Why does that make you upset?

LORI: I felt embarrassed that you said it in front of our friends. Then what I wonder is whether you said it on purpose to embarrass or hurt me. I don’t know why you’d want to do that, but that’s what I’m thinking when it happens.

LEO: Well, I’m certainly not doing it on purpose. I guess I didn’t realize it was so upsetting. I’m confused about what it is you want me to say if I see you going off your diet... 

The conversation is only beginning, but it is off to a better start.

~ ~ ~

When you share your assumptions about their intentions, simply be clear that you are sharing assumptions – guesses – and that you are sharing them for the purpose of testing whether they make sense to the other person.
When we find ourselves in Leo’s position – being accused of bad intentions – we have a strong tendency to want to defend ourselves: “That is not what I intended.” We are defending our intentions and our character.
Remember that the accusation about our bad intentions is always made up of two separate ideas: 
(1) we had bad intentions and 
(2) the other person was frustrated, hurt, or embarrassed.

And if you start by listening and acknowledging the feelings, and then return to the question of intentions, it will make your conversation significantly easier and more constructive.

Understanding how we distort others’ intentions, making difficult conversations even more difficult, is crucial to untangling what happened between us.
You blame your assistant, not just because she’s a convenient target for your frustration or because letting others know it was she and not you who screwed up may help salvage your reputation, but because it is the simple truth: this was her fault.
You can blame her explicitly, saying something like “I don’t know how you could have let this happen!” Or, if you tend to be less confrontational (or have been taught that blaming people isn’t helpful), you can blame her implicitly, with something less threatening, like “Let’s do better next time.” Either way, she’ll get the message: she’s to blame.
Who is the bad person in this relationship? Focusing on blame is a bad idea. Primarily, because blame is often irrelevant and unfair.
You can’t move away from blame until you understand what blame is, what motivates us to want to blame each other, and how to move toward something else that will better serve your purposes in difficult conversations.
When we ask the question “Who is to blame?” we are really asking three questions in one.
Did your assistant’s actions (or inaction) cause you to have the wrong storyboards? When blame is in play, you can expect defensiveness, strong emotion, interruptions, and arguments about what “good assistants,” “loving spouses,” or “any reasonable person” should or shouldn’t do.
When we blame someone, we are offering them the role of “the accused,” so they do what accused people do: they defend themselves any way they can.

The first question is “How did we each contribute to bringing about the current situation?” Or put another way: “What did we each do or not do to get ourselves into this mess?” The second question is “Having identified the contribution system, how can we change it? What can we do about it as we go forward?” In short, contribution is useful when our goal is to understand what actually happened so that we can improve how we work together in the future.
In the worlds of both business and personal relationships, too often we deal in blame when our real goals are understanding and change.
To illustrate, let’s return to the ExtremeSport story and imagine two contrasting conversations between you and your assistant.
The first conversation focuses on blame, the second on contribution.

YOU: I wanted to talk to you about my presentation at ExtremeSport. You packed the wrong storyboards. The situation was unbelievably awkward, and made me look terrible. We simply can’t work this way.

ASSISTANT: I heard. I’m so sorry. I just, well, you probably don’t want to hear my excuses.

YOU: I just don’t understand how you could let this happen.

ASSISTANT: I’m really sorry.

YOU: I know you didn’t do it on purpose, and I know you feel bad, but I don’t want this to happen again. You understand what I’m saying?

ASSISTANT: It won’t. I promise you. 

All three elements of blame are present: you caused this, I’m judging you negatively, and implicit in what I am saying is that one way or another you will be punished, especially if it happens again. In contrast, a conversation about contribution might sound like this:

YOU: I wanted to talk to you about my presentation at ExtremeSport. When I arrived I found the wrong storyboards in my briefcase.

ASSISTANT: I heard. I’m so sorry. I feel terrible.

YOU: I appreciate that. I’m feeling bad too. Let’s retrace our steps and think about how this happened. I suspect we may each have contributed to the problem. From your point of view, did I do anything differently this time?

ASSISTANT: I’m not sure. We were working on three accounts at once, and on the one just before this one, when I asked about which boards you wanted packed, you got angry. I know it is my responsibility to know which boards you want, but sometimes when things get hectic, it can get confusing.

YOU: If you’re unsure, you should always ask. But it sounds like you’re saying I don’t always make it easy to do that.

ASSISTANT: Well, I do feel intimidated sometimes. When you get really busy, it’s like you don’t want to be bothered. The day you left you were in that kind of mood. I was trying to stay out of your way, because I didn’t want to add to your frustration. I had planned to double-check which boards you wanted when you got off the phone, but then I had to run to the copy center. After you left I remembered, but I knew you usually double-checked your briefcase, so I figured it was okay.

YOU: Yeah, I do usually double-check, but this time I was so overwhelmed I forgot. I think we’d both better doublecheck every time. And I do get in those moods. I know it can be hard to interact with me when I’m like that. I need to work on being less impatient and abrupt. But if you’re unsure, I need you to ask questions no matter what kind of mood I’m in.

ASSISTANT: So you want me to ask questions even if I think it will annoy you? 

YOU: Yes, although I’ll try to be less irritable. Can you do that? 

ASSISTANT: Well, talking about it like this makes it easier. I realize it’s important.

YOU: You can even refer to this conversation. You can say, “I know you’re under pressure, but you made me promise I’d ask this...” Or just say, “Hey, you promised not to be such a jerk!”

ASSISTANT: [laughs] Okay, that works for me.

YOU: And we might also think about how you could track better which appointments are going to be for which campaigns.... 

In the second conversation, you and your assistant have begun to identify the contributions that you each brought to the problem, and the ways in which each of your reactions are part of an overall pattern: You feel anxious and distracted about an upcoming presentation, and snap at your assistant.
She assumes you want her out of your way, and withdraws. Something falls through the cracks, and then you are even more annoyed and worried the next time you are preparing, since you’re no longer sure you can trust your assistant to help you. So you become more abrupt, increasingly unapproachable, and the communication between you continues to erode.
Mistakes multiply.
As you get a handle on the interactive system the two of you have created, you can see what you each need to do to avoid or alter that system in the future. As a result, this second conversation is much more likely than the first to produce lasting change in the way you work together. Indeed, the first conversation runs the risk of reinforcing the problem. Since part of the system is that your assistant feels discouraged from talking to you because she fears provoking your anger, a conversation about blame is likely to make that tendency worse, not better. If you go that way, she’ll eventually conclude that you’re impossible to work with, and you’ll report that she’s incompetent.

Contribution Is Joint and Interactive 

A contribution system includes inputs from both people. Think about a baseball pitcher facing a batter. If the batter strikes out in a crucial situation, he might explain that he wasn’t seeing well, that his wrist injury was still bothering him, or perhaps that he simply failed to come through in the clutch. The pitcher, however, might describe the strikeout by saying, “I knew he was thinking curve, so I came in with a high fast-ball,” or, “I was in a zone. I knew I had him before he even got in the batter’s box.”
Who is right, the batter or the pitcher? Of course, the answer is both, at least in part. Whether the batter strikes out or hits a home run is a result of the interaction between the batter and the pitcher. Depending on your perspective, you might focus on the actions of one or the other, but the actions of both are required for the outcome.
It’s the same in difficult conversations. Other than in extreme cases, such as child abuse, almost every situation that gives rise to a conversation is the result of a joint contribution system. Focusing on only one or the other of the contributors obscures rather than illuminates that system.

~ ~ ~

After a car accident, for example, an automaker expecting to be sued may resist making safety improvements for fear it will seem an admission that the company should have done something before the accident.
“Truth commissions” often are created because of this trade-off between assigning blame and gaining an understanding of what really happened.

Focusing on Blame Hinders Problem-Solving 

When the dog disappears, who’s to blame? When your real goal is finding the dog, fixing the ceiling, and preventing such incidents in the future, focusing on blame is a waste of time.
It neither helps you understand the problem looking back, nor helps you fix it going forward.
Fundamentally, using the blame frame makes conversations more difficult, while understanding the contribution system makes a difficult conversation easier and more likely to be productive.

~ ~ ~

Contribution asks: “What did I do that helped cause the situation?” You can find contribution even in situations where you carry no blame; you did contribute to being mugged.
In his autobiography, A Long Walk to Freedom, Nelson Mandela provides an example of how people who have been overwhelmingly victimized can still seek to understand their own contribution to their problems.
One of the most common contributions to a problem, and one of the easiest to overlook, is the simple act of avoiding.

~ ~ ~

"Fighting to show love" has limitations. Yet it and many other less-than-ideal dynamics are surprisingly common, at home and in the workplace. Why? First, because despite its problems the familiar pattern is comfortable, and the members of the group work to keep each person playing their role. Second, because changing a contribution system requires more than just spotting it and recognizing its limitations. The people involved also have to find another way to provide its benefits. George and his parents need to find better ways to demonstrate affection and maintain closeness. And this is likely to require some tough work in their Feelings and Identity Conversations.
In an organization, this explains why people find it hard to change how they work together even when they see the limitations of common role assumptions, such as “Leaders set strategy; subordinates implement it.” To change how people interact, they need both an alternate model everyone thinks is better and the skills to make that model work at least as well as the current approach.

~ ~ ~

Two Tools for Spotting Contribution If you are still unable to see your contribution, try one of the following two approaches.

1. Role Reversal 

Ask yourself, “What would they say I’m contributing?” Pretend you are the other person and answer the question in the first person, using pronouns such as I, me, and my. Seeing yourself through someone else’s eyes can help you understand what you’re doing to feed the system.

2. The Observer’s Insight 

Step back and look at the problem from the perspective of a disinterested observer. Imagine that you are a consultant called in to help the people in this situation better understand why they are getting stuck. How would you describe, in a neutral, nonjudgmental way, what each person is contributing? If you have trouble getting out of your own shoes in this way, ask a friend to try for you. If what your friend comes up with surprises you, don’t reject it immediately. Rather, imagine that it is true. Ask how that could be, and what it would mean.

~ ~ ~

The 'Feelings' Conversation
From Chapter: 5
Have Your Feelings (Or They Will Have You) 

% Feelings Matter: They Are Often at the Heart of Difficult Conversations 

% We Try to Frame Feelings Out of the Problem. Do not do this.

Framing the feelings out of the conversation is likely to result in outcomes that are unsatisfying for both people. The real problem is not dealt with, and further, emotions have an uncanny knack for finding their way back into the conversation, usually in not very helpful ways.

1. Unexpressed Feelings Can Leak into the Conversation 

2. Unexpressed Feelings Can Burst into the Conversation 

3. Unexpressed Feelings Make It Difficult to Listen 

4. Unexpressed Feelings Take a Toll on Our Self-Esteem and Relationships

~ ~ ~

A Way Out of the Feelings Bind 

There are ways to manage the problem of feelings. Working to get feelings into the conversation is almost always helpful as long as you do so in a purposive way. While the drawbacks of avoiding feelings are inevitable, the drawbacks of sharing feelings are not. If you are able to share feelings with skill, you can avoid many of the potential costs associated with expressing feelings and even reap some unexpected benefits. This is the way out of the feelings bind.
By following a few key guidelines you can greatly increase your chances of getting your feelings into your conversations and into your relationships in ways that are healthy, meaningful, and satisfying: first, you need to sort out just what your feelings are; second, you need to negotiate with your feelings; and third, you need to share your actual feelings, not attributions or judgments about the other person.

Finding Your Feelings: Learn Where Feelings Hide

% Explore Your Emotional Footprint

% Accept That Feelings Are Normal and Natural.

% Recognize That Good People Can Have Bad Feelings.

% Learn That Your Feelings Are as Important as Theirs.

% Find the Bundle of Feelings Behind the Simple Labels

% Don’t Let Hidden Feelings Block Other Emotions.

~ ~ ~

A Landscape of Sometimes Hard-to-Find Feelings

Love: Affectionate, caring, close, proud, passionate

Anger: Frustrated, exasperated, enraged, indignant

Hurt: Let down, betrayed, disappointed, needy

Shame: Embarrassed, guilty, regretful, humiliated, self-loathing

Fear: Anxious, terrified, worried, obsessed, suspicious

Self-Doubt: Inadequate, unworthy, inept, unmotivated

Joy: Happy, enthusiastic, full, elated, content

Sadness: Bereft, wistful, joyless, depressed

Jealousy: Envious, selfish, covetous, anguished, yearning

Gratitude: Appreciative, thankful, relieved, admiring

Loneliness: Desolate, abandoned, empty, longing

~ ~ ~

% Find the Feelings Lurking Under Attributions, Judgments, and Accusations

% Lift the Lid on Attributions and Judgments.

% We Translate Our Feelings Into Judgments: “If you were a good friend you would have been there for me.”
Attributions: “Why were you trying to hurt me?”
Characterizations: “You’re just so inconsiderate.”
Problem-Solving: “The answer is for you to call me more often.”

% Use the Urge to Blame as a Clue to Find Important Feelings.

% Don’t Treat Feelings as Gospel: Negotiate with Them

% Don’t Vent: Describe Feelings Carefully:
  1. Frame Feelings Back into the Problem 
  2. Express the Full Spectrum of Your Feelings 
  3. Don’t Evaluate — Just Share

% Express Your Feelings Without Judging, Attributing, or Blaming.

% Don’t Monopolize: Both Sides Can Have Strong Feelings at the Same Time.

% An Easy Reminder: Say “I Feel...”

~ ~ ~

The Identity Conversation

From Chapter: 6
Ground Your Identity: Ask Yourself What’s at Stake 

% Difficult Conversations Threaten Our Identity

Three Core Identities 
There are probably as many identities as there are people. But three identity issues seem particularly common, and often underlie what concerns us most during difficult conversations: 
- Am I competent? 
- Am I a good person? 
- Am I worthy of love? 

• Am I Competent? 
“I agonized about whether to bring up the subject of my salary. Spurred on by my colleagues, I finally did.
Before I could even get started, my supervisor said, ‘I’m surprised you want to discuss this. The truth is, I’ve been disappointed by your performance this year.’ I felt nauseous. Maybe I’m not the talented chemist I thought I was.”

• Am I a Good Person? 
“I had intended to break up with Sandra that night. I began in a roundabout way, and as soon as she got the drift, she started to cry. It hurt me so much to see her in such pain.
The hardest thing for me in life is hurting people I care about; it goes against who I am spiritually and emotionally. I just couldn’t bear how I was feeling, and after a few moments I was telling her how much I loved her and that everything would work out between us.” 

• Am I Worthy of Love? 
“I began a conversation with my brother about the way he treats his wife. He talks down to her and I know it really bothers her. I was hugely nervous bringing it up, and my words were getting all twisted. Then he shouted, ‘Who are you to tell me how to act?! You’ve never had a real relationship in your whole life!’ After that, I could hardly breathe, let alone talk. All I could think about was how I wanted to get out of there.”
Suddenly, who we thought we were when we walked into the conversation is called into question. 

% An Identity Quake Can Knock Us Off Balance

% There’s No Quick Fix

% Vulnerable Identities: The All-or-Nothing Syndrome
Ex: If I am not good, am probably bad.

% Denial Clinging to a purely positive identity leaves no place in our self-concept for negative feedback. If I think of myself as a super-competent person who never makes mistakes, then feedback suggesting that I have made a mistake presents a problem. The only way to keep my identity intact is to deny the feedback — to figure out why it’s not really true, why it doesn’t really matter, or why what I did wasn’t actually a mistake.

% Exaggeration: The alternative to denial is exaggeration. In all-or-nothing thinking, taking in negative feedback requires us not just to adjust our self-image, but to flip it. If I’m not completely competent, then I’m completely incompetent: “Maybe I’m not as creative and special as I thought I was. I’ll probably never amount to anything. Maybe I’ll even get fired.”

% We Let Their Feedback Define Who We Are.

When we exaggerate, we act as if the other person’s feedback is the only information we have about ourselves. We put everything up for grabs, and let what they say dictate how we see ourselves. We may turn in a hundred memos on time, but if we are criticized for being late with the 101st memo, we think to ourselves, “I can never do anything right.” This one piece of information fills our whole identity screen.
This example may seem ridiculous, but we all think like this on occasion, and not only around dramatic or traumatic events. If the waitress gives you a funny look as she collects her tip, you’re cheap. If you don’t help your friends paint their house, you’re selfish. If your brother says you don’t visit his children enough, you’re an uncaring aunt. It’s easy to see why exaggeration is such a debilitating reaction.

% Ground Your Identity

% Step One: Become Aware of Your Identity Issues

% Step Two: Complexify Your Identity (Adopt the And Stance)

% Three Things to Accept About Yourself

  1. You Will Make Mistakes.
  
  2. Your Intentions Are Complex.
  
  3. You Have Contributed to the Problem.

% During the Conversation: Learn to Regain Your Balance

  1. Let Go of Trying to Control Their Reaction
  
  2. Prepare for Their Response
  
  3. Imagine That It’s Three Months Or Ten Years From Now
  
  4. Take a Break

% Their Identity Is Also Implicated

% Raising Identity Issues Explicitly

% Find the Courage to Ask for Help

~ ~ ~

Chapter: 7
Create a Learning Conversation 
What’s Your Purpose? When to Raise It and When to Let Go 

% To Raise or Not to Raise: How to Decide? - How Do I Know I’ve Made the Right Choice? - Work Through the Three Conversations

% Three Kinds of Conversations That Don’t Make Sense - Is the Real Conflict Inside You? - Is There a Better Way to Address the Issue Than Talking About It? - Do You Have Purposes That Make Sense? - Remember, You Can’t Change Other People.
- Don’t Focus on Short-Term Relief at Long-Term Cost.
- Don’t Hit-and-Run.

% Letting Go

% Adopt Some Liberating Assumptions - It’s Not My Responsibility to Make Things Better; It’s My Responsibility to Do My Best.

  - They Have Limitations Too.
  
  - This Conflict Is Not Who I Am.
  
  - Letting Go Doesn’t Mean I No Longer Care.

% If You Raise It: Three Purposes That Work

  1. Learning Their Story 
  2. Expressing Your Views and Feelings 
  3. Problem-Solving Together

% Stance and Purpose Go Hand in Hand These three purposes accommodate the fact that you and the other person see the world differently, that you each have powerful feelings about what is going on, and that you each have your own identity issues to work through. Each of you, in short, has your own story. You need purposes that can reckon with this reality.
These are the purposes that emerge from a learning stance, from working through the Three Conversations and shifting your internal orientation from certainty to curiosity, from debate to exploration, from simplicity to complexity, from “either/or” to “and.” They may seem simple – perhaps even simplistic. But their straightforwardness masks both the difficulty involved in doing them well and the power they have to transform the way you handle your conversations.
Working from a learning stance with these purposes in mind, the rest of this book explores in detail how to conduct a learning conversation, from getting started to getting unstuck.

~ ~ ~

Chapter: 8
Getting Started: Begin from the Third Story 

% Why Our Typical Openings Don’t Help 
  - We Begin Inside Our Own Story 
  - We Trigger Their Identity Conversation from the Start

% Step One: Begin from the Third Story 
  - Think Like a Mediator 
  - Not Right or Wrong, Not Better or Worse 
  - Just Different 
  - The Third Story.

Example:

Opening Lines From Inside Your Story: If you contest Dad’s will, it’s going to tear our family apart. 

From the Third Story: I wanted to talk about Dad’s will. You and I obviously have different understandings of what Dad intended, and of what’s fair to each of us. I wanted to understand why you see things the way you do, and to share with you my perspective and feelings. In addition, I have strong feelings and fears about what a court fight would mean for the family; I suspect you do too.

-- -- --

From Inside Your Story : I was very upset by what you said in front of our supervisor. 

From the Third Story: I wanted to talk to you about what happened in the meeting this morning. I was upset by something you said. I wanted to explain what was bothering me, and also hear your perspective on the situation. 

From Inside Your Story: Your son Nathan can be difficult in class — disruptive and argumentative. You’ve said in the past that things at home are fine, but something must be troubling him.

From the Third Story: I wanted to share with you my concerns about Nathan’s behavior in class, and hear more about your sense of what might be contributing to it. I know from our past conversation that you and I have different thinking on this. My sense is that if a child is having trouble at school, something is usually bothering him at home, and I know you’ve felt strongly that that’s not true in this case. Maybe together we can figure out what’s motivating Nathan and how to handle it.

% If They Start the Conversation, You Can Still Step to the Third Story

% Step Two: Extend an Invitation 
  1. Describe Your Purposes 
  2. Invite, Don’t Impose 
  3. Make Them Your Partner in Figuring It Out 4. Be Persistent

% Some Specific Kinds of Conversations 
  - Delivering Bad News 
  - Making Requests 
    : “I Wonder If It Would Make Sense...?”
  - Revisiting Conversations Gone Wrong 
    : Talk About How to Talk About It.

% A Map for Going Forward: Third Story, Their Story, Your Story 
- What to Talk About: The Three Conversations

- What to Talk About:
Explore where each story comes from: “My reactions here probably have a lot to do with my experiences in a previous job...”
Share the impact on you: “I don’t know whether you intended this, but I felt extremely uncomfortable when...”
Take responsibility for your contribution: “There are a number of things I’ve done that have made this situation harder...”
Describe feelings: “I’m anxious about bringing this up, but at the same time, it’s important to me that we talk about it...”
Reflect on the identity issues: “I think the reason this subject hooks me is that I don’t like thinking of myself as someone who...”

% How to Talk About It: Listening, Expression, and Problem-Solving

~ ~ ~

Chapter: 9
Learning: Listen from the Inside Out 

% Listening Transforms the Conversation

% Listening to Them Helps Them Listen to You

% The Stance of Curiosity: How to Listen from the Inside Out 
- Forget the Words, Focus on Authenticity 
- The Commentator in Your Head: Become More Aware of Your Internal Voice 
- Don’t Turn It Off, Turn It Up 
- Managing Your Internal Voice 
- Negotiate Your Way to Curiosity.
- Don’t Listen: Talk. Sometimes you’ll find that your internal voice is just too strong to take on. You try to negotiate your way to curiosity, but you just can’t get there. If you’re sitting on feelings of pain or outrage or betrayal, or, conversely, if you’re overcome with joy or love, then listening may be a hopeless task.

% Three Skills: Inquiry, Paraphrasing, and Acknowledgment 
 - Inquire to Learn 
 - Don’t Make Statements Disguised as Questions 
 - Don’t Use Questions to Cross-Examine 
 - Ask Open-Ended Questions 
 - Ask for More Concrete Information 
 - Ask Questions About the Three Conversations 
 - Make It Safe for Them Not to Answer

% Paraphrase for Clarity 
 - Check Your Understanding 
 - Show That You’ve Heard

% Acknowledge Their Feelings 
 - Answer the Invisible Questions 
 - How to Acknowledge 
 - Order Matters: Acknowledge Before Problem-Solving 
 - Acknowledging Is Not Agreeing

% A Final Thought: Empathy Is a Journey, Not a Destination

~ ~ ~

Chapter 10
Expression: Speak for Yourself with Clarity and Power 

% Orators Need Not Apply 
% You’re Entitled (Yes, You) 
  - No More, But No Less 
  - Beware Self-Sabotage 
  - Failure to Express Yourself Keeps You Out of the Relationship 
  - Feel Entitled, Feel Encouraged, But Don’t Feel Obligated

% Speak the Heart of the Matter 
  - Start with What Matters Most 
  - Say What You Mean: Don’t Make Them Guess 
  - Don’t Rely on Subtext.
  - Avoid Easing In.
  - Don’t Make Your Story Simplistic: Use the “Me-Me” And

% Telling Your Story with Clarity: Three Guidelines 
  1. Don’t Present Your Conclusions as The Truth 
  
  2. Share Where Your Conclusions Come From 
  
  3. Don’t Exaggerate with “Always” and “Never”
    : Give Them Room to Change

% Help Them Understand You 
  - Ask Them to Paraphrase Back 
  - Ask How They See It Differently 
  — and Why

~ ~ ~

Chapter 11
Problem-Solving: Take the Lead 

% Skills for Leading the Conversation If your conversations are going to get anywhere, you’re going to have to take the lead. There are a set of powerful “moves” you can make during the conversation – reframing, listening, and naming the dynamic – that can help keep the conversation on track, whether the other person is being cooperative or not.
When the other person heads in a destructive direction, reframing puts the conversation back on course. It allows you to translate unhelpful statements into helpful ones. Listening is not only the skill that lets you into the other person’s world; it is also the single most powerful move you can make to keep the conversation constructive. And naming the dynamic is useful when you want to address a troubling aspect of the conversation. It is a particularly good strategy if the other person is dominating the conversation and seems unwilling to follow your lead.

% Reframe, Reframe, Reframe

% You Can Reframe Anything

Truth -> Different Story

Accusations -> Intentions and impact

Blame -> Contributions

Judgements, characterizations -> Feelings

What's wrong with you -> What's going on for them

% The “You-Me” And

% It’s Always the Right Time to Listen

- Be Persistent About Listening 

% Name the Dynamic: Make the Trouble Explicit

% Now What? Begin to Problem-Solve 
  - It Takes Two to Agree 
  - Gather Information and Test Your Perceptions 
  - Propose Crafting a Test.
    : Say What Is Still Missing 
    : Say What Would Persuade You 
	: Ask What (If Anything) Would Persuade Them 
  - Ask Their Advice

  - Invent Options 
  - Ask What Standards Should Apply 
  - The Principle of Mutual Caretaking.
  - If You Still Can’t Agree, Consider Your Alternatives 
  - It Takes Time

~ ~ ~

Chapter: 12
Putting It All Together 

Five steps to a difficult conversation:

Step One: Prepare by Walking Through the Three Conversations

Step Two: Check Your Purposes and Decide Whether to Raise It

Step Three: Start from the Third Story

Step Four: Explore Their Story and Yours

Step Five: Problem-Solving

• Deciding: Is this the best way to address the issue and achieve your purposes? 

• Is the issue really embedded in your Identity Conversation? 

• Can you affect the problem by changing your contributions? 

• If you don’t raise it, what can you do to help yourself let go? 

Now with sub-steps:

Step 3: Start from the Third Story 

  1. Describe the problem as the difference between your stories. Include both viewpoints as a legitimate part of the discussion.
  2. Share your purposes.
  3. Invite them to join you as a partner in sorting out the situation together.
  
Step 4: Explore Their Story and Yours 

• Listen to understand their perspective on what happened. Ask questions. Acknowledge the feelings behind the arguments and accusations. Paraphrase to see if you’ve got it. Try to unravel how the two of you got to this place.

• Share your own viewpoint, your past experiences, intentions, feelings.

• Reframe, reframe, reframe to keep on track. From truth to perceptions, blame to contribution, accusations to feelings, and so on.

Step 5: Problem-Solving 

• Invent options that meet each side’s most important concerns and interests.

• Look to standards for what should happen. Keep in mind the standard of mutual caretaking; relationships that always go one way rarely last.

• Talk about how to keep communication open as you go forward.

Sentiment Analysis Testing on Some Difficult Sentences

We are going to test three sentiment analyzers:
1. TextBlob
2. BERT Based Sentiment Analyzer
3. vaderSentiment

The sentences are shown below (and link to Excel is given at the bottom):



Note: vaderSentiment could not be found in "conda-forge". "conda install vaderSentiment -c conda-forge" failed.

(temp) C:\Users\Ashish Jain>pip install vaderSentiment
Collecting vaderSentiment
  Downloading vaderSentiment-3.3.2-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 1.6 MB/s
Requirement already satisfied: requests in e:\programfiles\anaconda3\envs\temp\lib\site-packages (from vaderSentiment) (2.24.0)
Requirement already satisfied: idna<3,>=2.5 in e:\programfiles\anaconda3\envs\temp\lib\site-packages (from requests->vaderSentiment) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in e:\programfiles\anaconda3\envs\temp\lib\site-packages (from requests->vaderSentiment) (1.25.10)
Requirement already satisfied: chardet<4,>=3.0.2 in e:\programfiles\anaconda3\envs\temp\lib\site-packages (from requests->vaderSentiment) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in e:\programfiles\anaconda3\envs\temp\lib\site-packages (from requests->vaderSentiment) (2020.6.20)
Installing collected packages: vaderSentiment
Successfully installed vaderSentiment-3.3.2 

"vaderSentiment"
As of 23-Sep-2020, new changes in current version: Refactoring for Python 3 compatibility, improved modularity, and incorporation into [NLTK: current version 3.5].

Ref: nltk.org

About the Scoring of 'vaderSentiment':

1.The compound score is computed by summing the valence scores of each word in the lexicon, adjusted according to the rules, and then normalized to be between -1 (most extreme negative) and +1 (most extreme positive). This is the most useful metric if you want a single unidimensional measure of sentiment for a given sentence. Calling it a 'normalized, weighted composite score' is accurate.

  It is also useful for researchers who would like to set standardized thresholds for classifying sentences as either positive, neutral, or negative. Typical threshold values (used in the literature cited on this page) are:

  % positive sentiment: compound score >= 0.05
  % neutral sentiment: (compound score > -0.05) and (compound score < 0.05)
  % negative sentiment: compound score <= -0.05

2. The pos, neu, and neg scores are ratios for proportions of text that fall in each category (so these should all add up to be 1... or close to it with float operation). These are the most useful metrics if you want multidimensional measures of sentiment for a given sentence.

For example:
Not bad at all >> {'pos': 0.487, 'compound': 0.431, 'neu': 0.513, 'neg': 0.0}

This scoring can modified to taking the maximum of 'pos', 'neu' or 'neg'.

Python Code 

import pandas as pd
from textblob import TextBlob
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score
from collections import Counter
import requests
import DrawConfusionMatrix as dcm
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

df = pd.read_excel('files_1/Sentences for Sentiment Analysis.xlsx')

df_1 = df.drop_duplicates(subset='Sentence', keep="last")
df_2 = df_1[df_1.Sentiment.isin(['Positive', 'Negative', 'Neutral'])]

def get_sentiment_textblob(sentence):
    blob = TextBlob(sentence)
    sentiment = 'Positive'
    if blob.sentiment.polarity < 0:
        sentiment = 'Negative'
    elif blob.sentiment.polarity == 0:
        sentiment = 'Neutral'
    return sentiment
	
df_2['textblob'] = df_2.Sentence.apply(lambda x: get_sentiment_textblob(x))

cm = confusion_matrix(df_2['Sentiment'], df_2['textblob'])

print("Accuracy: {:.3f}".format(accuracy_score(df_2['Sentiment'], df_2['textblob'])))
print(Counter(df_2['textblob'])) 

Accuracy: 0.519
Counter({'Positive': 33, 'Neutral': 28, 'Negative': 18}) 

classes = ['Negative', 'Neutral', 'Positive']
dcm.plot_confusion_matrix(cm, classes = classes, use_seaborn = True)  

For TextBlob:


For BERT based code, we have to start a Flask server and get sentiment output from there (Ref 1):
Note: BERT based Sentiment Analyzer, we get only 'Positive' or 'Negative':

def get_sentiment_bert(sentence):
    return requests.get("http://127.0.0.1:5000/?text=" + sentence).json()['sentiment']
	
df_2['bert_sentiment'] = df_2.Sentence.apply(lambda x: get_sentiment_bert(x))

cm = confusion_matrix(df_2['Sentiment'], df_2['bert_sentiment'])
print("Accuracy: {:.3f}".format(accuracy_score(df_2['Sentiment'], df_2['bert_sentiment'])))
print(Counter(df_2['bert_sentiment'])) 

Accuracy: 0.722
Counter({'Negative': 43, 'Positive': 36}) 

classes = ['Negative', 'Neutral', 'Positive']
dcm.plot_confusion_matrix(cm, classes = classes, use_seaborn = True) 



print(f1_score(df_2['Sentiment'], df_2['bert_sentiment'], average='macro'))
print(f1_score(df_2['Sentiment'], df_2['bert_sentiment'], average='micro'))
print(f1_score(df_2['Sentiment'], df_2['bert_sentiment'], average='weighted')) 

macro: 0.5029855988760098
micro: 0.7215189873417721
weighted: 0.6898667484760773 

# vaderSentiment
analyzer = SentimentIntensityAnalyzer()
def get_sentiment_vader(sentence):
    vs = analyzer.polarity_scores(sentence)
    if vs['compound'] >= 0.05:
        rtn_val = 'Positive'
    elif vs['compound'] > -0.05 and vs['compound'] < 0.05:
        rtn_val = 'Neutral'
    else:
        rtn_val = 'Negative'
    return rtn_val

df_2['vader_sentiment'] = df_2.Sentence.apply(lambda x: get_sentiment_vader(x)) 

Accuracy: 0.557
Counter({'Positive': 41, 'Neutral': 23, 'Negative': 15})

Confusion matrix, without normalization
[[11 10  9]
 [ 0  4  3]
 [ 4  9 29]] 
 


BERT based code is performing the best with accuracy of about .72 

The link to Excel containining the setences: GitHub

Ref 1: Sentiment Analysis using BERT, DistilBERT and ALBERT

Ref 2: Sentiment Analysis Tutorial (2014, Bing Liu)

Flutter Notes (Week 2, Sep 2020)

Ques: How to fix initialization error for DefaultKotlinSourceSetKt?
  Error: Could not initialize class org.jetbrains.kotlin.gradle.plugin.sources.DefaultKotlinSourceSetKt

Answer:
This error comes on upgrading Android Studio (from October 2019 Release to Sep, 2020 (v4.0.1)) and the SDK Components.

On updating the Kotlin version on Android Studio -> Tools -> Kotlin -> Check For Update.

Then make changes on Kotlin Version as below on build.gradle 



After changing: ext.kotlin_version = '1.3.72'
Old value: 1.3.10

Issues appearing in "Build" process logs:


1.
  Deprecation Warning for 'Android Gradle' plugin:
  
  WARNING: API 'variant.getMergeAssets()' is obsolete and has been replaced with 'variant.getMergeAssetsProvider()'.
  It will be removed in version 5.0 of the Android Gradle plugin.
  For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
  To determine what is calling variant.getMergeAssets(), use -Pandroid.debug.obsoleteApi=true on the command line to display more information.
  WARNING: API 'variantOutput.getProcessResources()' is obsolete and has been replaced with 'variantOutput.getProcessResourcesProvider()'.
  It will be removed in version 5.0 of the Android Gradle plugin.
  For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
  To determine what is calling variantOutput.getProcessResources(), use -Pandroid.debug.obsoleteApi=true on the command line to display more information. 

  
2.
 > Task :app:compileFlutterBuildDebugArm
  ╔====
  ║ A new version of Flutter is available!
  ║
  ║ To update to the latest version, run "flutter upgrade".
  ╚====

  C:\Users\Ashish Jain>flutter --version
  Waiting for another flutter command to release the startup lock...
  Flutter 1.9.1+hotfix.2 • channel stable • https://github.com/flutter/flutter.git
  Framework • revision 2d2a1ffec9 (1 year, 1 month ago) • 2019-09-06 18:39:49 -0700
  Engine • revision b863200c37
  Tools • Dart 2.5.0 


3. 

  Manifest merger failed : Overlay manifest:package attribute declared at AndroidManifest.xml:2:5-37 value=(com.survival8.survival) has a different value=(com.survival8.one) declared in main manifest at AndroidManifest.xml:2:5-32
  	Suggestion: remove the overlay declaration at AndroidManifest.xml 	and place it in the build.gradle:
  		flavorName {
  			applicationId = "com.survival8.survival"
  		} 
  



4.
  On replacing the entire code from a back-up of code developed on 'Ubuntu':
  "\home\administrator\flutter" was our path in Ubuntu machine.
    Error:
    C:\Users\Ashish Jain\AndroidStudioProjects\survival\android\app\home\administrator\flutter\packages\flutter_tools\gradle\flutter.gradle (The system cannot find the path specified) 
  


We have a "local.properties" file with contents:
Path: C:\Users\Ashish Jain\AndroidStudioProjects\survival\android\local.properties
Contents from Ubuntu back-up code:

  ## This file must *NOT* be checked into Version Control Systems,
  # as it contains information specific to your local configuration.
  #
  # Location of the SDK. This is only used by Gradle.
  # For customization when using a Version Control System, please read the header note.
  # Mon Sep 21 19:31:06 IST 2020
  flutter.buildMode=release
  flutter.versionName=1.0.0
  flutter.sdk=/home/administrator/flutter
  sdk.dir=C\:\\Users\\Ashish Jain\\AppData\\Local\\Android\\Sdk
  flutter.versionCode=1 

Here, we need to change the property "flutter.sdk" according to our current system.

  --> flutter.sdk=E\:\\programfiles\\flutter

  C:\Users\Ashish Jain\AndroidStudioProjects>flutter upgrade
  Flutter 1.20.4 • channel stable • https://github.com/flutter/flutter.git
  Framework • revision fba99f6cf9 (7 days ago) • 2020-09-14 15:32:52 -0700
  Engine • revision d1bc06f032
  Tools • Dart 2.9.2
  
  Running flutter doctor...
  Doctor summary (to see all details, run flutter doctor -v):
  [√] Flutter (Channel stable, 1.20.4, on Microsoft Windows [Version 10.0.18363.1082], locale en-US)
  [!] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
      X Android license status unknown.
        Try re-installing or updating your Android SDK Manager.
        See https://developer.android.com/studio/#downloads or visit https://flutter.dev/docs/get-started/install/windows#android-setup for detailed instructions.
  [√] Android Studio (version 4.0) 
  
  C:\Users\Ashish Jain\AndroidStudioProjects>flutter --version
  Flutter 1.20.4 • channel stable • https://github.com/flutter/flutter.git
  Framework • revision fba99f6cf9 (7 days ago) • 2020-09-14 15:32:52 -0700
  Engine • revision d1bc06f032
  Tools • Dart 2.9.2 




5.

When opening a project, open the "android" folder and not the parent directory such as "survival" or "survival8".




6.
  Where to view the Gradle version running:




7.
  Way to clone 'stable' Flutter GitHub repository:
  $ git clone -b stable https://github.com/flutter/flutter.git

12 'ECMAScript 6' Tips (Sep 2020)

1. 
  clear()
  This function clears the 'output pane' in 'JS console'.

2. 
  You can define an array as follows:

  var arr = [
    'elem a',
    'elem b',
    'elem c'
  ] 

  'Shift + Enter' lets you add a line-feed in the code in Firefox JS Console.

3. 
  "arr.length" 
  Will return you '3'.

4.
  Get unique elements from an Array:
  With ES6:
  var a = ['a', 'b', 'c', 'c']
  console.log([...new Set(a)]) 
  
  Out:
   ["a", "b", "c"]
   
  OR
   Array.from(new Set(a)); 
  
  OR
   list = list.filter((x, i, a) => a.indexOf(x) === i) 
   
5.
  To concatenate two arrays:
  a = ['a', 'b', 'c', 'c']
  b = ['d', 'e']
  console.log(a.concat(b)) 
  
  Out:
   ["a", "b", "c", "c", "d", "e"]
  
6. 
  ES6 based counter:
    var cntr = 0;
    function myFunction() {
      console.log(arr[cntr]);
      cntr++;
    } 

7.
  Add an element to "DOM body" before other child elements.
    b = document.querySelector("body")
    theKid = document.createElement("p");
    theKid.innerHTML = '<button onclick="myFunction(chatroom_names_kenya)" style="position: sticky !important;top: 0;z-index: 999;">Submit</button>';
    b.insertBefore(theKid, b.firstChild) 

8.
  Ways to access an element of HTML DOM using JavaScript:
  
 Gets           Selector Syntax Method 
 ID                #demo           getElementById()
 Class             .demo           getElementsByClassName()
 Tag               demo            getElementsByTagName()
 Selector (single)                 querySelector()
 Selector (all)                    querySelectorAll()

 About Query Selectors: It is JavaScript's method of accessing the DOM with CSS selectors.

9.
  Finding HTML Elements by HTML Object Collections:
    var x = document.forms["frm1"];
    var text = "";
    var i;
    for (i = 0; i < x.length; i++) {
      text += x.elements[i].value + "<br>";
    }
    document.getElementById("demo").innerHTML = text;  

  The following HTML objects (and object collections) are also accessible:
  1. document.anchors
  2. document.body
  3. document.documentElement
  4. document.embeds
  5. document.forms
  6. document.head
  7. document.images
  8. document.links
  9. document.scripts
  10. document.title

10.
  Identifying type of a variable:
  The typeof operator returns the type of a variable, object, function or expression:
  
    typeof "John"                 // Returns string
    typeof 3.14                   // Returns number
    typeof NaN                    // Returns number
    typeof false                  // Returns boolean
    typeof [1, 2, 3, 4]           // Returns object
    typeof {name:'John', age:34}  // Returns object
    typeof new Date()             // Returns object
    typeof function () {}         // Returns function
    typeof myCar                  // Returns undefined (if myCar is not declared)
    typeof null                   // Returns object  

  Please observe:
    The data type of NaN is number
    The data type of an array is object
    The data type of a date is object
    The data type of null is object
    The data type of an undefined variable is undefined

11.
  The instanceof Operator
  The instanceof operator returns true if the specified object is an instance of the specified object:

    var cars = ["Saab", "Volvo", "BMW"];
    
    cars instanceof Array;          // Returns true
    cars instanceof Object;         // Returns true
    cars instanceof String;         // Returns false
    cars instanceof Number;         // Returns false 

12.
  Spread Operator
  
  Spread operator allows an iterable to expand in places where 0+ arguments are expected. It is mostly used in the variable array where there is more than 1 values are expected. It allows us the privilege to obtain a list of parameters from an array. Syntax of Spread operator is same as "Rest parameter" but it works completely opposite of it. 

  12.1    
    // spread operator doing the concat job 
    let arr = [1,2,3]; 
    let arr2 = [4,5]; 
      
    arr = [...arr,...arr2]; 
    console.log(arr); // [ 1, 2, 3, 4, 5 ]  

  12.2 
    // spread operator for copying  
    let arr = ['a','b','c']; 
    let arr2 = [...arr]; 
      
    console.log(arr); // [ 'a', 'b', 'c' ] 
      
    arr2.push('d'); //inserting an element at the end of arr2 
      
    console.log(arr2); // [ 'a', 'b', 'c', 'd' ] 
    console.log(arr); // [ 'a', 'b', 'c' ]  

Failing to launch Android Virtual Device on Ubuntu in VirtualBox on Windows 10

'Android Virtual Device' fails to run on Ubuntu in VirtualBox on Windows 10:
Your CPU does not support required features (VT-x or SVM).



Message you get on clicking 'Troubleshoot':



If your computer does not support hardware accelerated virtualization. Android Studio provides suggestions:

1. Use a physical device for testing.

2. Develop on a Windows/OSX computer with an Intel processor that supports VT-x and NX.

3. Develop on a Linux computer that supports VT-x or SVM.

4. Use an Android Virtual Device based on an ARM system image. (This is 10X slower than hardware accelerated virtualization.)

Our VirtualBox GuestOS setting:



How to enable the "nested vtx/amd-v" in Orable VirtualBox for Windows? 

In Windows, go to VirtualBox installation folders 
-> type 'cmd' in the 'address' bar (it will pop up 'cmd' in that folder) 
-> type VBoxManage modifyvm YourVirtualBoxName --nested-hw-virt on 
-> enter.



Now it should be ticked.

Warning we still get:



Performance Issues

1.


2. AVD never starts beyond this:

Web Security - Prevent Website from opening in an IFrame

For Web Security, prevent website from opening in an IFrame as 'WhatsApp Web' does.

<div>
    <p>WhatsApp Error: Prevention from opening WhatsApp Web in an IFrame.</p>
</div>
<iframe src="https://web.whatsapp.com/" title="My WhatsApp" width=900 height=400></iframe> 

View in Mozilla Firefox:

Setting up Ubuntu 20.04 for Flutter based Android app development

1.
Install Git.

  $ sudo apt install git


2.
Create a directory where we download the 'flutter':

(base) ashish@ashish-VirtualBox:~/Desktop/ws/programfiles/flutter$ pwd
/home/ashish/Desktop/ws/programfiles/flutter_box 


3.
Download 'flutter':

$ pwd
/home/ashish/Desktop/ws/programfiles/flutter_box 

$ git clone https://github.com/flutter/flutter.git 


4.
Add the flutter tool to your path:

 $export PATH="$PATH:`pwd`/flutter/bin" 
 
OR

Update this in "~/.bashrc" file.

$ nano ~/.bashrc
$ source ~/.bashrc 


5.
Optionally, pre-download development binaries:

The flutter tool downloads platform-specific development binaries as needed. For scenarios where pre-downloading these artifacts is preferable (for example, in hermetic build environments, or with intermittent network availability), iOS and Android binaries can be downloaded ahead of time by running:

$ flutter precache 


6.
Install "Android SDK" from 'Terminal'.

$ sudo apt update && sudo apt install android-sdk 


7. 
Install "Android Studio" from "Ubuntu Software".




8. 
When you launch 'Android Studio' for the first time, it gives the prompt for 'Import Android Studio Settings':




Set it to "Do not import 'Settings'."


9.
It will next launch the 'Android Studio Setup Wizard'.




10.
Default JDK location:




11.
Next, it downloads SDK components:




12. 
Prompt for 'Emulator Settings for Hardware Acceleration'




13.
Undate Android license status.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/linux#android-setup for more details.
	  
$ flutter doctor --android-licenses 


14.
Launch "Settings" as shown below. Then go to "Plugins".





If we launch installation of 'Flutter' plugin, it automatically prompts for the installation for 'Dart'.




Then, give 'Android Studio' a restart.


15. 
Installing 'Flutter Extension' in Visual Studio Code.

Go to 'Extensions' as shown below and search for 'flutter'.


...



16.
Test installation:

(base) ashish@ashish-VirtualBox:~/.../flutter_box$ flutter doctor 
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel master, 1.22.0-10.0.pre.264, on Linux, locale en_IN)
 
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
[✓] Android Studio (version 4.0)
[✓] VS Code (version 1.49.1)
[!] Connected device
    ! No devices available

! Doctor found issues in 1 category. 


17.
Common Issues that we notice from 'flutter doctor':

As of Flutter’s 1.19.0 dev release, the Flutter SDK contains the dart command alongside the flutter command so that you can more easily run Dart command-line programs. Downloading the Flutter SDK also downloads the compatible version of Dart, but if you’ve downloaded the Dart SDK separately, make sure that the Flutter version of dart is first in your path, as the two versions might not be compatible. 

$ flutter doctor 
Doctor summary (to see all details, run flutter doctor -v):

17.1.
  [!] Android toolchain - develop for Android devices (Android SDK version 27.0.1)
    ✗ Flutter requires Android SDK 29 and the Android BuildTools 28.0.3
      To update the Android SDK visit Flutter.dev: Android Setup on Linux for detailed instructions.

17.2.
  ✗ Android license status unknown.
    Run `flutter doctor --android-licenses` to accept the SDK licenses.
    See Flutter.dev: Android Setup on Linux for more details.

17.3.
  ✗ Android licenses not accepted.  To resolve this, run: flutter doctor --android-licenses

17.4.
  [!] Android Studio (not installed)

17.5.
  [!] Android Studio (version 4.0)
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.

17.6
  [!] Android Studio (version 4.0)
    ✗ Dart plugin not installed; this adds Dart specific functionality.

17.7.
  [!] VS Code (version 1.49.1)
    ✗ Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

17.8.
  [!] Connected device                          
    ! No devices available

! Doctor found issues in 4 categories. 


Dated: Sep 2020
Ref: https://flutter.dev/docs/get-started/install/linux

Binomial Probability Distribution (visualization using Seaborn)

Binomial Probability Distribution 



"pmf" is "Probability Mass Function" or "Probability Distribution".

"rv" is "Random Variable".



Note: Binomial Distribution is a Discrete Distribution.

Visualization of Binomial Distribution 



Difference Between Normal and Binomial Distribution 

The main difference is that normal distribution is continous whereas binomial is discrete, but if there are enough data points it will be quite similar to normal distribution with certain loc and scale.

We have code that produces overlapped "Normal" and "Binomial" distributions. We will show some of the best and some of the worst overlaps.

number_of_trials = 150

for s in range(1000, 100000000, 1000000): 
    print("size:", s)
    sns.distplot(random.binomial(n = number_of_trials, p=0.5, size=s), hist=False, label='binomial')

    sns.distplot(random.normal(loc = number_of_trials / 2, 
                               scale=5, size=s), hist=False, label='normal')

    plt.show() 
	
Best Overlaps



Worst Overlaps



References 
% numpy.org

Improving a Classifier (ML) Using Snorkel's Slicing Technique

The dataset we are using is the '150 datapoints strong' Iris flower species dataset (Download from here).

We have a dependency here to draw the confusion matrix. The code file name is: DrawConfusionMatrix.py

Content:

# Ref: Scikit-Learn 

import itertools
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

def plot_confusion_matrix(cm, classes,
                          normalize = False,
                          title = 'Confusion matrix',
                          cmap = plt.cm.Blues,
                          use_seaborn = False):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)
    
    if use_seaborn == False:
        plt.imshow(cm, interpolation='nearest', cmap=cmap)
        plt.colorbar()
        
        fmt = '.2f' if normalize else 'd'
        thresh = cm.max() / 2.
        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
            plt.text(j, i, format(cm[i, j], fmt),
                     horizontalalignment="center",
                     color="white" if cm[i, j] > thresh else "black")
            
        tick_marks = np.arange(len(classes) + 0)
    
    else:
        
        ax = sns.heatmap(cm, annot=True, fmt='d') #notation: "annot" not "annote"
        # fmt='d': print values as decimals
        
        bottom, top = ax.get_ylim()
        ax.set_ylim(bottom + 0.5, top - 0.5)
        tick_marks = np.arange(len(classes) + 1)

    plt.title(title)
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)
    plt.ylabel('True label')
    plt.xlabel('Predicted label') 
    
Now, the main problem: 

# Import libraries.

import DrawConfusionMatrix as dcm
import importlib # The imp module was deprecated in Python 3.4 in favor of the importlib module.
importlib.reload(dcm)

import pandas as pd
import numpy as np
from collections import Counter

from snorkel.augmentation import transformation_function
from snorkel.augmentation import RandomPolicy
from snorkel.augmentation import PandasTFApplier

from sklearn import svm
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix


df = pd.read_csv('datasets_19_420_Iris.csv') 

for i in set(df.Species):
    # ['count', 'mean', 'std', 'min', '25%', '50%', '75%', 'max']
    print(i)
    print(df[df.Species == i].describe().loc[['mean', 'std'], :], '\n') 
	
Iris-versicolor
            Id  SepalLengthCm  SepalWidthCm  PetalLengthCm  PetalWidthCm
mean  75.50000       5.936000      2.770000       4.260000      1.326000
std   14.57738       0.516171      0.313798       0.469911      0.197753 

Iris-virginica
             Id  SepalLengthCm  SepalWidthCm  PetalLengthCm  PetalWidthCm
mean  125.50000        6.58800      2.974000       5.552000       2.02600
std    14.57738        0.63588      0.322497       0.551895       0.27465 

Iris-setosa
            Id  SepalLengthCm  SepalWidthCm  PetalLengthCm  PetalWidthCm
mean  25.50000        5.00600      3.418000       1.464000       0.24400
std   14.57738        0.35249      0.381024       0.173511       0.10721  

 
features = ['SepalLengthCm', 'SepalWidthCm', 'PetalLengthCm', 'PetalWidthCm']

classes = ['Iris-setosa', 'Iris-virginica', 'Iris-versicolor']
desc_dict = {}
for i in classes:
    desc_dict[i] = df[df.Species == i].describe()
	
df['Train'] = 'Train'

# random.randint returns a random integer N such that a <= N <= b

@transformation_function(pre = [])
def get_new_instance_for_this_class(x):
    x.SepalLengthCm = np.random.normal(round(desc_dict[x.Species].loc[['mean'], ['SepalLengthCm']].iloc[0,0], 2) * 100, 
                  round(desc_dict[x.Species].loc[['std'], ['SepalLengthCm']].iloc[0,0], 2) * 100) / 100
    
    x.SepalWidthCm = np.random.normal(round(desc_dict[x.Species].loc[['mean'], ['SepalWidthCm']].iloc[0,0], 2) * 100, 
                  round(desc_dict[x.Species].loc[['std'], ['SepalWidthCm']].iloc[0,0], 2) * 100) / 100
    
    x.PetalLengthCm = np.random.normal(round(desc_dict[x.Species].loc[['mean'], ['PetalLengthCm']].iloc[0,0], 2) * 100, 
                  round(desc_dict[x.Species].loc[['std'], ['PetalLengthCm']].iloc[0,0], 2) * 100) / 100
    
    x.PetalWidthCm = np.random.normal(round(desc_dict[x.Species].loc[['mean'], ['PetalWidthCm']].iloc[0,0], 2) * 100, 
                  round(desc_dict[x.Species].loc[['std'], ['PetalWidthCm']].iloc[0,0], 2) * 100) / 100
    
    x.Train = 'Test'
    return x

tfs = [ get_new_instance_for_this_class ]

random_policy = RandomPolicy(
    len(tfs), sequence_length=2, n_per_original=5, keep_original=True
    # n_per_original (int) – Number of transformed data points per original
)

tf_applier = PandasTFApplier(tfs, random_policy)
df_train_augmented = tf_applier.apply(df)

print(f"Original training set size: {len(df)}")
print(f"Augmented training set size: {len(df_train_augmented)}") 

Original training set size: 150
Augmented training set size: 900 

df_test = df_train_augmented[df_train_augmented.Train == 'Test']

pred = clf.predict(df_test[features])

pred_probs = clf.predict_proba(df_test[features])
# Make Note Of >> AttributeError: predict_proba is not available when 'probability=False'

print(Counter(pred))
print("Accuracy: {:.3f}".format(accuracy_score(df_test['Species'], pred)))

cm = confusion_matrix(df_test['Species'], pred)
print("Confusion matrix:\n{}".format(cm))

Counter({'Iris-versicolor': 252, 'Iris-setosa': 250, 'Iris-virginica': 248})
Accuracy: 0.968
Confusion matrix:
[[250   0   0]
 [  0 239  11]
 [  0  13 237]] 

classes = ['setosa', 'versicolor', 'virginica']

dcm.plot_confusion_matrix(cm, classes = classes, use_seaborn = True) 

# This plot is for 'Support Vector Machine' based classifier.



# This plot is for 'Random Forest' based classifier.



Here we see that there are some misclassified data points for classes 'Versicolor' and 'Verginica'.
'Setosa' has not been misclassified by either SVM or RandomForest.

Next, we would slice the dataframe into 'setosa' and 'not setosa' dataframes. Because we are not having issues with 'setosa' data points, we would re-train a classifier on the other two classes viz. 'versicolor' and 'virginica'.

import re
from snorkel.slicing import slicing_function

@slicing_function()
def not_setosa(x):
    return x.Species != 'Iris-setosa'

sfs = [not_setosa]

# ~ ~ ~

#Store slice metadata in S
from snorkel.slicing import PandasSFApplier

applier = PandasSFApplier(sfs)
S_test = applier.apply(df_test)

# ~ ~ ~

from snorkel.analysis import Scorer

scorer = Scorer(metrics=["f1_micro", "f1_macro"])
# Make Note Of >> ValueError: f1 not supported for multiclass. 
# Try f1_micro or f1_macro instead.

# ~ ~ ~

from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(df_test['Species'])

scorer.score_slices(
    S=S_test, 
    golds=le.transform(df_test['Species']), 
    preds=le.transform(pred), 
    probs=pred_probs, 
    as_dataframe=True
) 



from snorkel.slicing import slice_dataframe

df_not_setosa = slice_dataframe(df_train_augmented, not_setosa)

from sklearn.ensemble import RandomForestClassifier

rfc = RandomForestClassifier(max_depth=4, random_state=0, n_estimators = 100) 

RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
	max_depth=4, max_features='auto', max_leaf_nodes=None,
	min_impurity_decrease=0.0, min_impurity_split=None,
	min_samples_leaf=1, min_samples_split=2,
	min_weight_fraction_leaf=0.0, n_estimators=100,
	n_jobs=None, oob_score=False, random_state=0, verbose=0,
	warm_start=False) 
	
df_test_rfc = df_not_setosa[df_not_setosa.Train == 'Test']
pred_rfc = rfc.predict(df_test_rfc[features])
print(Counter(pred_rfc))
print("Accuracy: {:.3f}".format(accuracy_score(df_test_rfc['Species'], pred_rfc)))

cm = confusion_matrix(df_test_rfc['Species'], pred_rfc)
print("Confusion matrix:\n{}".format(cm))

Counter({'Iris-versicolor': 251, 'Iris-virginica': 249})
Accuracy: 0.990 
Confusion matrix:
[[248   2]
 [  3 247]] 
 
 dcm.plot_confusion_matrix(cm, 
    classes = ['versicolor', 'virginica'], 
    use_seaborn = True) 

Using RandomForestClassifier on sliced dataset:


We also have the score for SVC, it is not as good as RandomForestClassifier:

svc = svm.SVC(gamma = 'auto', probability=True)
svc.fit(df_not_setosa[features], df_not_setosa['Species']) 

SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
    max_iter=-1, probability=True, random_state=None, shrinking=True, tol=0.001,
    verbose=False) 
	
pred_svc = svc.predict(df_test_rfc[features])
print(Counter(pred_svc))
print("Accuracy: {:.3f}".format(accuracy_score(df_test_rfc['Species'], pred_svc)))

cm = confusion_matrix(df_test_rfc['Species'], pred_svc)
print("Confusion matrix:\n{}".format(cm)) 

Counter({'Iris-versicolor': 251, 'Iris-virginica': 249})
Accuracy: 0.986 
Confusion matrix:
[[247   3]
 [  4 246]] 
 
Reference 
% Slice-based Learning: a Programming Model for Residual Learning in Critical Data Slices