Step 1: Create an API key for free by logging into Google AI Studio
A:
B:
C:
D:
E:
F: Your free API key is created. Copy it and save it somewhere.
Trying a couple of things...
import google.generativeai as genai
API_KEY = 'A...o'
genai.configure(api_key=API_KEY)
model = genai.GenerativeModel()
response = model.generate_content('Teach me about how an LLM works')
print(response.text)
**Understanding Large Language Models (LLMs)**
**Introduction:**
LLMs are advanced machine learning models trained on vast amounts of text data. They can generate human-like text, translate languages, write different types of content, and perform various other text-related tasks.
**Working Principle:**
LLMs are trained on billions of words of text using a technique called transformer neural networks. These networks process sequential data and learn the relationships between words and phrases within the text.
**Training Process:**
1. **Massive Data Collection:** LLMs are trained on enormous datasets of text from various sources, such as books, articles, news, and social media posts.
2. **Tokenization:** The text is broken down into individual words and phrases called "tokens."
3. **Encoding:** Tokens are converted into numerical representations using embedding techniques.
4. **Transformer Architecture:** The transformer neural network processes the encoded tokens, identifying patterns and learning relationships within the text.
5. **Parameter Optimization:** The model adjusts its internal parameters (weights and biases) to minimize a loss function, which measures how well it predicts the next words in the text.
6. **Fine-tuning:** After general training, the LLM can be fine-tuned for specific tasks, such as language translation or content generation.
**Capabilities:**
* **Text Generation:** LLMs can create original, fluent, and grammatically correct text in response to a given prompt.
* **Language Translation:** They can translate text from one language to another with high accuracy.
* **Question Answering:** LLMs can extract information from text and answer questions related to the content.
* **Summarization:** They can condense large amounts of text into concise and informative summaries.
* **Content Creation:** LLMs can generate various types of content, including articles, stories, poems, and website copy.
**Applications:**
* **Natural Language Processing (NLP):** Sentiment analysis, text classification, chatbots
* **Content Generation:** Creative writing, marketing materials, news articles
* **Education:** Language learning, essay writing assistance
* **Research:** Literature analysis, data extraction
* **Customer Service:** Automated response systems, chat support
**Limitations:**
* **Bias:** LLMs can exhibit biases present in their training data.
* **Factual Errors:** They are not immune to factual errors in the sources they are trained on.
* **Limited Contextual Understanding:** They may struggle to understand the full context of complex text.
* **Ethics:** The use of LLMs raises ethical concerns about authorship, misinformation, and deepfakes.
Listing all models
import pprint
for model in genai.list_models():
pprint.pprint(model)
Model(name='models/chat-bison-001',
base_model_id='',
version='001',
display_name='PaLM 2 Chat (Legacy)',
description='A legacy text-only model optimized for chat conversations',
input_token_limit=4096,
output_token_limit=1024,
supported_generation_methods=['generateMessage', 'countMessageTokens'],
temperature=0.25,
top_p=0.95,
top_k=40)
Model(name='models/text-bison-001',
base_model_id='',
version='001',
display_name='PaLM 2 (Legacy)',
description='A legacy model that understands text and generates text as an output',
input_token_limit=8196,
output_token_limit=1024,
supported_generation_methods=['generateText', 'countTextTokens', 'createTunedTextModel'],
temperature=0.7,
top_p=0.95,
top_k=40)
Model(name='models/embedding-gecko-001',
base_model_id='',
version='001',
display_name='Embedding Gecko',
description='Obtain a distributed representation of a text.',
input_token_limit=1024,
output_token_limit=1,
supported_generation_methods=['embedText', 'countTextTokens'],
temperature=None,
top_p=None,
top_k=None)
Model(name='models/gemini-1.0-pro',
base_model_id='',
version='001',
display_name='Gemini 1.0 Pro',
description='The best model for scaling across a wide range of tasks',
input_token_limit=30720,
output_token_limit=2048,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=0.9,
top_p=1.0,
top_k=None)
Model(name='models/gemini-1.0-pro-001',
base_model_id='',
version='001',
display_name='Gemini 1.0 Pro 001 (Tuning)',
description=('The best model for scaling across a wide range of tasks. This is a stable '
'model that supports tuning.'),
input_token_limit=30720,
output_token_limit=2048,
supported_generation_methods=['generateContent', 'countTokens', 'createTunedModel'],
temperature=0.9,
top_p=1.0,
top_k=None)
Model(name='models/gemini-1.0-pro-latest',
base_model_id='',
version='001',
display_name='Gemini 1.0 Pro Latest',
description=('The best model for scaling across a wide range of tasks. This is the latest '
'model.'),
input_token_limit=30720,
output_token_limit=2048,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=0.9,
top_p=1.0,
top_k=None)
Model(name='models/gemini-1.0-pro-vision-latest',
base_model_id='',
version='001',
display_name='Gemini 1.0 Pro Vision',
description='The best image understanding model to handle a broad range of applications',
input_token_limit=12288,
output_token_limit=4096,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=0.4,
top_p=1.0,
top_k=32)
Model(name='models/gemini-1.5-flash',
base_model_id='',
version='001',
display_name='Gemini 1.5 Flash',
description='Fast and versatile multimodal model for scaling across diverse tasks',
input_token_limit=1048576,
output_token_limit=8192,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=1.0,
top_p=0.95,
top_k=64)
Model(name='models/gemini-1.5-flash-001',
base_model_id='',
version='001',
display_name='Gemini 1.5 Flash 001',
description='Fast and versatile multimodal model for scaling across diverse tasks',
input_token_limit=1048576,
output_token_limit=8192,
supported_generation_methods=['generateContent', 'countTokens', 'createCachedContent'],
temperature=1.0,
top_p=0.95,
top_k=64)
Model(name='models/gemini-1.5-flash-latest',
base_model_id='',
version='001',
display_name='Gemini 1.5 Flash Latest',
description='Fast and versatile multimodal model for scaling across diverse tasks',
input_token_limit=1048576,
output_token_limit=8192,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=1.0,
top_p=0.95,
top_k=64)
Model(name='models/gemini-1.5-pro',
base_model_id='',
version='001',
display_name='Gemini 1.5 Pro',
description='Mid-size multimodal model that supports up to 1 million tokens',
input_token_limit=2097152,
output_token_limit=8192,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=1.0,
top_p=0.95,
top_k=64)
Model(name='models/gemini-1.5-pro-001',
base_model_id='',
version='001',
display_name='Gemini 1.5 Pro 001',
description='Mid-size multimodal model that supports up to 1 million tokens',
input_token_limit=2097152,
output_token_limit=8192,
supported_generation_methods=['generateContent', 'countTokens', 'createCachedContent'],
temperature=1.0,
top_p=0.95,
top_k=64)
Model(name='models/gemini-1.5-pro-latest',
base_model_id='',
version='001',
display_name='Gemini 1.5 Pro Latest',
description='Mid-size multimodal model that supports up to 1 million tokens',
input_token_limit=2097152,
output_token_limit=8192,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=1.0,
top_p=0.95,
top_k=64)
Model(name='models/gemini-pro',
base_model_id='',
version='001',
display_name='Gemini 1.0 Pro',
description='The best model for scaling across a wide range of tasks',
input_token_limit=30720,
output_token_limit=2048,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=0.9,
top_p=1.0,
top_k=None)
Model(name='models/gemini-pro-vision',
base_model_id='',
version='001',
display_name='Gemini 1.0 Pro Vision',
description='The best image understanding model to handle a broad range of applications',
input_token_limit=12288,
output_token_limit=4096,
supported_generation_methods=['generateContent', 'countTokens'],
temperature=0.4,
top_p=1.0,
top_k=32)
Model(name='models/embedding-001',
base_model_id='',
version='001',
display_name='Embedding 001',
description='Obtain a distributed representation of a text.',
input_token_limit=2048,
output_token_limit=1,
supported_generation_methods=['embedContent'],
temperature=None,
top_p=None,
top_k=None)
Model(name='models/text-embedding-004',
base_model_id='',
version='004',
display_name='Text Embedding 004',
description='Obtain a distributed representation of a text.',
input_token_limit=2048,
output_token_limit=1,
supported_generation_methods=['embedContent'],
temperature=None,
top_p=None,
top_k=None)
Model(name='models/aqa',
base_model_id='',
version='001',
display_name='Model that performs Attributed Question Answering.',
description=('Model trained to return answers to questions that are grounded in provided '
'sources, along with estimating answerable probability.'),
input_token_limit=7168,
output_token_limit=1024,
supported_generation_methods=['generateAnswer'],
temperature=0.2,
top_p=1.0,
top_k=40)
Question 1: Write a CREATE query having the following nodes and the relationship from ROOT to other nodes is 'HAS_CHILD'.
ROOT
|--BROKER
|--PROVIDER
|--MEMBER
Answer:
CREATE (root:ROOT),
(broker:BROKER),
(provider:PROVIDER),
(member:MEMBER),
(root)-[:HAS_CHILD]->(broker),
(root)-[:HAS_CHILD]->(provider),
(root)-[:HAS_CHILD]->(member)
~~~
Question 2: Write a DELETE query to delete all nodes and relationships in a graph.
Answer:
MATCH (n) DETACH DELETE n
Ref
~~~
Question 3: Write a query to get a count for all nodes of a given label:
Answer:
MATCH (n:Person)
RETURN count(n) as count
Ref
~~~
Question 4: There are three EPIC nodes in my graph.
Each node has a numerical property CUSTOM_ID.
Now, I want to retrieve the node with the largest CUSTOM_ID.
Answer:
MATCH (n:EPIC)
RETURN n
ORDER BY n.CUSTOM_ID DESC
LIMIT 1
~~~
Question 5: Write query to get a node by property value in Neo4j.
Answer:
MATCH (n)
WHERE n.name = 'Mark'
RETURN n
Ref
~~~
Question 6: Delete a node with a given property.
Answer:
MATCH (n:Person {name: 'Tom Hanks'})
DELETE n
Ref
~~~
Question 7: Delete ONLY nodes having label of ENTITY:
Answer:
MATCH (n:ENTITY)
DELETE n
~~~
Question 8: Return number of EPIC nodes in the knowledge graph.
Answer:
MATCH (epic:EPIC)
RETURN count(epic) as count
~~~
Question 9: Write a query to get the EPIC node with largest numerical property of CUSTOM_ID.
Answer:
MATCH (epic:EPIC)
RETURN epic
ORDER BY epic.CUSTOM_ID DESC
LIMIT 1
~~~
Question 10:
What are some of the use cases where Between Centrality Algorithm is used?
Answer:
The Betweenness Centrality Algorithm is a powerful tool used to understand the roles of nodes in a graph and their impact on the network. Here are some use cases where it finds application:
Supply Chain Risk Analysis: In supply chain processes, Betweenness Centrality helps identify critical nodes that act as bridges between different parts of the network. For example, when transporting a product internationally, it can pinpoint bottleneck nodes during cargo ship stops in intermediate ports1.
Power Grid Contingency Analysis: The algorithm is used to analyze power grid networks, identifying critical nodes that affect the flow of electricity. Due to its computational intensity, this application often requires supercomputers2.
Community Detection and Network Routing: Betweenness Centrality assists in Girvan–Newman community detection and network routing tasks. It helps find influential nodes that connect different communities or guide information flow2.
Artificial Intelligence and Skill Characterization: Skill characterization in AI relies on identifying influential nodes. Betweenness Centrality helps determine which nodes play a crucial role in spreading information or resources2.
Epidemiology and Rumor Spreading: In epidemiology, it identifies nodes that influence the spread of diseases. Similarly, it helps analyze rumor propagation in social networks1.
Transportation Networks: The algorithm is applied to transportation networks, such as road or rail systems, to find critical nodes affecting traffic flow or resource distribution1.
Remember, Betweenness Centrality is about detecting nodes that serve as bridges, allowing information or resources to flow efficiently across a graph.
1: graphable.ai
2: computationalsocialnetworks.springeropen.com
3: nature.com
---
1.
Graph Machine Learning: Take Graph Data to the Next Level by Applying Machine Learning Techniques and Algorithms
Enrico Deusebio, 2021
2.
Graph-Powered Machine Learning
Alessandro Negro, 2021
3.
Graph Representation Learning
William L. Hamilton, 2020
4.
Deep Learning on Graphs
Jiliang Tang, 2021
5.
Graph-Powered Analytics and Machine Learning with TigerGraph
Alexander Thomas, 2023
6.
Graph Neural Networks: Foundations, Frontiers, and Applications
2022
7.
Graph Algorithms: Practical Examples in Apache Spark and Neo4j
Amy E. Hodler, 2019
8.
Building Knowledge Graphs
Jim Webber, 2023
9.
Graph Algorithms for Data Science: With Examples in Neo4j
Tomaž Bratanic, 2024
10.
Graph Neural Networks in Action
Keita Broadwater, 2024
11.
Hands-On Graph Neural Networks Using Python: Practical Techniques and Architectures for Building Powerful Graph and Deep Learning Apps with PyTorch
Maxime Labonne, 2023
12.
The Practitioner's Guide to Graph Data: Applying Graph Thinking and Graph Technologies to Solve Complex Problems
Denise Koessler Gosnell, 2020
13.
Algorithms in C, Part 5: Graph Algorithms
Robert Sedgewick, 2001
14.
Mining of Massive Datasets
Jeffrey Ullman, 2011
15.
Machine Learning for Text
Charu C. Aggarwal, 2018
16.
Knowledge Graphs: Fundamentals, Techniques, and Applications
Craig A. Knoblock, 2021
17.
Networks, Crowds, and Markets: Reasoning about a Highly Connected World
Jon Kleinberg, 2010
18.
Graph-based Natural Language Processing and Information Retrieval
Dragomir R. Radev, 2011
19.
Designing and Building Enterprise Knowledge Graphs
(Synthesis Lectures on Data, Semantics, and Knowledge)
Juan Sequeda, Ora Lassila
Morgan & Claypool (2021)
1. How comfortable are you in Python?
2. How comfortable are you in PySpark?
3. How comfortable are you in Scala?
4. And shell scripting?
---
1. What is the difference between list and tuple?
2. What are the 3 ways to work on a dataset in PySpark? (RDD, Spark SQL, and Pandas Dataframe)
3. What is lazy evaluation?
4. What is the opposite of lazy evaluation? (Eager evaluation)
5. What is the regular expression?
6. What does grep command do?
7. What does find command do?
8. What is the difference between find and grep?
9. What does sed command do?
10. What does awk command do?
11. What is narrow transformation? (Like map())
12. What is wide transformation? (Like groupby and reduceby)
13. What is the difference between narrow transformation and wide transformation?
14. How much would you give yourself in Hive?
15. Write SQL query to get current date from Hive SQL interface? (getdate(), now())
16. Take out the year from the date. (year(date_col))
17. How would you get a;b;c into:
a
b
c
Into three rows.
18. What is Spark session? (Entry point to create Spark context)
19. What is spark context?
20. Scope of which one is bigger?
21. Is there any other context object we need to know about?
22. There is a CSV file. You have to load this CSV data into an RDD, SQL dataframe, and Pandas dataframe.
THE PSYCHOLOGIST GARY Klein once told me a story about a woman who attended a family gathering. She had spent years working as a paramedic and, upon arriving at the event, took one look at her father- in-law and got very concerned.
“I don’t like the way you look,” she said.
Her father-in-law, who was feeling perfectly fine, jokingly replied,
“Well, I don’t like your looks, either.”
“No,” she insisted. “You need to go to the hospital now.”
A few hours later, the man was undergoing lifesaving surgery after an examination had revealed that he had a blockage to a major artery and was at immediate risk of a heart attack. Without his daughter-in- law’s intuition, he could have died.
What did the paramedic see? How did she predict his impending heart attack? When major arteries are obstructed, the body focuses on sending blood to critical organs and away from peripheral locations near the surface of the skin. The result is a change in the pattern of distribution of blood in the face. After many years of working with people with heart failure, the woman had unknowingly developed the ability to recognize this pattern on sight. She couldn’t explain what it was that she noticed in her father-in-law’s face, but she knew something was wrong.
~~~
...we must begin the process of behavior change with awareness
The human brain is a prediction machine. It is continuously taking in your surroundings and analyzing the information it comes across.
Whenever you experience something repeatedly—like a paramedic seeing the face of a heart attack patient or a military analyst seeing a missile on a radar screen—your brain begins noticing what is important, sorting through the details and highlighting the relevant cues, and cataloging that information for future use.
With enough practice, you can pick up on the cues that predict certain outcomes without consciously thinking about it. Automatically, your brain encodes the lessons learned through experience. We can’t always explain what it is we are learning, but learning is happening all along the way, and your ability to notice the relevant cues in a given situation is the foundation for every habit you have.
We underestimate how much our brains and bodies can do without thinking. You do not tell your hair to grow, your heart to pump, your lungs to breathe, or your stomach to digest. And yet your body handles all this and more on autopilot. You are much more than your conscious self.
Consider hunger. How do you know when you’re hungry? You don’t necessarily have to see a cookie on the counter to realize that it is time to eat. Appetite and hunger are governed nonconsciously. Your body has a variety of feedback loops that gradually alert you when it is time to eat again and that track what is going on around you and within you.
Cravings can arise thanks to hormones and chemicals circulating through your body. Suddenly, you’re hungry even though you’re not quite sure what tipped you off.
This is one of the most surprising insights about our habits: you don’t need to be aware of the cue for a habit to begin. You can notice an opportunity and take action without dedicating conscious attention to it. This is what makes habits useful.
It’s also what makes them dangerous. As habits form, your actions come under the direction of your automatic and nonconscious mind.
You fall into old patterns before you realize what’s happening. Unless someone points it out, you may not notice that you cover your mouth with your hand whenever you laugh, that you apologize before asking a question, or that you have a habit of finishing other people’s sentences.
And the more you repeat these patterns, the less likely you become to question what you’re doing and why you’re doing it.
Over time, the cues that spark our habits become so common that they are essentially invisible: the treats on the kitchen counter, the remote control next to the couch, the phone in our pocket. Our responses to these cues are so deeply encoded that it may feel like the urge to act comes from nowhere. For this reason, we must begin the process of behavior change with awareness.
THE HABITS SCORECARD
The Japanese railway system
The Japanese railway system is regarded as one of the best in the world. If you ever find yourself riding a train in Tokyo, you’ll notice that the conductors have a peculiar habit.
As each operator runs the train, they proceed through a ritual of pointing at different objects and calling out commands. When the train approaches a signal, the operator will point at it and say, “Signal is green.” As the train pulls into and out of each station, the operator will point at the speedometer and call out the exact speed. When it’s time to leave, the operator will point at the timetable and state the time. Out on the platform, other employees are performing similar actions.
Before each train departs, staff members will point along the edge of the platform and declare, “All clear!” Every detail is identified, pointed at, and named aloud.* This process, known as Pointing-and-Calling, is a safety system designed to reduce mistakes. It seems silly, but it works incredibly well. Pointing-and-Calling reduces errors by up to 85 percent and cuts accidents by 30 percent. The MTA subway system in New York City adopted a modified version that is “point-only,” and “within two years of implementation, incidents of incorrectly berthed subways fell 57 percent.”
Pointing-and-Calling is so effective because it raises the level of awareness from a nonconscious habit to a more conscious level.
Because the train operators must use their eyes, hands, mouth, and ears, they are more likely to notice problems before something goes wrong.
The more automatic a behavior becomes, the less likely we are to consciously think about it. And when we’ve done something a thousand times before, we begin to overlook things. We assume that the next time will be just like the last. We’re so used to doing what we’ve always done that we don’t stop to question whether it’s the right thing to do at all. Many of our failures in performance are largely attributable to a lack of self-awareness.
One of our greatest challenges in changing habits is maintaining awareness of what we are actually doing. This helps explain why the consequences of bad habits can sneak up on us. We need a “point-and- call” system for our personal lives. That’s the origin of the Habits Scorecard, which is a simple exercise you can use to become more aware of your behavior. To create your own, make a list of your daily habits.
Here’s a sample of where your list might start:
Wake up
Turn off alarm
Check my phone
Go to the bathroom
Weigh myself
Take a shower
Brush my teeth
Floss my teeth
Put on deodorant
Hang up towel to dry
Get dressed
Make a cup of tea . . . and so on.
Once you have a full list, look at each behavior, and ask yourself, “Is this a good habit, a bad habit, or a neutral habit?” If it is a good habit, write “+” next to it. If it is a bad habit, write “–”. If it is a neutral habit, write “=”.For example, the list above might look like this:
Wake up =
Turn off alarm =
Check my phone –
Go to the bathroom =
Weigh myself +
Take a shower +
Brush my teeth +
Floss my teeth +
Put on deodorant +
Hang up towel to dry =
Get dressed =
Make a cup of tea +
The marks you give to a particular habit will depend on your situation and your goals. For someone who is trying to lose weight, eating a bagel with peanut butter every morning might be a bad habit.
For someone who is trying to bulk up and add muscle, the same behavior might be a good habit. It all depends on what you’re working toward.* Scoring your habits can be a bit more complex for another reason as well. The labels “good habit” and “bad habit” are slightly inaccurate.
There are no good habits or bad habits. There are only effective habits.
That is, effective at solving problems. All habits serve you in some way —even the bad ones—which is why you repeat them. For this exercise, categorize your habits by how they will benefit you in the long run.
Generally speaking, good habits will have net positive outcomes. Bad habits have net negative outcomes. Smoking a cigarette may reduce stress right now (that’s how it’s serving you), but it’s not a healthy long-term behavior.
If you’re still having trouble determining how to rate a particular habit, here is a question I like to use: “Does this behavior help me become the type of person I wish to be? Does this habit cast a vote foror against my desired identity?” Habits that reinforce your desired identity are usually good. Habits that conflict with your desired identity are usually bad.
As you create your Habits Scorecard, there is no need to change anything at first. The goal is to simply notice what is actually going on.
Observe your thoughts and actions without judgment or internal criticism. Don’t blame yourself for your faults. Don’t praise yourself for your successes.
If you eat a chocolate bar every morning, acknowledge it, almost as if you were watching someone else. Oh, how interesting that they would do such a thing. If you binge-eat, simply notice that you are eating more calories than you should. If you waste time online, notice that you are spending your life in a way that you do not want to.
The first step to changing bad habits is to be on the lookout for them. If you feel like you need extra help, then you can try Pointing- and-Calling in your own life. Say out loud the action that you are thinking of taking and what the outcome will be. If you want to cut back on your junk food habit but notice yourself grabbing another cookie, say out loud, “I’m about to eat this cookie, but I don’t need it.
Eating it will cause me to gain weight and hurt my health.”
Hearing your bad habits spoken aloud makes the consequences seem more real. It adds weight to the action rather than letting yourself mindlessly slip into an old routine. This approach is useful even if you’re simply trying to remember a task on your to-do list. Just saying out loud, “Tomorrow, I need to go to the post office after lunch,” increases the odds that you’ll actually do it. You’re getting yourself to acknowledge the need for action—and that can make all the difference.
The process of behavior change always starts with awareness.
Strategies like Pointing-and-Calling and the Habits Scorecard are focused on getting you to recognize your habits and acknowledge the cues that trigger them, which makes it possible to respond in a way that benefits you.
Key Points
# With enough practice, your brain will pick up on the cues that predict certain outcomes without consciously thinking about it.
# Once our habits become automatic, we stop paying attention to what we are doing.
# The process of behavior change always starts with awareness. You need to be aware of your habits before you can change them.
# Pointing-and-Calling raises your level of awareness from a nonconscious habit to a more conscious level by verbalizing your actions.
# The Habits Scorecard is a simple exercise you can use to become more aware of your behavior.
