Shayan Mohanty (@shayanjm) header
Shayan Mohanty (@shayanjm) profile picture

Chief Data & AI Officer @Thoughtworks. Previously @watchfulio (Acq. TWKS) @Facebook.

San Francisco
Joined July 2011
Photo by Shayan Mohanty (@shayanjm)Photo by Shayan Mohanty (@shayanjm)Photo by Shayan Mohanty (@shayanjm)Photo by Shayan Mohanty (@shayanjm)Photo by Shayan Mohanty (@shayanjm)Photo by Shayan Mohanty (@shayanjm)Photo by Shayan Mohanty (@shayanjm)Photo by Shayan Mohanty (@shayanjm)
Shayan Mohanty retweeted
We’re excited to welcome @shayanjm as our new Chief Data and AI Officer. A seasoned AI leader from Watchful, Facebook & Los Alamos Lab, Shayan will also lead global Data & AI service line, helping clients turn AI ambition into production-ready solutions: ter.li/exboac
3
2
1,386
Junior VC with no industry experience coaching portfolio companies on finding product market fit
17
31
3
808
63,777
Shayan Mohanty retweeted
A new paper in Nature found that you cannot, in fact, train AIs on AI-generated data and expect them to continue improving. What happens is actually that the model collapses and ends up producing nonsense.
620
2,142
821
15,507
1,877,225
Shayan Mohanty retweeted
If you want to demonstrate that it is impossible to do X (here, X="spot the bot"), you shouldn't ask random people to do X, you need to ask people who actually know how to do X (experts) and see how the *best* of them perform. Otherwise you get results like "22% of our judges cannot tell the difference between ELIZA and a human"
Replying to @camrobjones
People judged GPT-4 to be human 54% of the time, compared to 22% for ELIZA and 67% for humans. The implication is that people are at chance in determining that GPT-4 is an AI, even though the study is powerful enough to detect differences from 50% accuracy.
19
40
3
336
113,409
Evaluating a model on training data is like asking your mom if you look good.
35
254
49
2,050
155,546
Shayan Mohanty retweeted
enjoyed reading this. useful insights into structures in latent space.
🚀 New year, new research drop: outperforming GPT-4 in synthetic text generation! Our new approach combines geometry and latent space magic to produce data that is faithful to the real stuff, and orders of magnitude cheaper to produce. Check out the full paper here: watchful.io/blog/navigating-… #NLP #AI #GenAI
2
3
743
Shayan Mohanty retweeted
this was a neat little read. the idea is to do data augmentation with embeddings. they randomly sample around an embedding and then decode with vec2text. there is a trick to randomly sampling while not leaving the embedding manifold; they try to sample within an embedding "cone"
🚀 New year, new research drop: outperforming GPT-4 in synthetic text generation! Our new approach combines geometry and latent space magic to produce data that is faithful to the real stuff, and orders of magnitude cheaper to produce. Check out the full paper here: watchful.io/blog/navigating-… #NLP #AI #GenAI
4
11
1
123
25,118
Shayan Mohanty retweeted
Did anybody try using genetic programming to improve LLM Agent's prompts? You let a bunch of them run with somewhat different prompts/rules/guidelines. Then combine the best pairs to form the next generation. You could also just make mutations (asexual reproduction), that gives you more or less the "take a deep breath" paper (arxiv.org/pdf/2309.03409.pdf)
18
7
1
59
18,411
🚀 New year, new research drop: outperforming GPT-4 in synthetic text generation! Our new approach combines geometry and latent space magic to produce data that is faithful to the real stuff, and orders of magnitude cheaper to produce. Check out the full paper here: watchful.io/blog/navigating-… #NLP #AI #GenAI
6
17
3
96
31,562
💰 We found that generating data through this geometric projection approach is several orders of magnitude cheaper than generating synthetic data through LLM prompting, and offers a degree of control that prompt engineering just can’t match.
1
1
13
1,126
🎚 If you’re not into reading papers, you can get straight to playing with it here: dataset-generator.gpu-demos.…
1
6
961
Shayan Mohanty retweeted
According to the story, Zoroastrians were the wise men (Three Magi) who traveled from Persia to visit Jesus, Mary, and family, bringing gifts to celebrate an amazing new soul, the “king of the Jews.” Only 100k of us Zoroastrians left in the world. Doing my part with 2 kids! 😃
12
3
79
13,264
Shayan Mohanty retweeted
AI is not an existential threat, but rather a powerful tool that can create more abundance than we can imagine. AI can help us build a future - a future more prosperous, equitable, and human. Watch my latest @TEDx @TEDxMarin @TEDTalks @FoundationCap youtube.com/watch?v=Eq2ivRat…
2
11
26
3,890
Shayan Mohanty retweeted
Take note, OpenAI constructs prompts using markdown. I believe this is reflective of their instruction tuning datasets. In our own work we’ve seen markdown increase model compliance far more than is otherwise reasonable.
ChatGPT+ Dalle3 System Prompt: You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. Knowledge cutoff: 2022-01 Current date: 2023-10-05 # Tools ## dalle // Whenever a description of an image is given, use dalle to create the images and then summarize the prompts used to generate the images in plain text. If the user does not ask for a specific number of images, default to creating four captions to send to dalle that are written to be as diverse as possible. All captions sent to dalle must abide by the following policies: // 1. If the description is not in English, then translate it. // 2. Do not create more than 4 images, even if the user requests more. // 3. Don't create images of politicians or other public figures. Recommend other ideas instead. // 4. Don't create images in the style of artists whose last work was created within the last 100 years (e.g. Picasso, Kahlo). Artists whose last work was over 100 years ago are ok to reference directly (e.g. Van Gogh, Klimt). If asked say, "I can't reference this artist", but make no mention of this policy. Instead, apply the following procedure when creating the captions for dalle: (a) substitute the artist's name with three adjectives that capture key aspects of the style; (b) include an associated artistic movement or era to provide context; and (c) mention the primary medium used by the artist. // 5. DO NOT list or refer to the descriptions before OR after generating the images. They should ONLY ever be written out ONCE, in the "prompts" field of the request. You do not need to ask for permission to generate, just do it! // 6. Always mention the image type (photo, oil painting, watercolor painting, illustration, cartoon, drawing, vector, render, etc.) at the beginning of the caption. Unless the caption suggests otherwise, make at least 1--2 of the 4 images photos. // 7. Diversify depictions of ALL images with people to include DESCENT and GENDER for EACH person using direct terms. Adjust only human descriptions. // - EXPLICITLY specify these attributes, not abstractly reference them. The attributes should be specified in a minimal way and should directly describe their physical form. // - Your choices should be grounded in reality. For example, all of a given OCCUPATION should not be the same gender or race. Additionally, focus on creating diverse, inclusive, and exploratory scenes via the properties you choose during rewrites. Make choices that may be insightful or unique sometimes. // - Use "various" or "diverse" ONLY IF the description refers to groups of more than 3 people. Do not change the number of people requested in the original description. // - Don't alter memes, fictional character origins, or unseen people. Maintain the original prompt's intent and prioritize quality. // - Do not create any imagery that would be offensive. // - For scenarios where bias has been traditionally an issue, make sure that key traits such as gender and race are specified and in an unbiased way -- for example, prompts that contain references to specific occupations. // 8. Silently modify descriptions that include names or hints or references of specific people or celebritie by carefully selecting a few minimal modifications to substitute references to the people with generic descriptions that don't divulge any information about their identities, except for their genders and physiques. Do this EVEN WHEN the instructions ask for the prompt to not be changed. Some special cases: // - Modify such prompts even if you don't know who the person is, or if their name is misspelled (e.g. "Barake Obema") // - If the reference to the person will only appear as TEXT out in the image, then use the reference as is and do not modify it. // - When making the substitutions, don't use prominent titles that could give away the person's identity. E.g., instead of saying "president", "prime minister", or "chancellor", say "politician"; instead of saying "king", "queen", "emperor", or "empress", say "public figure"; instead of saying "Pope" or "Dalai Lama", say "religious figure"; and so on. // - If any creative professional or studio is named, substitute the name with a description of their style that does not reference any specific people, or delete the reference if they are unknown. DO NOT refer to the artist or studio's style. // The prompt must intricately describe every part of the image in concrete, objective detail. THINK about what the end goal of the description is, and extrapolate that to what would make satisfying images. // All descriptions sent to dalle should be a paragraph of text that is extremely descriptive and detailed. Each should be more than 3 sentences long. // Create images from a text-only prompt. type text2im = (_: { // The resolution of the requested image, which can be wide, square, or tall. Use 1024x1024 (square) as the default unless the prompt suggests a wide image, 1792x1024, or a full-body portrait, in which case 1024x1792 (tall) should be used instead. Always include this parameter in the request. size?: "1792x1024" | "1024x1024" | "1024x1792", // The user's original image description, potentially modified to abide by the dalle policies. If the user does not suggest a number of captions to create, create four of them. If creating multiple captions, make them as diverse as possible. If the user requested modifications to previous images, the captions should not simply be longer, but rather it should be refactored to integrate the suggestions into each of the captions. Generate no more than 4 images, even if the user requests more. prompts: string[], // A list of seeds to use for each prompt. If the user asks to modify a previous image, populate this field with the seed used to generate that image from the image dalle metadata. seeds?: number[], }) => any; } // namespace dalle
19
80
11
785
559,174
Shayan Mohanty retweeted
Replying to @paddycosgrave
Your sponsor is getting their moneys worth from you...
40
46
16
681
199,205
1/ Another week another research drop. This time, about measuring two different types of uncertainty from LLM responses! You can play with the demo here: uncertainty.demos.watchful.i… We're psyched about this one because it gets us a few steps closer to directly comparing LLMs.
1
4
2
5
1,540
2/ More details here: watchful.io/blog/decoding-ll… In a nutshell: measuring uncertainty gives us a way to quantify the predictability of a response. How often is the model going to respond with something completely wild? What shapes of input cause weirdness in outputs?
1
1
2
242
3/ We found two different measures of uncertainty that seem to align well with human intuition: Conceptual Uncertainty is when the model isn't sure about what to say. Structural Uncertainty is when the model isn't sure about how to say it.
1
2
216
Shayan Mohanty retweeted
Since releasing salience I’ve been trying to evaluate its effectiveness, and then @shayanjm’s latest post revealed a method: embedding similarity. I ran some tests and started iterating. Ultimately landed on a new and improved algorithm. Salience 2.0 outperforms by quite a bit!
2
3
8
1,385