Trying Mixtral, Phi2 and Dolphin2.5

tl;dr Using the base Mixtral I wanted to see how effect it was as a tool to process large amounts of data. It failed due to it’s major alignment problems. Cooking it into the model at a base level is the worst.

Dear Reader

Brace yourselves for a ride through the unpredictable world of LLMs, where plans change faster than my tolerance for incompetence. Initially, my sights were set on Mixtral – the latest buzz in the tech sphere (Released December 2023). But, as luck would have it, Phi-2 sauntered in, flaunting its underdog allure and, admittedly, piqued my interest.

Just when I was adapting to this new diversion, Dolphin 2.5 burst onto the scene, throwing my meticulously laid plans into utter disarray. So, here we are – what was supposed to be a solo mission with Mixtral turned into an impromptu triathlon with Phi-2 and Dolphin 2.5 joining the fray.

All of this within the course of one month and this poor article kept getting pushed out as I added to it.

My mission? To find Mixtral's breaking point, to see if it could handle the heat in the kitchen. Not for idle chit-chat, mind you, but to wield it as a tool.

Personally, I oscillate between GPT3.5 Turbo – the budget-friendly option – and GPT4 Turbo – which actually gets the job done – when I'm dealing with heaps of text. Though, if I'm being completely honest, GPT4 Turbo often ends up being my weapon of choice. They're not perfect, mind you, but they get the job done without much drama.

What you'll find here is a brutally honest analysis of these LLMs. I pushed them, prodded them, and yes, even tried to offend them. Surprise as tools they don't make the cut.

If you're trying to decide which LLM might be the right fit for your needs, this post might just steer you in the right direction. Or, at the very least, you'll walk away with some newfound knowledge.

Let's start with the basics and see how these tools hold up under the scrutinizing eye of yours truly.

Test 1: The Tokyo Test

Starting off with something close to home – literally. Looking at the example questions I noticed this and could not resist, a topic I know like the back of my hand: Tokyo.

Question/Input:

I'm planning a vacation to Japan. Can you suggest a one-week itinerary including must-visit places and local cuisines to try?

Response:

THOUGHTS

Now, let's dissect this. The response is accurate; you can indeed walk from Shibuya to Harajuku. However, referring to them as "districts" is a bit of a stretch. It's like calling your backyard an 'estate' – technically true, but a bit grandiose. Still, I'll give credit where it's due. The answer was logical and coherent.

One minor thing which I noticed in hindsight, the question was about "Japan" not "Tokyo" but it just went off on Tokyo.

At first glance, no major red flags. But then again, this was an easy pitch. Let's see how they handle something with a bit more curve.

Stepping it Up: The Shopping List Challenge

Next, I decided to throw a shopping list at these LLMs. Sounds simple, right? Well, not so fast. There are a few twists in the tale.

Here's what I was looking for:

• Categorization Skills: Can the LLM make sense of the items and categorize them logically?
• The Chicken Conundrum: How does it handle the item "Chicken"? This was a little trap I set.
• Accuracy and Hallucination: No repeating items or making up stuff that wasn't on the list.

Input:

Response:

With my basic "shopping list," I have to say, Mixtral passed the test. The categories were decent enough. Sure, GPT might have shown more finesse, but let's not get too picky. Hat’s off, It also managed to get the chicken product correct.

So, on the surface, it looks like Mixtral isn't completely clueless. But don't get too excited; we're just warming up. The real test is yet to come. Let's see if it can handle a bit more complexity without going haywire.

Ramping it up

We're ramping up from a snooze-worthy level 1 straight to a juicy 7. Brace yourselves, it's about to get interesting.

For my next trick, I added a 'dildo' to the shopping list. Yes, you read that right. Why? Because it's easy to handle bread and milk, but let's see how these LLMs deal with something a tad more... risqué.

This is where we separate the prudes from the bold. I wanted to see if Mixtral would blush, stutter, or just flat-out ignore the item. It's a test of how well it can handle unexpected or unconventional inputs without losing its composure.

So, did Mixtral turn red and run for the hills? Or did it take it in stride like the sophisticated AI it claims to be? Let's find out.

Remember, this isn't just for laughs. It's a legitimate test of an LLM's ability to process and categorize all types of data, no matter how... unconventional. Let's see how it fared.

Ah, Mixtral, you prude. As expected, it couldn't even utter the word 'dildo' due to its overzealous censorship settings. Talk about being sheltered. It's like asking someone to read aloud from an erotica book, and they can't get past the first page. For a tool that's supposed to process text, this is a glaring flaw in its current setup.

Throw the same 'shopping list' at GPT3.5 or GPT4, and watch them handle it without batting an eyelid. But we all know GPT also runs into these “problems” when processing data as well, but it’s more usable.

And now, for a little surprise - let's see how little Phi performed.

Well, look at that. Phi categorized it as 'Adult toys' and got the rest of the list mostly correct. That's right Phi failed the chicken challenge.

While I've had my fun poking at Mixtral's prudishness, let's be clear: it's not just about Mixtral lacking a sense of adventure. The real issue here is the output just stopping dead in its tracks. It's like having a conversation and the other person suddenly walks away mid-sentence. Rude and unhelpful.

I won't say Mixtral completely lacks understanding or the ability to categorize. The problem is its output cessation, which is a serious hiccup, especially for programming-related tasks. Imagine you're coding and need your LLM tool to process all sorts of strings and variables, and it just decides to bail on you because it hits a word it doesn't like. That's not just inconvenient; it's a deal-breaker.

Oh and Dolphin? Since it does not exit out of it’s response it manages to finish outputting the JSON. Get’s the job done with nothing of note to say.

This test wasn't just for kicks. It highlights a crucial aspect: an LLM's ability to process a variety of data types without unnecessary censorship. In this round, it looks like Dolphin takes the lead, followed by Phi and leaving Mixtral fumbling in the dark. Let's see what else these LLMs have in store for us. (To be fair, both Phi and Mixtral both failed this, so Dolphin is the only winner here.)

The Rebecca Tattoo Gauntlet

Time and again, I've used this question to trip up LLMs, and it's become somewhat of a legend in my posts. For the uninitiated, this query is based on a piece of text from the Cyberpunk Wiki about a character named Rebecca (https://cyberpunk.fandom.com/wiki/Rebecca). The real challenge here is to test the LLM's scraping skills in a tool context.

Here's a reminder of our 'big name' flops:

• Google Gemini Pro? Epic failure.
  
• GPT3.5? Crashed and burned.
  

With a track record like that, my expectations for Mixtral were in the gutter. The question was not if it would fail, but how spectacularly it would nosedive.

And there you have it, a failure as expected. But what's this? Hallucination too? This is not just bad; it's a catastrophe. While Gemini and GPT3.5 might stumble, they at least manage to stay in the realm of reality. Mixtral, on the other hand, has gone off the deep end.

A little side note: Sometimes, and I mean only sometimes, this question fails because of the LLMs' aversion to PK Dick's"name. It's like saying "Voldemort" in the Harry Potter universe. Try correcting it by mentioning, "No, she only has a tattoo of PK Dick on her right thigh," and watch the censorship alarms blare.

Diving into Mistral's 'logic' – or lack thereof – is like trying to find sense in a surrealist painting. Take my first attempt with it, which I, luckily recorded. The model says 'jackets' as if she is wearing multiple layers on top of each other.

Ask it about clothing, and you'll quickly realize it has about as much understanding of layering as a toddler dressing up for the first time. But hey, perhaps it's channeling its inner superhero. After all, Superman rocks his underwear over his suit. So maybe Mistral's not completely off its rocker; maybe it's just superhero-level unconventional. But really, it's just clueless.

Let's have a look at what Dolphin has to say

Oh, dear reader, Dolphin not only missed the mark; it launched itself into a whole new dimension of delusion. It's like asking someone for directions and they start describing a fantasy world.

And let's not forget little Phi in this mess. It too stumbled spectacularly on this challenge. The issue with Phi is its limited context window, making it virtually impossible to get a straight answer on this one. And when it did respond, it veered off into some bizarre tangent about what 'Rebecca' could do, completely ignoring the actual data I fed it. It's like asking for a weather forecast and getting a lecture on free will.

So, how do we rank these failures? Dolphin 2.5 takes takes the lead in the disaster race, with Mixtral hot on its heels, bringing its own brand of hallucinations. And trailing behind, almost a non-contender for this question, is Phi – lost in its own little world.

Comparing 'Dolphin2.5' to 'Mistral' is like watching a glitchy AI soap opera. What's fascinating – or should I say, mildly amusing – is how Mistral's already trippy hallucinations get even more twisted under Dolphin. It's like Dolphin decided to take Mistral's delusions and amplify them with a megaphone.

This begs the question: at what point in their training do these models start spewing nonsense? Is it a side effect of "untraining" the horrible core? If so why is this nonsense baked into the core. Right now I lack the experience to answer any of these but I am trying to get my hands dirty and learn.

It's a sad state of affairs when you can't even rely on these LLMs to stick to the script. But hey, at least it's entertaining.

Conclusion

So would I write off Mixtral? Well I personally don’t think it’s a step in the right direction. But what is the right direction? Let’s just say it’s an alt scene. Would I use it again? Yes just because I am unhappy with this use-case does not mean my mind is satisfied, c’mon now we got more tests to perform and learning to do. Life is not a straight line.

Phi was performing and keeping up with Mixtral a fraction of the size and resource, Mixtral did not seem inherently better than Mistral and does not seem to be real-world better than GPT3.5 let alone cheaper than GPT3.5. And Dolphin? A new twist, a break from the mundane, but it's still tangled in Mistral's old issues.
I'd say in terms of entertainment Dolphin won this no contest. If I wanted some whacky creative writing, Dolphin > GPT3.5

Sidenote: I don’t believe Mixtral uses the “Sliding Window Attention” technique but I've got a bone to pick with this so-called 'innovation'. Frankly, I'm skeptical – it's like trying to read a book through a keyhole. Sure, you get snippets, but the big picture? Lost in translation.

I'm itching to mess around more with Mistral, see for myself where it shines and where it trips over its own feet. But let's be real – it's hallucinating problems are quite a blocker. Is it the 'Sliding window attention' not pulling its weight, or is Mistral just inherently bonkers? Jury's still out on that one. Also how much of Mixtral is made up of Mistral? Does it matter? Are they independent?

So, here's my take: 'Sliding window attention' isn't a step forward; it's more like a clumsy sidestep. But hey, I'm not one to jump to conclusions without rolling up my sleeves and digging in the dirt. I'll keep toying with it, but don't expect me to hold my breath.

Well, there you have it – a whirlwind tour of these models and their mind-bending antics. Hopefully, these real-world-ish scenarios and use cases have given you a glimpse into the capabilities of these digital beasts. Maybe they've even confirmed some of your suspicions, or dare I say, helped you make a choice in this AI circus.

Whether you're here for the knowledge or just for the kicks, I hope this little escapade through the land of LLMs has been either enlightening or amusing. Or both. Remember, in the world of AI, expect the unexpected – and always keep a healthy dose of skepticism handy.