[Bug]: Cannot run a LMStudio local model #14502
Description
What happened?
I got a LMStudio model running locally. I ensured that it worked fine by doing a curl request:
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-8b",
"messages": [
{ "role": "system", "content": "Always answer in rhymes. Today is Thursday" },
{ "role": "user", "content": "What day is it today?" }
],
"temperature": 0.7,
"max_tokens": -1,
"stream": false
}'
However, when I try to get a completion using LiteLLM its fails. Here is the code:
import os
from litellm import completion
os.environ['LM_STUDIO_API_BASE'] = "http://127.0.0.1:1234"
messages = [{ "content": "Hello, how are you?","role": "user"}]
response = completion(model="lm_studio/qwen3-8b", messages=messages)The model name and endpoint base URL are the same.
This fails with the following error:
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
Provider List: https://docs.litellm.ai/docs/providers
Traceback (most recent call last):
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_transports/default.py", line 250, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
raise exc from None
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
response = connection.handle_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
return self._connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
raise exc
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 86, in handle_request
self._send_request_headers(**kwargs)
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 144, in _send_request_headers
with map_exceptions({h11.LocalProtocolError: LocalProtocolError}):
File "/Users/louislac/.local/share/uv/python/cpython-3.11.9-macos-aarch64-none/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.LocalProtocolError: Illegal header value b'Bearer '
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/openai/_base_client.py", line 982, in request
response = self._client.send(
^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_client.py", line 1014, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_transports/default.py", line 249, in handle_request
with map_httpcore_exceptions():
File "/Users/louislac/.local/share/uv/python/cpython-3.11.9-macos-aarch64-none/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.LocalProtocolError: Illegal header value b'Bearer '
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/llms/openai/openai.py", line 736, in completion
raise e
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/llms/openai/openai.py", line 664, in completion
) = self.make_sync_openai_chat_completion_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/litellm_core_utils/logging_utils.py", line 237, in sync_wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/llms/openai/openai.py", line 482, in make_sync_openai_chat_completion_request
raise e
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/llms/openai/openai.py", line 464, in make_sync_openai_chat_completion_request
raw_response = openai_client.chat.completions.with_raw_response.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/openai/_legacy_response.py", line 364, in wrapped
return cast(LegacyAPIResponse[R], func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/openai/_utils/_utils.py", line 286, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/openai/resources/chat/completions/completions.py", line 1147, in create
return self._post(
^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/openai/_base_client.py", line 1259, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/openai/_base_client.py", line 1014, in request
raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/main.py", line 2070, in completion
raise e
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/main.py", line 2043, in completion
response = openai_chat_completions.completion(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/llms/openai/openai.py", line 747, in completion
raise OpenAIError(
litellm.llms.openai.common_utils.OpenAIError: Connection error.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/utils.py", line 1343, in wrapper
raise e
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/utils.py", line 1218, in wrapper
result = original_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/main.py", line 3617, in completion
raise exception_type(
^^^^^^^^^^^^^^^
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2301, in exception_type
raise e
File "/Users/louislac/Library/Caches/uv/archive-v0/KY6HUjgtA_HiZxzY_HvFv/lib/python3.11/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 501, in exception_type
raise InternalServerError(
litellm.exceptions.InternalServerError: litellm.InternalServerError: InternalServerError: Lm_studioException - Connection error.
Nothings shows up in LMStudio logs.
I tried to turn on litellm._turn_on_debug() but this adds little details:
21:02:03 - LiteLLM:DEBUG: utils.py:350 -
21:02:03 - LiteLLM:DEBUG: utils.py:350 - Request to litellm:
21:02:03 - LiteLLM:DEBUG: utils.py:350 - litellm.completion(model='lm_studio/qwen3-8b', messages=[{'content': 'Hello, how are you?', 'role': 'user'}])
21:02:03 - LiteLLM:DEBUG: utils.py:350 -
21:02:03 - LiteLLM:DEBUG: litellm_logging.py:474 - self.optional_params: {}
21:02:03 - LiteLLM:DEBUG: utils.py:350 - SYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache')['no-cache']: False
21:02:03 - LiteLLM:INFO: utils.py:3347 -
LiteLLM completion() model= qwen3-8b; provider = lm_studio
21:02:03 - LiteLLM:DEBUG: utils.py:3350 -
LiteLLM: Params passed to completion() {'model': 'qwen3-8b', 'functions': None, 'function_call': None, 'temperature': None, 'top_p': None, 'n': None, 'stream': None, 'stream_options': None, 'stop': None, 'max_tokens': None, 'max_completion_tokens': None, 'modalities': None, 'prediction': None, 'audio': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'custom_llm_provider': 'lm_studio', 'response_format': None, 'seed': None, 'tools': None, 'tool_choice': None, 'max_retries': None, 'logprobs': None, 'top_logprobs': None, 'extra_headers': None, 'api_version': None, 'parallel_tool_calls': None, 'drop_params': None, 'allowed_openai_params': None, 'reasoning_effort': None, 'additional_drop_params': None, 'messages': [{'content': 'Hello, how are you?', 'role': 'user'}], 'thinking': None, 'web_search_options': None, 'safety_identifier': None}
21:02:03 - LiteLLM:DEBUG: utils.py:3353 -
LiteLLM: Non-Default params passed to completion() {}
21:02:03 - LiteLLM:DEBUG: utils.py:350 - Final returned optional params: {'extra_body': {}}
21:02:03 - LiteLLM:DEBUG: litellm_logging.py:474 - self.optional_params: {'extra_body': {}}
21:02:03 - LiteLLM:DEBUG: utils.py:4730 - checking potential_model_names in litellm.model_cost: {'split_model': 'qwen3-8b', 'combined_model_name': 'lm_studio/qwen3-8b', 'stripped_model_name': 'qwen3-8b', 'combined_stripped_model_name': 'lm_studio/qwen3-8b', 'custom_llm_provider': 'lm_studio'}
21:02:03 - LiteLLM:DEBUG: utils.py:4945 - Error getting model info: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
21:02:03 - LiteLLM:DEBUG: main.py:868 - Error getting model info: This model isn't mapped yet. model=qwen3-8b, custom_llm_provider=lm_studio. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.
21:02:03 - LiteLLM:DEBUG: litellm_logging.py:929 -
POST Request Sent from LiteLLM:
curl -X POST \
http://127.0.0.1:1234 \
-d '{'model': 'qwen3-8b', 'messages': [{'content': 'Hello, how are you?', 'role': 'user'}], 'extra_body': {}}'
21:02:05 - LiteLLM:DEBUG: main.py:5849 - openai.py: Received openai error - Connection error.
21:02:05 - LiteLLM:DEBUG: utils.py:350 - RAW RESPONSE:
Connection error.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
Provider List: https://docs.litellm.ai/docs/providers
21:02:05 - LiteLLM:DEBUG: get_api_base.py:62 - Error occurred in getting api base - litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=qwen3-8b
Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers
21:02:05 - LiteLLM:DEBUG: exception_mapping_utils.py:2330 - Logging Details: logger_fn - None | callable(logger_fn) - False
21:02:05 - LiteLLM:DEBUG: litellm_logging.py:2455 - Logging Details LiteLLM-Failure Call: []
Relevant log output
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
v1.74.9
Twitter / LinkedIn details
No response