WriteTimeout error when trying to run kbatch job submit

[Drew Terasaki Hart (erthward)]

Hi, All!

I used to be able to submit kbatch jobs just fine. Then, seemingly out of nowhere, I became unable to continue submitting jobs. The error I get is recondite:

Traceback (most recent call last):
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/backends/sync.py", line 36, in write
    n = self._sock.send(buffer)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
    raise exc
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
    response = connection.handle_request(request)
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 90, in handle_request
    return self._connection.handle_request(request)
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 102, in handle_request
    raise exc
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 72, in handle_request
    self._send_request_body(**kwargs)
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 125, in _send_request_body
    self._send_event(event, timeout=timeout)
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 132, in _send_event
    self._network_stream.write(bytes_to_send, timeout=timeout)
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/backends/sync.py", line 37, in write
    buffer = buffer[n:]
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/deth/.local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.WriteTimeout: The write operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/deth/miniconda3/envs/sci/bin/kbatch", line 8, in <module>
    sys.exit(cli())
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/site-packages/kbatch/cli.py", line 281, in submit_job
    result = _core.submit_job(
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/site-packages/kbatch/_core.py", line 170, in submit_job
    return _request_action(kbatch_url, token, "POST", model, json_data=data)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/site-packages/kbatch/_core.py", line 110, in _request_action
    r = client.request(
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_client.py", line 802, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_client.py", line 889, in send
    response = self._send_handling_auth(
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_client.py", line 917, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_client.py", line 954, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_client.py", line 990, in _send_single_request
    response = transport.handle_request(request)
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.WriteTimeout: The write operation timed out

I have uninstalled and reinstalled kbatch, and reconfigured with a new token. None of that fixed or changed the problem in any way.

Before I waste more time on this, any quick ideas about what might be going wrong?

Thanks!

Drew

[Tom Augspurger (TomAugspurger)]

I don't think I've changed anything on the backend that would have affected this.

[Jesse Anderson (jessjaco)]

Sometimes this happens if I accidentally left some big files in the directory I am using as source (assuming you're passing a directory and not a single file).

[Drew Terasaki Hart (erthward)]

Thanks so much! This seems like it may be moving me in the right direction. I had a Jupyter notebook file (small, but still on the order of a few MB) left in the directory. Removing that changed the problem to a new and equally opaque error, however:

Traceback (most recent call last):
  File "/home/deth/miniconda3/envs/sci/bin/kbatch", line 8, in <module>
    sys.exit(cli())
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/deth/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/site-packages/kbatch/cli.py", line 281, in submit_job
    result = _core.submit_job(
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/site-packages/kbatch/_core.py", line 170, in submit_job
    return _request_action(kbatch_url, token, "POST", model, json_data=data)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/site-packages/kbatch/_core.py", line 119, in _request_action
    logger.exception(r.json())
  File "/home/deth/.local/lib/python3.9/site-packages/httpx/_models.py", line 1517, in json
    return jsonlib.loads(self.text, **kwargs)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/deth/miniconda3/envs/sci/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

As always, any insight greatly appreciated!

[Jesse Anderson (jessjaco)]

In my experience, that is usually due to bad characters in the yaml or command line arguments. See kbatch-dev/kbatch#52

[Drew Terasaki Hart (erthward)]

Jesse Anderson (@jessjaco), you are a genius and a rock star. I somehow missed the link at the end of your last reply, and have been scratching my chin for two months, then just circled back to retrace my steps and saw it sitting there, that gem of an answer: I used an underscore in the name of the directory of code I was passing.

The kbatch developers should definitely mention the fact that capitals and underscores break things and give a totally uninformative error, either in bold, red warnings in the MSPC docs (... maybe it's there and I missed it?) or ideally with an informative/helpful error message. I'm definitely not used to having to avoid underscores in my filenames!

Anyhow, case closed, and I seriously owe you one!

[Drew Terasaki Hart (erthward)]

Thanks again! That's really good to know. Unfortunately, I've gone back and carefully double-checked the yaml file, but to no avail. It's identical to what I was able to successfully run before. Tom, any chance you have any experience with or insight into that JSONDecodeError?

…

On Wed, Sep 20, 2023 at 8:09 AM Jesse Anderson ***@***.***> wrote: In my experience, that is usually due to bad characters in the yaml or command line arguments. See kbatch-dev/kbatch#52 <kbatch-dev/kbatch#52> — Reply to this email directly, view it on GitHub <#265 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOSTJU3VSLMH5V5DZ3SOE3X3IQ2NANCNFSM6AAAAAA4XKGPYU> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>