menu

Questions & Answers

the log fd was flush to write to mq socket fd by system

I use python uwsgi to make my code. there is a error occur sometime when i send message to rabbbitmq and didn't use confirm mode. the error like this:

rabbmitmq server log error =ERROR REPORT==== 21-Jul-2022::15:23:04 === Error on AMQP connection <0.23590.991> (172.198.12.10:59211 -> 10.0.12.1:5672, vhost: 'host', user: 'host', state: running), channel 12848: operation none caused a connection exception frame_error: "type 91, all octets = <<>>: {frame_too_large,842149168,131064}"

in other way, i find my python create log file fd also put some error logs :

python log error IOError: [Errno 9] Bad file descriptor Traceback (most recent call last): File "/usr/lib64/python2.6/logging/init.py", line 800, in emit self.flush() File "/usr/lib64/python2.6/logging/init.py", line 762, in flush self.stream.flush()

this is create python log file code :

   def init_logger(self):
    self._logger = logging.getLogger(self._proj)
    self._logger.setLevel(logging.DEBUG)
    formatter = logging.Formatter('[%(asctime)s] [%(process)d] [%(levelname)s] %(message)s')
    if not self._b_stream_init:
        stream_handler = logging.StreamHandler(sys.stderr)
        stream_handler.setFormatter(formatter)
        stream_handler.setLevel(logging.DEBUG)
        self._logger.addHandler(stream_handler)
        self._b_stream_init = True
    ret = self.check_log_name()
    if ret[0]:
        return 0
    try:
        log_file_handler = logging.FileHandler(ret[1])
        log_file_handler.setFormatter(formatter)
        log_file_handler.setLevel(CLog.LEVEL_MAP[self._log_level])
        self._logger.addHandler(log_file_handler)
        if self._last_file_handle is not None:
            self._logger.removeHandler(self._last_file_handle)
            self._last_file_handle.close()
        self._last_file_handle = log_file_handler
        self._last_log_name = ret[1]
    except:
        pass
def check_log_name(self):
    if self._log_dir is None or self._log_prefix is None:
        return True, None
    log_name_arr = [self._log_dir, self._log_prefix, '_', time.strftime('%Y%m%d_%H'), '.log']
    log_name = ''.join(log_name_arr)
    if self._last_log_name != log_name or not os.path.exists(log_name):
        return False, log_name
    else:
        return True, log_name

this code raise exception but i can't find out the reason why.

this code i use client send message to rabbitmq server:
@trace_report(switch=True)
def send(self, key, message, declare=False, expiration=None):

    if declare:
        self._declare_exchange()
        self._declare_queue_with_key(key)
    if isinstance(expiration, int):
        expiration = str(expiration)
    properties = pika.BasicProperties(delivery_mode=2,
                                      expiration=expiration)
    self.channel.basic_publish(exchange=self.exchange, routing_key=key,
                               body=message, properties=properties)

I guess that the file system write log content to the rabbitmq server socket fd by mistake. what i can do in this situation.

ps: socket_wait was so much(4w) but without no any kernel log

Comments:
2023-01-18 23:52:07
There's too much information you haven't told us. How did you create your logging? Are you using fork? Somehow, your logging file handle got closed. This has nothing to do with the rabbitmq socket.
2023-01-18 23:52:07
You might need to put some debug prints in init_logger to trace what's going on. Somehow, your logging file handle is getting closed. Do ANY log messages work? Do you know what check_log_name returns?
2023-01-18 23:52:07
actually my python code create log fd per hour , and error occur in this point. i guess the system flush the log content to mq server so the channel close.
2023-01-18 23:52:07
yes i must do it . in this point couldn't find out any python logs to print.
2023-01-18 23:52:07
Please trim your code to make it easier to find your problem. Follow these guidelines to create a minimal reproducible example.
Answers(1) :
{frame_too_large,842149168,131064}

means that you sent a message larger than what the RabbitMQ server is configured for.

You can try increasing the max-frame configuration of the server but it can impact performance of it as well.